Image processing, image processing network training method, device, equipment and medium

Abstract

The application discloses an image processing method and device, an image processing network training method and device, equipment and a medium, and belongs to the technical field of artificial intelligence. In the embodiment of the application, for an image sequence, in addition to extracting time sequence features, spatial features of the image sequence are innovatively obtained. The extracted features can not only reflect the correlation in time sequence of at least two images in the image sequence, but also reflect the correlation in space of at least two images in the image sequence. The image is reconstructed by comprehensively combining the two features, so that more features in the image sequence are fused in the target image, thereby improving the signal-to-noise ratio and accuracy of the target image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing, image processing network training method, apparatus, device and medium. Background Technology

[0002] Image stacking is an image processing method that merges the contents of multiple images in an image sequence into a single image, making the contents of the image stand out more.

[0003] Currently, image processing methods typically extract temporal features from multiple images in an image sequence, and then reconstruct the image based on these temporal features to obtain the target image.

[0004] The image processing method described above is only applicable to image sequences with high signal-to-noise ratio. For image sequences with low signal-to-noise ratio, the correlation between multiple images in the sequence is relatively weak, and accurate temporal features cannot be extracted. Therefore, the accuracy of the final target image is relatively poor. Summary of the Invention

[0005] This application provides an image processing method, apparatus, device, and medium for training an image processing network, thereby improving the accuracy of image processing. The technical solution is as follows:

[0006] On the one hand, an image processing method is provided, the method comprising:

[0007] The temporal features of the image sequence are obtained based on the order of at least two images in the image sequence;

[0008] Based on the positional changes of pixels between at least two images in the image sequence, the spatial features of the image sequence are obtained;

[0009] Based on the temporal and spatial features of the image sequence, image reconstruction is performed to obtain a target image, which incorporates the content of at least two images from the image sequence.

[0010] On the one hand, an image processing network training method is provided, the method comprising:

[0011] Based on an image processing network, the temporal features of the sample image sequence are obtained according to the order of at least two sample images in the sample image sequence;

[0012] Based on the image processing network, the spatial features of the sample image sequence are obtained according to the positional changes of pixels between at least two sample images in the sample image sequence;

[0013] Based on the image processing network, and based on the temporal and spatial features of the sample image sequence, image reconstruction is performed to obtain the predicted image;

[0014] The image processing network is trained based on the similarity between the predicted image and the target image corresponding to the sample image sequence.

[0015] On one hand, an image processing apparatus is provided, the apparatus comprising:

[0016] The acquisition module is used to acquire the temporal features of the image sequence based on the order of at least two images in the image sequence;

[0017] The acquisition module is further configured to acquire spatial features of the image sequence based on the positional changes of pixels between at least two images in the image sequence;

[0018] The reconstruction module is used to reconstruct the image based on the temporal features and spatial features of the image sequence to obtain a target image, wherein the target image incorporates the content of at least two images in the image sequence.

[0019] In some embodiments, the acquisition module is configured to:

[0020] The first image and the second image in the image sequence are downsampled to obtain the features of the first image and the second image, wherein the first image is an image in the image sequence and the second image is an image in the image sequence other than the first image;

[0021] The features of the first image and the second image are convolved to obtain the displacement of the pixel in the second image relative to the corresponding pixel in the first image;

[0022] Based on the features of the first image and the second image in the image sequence, and the displacement of pixels in the second image relative to the corresponding pixels in the first image, the pixels in the second image and the first image are aligned to obtain the spatial features of the image sequence.

[0023] In some embodiments, the acquisition module is configured to:

[0024] The features of the first image and the second image are concatenated to obtain the first feature;

[0025] The first feature is convolved to obtain the displacement of a pixel in the second image relative to the corresponding pixel in the first image;

[0026] The step of aligning pixels in the second image and the first image based on the features of the first image and the second image in the image sequence, and the displacement of pixels in the second image relative to corresponding pixels in the first image, to obtain the spatial features of the image sequence includes:

[0027] The spatial features of the second image are obtained by performing deformable convolution on the features of the first image and the second image, as well as the displacement of the pixels in the second image relative to the corresponding pixels in the first image.

[0028] The features of the first image are used as the spatial features of the first image.

[0029] In some embodiments, the acquisition module is used to process the features of the first image and the second image based on a filtered cascaded deformable convolutional network to obtain the spatial features of the image sequence.

[0030] In some embodiments, the filtered cascaded deformable convolutional network includes at least two cascaded deformable convolutional networks.

[0031] The acquisition module is used to input the features of the first image and the second image into a deformable convolutional network, and the deformable convolutional network processes the input features to obtain the second feature;

[0032] The features of the first image and the second image are denoised, and the denoised features are input into other layers of deformable convolutional networks. The other layers of deformable convolutional networks process the input features to obtain the third feature.

[0033] Based on the second feature and the third feature, the spatial features of the second image are obtained;

[0034] The features of the first image are used as the spatial features of the first image.

[0035] In some embodiments, the acquisition module is used to process the features of the first image and the second image based on an optical flow estimation network to obtain the spatial features of the image sequence.

[0036] In some embodiments, the reconstruction module is configured to:

[0037] The temporal features and spatial features of the image sequence are fused to obtain the target features;

[0038] The target image is obtained by reconstructing the image based on the target features.

[0039] In some embodiments, the reconstruction module is configured to:

[0040] For any image in the image sequence, the temporal features and spatial features of the image are concatenated to obtain the image features of the image;

[0041] The image features of at least two images in the image sequence are weighted to obtain the target features.

[0042] In some embodiments, the acquisition module is used to process at least two images in the image sequence based on any one of a recurrent neural network, a recurrent neural network, or a long short-term memory neural network that includes at least two residual dense blocks, to obtain the temporal features of the image sequence.

[0043] In some embodiments, the acquisition module and the reconstruction module are used to input the image sequence into an image processing network, whereby the temporal module of the image processing network acquires the temporal features of the image sequence, the spatial module of the image processing network acquires the spatial features of the image sequence, and the image processing network performs image reconstruction based on the temporal features and the spatial features of the image sequence to output a target image.

[0044] On one hand, an image processing network training device is provided, the device comprising:

[0045] The acquisition module is used to acquire the temporal features of the sample image sequence based on the image processing network and according to the order of at least two sample images in the sample image sequence;

[0046] The acquisition module is used to acquire the spatial features of the sample image sequence based on the image processing network and according to the positional changes of pixels between at least two sample images in the sample image sequence;

[0047] The reconstruction module is used to reconstruct the image based on the image processing network and the temporal and spatial features of the sample image sequence to obtain the predicted image.

[0048] The training module is used to train the image processing network based on the similarity between the predicted image and the target image corresponding to the sample image sequence.

[0049] On one hand, an electronic device is provided, comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement various optional implementations of the above-described image processing method or image processing network training method.

[0050] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to implement various optional implementations of the above-described image processing method or image processing network training method.

[0051] In one aspect, a computer program product or computer program is provided, the computer program product or computer program comprising one or more lines of program code stored in a computer-readable storage medium. One or more processors of an electronic device read the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, causing the electronic device to perform an image processing method or an image processing network training method according to any of the above possible embodiments.

[0052] In this embodiment of the application, in addition to extracting temporal features, spatial features of the image sequence are also innovatively obtained for the image sequence. The extracted features can not only reflect the temporal correlation between at least two images in the image sequence, but also reflect the spatial correlation between at least two images in the image sequence. By combining the two features for image reconstruction, the target image can incorporate more features from the image sequence, thereby improving the signal-to-noise ratio and accuracy of the target image. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the implementation environment of an image processing method or an image processing network training method provided in the embodiments of this application;

[0055] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application;

[0056] Figure 3 This is a flowchart of an image processing method provided in an embodiment of this application;

[0057] Figure 4 This is a flowchart of an image processing method provided in an embodiment of this application;

[0058] Figure 5 This is a schematic diagram of an image processing network provided in an embodiment of this application;

[0059] Figure 6This is a schematic diagram of an FCD module provided in an embodiment of this application;

[0060] Figure 7 This is a schematic diagram of an STF module provided in an embodiment of this application;

[0061] Figure 8 This is a flowchart of an image processing network training method provided in an embodiment of this application;

[0062] Figure 9 This is a schematic diagram of the experimental results of an image processing method provided in an embodiment of this application;

[0063] Figure 10 This is a schematic diagram of the experimental results of an image processing method provided in an embodiment of this application;

[0064] Figure 11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0065] Figure 12 This is a schematic diagram of the structure of an image processing network training device provided in an embodiment of this application;

[0066] Figure 13 This is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0067] Figure 14 This is a structural block diagram of a terminal provided in an embodiment of this application;

[0068] Figure 15 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0070] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, a first image is referred to as a second image, and similarly, a second image is referred to as a first image. Both the first image and the second image are images, and in some cases, they are separate and distinct images.

[0071] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple data packets means two or more data packets.

[0072] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0073] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating the existence of three relationships; for example, A and / or B means: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0074] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0075] It should also be understood that determining B based on A does not mean determining B solely based on A, but also based on A and / or other information.

[0076] It should also be understood that the term “comprising” (also referred to as “inCludes”, “inCluding”, “Comprises”, and / or “Comprising”) as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0077] It should also be understood that the term "if" can be interpreted as meaning "when" or "upon" or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrases "if determination..." or "if detection [the stated condition or event]" can be interpreted as meaning "when determination..." or "in response to determination..." or "when detection [the stated condition or event]" or "in response to detection [the stated condition or event]."

[0078] The following is an explanation of the terms used in this application.

[0079] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0080] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0081] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0082] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0083] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0084] The solutions provided in this application involve artificial intelligence technologies such as computer vision and machine learning, which are specifically illustrated through the following embodiments.

[0085] The implementation environment of this application is described below.

[0086] Figure 1 This is a schematic diagram of an implementation environment for an image processing method or an image processing network training method provided in this application embodiment. The implementation environment includes a terminal 101, or it includes a terminal 101 and an image processing platform 102. The terminal 101 is connected to the image processing platform 102 via a wireless network or a wired network.

[0087] Terminal 101 is at least one of a smartphone, game console, desktop computer, tablet computer, e-book reader, MP3 (Moving Picture Experts Group Audio Layer III) player or MP4 (Moving Picture Experts Group Audio Layer IV) player, and laptop computer. Terminal 101 has installed and runs applications that support image processing, such as system applications, instant messaging applications, news push applications, shopping applications, online video applications, and social applications.

[0088] For example, the terminal 101 has image acquisition and image processing functions, processes the acquired images, and executes corresponding functions based on the processing results. The terminal 101 performs this work independently, and also receives data services from the image processing platform 102. This application embodiment does not limit this aspect.

[0089] For example, the terminal 101 or the image processing platform 102 can train an image processing network based on a sequence of sample images.

[0090] Image processing platform 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Image processing platform 102 provides background services for applications supporting image processing. Optionally, image processing platform 102 performs the primary processing task, and terminal 101 performs secondary processing task; or, image processing platform 102 performs secondary processing task, and terminal 101 performs primary processing task; or, image processing platform 102 or terminal 101 each performs processing task independently. Alternatively, image processing platform 102 and terminal 101 collaborate using a distributed computing architecture.

[0091] Optionally, the image processing platform 102 includes at least one server 1021 and a database 1022. The database 1022 is used to store data. In this embodiment, the database 1022 stores sample image sequences or sample videos to provide data services to at least one server 1021.

[0092] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. A terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0093] Those skilled in the art will know that the number of terminals 101 and servers 1021 may be more or less. For example, there may be only one terminal 101 or server 1021, or there may be dozens or hundreds of terminals 101 or servers 1021, or even more. The embodiments of this application do not limit the number or type of terminals or servers.

[0094] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application. The method is applied in an electronic device, which may be a terminal or a server. See also... Figure 2 The method includes the following steps.

[0095] 201. An electronic device obtains the temporal characteristics of an image sequence based on the order of at least two images in the image sequence.

[0096] A feature is a characteristic that distinguishes one thing from another. Temporal features refer to features that can reflect the temporal correlation between at least two images.

[0097] When processing image sequences, feature extraction is required to extract the image features, which are then used for image reconstruction.

[0098] In this embodiment, during image feature extraction, not only temporal features but also spatial features are extracted. This ensures that the temporal and spatial correlations between at least two images in the image sequence are extracted, better guiding the fusion of the image sequence into a single image. This fusion process is an image stacking process, used to superimpose the content of multiple images in the image sequence into one image, thereby highlighting the content within the image.

[0099] The image processing method provided in this application can be applied to electron microscopy image processing scenarios. For example, cryo-electron microscopy (cryo-EM) can determine the structure of biomolecules at near-atomic resolution. In this technique, K2 Summit electron counting is used to directly detect a segment of low signal-to-noise ratio (SNR) video data of biomolecules recorded by a camera during exposure. Because biomolecules move due to electron impacts during imaging, different frames of this video data need to be aligned and stacked to obtain a high SNR image, which helps to obtain higher resolution biomolecule structures. Furthermore, electron microscopy images generally have a low SNR. If image reconstruction is performed solely based on temporal features, as in related techniques, accurate temporal features cannot be extracted, and the final reconstructed image may also have a low SNR, resulting in poor performance. The method in this application extracts both temporal and spatial features to guide image reconstruction, effectively overcoming the above problems and improving the SNR and accuracy of the reconstructed image.

[0100] 202. The electronic device obtains the spatial features of the image sequence based on the positional changes of pixels between at least two images in the image sequence.

[0101] This spatial feature can reflect the positional changes of pixels between at least two images in the image sequence.

[0102] 203. The electronic device reconstructs an image based on the temporal and spatial features of the image sequence to obtain a target image, wherein the target image incorporates the content of at least two images from the image sequence.

[0103] The electronic device extracts two types of features, and combines the features that reflect temporal and spatial correlations to reconstruct the image. By reconstructing the image based on more features, a better target image can be obtained.

[0104] In this embodiment of the application, in addition to extracting temporal features, spatial features of the image sequence are also innovatively obtained for the image sequence. The extracted features can not only reflect the temporal correlation between at least two images in the image sequence, but also reflect the spatial correlation between at least two images in the image sequence. By combining the two features for image reconstruction, the target image can incorporate more features from the image sequence, thereby improving the signal-to-noise ratio and accuracy of the target image.

[0105] Figure 3 This is a flowchart of an image processing network training method provided in an embodiment of this application. See also... Figure 3 The method includes the following steps.

[0106] 301. An electronic device based on an image processing network obtains the temporal features of a sample image sequence according to the order of at least two sample images in the sample image sequence.

[0107] 302. The electronic device, based on the image processing network, obtains the spatial features of the sample image sequence according to the positional changes of pixels between at least two sample images in the sample image sequence.

[0108] 303. The electronic device reconstructs the image based on the image processing network and the temporal and spatial features of the sample image sequence to obtain the predicted image.

[0109] Steps 301 to 303 are similar to steps 201 to 203 above, except that the relevant image processing steps are performed through an image processing network.

[0110] 304. The electronic device trains the image processing network based on the similarity between the predicted image and the target image corresponding to the sample image sequence.

[0111] The predicted image obtained by the image processing network is a predicted value or estimate, while the target image is the true value or correct value. Electronic devices can determine whether the predicted image is accurate by comparing the predicted image and the target image and by measuring the similarity between the predicted image and the target image.

[0112] Understandably, if the image processing network is inaccurate, its parameters can be updated to make the predicted image closer to the target image. If the image is accurate, training can end, at which point the network parameters are relatively accurate.

[0113] In this embodiment of the application, in addition to extracting temporal features, spatial features of the image sequence are also innovatively obtained for the image sequence. The extracted features can not only reflect the temporal correlation between at least two images in the image sequence, but also reflect the spatial correlation between at least two images in the image sequence. By combining the two features for image reconstruction, the target image can incorporate more features from the image sequence, thereby improving the signal-to-noise ratio and accuracy of the target image.

[0114] Figure 4 This is a flowchart of an image processing method provided in an embodiment of this application. See also... Figure 4 The method includes the following steps.

[0115] 401. Electronic devices acquire image sequences.

[0116] In some embodiments, the image sequence can be captured by other electronic devices and sent to those devices; that is, when other electronic devices have image processing needs, the image sequence can be sent to those devices.

[0117] In other embodiments, the image sequence can be captured by the electronic device. For example, a video of an object can be captured, resulting in multiple video frames.

[0118] The image sequence includes at least two images, and the number of these at least two images can be set by relevant technicians according to their needs, such as 3, 5, 7, etc. This application embodiment does not limit this.

[0119] 402. The electronic device obtains the temporal characteristics of the image sequence based on the order of at least two images in the image sequence.

[0120] Electronic devices can extract features from at least two images in an image sequence based on the order of at least two images in the sequence, thereby obtaining temporal features.

[0121] In some embodiments, the electronic device first preprocesses at least two images in the image sequence, and then performs the above-described steps of acquiring temporal features on the preprocessed image sequence.

[0122] In some embodiments, the electronic device processes at least two images in the image sequence based on any one of a recurrent neural network, a recurrent neural network, or a long short-term memory neural network that includes at least two residual dense blocks, to obtain the temporal features of the image sequence.

[0123] This image processing method can be implemented using an image processing network. The image processing network may include a temporal module, which can be any of a recurrent neural network, a recurrent neural network, or a long short-term memory neural network comprising at least two residual dense blocks.

[0124] For example, in a specific example, such as Figure 5 As shown, for an image sequence, after preprocessing, it can be input into a recurrent neural network that includes multiple residual dense blocks (RDBs), hence it can also be called an RDB-RNN module, which is simply referred to as RDB here. Through this RDB, the features of the image can be converted into temporal features T.

[0125] 403. The electronic device obtains the spatial features of the image sequence based on the positional changes of pixels between at least two images in the image sequence.

[0126] When acquiring spatial features, electronic devices can also extract spatial features from preprocessed image sequences.

[0127] Specifically, the electronic device can perform the spatial feature extraction process through the following steps one through three.

[0128] Step 1: The electronic device downsamples the first image and the second image in the image sequence to obtain the features of the first image and the second image. The first image is one image in the image sequence, and the second image is the image in the image sequence other than the first image.

[0129] In some embodiments, in step one, the electronic device can downsample each image in the image sequence through a convolutional layer to obtain the features of each image.

[0130] Step 2: The electronic device performs convolution processing on the features of the first image and the second image to obtain the displacement of the pixels in the second image relative to the corresponding pixels in the first image.

[0131] The displacement of this pixel is actually an estimated value obtained from the convolution process. Once the electronic device obtains the displacement of the pixel, it can align the pixels in the two images based on this displacement. In this way, the positions of corresponding pixels in the two images can be adjusted to be consistent, and the features of the pixel can be highlighted during subsequent image fusion.

[0132] Step 3: Based on the features of the first image and the second image in the image sequence, and the displacement of pixels in the second image relative to the corresponding pixels in the first image, the electronic device aligns the pixels in the second image and the first image to obtain the spatial features of the image sequence.

[0133] After determining the displacement of a pixel in the second image relative to the corresponding pixel in the first image, the features of the pixels in the second image can be moved according to the displacement to align the two images and obtain the spatial features of the second image.

[0134] For the first image, which is a reference image, the features of the first image can be used as the spatial features of the first image.

[0135] In some embodiments, after obtaining the features of the first image and the second image, the process of obtaining the above-mentioned spatial features can be implemented in a variety of ways.

[0136] Method 1: The electronic device can concatenate the features of the first image and the second image to obtain a first feature, and then perform convolution processing on the first feature to obtain the displacement of a pixel in the second image relative to the corresponding pixel in the first image. Furthermore, the electronic device performs deformable convolution processing on the features of the first image and the second image, as well as the displacement of a pixel in the second image relative to the corresponding pixel in the first image, to obtain the spatial features of the second image, and uses the features of the first image as the spatial features of the first image.

[0137] In this first method, displacement can be obtained through convolution processing, and spatial features can be determined through deformable convolution processing. This process can be implemented using a filtered cascaded deformable convolutional network. For example, an electronic device can process the features of the first image and the second image based on a filtered cascaded deformable convolutional network to obtain the spatial features of the image sequence.

[0138] Specifically, the filtered cascaded deformable convolutional network includes at least two cascaded deformable convolutional networks. The electronic device can input the features of the first image and the second image into a single deformable convolutional network, where the network processes the input features to obtain a second feature. Then, the features of the first image and the second image are denoised, and the denoised features are input into other layers of deformable convolutional networks. These other layers process the input features to obtain a third feature. Based on the second and third features, the spatial features of the second image are obtained, and the features of the first image are used as the spatial features of the first image.

[0139] like Figure 6As shown, the Filtering Cascading Deformable Network (FCD) consists of three layers of deformable convolutional networks, denoted as L1, L2, and L3. The features of the input image are first convolved by convolutional layers with a stride of 2 to obtain an L-layer pyramid. For the first and second images, similar operations are performed at each layer of the pyramid: the features of the two images are concatenated and convolved to obtain the offsets of the deformable convolution. The features of the second image are input to the deformable convolutional layer and output to obtain new features. In addition, the offsets output of the lower layer of the pyramid are used as the input of the offsets convolution of the upper layer for more accurate offset estimation. The features output by the deformable convolutional layer are also fused with the corresponding features of the upper layer. Up to L1 of the pyramid, the output of the deformable convolutional layer and the features fused with the bottom layer are concatenated with the features of the first image as offsets of the new deformable convolutional layer, which can then predict the spatial features aligned with the second image.

[0140] This image processing method can be implemented using an image processing network. This image processing network may include a spatial module, which can employ a filtered, cascaded, deformable convolutional network.

[0141] For example, in a specific example, such as Figure 5 As shown, for an image sequence, after preprocessing, the image features can be obtained by downsampling and then input into an FCD (Full Detector). The FCD can then convert the image features into spatial features S.

[0142] For FCD, given two images, their pre-defined... Displacement between sampling points Then use this displacement to move any position in the two images. To perform spatial alignment:

[0143]

[0144] in, and Indicates the pre-set number The weights and displacements of each sampling point. Let... Indicates all The set of neighbor frame features and target frame features Displacement between It can be obtained through the following formula:

[0145]

[0146] in, It is a convolutional neural network (CNN). This is a serial connection operation.

[0147] To better align low signal-to-noise ratio images spatially, this invention employs a coarse-to-fine learning process and proposes... The FCD module of the layer is used to achieve spatial alignment. To generate the first... Layer characteristics , for the The features of the layer are then subjected to Gaussian filtering. Finally, the first... The output of the layer will be returned to the first layer. layer:

[0148]

[0149]

[0150] in, and For CNN. It represents DCN.

[0151] In method two, the electronic device can process the features of the first and second images based on an optical flow estimation network to obtain the spatial features of the image sequence. Specifically, it first determines the pixel displacements and then spatially aligns the images using these displacements.

[0152] 404. The electronic device reconstructs an image based on the temporal and spatial features of the image sequence to obtain a target image, wherein the target image incorporates the content of at least two images from the image sequence.

[0153] After obtaining temporal and spatial features, electronic devices can combine the two features to reconstruct images.

[0154] In some embodiments, the electronic device may first fuse two features into a target feature, and then reconstruct the image based on the target feature. Specifically, the electronic device may fuse the temporal feature and the spatial feature of the image sequence to obtain the target feature, and then reconstruct the image based on the target feature to obtain the target image.

[0155] Specifically, when fusing to obtain target features, temporal and spatial features can be connected to obtain new target features. Specifically, for any image in the image sequence, the electronic device can connect the temporal and spatial features of that image to obtain its image features, and then weight the image features of at least two images in the image sequence to obtain the target features.

[0156] In some embodiments, the image processing method can be implemented through an image processing network. Specifically, the electronic device can input the image sequence into the image processing network, where the temporal module of the image processing network obtains the temporal features of the image sequence, the spatial module of the image processing network obtains the spatial features of the image sequence, and the image processing network performs image reconstruction based on the temporal and spatial features of the image sequence, outputting the target image.

[0157] This image processing method can be implemented using an image processing network. This network can include a spatial-temporal fusion module (STF). For example, in a specific example, such as... Figure 5 As shown, for an image sequence, after obtaining spatial features S and temporal features T, the electronic device can input them into a spatial-temporal fusion module (STF), which fuses the two features. Specifically, this STF module can be as follows: Figure 7 As shown, for the temporal and spatial features of an image, they can be concatenated to obtain the image feature C of that image. Then, for the image features of multiple images, they can be weighted and summed to obtain the final target feature F. When fusing the image features of multiple images, an attention mechanism can be used, that is, the weight of the image feature of each image is the attention score of that image. This attention score (weight) can be determined during training, and this embodiment of the application does not limit it.

[0158] In this embodiment of the application, in addition to extracting temporal features, spatial features of the image sequence are also innovatively obtained for the image sequence. The extracted features can not only reflect the temporal correlation between at least two images in the image sequence, but also reflect the spatial correlation between at least two images in the image sequence. By combining the two features for image reconstruction, the target image can incorporate more features from the image sequence, thereby improving the signal-to-noise ratio and accuracy of the target image.

[0159] Figure 8 This is a flowchart of an image processing network training method provided in an embodiment of this application. See also... Figure 8 The method includes the following steps.

[0160] 801. An electronic device acquires a sequence of sample images, which corresponds to a target image.

[0161] The image processing network training method can be applied to two-dimensional image processing as well as three-dimensional image processing. That is, the images in the sample image sequence can be two-dimensional images or three-dimensional images. This application does not limit this.

[0162] The sample image sequence may be stored in different locations, and correspondingly, electronic devices can acquire the sample image sequence in different ways.

[0163] In some embodiments, the sample image sequence may be stored in an electronic device, and the electronic device may retrieve the sample image sequence from the stored data.

[0164] In other embodiments, the sample image sequence may be stored in an image database. Accordingly, the electronic device can retrieve the sample image sequence from the image database.

[0165] In other embodiments, the sample image sequence can be extracted from a video. Accordingly, the electronic device can acquire the sample video, extract frames from the sample video, and obtain the sample image sequence.

[0166] The above provides three possible methods for obtaining sample image sequences, and this application embodiment does not limit these methods.

[0167] The target image is the correct and realistic image stacking result. In some embodiments, the target image may be set by a person skilled in the art for the sample image sequence, and the target image can be understood as the annotation data or label of the sample image pair. Specifically, the target image may be obtained by a person skilled in the art through other image stacking methods, and when the person skilled in the art determines that the effect of the target image is good, it is used as the target image corresponding to the sample image sequence.

[0168] In other embodiments, the target image is stored in association with a sample image sequence, and the electronic device acquires the target image synchronously when the sample image sequence is acquired.

[0169] 802. Electronic devices based on image processing networks acquire temporal features of a sample image sequence according to the order of at least two sample images in the sample image sequence.

[0170] Step 802 is similar to step 402 above, and will not be elaborated further here, except that the temporal feature is obtained by the image processing network. Specifically, the image processing network includes a temporal module, and the electronic device can obtain the temporal feature of the sample image sequence based on the temporal module of the image processing network according to the order of at least two sample images in the sample image sequence.

[0171] 803. An electronic device, based on the image processing network, obtains the spatial features of the sample image sequence according to the positional changes of pixels between at least two sample images in the sample image sequence.

[0172] Step 803 is similar to step 403 above, and will not be elaborated further here, except that the spatial features are obtained by an image processing network. Specifically, the image processing network includes a spatial module, and the electronic device can obtain the spatial features of the sample image sequence based on the positional changes of pixels between at least two sample images in the sample image sequence using the spatial module of the image processing network.

[0173] 804. The electronic device reconstructs the image based on the image processing network and the temporal and spatial features of the sample image sequence to obtain the predicted image.

[0174] Step 804 is similar to step 404 above, and will not be elaborated further here, except that the spatial features are obtained by an image processing network. Specifically, the image processing network includes a spatial module, and the electronic device can obtain the spatial features of the sample image sequence based on the positional changes of pixels between at least two sample images in the sample image sequence using the spatial module of the image processing network.

[0175] 805. The electronic device trains the image processing network based on the similarity between the predicted image and the target image corresponding to the sample image sequence.

[0176] The training process described above is an iterative process. In each iteration, the image processing network processes the input sample image sequence and obtains the similarity between the predicted image and the target image corresponding to the sample image sequence. If the similarity does not meet the target condition, the electronic device can optimize the parameters of the image processing network based on the similarity. If the similarity meets the target condition, the electronic device can determine that the training has ended and use the network parameters used in this iteration as the final network parameters of the image processing network.

[0177] This optimization process can be implemented in various ways. Taking gradient descent as an example, the electronic device can obtain the gradient of the network parameters based on similarity, and update the network parameters based on this gradient to obtain the updated network parameters. In the next iteration, when the image processing network processes the input sample image sequence, it can use the updated network parameters.

[0178] The specific similarity can be determined by a similarity metric function or a loss function. For example, an L2-norm loss function, or an L1-norm loss function, or other regularization terms can be used. Of course, any other form of supervised loss function can also be used, and this application does not limit this.

[0179] In a specific example, the loss function for the entire training process It can be represented as:

[0180]

[0181] in, This represents the predicted image from the reconstructed output. This represents the target image.

[0182] For example, in a specific example, the input N images are first preprocessed, including image normalization and random cropping into 256*256 patches. N is a positive integer. After preprocessing, the cropped patches are used as input to the spatial and temporal modules for parallel training. The spatial module, based on a deformable convolutional network, spatially aligns each neighboring frame to the target frame, thus obtaining aligned spatial features. The temporal module, based on a recurrent neural network, extracts important temporal features from the input frames. The aligned spatial and temporal features are then merged and fed into a spatial-temporal attention mechanism network for learning, resulting in fused features from the N images. Finally, these features are reconstructed into an image using a deconvolutional network. For example, training data can use 256×256 patches extracted from the EMPIAR (Electron Microscopy Public Image Archive)-10543 and EMPIAR-10428 datasets as input data. Training the model in this manner for approximately 5000 epochs yields good results. An epoch refers to one iteration.

[0183] In some embodiments, the model training process can be performed by an electronic device based on a GPU (Graphics Processing Unit) to accelerate training speed and improve training efficiency.

[0184] After the image processing network is trained, if an electronic device requires image processing, it can call upon the trained network to perform image processing. The specific image processing procedure can be found above. Figure 4 The embodiments shown are not described in detail here.

[0185] As can be seen from the above, this application significantly reduces the signal-to-noise ratio (SNR) requirement for the dataset. Compared with other deep learning methods, it no longer requires the dataset to be high-SNR natural images, but can instead use low-SNR images. Therefore, it can be applied to various electron microscopy datasets.

[0186] This application uses end-to-end training. The training process is simple, requiring no additional preprocessing or pretraining, nor any other input information to assist model training.

[0187] This application can be accelerated using a GPU (Graphics Processing Unit), offering high cost-effectiveness in practical applications: because this invention uses deep learning-related methods, various acceleration schemes optimized for deep learning can be used. In contrast, commonly used algorithms in the field of electron microscopy, such as MotionCor2, require longer computation times than this invention.

[0188] The following experiments compare the method provided in this application with other related methods on different datasets, specifically using 256×256 patches derived from the EMPIAR-10543 and EMPIAR-10428 datasets as input. Image stacking was performed using various methods, including the direct addition method Sum, the traditional algorithm ECC (Entropy Correlation Coefficient), and deep learning algorithms CDVD, EDVR (Video Restoration with Enhanced Deformable Convolutional Networks), and ESTRNN (an efficient spatiotemporal recurrent neural network for video deblurring). The performance of image stacking was reflected by comparing the PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity) of the output results of various models with reference standards.

[0189] Table 1 below shows the performance comparison results of the present invention with other stacking methods on the EMPIAR-10543 dataset, and Table 2 shows the performance comparison results of the present invention with other stacking methods on the EMPIAR-10428 dataset.

[0190] Table 1

[0191]

[0192] Table 2

[0193]

[0194] As can be seen from Tables 1 and 2, the method of this application can achieve optimal PSNR and SSIM in most cases when performing image stacking.

[0195] like Figure 9 and Figure 10As shown in the figure, this application is referred to as "this invention," and the input frame refers to the reference image in the input image sequence. The reference standard refers to the result with better stacking performance. As can be seen from the figure, the image stacking effect of this application is the best among the methods compared: on the EMPIAR-10543 and EMPIAR-10428 datasets, the performance of this application is better than the methods proposed by Sum, ECC, CDVD, EDVR, and ESTRNN. Compared with other methods, the results of this application have a higher signal-to-noise ratio, clearer images, and are closer to the reference standard.

[0196] In this embodiment of the application, in addition to extracting temporal features, spatial features of the image sequence are also innovatively obtained for the image sequence. The extracted features can not only reflect the temporal correlation between at least two images in the image sequence, but also reflect the spatial correlation between at least two images in the image sequence. By combining the two features for image reconstruction, the target image can incorporate more features from the image sequence, thereby improving the signal-to-noise ratio and accuracy of the target image.

[0197] All the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0198] Figure 11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. See also... Figure 11 The device includes:

[0199] The acquisition module 1101 is used to acquire the temporal features of the image sequence based on the order of at least two images in the image sequence;

[0200] The acquisition module 1101 is further configured to acquire the spatial features of the image sequence based on the positional changes of pixels between the at least two images in the image sequence;

[0201] The reconstruction module 1102 is used to reconstruct the image based on the temporal and spatial features of the image sequence to obtain a target image, which incorporates the content of at least two images in the image sequence.

[0202] In some embodiments, the acquisition module 1101 is configured to:

[0203] The first image and the second image in the image sequence are downsampled to obtain the features of the first image and the second image. The first image is an image in the image sequence, and the second image is an image in the image sequence other than the first image.

[0204] The features of the first image and the second image are convolved to obtain the displacement of the pixels in the second image relative to the corresponding pixels in the first image;

[0205] Based on the features of the first image and the second image in the image sequence, and the displacement of pixels in the second image relative to the corresponding pixels in the first image, the pixels in the second image and the first image are aligned to obtain the spatial features of the image sequence.

[0206] In some embodiments, the acquisition module 1101 is configured to:

[0207] The features of the first image and the second image are concatenated to obtain the first feature;

[0208] The first feature is convolved to obtain the displacement of the pixel in the second image relative to the corresponding pixel in the first image;

[0209] Based on the features of the first and second images in the image sequence, and the displacement of pixels in the second image relative to corresponding pixels in the first image, the pixels in the second image and the first image are aligned to obtain the spatial features of the image sequence, including:

[0210] The spatial features of the second image are obtained by performing deformable convolution on the features of the first image and the second image, as well as the displacement of pixels in the second image relative to the corresponding pixels in the first image.

[0211] The features of the first image are used as the spatial features of the first image.

[0212] In some embodiments, the acquisition module 1101 is used to process the features of the first image and the second image based on a filtered cascaded deformable convolutional network to obtain the spatial features of the image sequence.

[0213] In some embodiments, the filtered cascaded deformable convolutional network includes at least two cascaded deformable convolutional networks.

[0214] The acquisition module 1101 is used to input the features of the first image and the second image into a deformable convolutional network, and the deformable convolutional network processes the input features to obtain the second feature;

[0215] The features of the first image and the second image are denoised, and the denoised features are input into other deformable convolutional networks. The other deformable convolutional networks process the input features to obtain the third feature.

[0216] Based on the second feature and the third feature, the spatial features of the second image are obtained;

[0217] The features of the first image are used as the spatial features of the first image.

[0218] In some embodiments, the acquisition module 1101 is used to process the features of the first image and the second image based on an optical flow estimation network to obtain the spatial features of the image sequence.

[0219] In some embodiments, the reconstruction module 1102 is configured to:

[0220] The temporal and spatial features of the image sequence are fused to obtain the target features;

[0221] The target image is obtained by reconstructing the image based on the target features.

[0222] In some embodiments, the reconstruction module 1102 is configured to:

[0223] For any image in the image sequence, the temporal feature and spatial feature of the image are concatenated to obtain the image feature of the image;

[0224] The target features are obtained by weighting the image features of at least two images in the image sequence.

[0225] In some embodiments, the acquisition module 1101 is used to process at least two images in the image sequence based on any one of a recurrent neural network, a recurrent neural network, or a long short-term memory neural network that includes at least two residual dense blocks, to obtain the temporal features of the image sequence.

[0226] In some embodiments, the acquisition module 1101 and the reconstruction module 1102 are used to input the image sequence into an image processing network, where the temporal module of the image processing network acquires the temporal features of the image sequence, the spatial module of the image processing network acquires the spatial features of the image sequence, and the image processing network performs image reconstruction based on the temporal and spatial features of the image sequence to output a target image.

[0227] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules during image processing. In actual applications, the above functions may be assigned to different functional modules as needed, that is, the internal structure of the image processing apparatus may be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus and the image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0228] Figure 12 This is a schematic diagram of the structure of an image processing network training device provided in an embodiment of this application. See also... Figure 12 The device includes:

[0229] The acquisition module 1201 is used to acquire the temporal features of the sample image sequence based on the image processing network and according to the order of at least two sample images in the sample image sequence;

[0230] The acquisition module 1201 is used to acquire the spatial features of the sample image sequence based on the image processing network and according to the positional changes of pixels between at least two sample images in the sample image sequence;

[0231] The reconstruction module 1202 is used to reconstruct the image based on the image processing network and the temporal and spatial features of the sample image sequence to obtain the predicted image.

[0232] The training module 1203 is used to train the image processing network based on the similarity between the predicted image and the target image corresponding to the sample image sequence.

[0233] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules during image processing. In actual applications, the above functions may be assigned to different functional modules as needed, that is, the internal structure of the image processing apparatus may be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus and the image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0234] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 1300 can vary significantly due to differences in configuration or performance. It includes one or more Central Processing Units (CPUs) 1301 and one or more memories 1302. The memory 1302 stores at least one computer program, which is loaded and executed by the processor 1301 to implement the image processing methods provided in the various method embodiments described above. The electronic device also includes other components for implementing device functions. For example, the electronic device also has wired or wireless network interfaces and input / output interfaces for input and output. Details of these embodiments are not elaborated upon here.

[0235] The electronic device in the above method embodiments is implemented as a terminal. For example, Figure 14This is a structural block diagram of a terminal provided in an embodiment of this application. The terminal 1400 can be a portable mobile terminal, such as a smartphone, tablet computer, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player, laptop computer, or desktop computer. The terminal 1400 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0236] Typically, terminal 1400 includes a processor 1401 and a memory 1402.

[0237] Processor 1401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0238] The memory 1402 may include one or more computer-readable storage media, which may be non-transitory. The memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1402 are used to store at least one instruction, which is executed by the processor 1401 to implement the image processing method provided in the method embodiments of this application.

[0239] In some embodiments, the terminal 1400 may also optionally include a peripheral device interface 1403 and at least one peripheral device. The processor 1401, memory 1402, and peripheral device interface 1403 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1403 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1404, a display screen 1405, a camera assembly 1406, an audio circuit 1407, and a power supply 1409.

[0240] Peripheral interface 1403 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1401 and memory 1402. In some embodiments, processor 1401, memory 1402 and peripheral interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1401, memory 1402 and peripheral interface 1403 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0241] The radio frequency (RF) circuit 1404 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1404 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1404 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1404 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1404 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1404 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0242] Display screen 1405 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1405 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1401 for processing. In this case, display screen 1405 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1405, disposed on the front panel of terminal 1400; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1400 or in a folded design; in still other embodiments, display screen 1405 may be a flexible display screen, disposed on a curved or folded surface of terminal 1400. Furthermore, display screen 1405 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1405 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0243] The camera assembly 1406 is used to acquire images or videos. Optionally, the camera assembly 1406 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1406 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0244] The audio circuit 1407 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1401 for processing, or input to the radio frequency circuit 1404 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1400. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1401 or the radio frequency circuit 1404 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1407 may also include a headphone jack.

[0245] Power supply 1409 is used to power the various components in terminal 1400. Power supply 1409 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1409 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0246] In some embodiments, the terminal 1400 further includes one or more sensors 1410. The one or more sensors 1410 include, but are not limited to: an accelerometer 1411, a gyroscope 1412, a pressure sensor 1413, an optical sensor 1415, and a proximity sensor 1416.

[0247] Accelerometer 1411 can detect the magnitude of acceleration along the three axes of a coordinate system established with terminal 1400. For example, accelerometer 1411 can be used to detect the components of gravitational acceleration along the three axes. Processor 1401 can control display screen 1405 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1411. Accelerometer 1411 can also be used for games or for acquiring user motion data.

[0248] The gyroscope sensor 1412 can detect the orientation and rotation angle of the terminal 1400. The gyroscope sensor 1412 can work in conjunction with the accelerometer sensor 1411 to collect the user's 3D movements on the terminal 1400. Based on the data collected by the gyroscope sensor 1412, the processor 1401 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0249] The pressure sensor 1413 can be disposed on the side bezel of the terminal 1400 and / or on the lower layer of the display screen 1405. When the pressure sensor 1413 is disposed on the side bezel of the terminal 1400, it can detect the user's grip signal on the terminal 1400, and the processor 1401 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1413. When the pressure sensor 1413 is disposed on the lower layer of the display screen 1405, the processor 1401 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1405. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0250] An optical sensor 1415 is used to collect ambient light intensity. In one embodiment, the processor 1401 can control the display brightness of the display screen 1405 based on the ambient light intensity collected by the optical sensor 1415. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1405 is increased; when the ambient light intensity is low, the display brightness of the display screen 1405 is decreased. In another embodiment, the processor 1401 can also dynamically adjust the shooting parameters of the camera assembly 1406 based on the ambient light intensity collected by the optical sensor 1415.

[0251] The proximity sensor 1416, also known as a distance sensor, is typically located on the front panel of the terminal 1400. The proximity sensor 1416 is used to detect the distance between the user and the front of the terminal 1400. In one embodiment, when the proximity sensor 1416 detects that the distance between the user and the front of the terminal 1400 is gradually decreasing, the processor 1401 controls the display screen 1405 to switch from a screen-on state to a screen-off state; when the proximity sensor 1416 detects that the distance between the user and the front of the terminal 1400 is gradually increasing, the processor 1401 controls the display screen 1405 to switch from a screen-off state to a screen-on state.

[0252] Those skilled in the art will understand that Figure 14 The structure shown does not constitute a limitation on terminal 1400 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0253] The electronic device in the above method embodiments is implemented as a server. For example, Figure 15This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1500 can vary significantly depending on its configuration or performance. It includes one or more Central Processing Units (CPUs) 1501 and one or more memories 1502. The memories 1502 store at least one computer program, which is loaded and executed by the processor 1501 to implement the image processing methods provided in the various method embodiments described above. Of course, the server also has wired or wireless network interfaces and input / output interfaces for input and output. The server also includes other components for implementing device functions, which will not be elaborated here.

[0254] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, which is executable by a processor to perform the image processing method described above. For example, the computer-readable storage medium is a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0255] In an exemplary embodiment, a computer program product or computer program is also provided, comprising one or more lines of program code stored in a computer-readable storage medium. One or more processors of an electronic device read the one or more lines of program code from the computer-readable storage medium and execute the one or more lines of program code, causing the electronic device to perform the image processing method described above.

[0256] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0257] Those skilled in the art will understand that all or part of the steps of the above embodiments are implemented by hardware, or by a program instructing related hardware to implement them. The program is stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0258] The above description is only an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: The temporal features of the image sequence are obtained based on the order of at least two images in the image sequence; The first image and the second image in the image sequence are downsampled to obtain the features of the first image and the second image, where the first image is one image in the image sequence and the second image is the image in the image sequence other than the first image; the features of the first image and the second image are concatenated to obtain a first feature; the first feature is convolved to obtain the displacement of the pixel in the second image relative to the corresponding pixel in the first image; The features of the first image and the second image, as well as the displacement of pixels in the second image relative to the corresponding pixels in the first image, are subjected to deformable convolution processing to obtain the spatial features of the second image; the features of the first image are used as the spatial features of the first image. The temporal and spatial features of any image in the image sequence are concatenated to obtain the initial image features of the image; the initial image features of the second image are fused with the initial image features of the first image to obtain the image features of the second image, and the initial image features of the first image are used as the image features of the first image; The image features of at least two images in the image sequence are concatenated, and the concatenated features are convolved to obtain target features. The image is reconstructed based on the target features to obtain a target image, and the target image contains the content of the at least two images in the image sequence.

2. The method according to claim 1, characterized in that, The method further includes: Based on a filtered cascaded deformable convolutional network, the features of the first image and the second image are processed to obtain the spatial features of the image sequence.

3. The method according to claim 2, characterized in that, The filtered cascaded deformable convolutional network includes at least two cascaded deformable convolutional networks. The filter-based cascaded deformable convolutional network processes the features of the first image and the second image to obtain the spatial features of the image sequence, including: The features of the first image and the second image are input into a deformable convolutional network, and the deformable convolutional network processes the input features to obtain the second feature; The features of the first image and the second image are denoised, and the denoised features are input into other layers of deformable convolutional networks. The other layers of deformable convolutional networks process the input features to obtain the third feature. Based on the second feature and the third feature, the spatial features of the second image are obtained; The features of the first image are used as the spatial features of the first image.

4. The method according to claim 1, characterized in that, The method further includes: Based on the optical flow estimation network, the features of the first image and the second image are processed to obtain the spatial features of the image sequence.

5. The method according to claim 1, characterized in that, The method further includes: For any image in the image sequence, the temporal features and spatial features of the image are concatenated to obtain the image features of the image; The target features are obtained by weighting the image features of at least two images in the image sequence.

6. The method according to claim 1, characterized in that, The step of obtaining the temporal features of the image sequence based on the order of at least two images in the image sequence includes: Based on any one of a recurrent neural network, a recurrent neural network, or a long short-term memory neural network that includes at least two residual dense blocks, at least two images in the image sequence are processed to obtain the temporal features of the image sequence.

7. The method according to claim 1, characterized in that, The method further includes: The image sequence is input into an image processing network. The temporal module of the image processing network obtains the temporal features of the image sequence, and the spatial module of the image processing network obtains the spatial features of the image sequence. Based on the temporal and spatial features of the image sequence, the image processing network reconstructs the image and outputs the target image.

8. A method for training an image processing network, characterized in that, The method includes: Based on an image processing network, the temporal features of the sample image sequence are obtained according to the order of at least two sample images in the sample image sequence; Based on the image processing network, the first sample image and the second sample image in the sample image sequence are downsampled to obtain the features of the first sample image and the second sample image. The first sample image is a sample image in the sample image sequence, and the second sample image is a sample image in the sample image sequence other than the first sample image. The features of the first sample image and the second sample image are concatenated to obtain a first feature. The first feature is then convolved to obtain the displacement of a pixel in the second sample image relative to the corresponding pixel in the first sample image. Based on the image processing network, deformable convolution processing is performed on the features of the first sample image and the second sample image, as well as the displacement of pixels in the second sample image relative to the corresponding pixels in the first sample image, to obtain the spatial features of the second sample image; the features of the first sample image are used as the spatial features of the first sample image. Based on the image processing network, the temporal and spatial features of any sample image in the sample image sequence are connected to obtain the initial image features of the sample image; the initial image features of the second sample image are fused with the initial image features of the first sample image to obtain the image features of the second sample image, and the initial image features of the first sample image are used as the image features of the first sample image. Based on the image processing network, the image features of at least two sample images in the sample image sequence are connected, the connected features are convolved to obtain target features, and the image is reconstructed based on the target features to obtain the predicted image. The image processing network is trained based on the similarity between the predicted image and the target image corresponding to the sample image sequence.

9. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the temporal features of the image sequence based on the order of at least two images in the image sequence; The acquisition module is further configured to downsample the first image and the second image in the image sequence to obtain features of the first image and the second image, wherein the first image is an image in the image sequence and the second image is an image in the image sequence other than the first image; to concatenate the features of the first image and the second image to obtain a first feature; and to perform convolution processing on the first feature to obtain the displacement of a pixel in the second image relative to the corresponding pixel in the first image. The acquisition module is further configured to perform deformable convolution processing on the features of the first image and the second image, as well as the displacement of pixels in the second image relative to corresponding pixels in the first image, to obtain the spatial features of the second image; and use the features of the first image as the spatial features of the first image. The reconstruction module is used to connect the temporal and spatial features of any image in the image sequence to obtain the initial image features of the image; to fuse the initial image features of the second image with the initial image features of the first image to obtain the image features of the second image; and to use the initial image features of the first image as the image features of the first image. The reconstruction module is further configured to connect the image features of at least two images in the image sequence, perform convolution processing on the connected features to obtain target features, and perform image reconstruction based on the target features to obtain a target image, wherein the target image contains the content of at least two images in the image sequence.

10. The apparatus according to claim 9, characterized in that, The acquisition module is also used for: Based on a filtered cascaded deformable convolutional network, the features of the first image and the second image are processed to obtain the spatial features of the image sequence.

11. The apparatus according to claim 10, characterized in that, The filtered cascaded deformable convolutional network comprises at least two cascaded deformable convolutional networks; the acquisition module is used for: The features of the first image and the second image are input into a deformable convolutional network, and the deformable convolutional network processes the input features to obtain the second feature; The features of the first image and the second image are denoised, and the denoised features are input into other layers of deformable convolutional networks. The other layers of deformable convolutional networks process the input features to obtain the third feature. Based on the second feature and the third feature, the spatial features of the second image are obtained; The features of the first image are used as the spatial features of the first image.

12. The apparatus according to claim 9, characterized in that, The acquisition module is also used for: Based on the optical flow estimation network, the features of the first image and the second image are processed to obtain the spatial features of the image sequence.

13. The apparatus according to claim 9, characterized in that, The reconstruction module is also used for: For any image in the image sequence, the temporal features and spatial features of the image are concatenated to obtain the image features of the image; The target features are obtained by weighting the image features of at least two images in the image sequence.

14. The apparatus according to claim 9, characterized in that, The acquisition module is used for: Based on any one of a recurrent neural network, a recurrent neural network, or a long short-term memory neural network that includes at least two residual dense blocks, at least two images in the image sequence are processed to obtain the temporal features of the image sequence.

15. The apparatus according to claim 9, characterized in that, The acquisition module and the reconstruction module are used for: The image sequence is input into an image processing network. The temporal module of the image processing network obtains the temporal features of the image sequence, and the spatial module of the image processing network obtains the spatial features of the image sequence. Based on the temporal and spatial features of the image sequence, the image processing network reconstructs the image and outputs the target image.

16. An image processing network training device, characterized in that, The device includes: The acquisition module is used to acquire the temporal features of the sample image sequence based on the image processing network and according to the order of at least two sample images in the sample image sequence; The acquisition module is configured to downsample the first sample image and the second sample image in the sample image sequence based on the image processing network to obtain features of the first sample image and the second sample image, wherein the first sample image is a sample image in the sample image sequence and the second sample image is a sample image in the sample image sequence other than the first sample image; concatenate the features of the first sample image and the second sample image to obtain a first feature; and perform convolution processing on the first feature to obtain the displacement of a pixel in the second sample image relative to the corresponding pixel in the first sample image. The acquisition module is used to perform deformable convolution processing on the features of the first sample image and the second sample image, as well as the displacement of pixels in the second sample image relative to the corresponding pixels in the first sample image, based on the image processing network, to obtain the spatial features of the second sample image; and to use the features of the first sample image as the spatial features of the first sample image. The reconstruction module is used to connect the temporal and spatial features of any sample image in the sample image sequence based on the image processing network to obtain the initial image features of the sample image; fuse the initial image features of the second sample image with the initial image features of the first sample image to obtain the image features of the second sample image; and use the initial image features of the first sample image as the image features of the first sample image. The reconstruction module is further configured to connect the image features of at least two sample images in the sample image sequence based on the image processing network, perform convolution processing on the connected features to obtain target features, and perform image reconstruction based on the target features to obtain a predicted image; The training module is used to train the image processing network based on the similarity between the predicted image and the target image corresponding to the sample image sequence.

17. An electronic device, characterized in that, The electronic device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the image processing method as described in any one of claims 1 to 7, or the image processing network training method as described in claim 8.

18. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer program, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1 to 7, or the image processing network training method as described in claim 8.

19. A computer program product, characterized in that, The computer program product includes one or more lines of program code stored in a computer-readable storage medium. One or more processors of an electronic device read the one or more lines of program code from the computer-readable storage medium and execute the one or more lines of program code, causing the electronic device to perform the image processing method as described in any one of claims 1 to 7, or to perform the image processing network training method as described in claim 8.

Images