Image processing method, device, equipment and readable storage medium

By acquiring and accumulating image correlation features and combining the image reconstruction model, the problem of low image quality in the existing technology is solved, and high-quality image reconstruction and time-domain stability are achieved.

CN114693551BActive Publication Date: 2024-11-01TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210300650.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-25
Publication Date
2024-11-01
Estimated Expiration
2042-03-25

AI Technical Summary

Technical Problem

The prior art is not high when denoising and reconstructing images with low pixel sample counts, and the features extracted by neural network models are single, which cannot meet the requirements of high-quality rendering.

Method used

By acquiring the correlation features of the target image and the historical image, generating mixed embedded features and performing feature accumulation, image reconstruction is carried out in combination with image reconstruction models, and time domain information is fully utilized.

Benefits of technology

Improve image reconstruction quality, suitable for high and low pixel sampling images, and enhance the time domain stability and quality of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114693551B_ABST
    Figure CN114693551B_ABST
Patent Text Reader

Abstract

The present application discloses an image processing method, apparatus, device and readable storage medium, the method comprising: obtaining a target image and a historical image in a target image sequence; obtaining a target image association feature corresponding to the target image, and a historical image association feature corresponding to the historical image; generating a first hybrid embedding feature for the target image according to the target image association feature, generating a second hybrid embedding feature for the historical image according to the target image association feature and the historical image association feature, accumulating features of the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature to obtain a target accumulation feature corresponding to the target image; determining a reconstructed result image corresponding to the target image according to the target image association feature, the target accumulation feature and the historical image. The present application can be used to improve the image quality of the reconstructed image in the image rendering business.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, device and readable storage medium. Background Art

[0002] With the rapid development of computer technology, game production, animation production and virtual reality technology are gradually emerging, and the requirements for rendering images (reconstructed noise-free images) of virtual scenes (such as game scenes, animation scenes, social scenes, etc.) are getting higher and higher. However, in actual rendering, the rendered images often have serious noise, so it is necessary to denoise the images and reconstruct the rendering results in the image space.

[0003] Existing denoising (i.e., removing noise from an image) and reconstruction methods are all aimed at images with a high number of samples per pixel (SPP) (e.g., SPP greater than 1), and mainly use some neural network models (e.g., Transformer network models) to denoise rendered images. For example, the existing method mainly inputs a scene rendering image with a high SPP into a neural network model, extracts the features of the current scene image through the neural network model, and then infers based on the extracted features and outputs a noise-free reconstructed image corresponding to the current image.

[0004] It can be seen that the existing methods are highly targeted. For images with low SPP, the image quality obtained by the existing methods is not high (the noise removal effect is not ideal); at the same time, the features of the current image extracted by automatic learning of the neural network model are relatively simple and can only represent one aspect of the current image. The quality of the image inferred and output based on this simple feature may not meet the image quality requirements. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device, and readable storage medium, which can improve the image quality of a reconstructed image in an image rendering service.

[0006] An embodiment of the present application provides an image processing method, including:

[0007] A target image and a historical image are obtained in a target image sequence; the target image sequence is an image sequence obtained after image acquisition of a target scene based on the target unit pixel sampling number; the historical image refers to the previous image of the target image in the target image sequence;

[0008] Obtaining target image associated features corresponding to the target image and historical image associated features corresponding to the historical image;

[0009] Generate a first hybrid embedding feature for the target image according to the target image association feature, generate a second hybrid embedding feature for the historical image according to the target image association feature and the historical image association feature, and perform feature accumulation of the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature to obtain a target accumulated feature corresponding to the target image;

[0010] According to the target image associated features, target accumulated features and historical images, a reconstructed result image corresponding to the target image is determined.

[0011] The present application provides another image processing method, including:

[0012] A target sample noise image and a historical sample noise image are obtained in a sample image sequence; the sample image sequence is an image sequence obtained after image acquisition of a sample scene based on a first unit pixel sampling number; the historical sample noise image refers to a previous sample noise image of the target sample noise image in the sample image sequence; the first unit pixel sampling number is less than a unit pixel sampling number threshold;

[0013] Obtaining target sample image correlation features corresponding to the target sample noise image and historical sample image correlation features corresponding to the historical sample noise image;

[0014] Generate a first sample mixed embedding feature for the target sample noise image according to the target sample image association feature, generate a second sample mixed embedding feature for the historical sample noise image according to the target sample image association feature and the historical sample image association feature, and perform feature accumulation on the target sample noise image and the historical sample noise image based on the first sample mixed embedding feature and the second sample mixed embedding feature to obtain a target sample accumulated feature corresponding to the target sample noise image;

[0015] Determine the target sample fusion cascade feature corresponding to the target sample noise image according to the target sample image association feature, the target sample accumulation feature and the historical sample noise image;

[0016] The target sample fusion cascade feature is input into the image reconstruction model, and in the image reconstruction model, a target sample reconstructed image corresponding to the target sample noise image is output according to the target sample fusion cascade feature;

[0017] Obtaining a target label sampling image corresponding to the target sample noise image; the target label sampling image refers to an image obtained after image acquisition of the sample scene based on the second unit pixel sampling number; the first unit pixel sampling number is less than the second unit pixel sampling number;

[0018] The image reconstruction model is trained according to the target label sampling image and the target sample reconstructed image to obtain a target image reconstruction model for performing image reconstruction processing on the target image in the target image sequence.

[0019] An embodiment of the present application provides an image processing device, including:

[0020] An image acquisition module is used to acquire a target image and a historical image in a target image sequence; the target image sequence is an image sequence obtained after image acquisition of a target scene based on the target unit pixel sampling number; the historical image refers to the previous image of the target image in the target image sequence;

[0021] A feature acquisition module, used to acquire target image associated features corresponding to the target image, and historical image associated features corresponding to the historical image;

[0022] A feature generation module, used for generating a first hybrid embedding feature for a target image according to a target image associated feature;

[0023] The feature generation module is further used to generate a second hybrid embedding feature for the historical image according to the target image associated feature and the historical image associated feature;

[0024] A feature accumulation module, used to accumulate features of the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature to obtain a target accumulated feature corresponding to the target image;

[0025] The image reconstruction module is used to determine the reconstruction result image corresponding to the target image according to the target image association features, target cumulative features and historical images.

[0026] In one embodiment, the target image associated features include a first normal dimension feature and a first depth dimension feature corresponding to the target image;

[0027] The feature generation module includes:

[0028] A first feature input unit, used to input a first normal dimension feature and a first depth dimension feature into a temporal accumulation model;

[0029] A first feature convolution unit, configured to perform convolution processing on the first normal dimension feature and the first depth dimension feature through a first convolutional network layer of a temporal accumulation model to obtain a first image embedding feature corresponding to the target image;

[0030] The first feature convolution unit is further used to perform convolution processing on the first normal dimension feature and the first depth dimension feature through a second convolution network layer of the temporal accumulation model to obtain a second image embedding feature corresponding to the target image;

[0031] The first feature operation unit is used to perform pixel multiplication operation on the first image embedding feature and the second image embedding feature to obtain a first mixed embedding feature for the target image.

[0032] In one embodiment, the target image associated features include a first normal dimension feature and a first depth dimension feature corresponding to the target image, and the historical image associated features include a second normal dimension feature and a second depth dimension feature corresponding to the historical image;

[0033] The feature generation module includes:

[0034] A feature transformation unit, used to perform affine transformation processing on the second normal dimension feature and the second depth dimension feature respectively, to obtain a normal transformation feature corresponding to the second normal dimension feature, and a depth transformation feature corresponding to the second depth dimension feature;

[0035] A second feature input unit, used for inputting the first normal dimension feature, the first depth dimension feature, the normal transformation feature and the depth transformation feature into the temporal accumulation model;

[0036] A second feature convolution unit is used to perform convolution processing on the first normal dimension feature and the first depth dimension feature through the first convolution network layer of the temporal accumulation model to obtain a target image embedding feature corresponding to the target image;

[0037] The second feature convolution unit is further used to perform convolution processing on the normal transformation feature and the depth transformation feature through the second convolution network layer of the temporal accumulation model to obtain the historical image embedding feature corresponding to the historical image;

[0038] The second feature operation unit is used to perform pixel multiplication operation on the target image embedding feature and the historical image embedding feature to obtain a second mixed embedding feature for the historical image.

[0039] In one embodiment, the feature accumulation module includes:

[0040] A fusion coefficient determination unit, used to obtain a logistic regression function, and determine a first feature fusion coefficient for a target image through the logistic regression function and the first hybrid embedding feature;

[0041] The fusion coefficient determination unit is further used to determine the second feature fusion coefficient for the historical image through a logistic regression function and a second mixed embedding feature;

[0042] The feature accumulation unit is used to accumulate features of the target image and the historical image according to the first feature fusion coefficient and the second feature fusion coefficient to obtain the target accumulated feature corresponding to the target image.

[0043] In one embodiment, the feature accumulation unit includes:

[0044] An image feature acquisition subunit is used to acquire target noise image features, target shadow features and target albedo features corresponding to the target image, and to acquire historical accumulated noise image features, historical accumulated shadow features and historical accumulated albedo features corresponding to the historical image;

[0045] A shadow accumulation subunit, configured to accumulate the target shadow feature and the historical accumulated shadow feature according to the first feature fusion coefficient and the second feature fusion coefficient, to obtain the target accumulated shadow feature corresponding to the target image;

[0046] A noise accumulation subunit, configured to accumulate the target noise image feature and the historical noise image feature according to the first feature fusion coefficient and the second feature fusion coefficient, to obtain a target accumulated noise image feature corresponding to the target acoustic image;

[0047] an albedo accumulation subunit, configured to accumulate the target albedo feature and the historical accumulated albedo feature according to the first feature fusion coefficient and the second feature fusion coefficient, so as to obtain the target accumulated albedo feature corresponding to the target image;

[0048] The cumulative feature determination subunit is used to determine the target cumulative shadow feature, the target cumulative noise image feature and the target cumulative albedo feature as the target cumulative feature corresponding to the target image.

[0049] In one embodiment, the image reconstruction module comprises:

[0050] A cascade feature determination unit, used to determine a target fusion cascade feature corresponding to the target image according to the target image association feature, the target cumulative feature and the historical image;

[0051] The reconstructed image determination unit is used to input the target fusion cascade features into the target image reconstruction model, and in the target image reconstruction model, output a reconstructed result image corresponding to the target image according to the target fusion cascade features.

[0052] In one embodiment, the target image associated features include a first normal dimension feature and a first depth dimension feature; the historical image associated features include a second normal dimension feature and a second depth dimension feature corresponding to the historical image;

[0053] The cascade feature determination unit includes:

[0054] A feature fusion subunit is used to obtain auxiliary features corresponding to the target image; the auxiliary features include a first normal dimension feature and a first depth dimension feature;

[0055] The feature fusion subunit is further used to determine the dimensional features other than the first normal dimensional feature and the first depth dimensional feature in the auxiliary features as the remaining dimensional features;

[0056] The feature fusion subunit is further used to fuse the target accumulated features, the first normal dimension features, the first depth dimension features and the remaining dimension features to obtain an image fusion feature corresponding to the target image;

[0057] A feature transformation processing subunit, used to perform affine transformation processing on the second normal dimension feature and the second depth dimension feature respectively, to obtain a normal transformation feature corresponding to the second normal dimension feature, and a depth transformation feature corresponding to the second depth dimension feature;

[0058] The feature cascade subunit is used to cascade the image fusion features, normal transformation features and depth transformation features to obtain the target fusion cascade features corresponding to the target image.

[0059] In one embodiment, the reconstructed image determination unit comprises:

[0060] The parameter output subunit is used to determine the predicted high-quality image, the predicted low-quality image, the first image reconstruction parameter and the second image reconstruction parameter corresponding to the target fusion cascade feature through the target image reconstruction model; the predicted high-quality image does not contain noise, and the resolution of the predicted high-quality image is greater than the resolution of the predicted low-quality image;

[0061] An image fusion subunit, used to fuse the predicted high-quality image with the predicted low-quality image in a target image reconstruction model by using a first image reconstruction parameter to obtain an initial reconstructed image corresponding to the target image;

[0062] The image fusion subunit is further used to obtain a historical reconstructed image corresponding to the historical image, and to fuse the initial reconstructed image with the historical reconstructed image through a second image reconstruction parameter to obtain a reconstructed result image corresponding to the target image.

[0063] On the one hand, an embodiment of the present application provides another image processing device, including:

[0064] The sample image acquisition module is used to acquire a target sample noise image and a historical sample noise image in a sample image sequence; the sample image sequence is an image sequence obtained after image acquisition of a sample scene based on a first unit pixel sampling number; the historical sample noise image refers to a sample noise image previous to the target sample noise image in the sample image sequence; the first unit pixel sampling number is less than a unit pixel sampling number threshold;

[0065] A sample feature acquisition module, used to acquire target sample image correlation features corresponding to the target sample noise image, and historical sample image correlation features corresponding to the historical sample noise image;

[0066] A sample feature generation module, used to generate a first sample mixed embedding feature for a target sample noise image according to a target sample image association feature;

[0067] The sample feature generation module is further used to generate a second sample mixed embedding feature for the historical sample noise image according to the target sample image association feature and the historical sample image association feature;

[0068] A sample feature accumulation module, used to accumulate features of a target sample noise image and a historical sample noise image based on the first sample mixed embedding feature and the second sample mixed embedding feature, so as to obtain a target sample accumulated feature corresponding to the target sample noise image;

[0069] A sample feature cascade module is used to determine the target sample fusion cascade feature corresponding to the target sample noise image according to the target sample image association feature, the target sample accumulation feature and the historical sample noise image;

[0070] A sample reconstruction image output module is used to input the target sample fusion cascade feature into the image reconstruction model, and in the image reconstruction model, output a target sample reconstructed image corresponding to the target sample noise image according to the target sample fusion cascade feature;

[0071] The label image acquisition module is used to acquire a target label sampling image corresponding to the target sample noise image; the target label sampling image refers to an image obtained after image acquisition of the sample scene based on the second unit pixel sampling number; the first unit pixel sampling number is less than the second unit pixel sampling number;

[0072] The model training module is used to train the image reconstruction model according to the target label sampling image and the target sample reconstructed image, and obtain the target image reconstruction model used for image reconstruction processing of the target image in the target image sequence.

[0073] In one embodiment, the model training module includes:

[0074] The image to be calculated determination unit is used to determine the sample reconstructed image corresponding to the remaining sample noise image in the sample image sequence as the reconstructed image to be calculated, and determine the label sampled image corresponding to the remaining sample noise image as the label sampled image to be calculated; the remaining sample noise image refers to the sample noise image other than the target sample noise image in the sample image sequence;

[0075] A loss value determination unit, used to determine a target loss value for the image reconstruction model according to the reconstructed image to be calculated, the label sampled image to be calculated, the target label sampled image and the target sample reconstructed image;

[0076] The model training unit is used to train the image reconstruction model according to the target loss value to obtain the target image reconstruction model.

[0077] In one embodiment, the loss value determination unit includes:

[0078] A loss value determination subunit, used to determine a spatial loss value for an image reconstruction model according to a reconstructed image to be calculated, a label sampling image to be calculated, a target label sampling image, and a target sample reconstructed image;

[0079] The loss value determination subunit is further used to determine the temporal loss value for the image reconstruction model according to the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image;

[0080] The loss value determination subunit is further used to determine the relative edge loss value for the image reconstruction model according to the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image;

[0081] The target loss value determination subunit is also used to determine the target loss value for the image reconstruction model based on the spatial loss value, the temporal loss value and the relative edge loss value.

[0082] In one embodiment, the loss value determination subunit is further specifically used to obtain a spatial loss function, and determine a first spatial sub-loss value for the remaining sample noise image according to the spatial loss function, the reconstructed image to be calculated, and the label sampling image to be calculated;

[0083] The loss value determination subunit is further specifically used to determine a second spatial sub-loss value for the target sample noise image according to the spatial loss function, the target label sampling image, and the target sample reconstructed image;

[0084] The loss value determination subunit is also specifically used to fuse the first spatial sub-loss value with the second spatial sub-loss value to obtain the spatial loss value of the image reconstruction model.

[0085] In one embodiment, the loss value determination subunit is further specifically used to obtain a first training weight for the residual sample noise image and a second training weight for the target sample noise image;

[0086] The loss value determination subunit is further specifically used to perform operation processing on the first training weight and the first spatial sub-loss value to obtain the first operation space sub-loss value;

[0087] The loss value determination subunit is further specifically used to perform operation processing on the second training weight and the second space sub-loss value to obtain the second operation space sub-loss value;

[0088] The loss value determination subunit is also specifically used to add the first operation space sub-loss value and the second operation space sub-loss value to obtain the spatial loss value of the image reconstruction model.

[0089] In one embodiment, the target sample noise image is the last sample noise image in the sample image sequence;

[0090] The target loss value determination subunit is further specifically used to obtain a historical sample reconstructed image and a historical label sampling image corresponding to the historical sample noise image, and determine an affine transformation loss value for the image reconstruction model according to the historical sample reconstructed image, the target sample reconstructed image, the historical label sampling image and the target label sampling image;

[0091] The target loss value determination subunit is further specifically used to obtain the target sample albedo feature corresponding to the target sample noise image, obtain the target sample cumulative albedo feature corresponding to the target sample noise image in the target sample cumulative feature, and determine the albedo loss value for the image reconstruction model according to the albedo loss function, the target sample albedo feature and the target sample cumulative albedo feature;

[0092] The target loss value determination subunit is also specifically used to fuse the spatial loss value, the temporal loss value, the relative edge loss value, the affine transformation loss value and the albedo loss value to obtain the target loss value of the image reconstruction model.

[0093] An embodiment of the present application provides a computer device, including: a processor and a memory;

[0094] The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method in the embodiment of the present application.

[0095] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the method in the embodiment of the present application is executed.

[0096] In one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in one aspect of the embodiments of the present application.

[0097] In an embodiment of the present application, after image acquisition of the target scene is performed using the target unit pixel sampling number, the resulting target image sequence, whether it is a high pixel sampling number or a low pixel sampling number, can be based on the image association features of the two images in the target image sequence (which can be referred to as the target image and the historical image, where the target image can be the current image and the historical image is the previous image of the current image) to calculate the mixed embedding features of the two images, and based on the mixed embedding features, the features of the two images are accumulated to obtain the accumulated features which can be used as the target accumulated features corresponding to the target image; subsequently, the reconstructed image corresponding to the target image can be determined based on the image association features of the target image, the target accumulated features and the historical images. It can be understood that the present application can calculate the hybrid embedded features corresponding to the two images respectively according to the associated features of the current image and the previous frame image, and accumulate features of the two images based on the hybrid embedded features. The obtained target accumulated features include the features of the previous and next frames after fusion. The fused features can characterize the correlation between the previous and next frames, and the features contain a wider dimension. At the same time, when determining the reconstructed result image of the target image, it is also determined based on the target image associated features of the target image, the target accumulated features and the previous frame image (historical image). When determining the reconstructed result image of the target image, not only the features of the current frame are taken into account, but also the features of the historical frames before time. The time domain information of the current frame can be fully utilized, and the obtained reconstructed result image has higher time domain stability and higher image quality. In summary, regardless of whether the image is obtained with a high pixel sampling number or a low pixel sampling number, the present application can accumulate features based on the image correlation features of the two frames before and after in the target image sequence, and determine the reconstructed result image of the target image together with the accumulated features and the historical images, which has universal applicability; by calculating the accumulated features of the two frames before and after, and then reconstructing the image based on the accumulated features, the method can make full use of the time domain information of the image, thereby greatly improving the image quality of the reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0099] Figure 1 This is a network architecture diagram provided by an embodiment of the present application;

[0100] Figure 2 is a schematic diagram of a scene for reconstructing an image provided by an embodiment of the present application;

[0101] Figure 3 is a method flow chart of an image processing method provided by an embodiment of the present application;

[0102] Figure 4 It is a flow chart of a model training process provided by an embodiment of the present application;

[0103] Figure 5 It is a system architecture diagram provided by an embodiment of the present application;

[0104] Figure 6 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0105] Figure 7 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0106] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0107] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0108] This application involves artificial intelligence and other related technologies. For ease of understanding, the following will give priority to explaining related concepts such as artificial intelligence.

[0109] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0110] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0111] The solution provided in the embodiment of the present application belongs to the computer vision technology (Computer Vision, CV) and machine learning (Machine Learning, ML) under the field of artificial intelligence.

[0112] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to identify and measure targets, and further perform graphics processing to make computer processing into images that are more suitable for observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies.

[0113] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0114] In addition, before further describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application will be explained below. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0115] 1) Three-dimensional scene: A scene in a three-dimensional space constructed based on three-dimensional modeling technology. Objects in the three-dimensional scene (i.e., three-dimensional objects) can be described by three-dimensional coordinates. Among them, three-dimensional coordinates may refer to coordinates in a three-dimensional coordinate system including an x-axis, a y-axis, and a z-axis. In some embodiments, a three-dimensional scene may be a virtual scene (or a three-dimensional virtual scene). A virtual scene is a scene that is different from the real world and output by an electronic device. Visual perception of the virtual scene can be formed with the assistance of the naked eye or specific equipment. A virtual scene can be a simulation of the real world, a semi-simulated and semi-fictional virtual environment, or a purely fictional virtual environment.

[0116] 2) Rendering: In the embodiment of the present application, it refers to the process of rendering an image to a human-computer interaction interface (such as the human-computer interaction interface provided by a browser) to present the image in the human-computer interaction interface.

[0117] 3) Pixel: refers to the element in the image that cannot be further divided.

[0118] 4) Image channel: used to describe the pixels in the image. In the embodiment of the present application, the image channel may include at least one of a color channel and a transparency channel (also called an Alpha channel). The color channel is used to describe the color of the pixel. There are differences in the color channels in different color spaces (also called color modes). For example, the color channels in the RGB color space include a red channel, a green channel, and a blue channel; for example, the color channels in the HSL color space include a hue channel, a saturation channel, and a lightness channel. The transparency channel is used to describe the transparency of the pixel.

[0119] See also Figure 1 , Figure 1 This is a network architecture diagram provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a service server 1000 and a terminal device cluster (i.e., a terminal device cluster). The terminal device cluster may include one or more terminal devices, and the number of terminal devices is not limited here. Figure 1 As shown, the multiple terminal devices may specifically include terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n. Figure 1As shown, the terminal device 100a, the terminal device 100b, the terminal device 100c, ..., the terminal device 100n can be respectively connected to the above-mentioned business server 1000 through a network, so that each terminal device can exchange data with the business server 1000 through the network connection. The network connection here does not limit the connection method, and can be directly or indirectly connected through a wired communication method, directly or indirectly connected through a wireless communication method, or through other methods, which are not limited in this application.

[0120] like Figure 1 Each terminal device shown in the figure can be integrated with a target application. When the target application is running in each terminal device, the background server corresponding to each terminal device can store the business data in the application and Figure 1 Data interaction is performed between the business servers 1000 shown. Among them, the target application may include an application with data information functions such as displaying text, images, audio and video. For example, the application may be a multimedia application (such as a video application), which can be used for users to upload pictures or videos, or for users to play and watch images or videos uploaded by others; the application may also be an entertainment application (such as a game application), which can be used for users to play games. The application may also be other applications with data information processing functions, such as browser applications, real-life scene simulation applications, social applications, etc., and examples of the applications will not be given one by one here. The target application may also be a small program, that is, an independent program that only needs to be downloaded to the browser environment to run. Of course, the target application may be an independent application, or a sub-application (such as a small program) embedded in a certain application, which can be controlled to run or closed by the user. In short, the target application can be an application, module or plug-in in any form. For game applications, it can be any type of first-person game (such as first-person shooter game, First-Person Shooting, FPS), third-person game, multiplayer online tactical competitive game (Multiplayer Online Battle Arena, MOBA) and multiplayer survival game, without limitation.

[0121] In the embodiment of the present application, a terminal device can be selected from multiple terminal devices as the target terminal device, and the terminal device can include: smart phones, tablet computers, laptops, desktop computers, smart TVs, smart speakers, desktop computers, smart watches, smart cars and other smart terminals with data processing functions (such as image data processing functions), but are not limited to these. Figure 1The terminal device 100a shown is used as the target terminal device. The target application mentioned above can be integrated into the target terminal device. At this time, the target terminal device can exchange data with the business server 1000.

[0122] The business server 1000 in the present application can interact with terminal devices (such as target terminal devices) through these applications. For example, after the target terminal device performs data acquisition and processing on the three-dimensional scene to obtain the data to be rendered, the business server 1000 can store the data to be rendered in the image buffer of the business server 1000. Then, the business server 1000 can send the data to be rendered to the target terminal device, and the target terminal device can render the received data to be rendered. The business server 1000 can send the rendered image obtained after the rendering process to the target terminal device, and the target terminal device can remove noise and reconstruct the received rendered image to present the reconstructed image in the human-computer interaction interface of the target terminal device, and the user can view the reconstructed image through the human-computer interaction interface. Of course, the business server 1000 can also remove noise and reconstruct the rendered image after the rendering process, and then send the reconstructed image to the target terminal device, so that the target terminal device does not need to perform reconstruction processing, but directly presents the reconstructed image in the human-computer interaction interface.

[0123] In some embodiments, the terminal device can calculate the data required for display through image computing hardware (such as GPU), and complete the loading, parsing, rendering and reconstruction of display data (such as scene images), and output images that can form visual perception of three-dimensional scenes with the help of image output hardware (such as screen), for example, presenting rendered and reconstructed scene images on the display screen of a smartphone. Taking the target terminal device as an example, after the target terminal device performs image acquisition processing on the three-dimensional scene to obtain the data to be rendered, the target terminal device can render and reconstruct the data to be rendered, and present the reconstructed scene image (i.e., the reconstructed image) in the human-computer interaction interface.

[0124] In some embodiments, the business server 1000 or the terminal device can perform data collection and processing on the three-dimensional scene when the three-dimensional scene meets the visual change condition to obtain the data to be rendered of the three-dimensional scene. The visual change condition will not be limited here. For example, when a trigger operation for a three-dimensional scene is received in the human-computer interaction interface, it can be determined that the visual change condition is met. When the visual change condition is met, the terminal device can perform data collection and processing on the three-dimensional scene being presented, or send relevant data of the three-dimensional scene being presented to the business server 1000, so that the business server performs data collection and processing on the three-dimensional scene being presented. Among them, the trigger operation here can be a contact operation, such as a click operation or a long press operation, etc.; it can also be a non-contact operation, such as a voice input operation or a gesture input operation, etc. In other words, the data to be rendered (also referred to as business data) can be business data triggered by a user through a trigger operation in the target application. For example, when a user uses a target application (such as a game application) in a target terminal device, the user generates a trigger operation for opening a parachute control in the game application, and the trigger operation satisfies the visual change condition. The terminal device can obtain the business data triggered by opening the parachute control based on the trigger action for opening the parachute control. The business data is the data to be rendered corresponding to the opening of the parachute control in the game scene. The terminal device or the business server can render it to obtain a rendering layer (also called a rendering image, for example, an image after the parachute is opened) with a certain picture quality.

[0125] It is understandable that a set of rendered image sequences can usually be obtained through image rendering, and when the present application reconstructs the current frame in the image sequence, the reconstructed result image of the current frame can be determined according to the current frame and the previous frame. The specific implementation method can be referred to in the subsequent Figure 3 The description in the corresponding embodiment.

[0126] It is understandable that the method provided in the embodiments of the present application can be executed by a computer device, including but not limited to a terminal device or a business server. Among them, the business server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0127] The terminal device and the service server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0128] Optionally, it can be understood that the above-mentioned computer device (such as the above-mentioned business server 1000, terminal device 100a, terminal device 100b, etc.) can be a node in a distributed system, wherein the distributed system can be a blockchain system, and the blockchain system can be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes can form a peer-to-peer (P2P, Peer To Peer) network, and the P2P protocol is an application layer protocol running on the transmission control protocol (TCP, Transmission Control Protocol) protocol. In a distributed system, any form of computer equipment, such as business servers, terminal devices and other electronic devices, can become a node in the blockchain system by joining the peer-to-peer network. For ease of understanding, the concept of blockchain will be explained below: Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism and encryption algorithm, which is mainly used to organize data in chronological order and encrypt it into an account book so that it cannot be tampered with or forged, and can verify, store and update data at the same time. When the computer device is a blockchain node, due to the tamper-proof and anti-counterfeiting characteristics of the blockchain, the data in the present application (such as the collected data to be rendered, the rendered scene image and the reconstructed image, etc.) can be authentic and secure, so that the results obtained after relevant data processing based on these data can be more reliable.

[0129] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. For ease of understanding, please refer to Figure 2 , Figure 2 is a schematic diagram of a scene for reconstructing an image provided by an embodiment of the present application. Figure 2 The business server 2000 shown can be the above Figure 1 The business server 1000 shown in FIG. Figure 2 The terminal device 2a shown can be Figure 1 Any terminal device selected from the terminal device cluster of the corresponding embodiment, for example, the terminal device may be the above-mentioned terminal device 100b.

[0130] like Figure 2 As shown, user a can be the target user. User a can start and run the target application (here a multiplayer competitive game application is used as an example) through terminal device 2a. Terminal device 2a needs to present the game scene screen of the multiplayer competitive game (hereinafter referred to as the target game) in the display interface. Figure 2As shown, after the terminal device 2a collects data of the scene of the target game and renders it, usually, these rendered images are often accompanied by noise. In order to improve the quality of the presented game image, a series of image processing such as noise removal and reconstruction of the rendering result can be performed on the rendered image. In the embodiment of the present application, the terminal device 2a can send the rendered image sequence to the business server 2000, and the business server 2000 performs noise removal and reconstruction on these images.

[0131] like Figure 2 As shown, the image sequence 200 can be an image sequence obtained after the terminal device 2a performs image acquisition and rendering on the target game scene. The terminal device 2a can send the image sequence 200 to the business server 2000, and the business server 2000 can reconstruct the image sequence. Among them, when reconstructing each image in the image sequence, the image features of the image and the image of the previous image can be used to jointly predict and output the reconstructed result image of the image. For ease of understanding, when reconstructing an image, the image can be referred to as the current frame (or the current image, or the target image), and the previous image of the current frame in the image sequence can be referred to as the historical frame (or the historical image).

[0132] like Figure 2 As shown, taking the image 20a in the image sequence 200 as the target image and the previous image 20b of the image 20b as the historical image as an example, the specific process for predicting the reconstructed result image of the output target image 20a can be the following process. First, the business server 2000 can obtain the image-related features of the target image 20a (which can be referred to as the target image-related features). It can be understood that when the target game scene is imaged and rendered, during the image rendering process, the auxiliary features corresponding to each image can be obtained at the same time (specifically, they can include normal features, depth features, albedo features, motion vector features, metalness features, and roughness and other dimensional features). The image-related features here can refer to some features of the auxiliary features obtained during the image rendering process (such as normal dimension features and depth dimension features), so the target image-related features here can specifically include the normal dimension features and depth dimension features corresponding to the target image 20a. The business server 2000 can also obtain image association features of the historical image 20b (for ease of distinction, they can be referred to as historical image association features). Similarly, the historical image association features can specifically include normal dimension features and depth dimension features corresponding to the historical image 20b.

[0133] Furthermore, the business server 2000 can generate a hybrid embedding feature (embedding feature, which can be referred to as the first hybrid embedding feature) for the target image 20a based on the target image association feature; it can also generate a hybrid embedding feature (embedding feature, which can be referred to as the second hybrid embedding feature for the sake of distinction) for the historical image 20b based on the historical image association feature and the target image. According to the above-mentioned first hybrid embedding feature and the second hybrid embedding feature, the business server 2000 can accumulate features of the target image 20a and the historical image 20b, and the accumulated features (which can be referred to as accumulated features) can be used as the accumulated features of the target image 20a (which can be referred to as target accumulated features). The target accumulated features fuse the features of the historical image and the features of the target image, and can reflect the correlation between the two images to a certain extent. Subsequently, the reconstructed result image of the target image 20a can be determined based on the target accumulated features, the above-mentioned target image association features and the historical image. Among them, for the specific implementation methods of determining the hybrid embedding features, accumulating features based on the hybrid embedding features, and determining the reconstructed result image based on the target accumulated features, please refer to the subsequent Figure 3 The description in the corresponding embodiment.

[0134] The above only describes a method of predicting the reconstructed result image of the current frame by taking image 20a as the current frame. For each image in the image sequence 200, the business server 2000 can determine its corresponding reconstructed result image. The business server can return each reconstructed result image in the image sequence 200 to the terminal device 2a, and the terminal device 2a can display the reconstructed result images in chronological order (in fact, the reconstructed result images can constitute a reconstructed video, and what is displayed by the terminal device 2a can actually be understood as a high-quality game video).

[0135] For further information, see Figure 3 , Figure 3 is a flowchart of an image processing method provided by an embodiment of the present application. The method can be performed by a terminal device (for example, the above Figure 1 or Figure 2 The terminal device shown in the figure) can also be executed by the service server (as shown in the figure above) Figure 1 The image processing method may be executed by the service server 1000 in the corresponding embodiment, or may be executed by the terminal device and the service server together. For ease of understanding, this embodiment takes the method executed by the above service server as an example for explanation. The image processing method may include at least the following steps S101 to S103:

[0136] Step S101, obtaining a target image and a historical image in a target image sequence; the target image sequence is an image sequence obtained after image acquisition of a target scene based on a target unit pixel sampling number; the historical image refers to the previous image of the target image in the target image sequence.

[0137] In this application, the target scene may refer to a virtual scene, wherein the virtual scene may be a simulation environment of the real world, a semi-simulation and semi-fictional virtual environment, or a purely fictional virtual environment. For example, a game virtual scene of a game application may be used as the target scene.

[0138] In the present application, data collection can be performed on the target scene, and the collected data to be rendered can be rendered. The target image sequence in the present application can refer to the rendered image sequence obtained after the collected data to be rendered is rendered. Among them, data collection for the target scene can include: data collection can be performed on the target scene being presented in the human-computer interaction interface (such as the terminal display interface presenting the game scene in the terminal device) to obtain the data to be rendered in the current field of view (referring to the field of view when presented) or a specific field of view; or when the target scene is not presented, the target scene can be collected and processed according to a specific field of view to obtain the data to be rendered, wherein the field of view used in the data collection and processing process can be preset or automatically determined. For example, when the target application is a game application, the user can run the target application by clicking the start control of the target application. Among them, the start control can be understood as a target control. When the user clicks on the start control, the terminal device can respond to the user's click trigger operation on the start control, and obtain the business data corresponding to the start control (that is, the data to be rendered, for example, the home page data of the game application). The data to be rendered is the data to be rendered obtained by collecting data for the game scene.

[0139] It can be understood that in rendering, the present application can sample and render the target scene based on the target unit pixel sampling number, wherein the target unit pixel sampling number here can refer to the number of samples per pixel (Sample Per Pixel, SPP). The target unit pixel sampling number in the present application can refer to a value lower than the unit pixel sampling number threshold, wherein the unit pixel sampling number threshold can be an artificially specified value (such as 0.3, 0.5...), and when the value of the unit pixel sampling number threshold is set to be small (such as less than 1), the image acquisition and rendering of the target scene based on the target unit pixel sampling number can actually be understood as sparse sampling. That is, the target image sequence in the present application can refer to the rendered image sequence obtained after sampling and rendering the target scene using sparse sampling. When the value of the unit pixel sampling number threshold is set to be small (such as less than 1), the target unit pixel sampling number can be a value less than 1. It should be noted that when the target unit pixel sampling number is less than 1, it can be characterized that some pixels will not be collected for the target scene.

[0140] The target image in this application may refer to any image in the target image sequence, and the historical image may refer to the previous image of the target image in the target image sequence. When the target image is the first image in the target image sequence, it can be considered that there is no historical image for the target image.

[0141] Step S102, obtaining target image associated features corresponding to the target image and historical image associated features corresponding to the historical image.

[0142] In the present application, a renderer (such as a mixed light renderer) can be used to render the target scene and generate a target image sequence. During rendering, the scene can be rasterized, thereby obtaining the image auxiliary features corresponding to each rendered image, wherein the image auxiliary features can specifically include normal dimension features, depth dimension features, motion vectors, albedo dimensions, metalness dimension features, roughness dimension features, shadow features, transparency dimension features, and the like. The image association features here may refer to some features in the image auxiliary features (such as normal dimension features and depth dimension features), then the image association features corresponding to the target image may be referred to as target image association features, and the image association features corresponding to the historical image may be referred to as historical image association features. In other words, the image association features of an image in the present application can be determined based on its corresponding image auxiliary features, and the image association features may refer to some features in the image auxiliary features (the specific features included may be artificially specified, such as normal dimension features and depth dimension features) or all features, and the image auxiliary features may refer to the image features of each rendered image obtained during the image acquisition and rendering of the target scene. Taking the target image as an example, the target image can be obtained by rendering the target scene, and during the rendering process, the image features corresponding to the target image (such as the normal dimension features, depth dimension features, metalness dimension features, etc.) can also be obtained. The image features of the target image obtained during the rendering process can be used as the image auxiliary features of the target image (which can be called target image auxiliary features), and the target image associated features can be some features (or all features) in the image auxiliary features of the target image.

[0143] Step S103, generating a first hybrid embedding feature for the target image according to the target image association feature, generating a second hybrid embedding feature for the historical image according to the target image association feature and the historical image association feature, and accumulating features of the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature to obtain a target accumulated feature corresponding to the target image.

[0144] In the present application, it can be known from the above that the image association features may include normal dimension features and depth dimension features. For the convenience of distinction, the normal dimension features of the target image may be referred to as the first normal dimension features, and the depth dimension features of the target image may be referred to as the first depth dimension features. Then, the target image association features may include the first normal dimension features and the first depth dimension features corresponding to the target image. The specific implementation method for generating the first mixed embedding features for the target image according to the target image association features may be as follows: the first normal dimension features and the first depth dimension features may be input into the temporal accumulation model; through the first convolutional network layer of the temporal accumulation model, the first normal dimension features and the first depth dimension features may be convolved to obtain the first image embedding features corresponding to the target image; through the second convolutional network layer of the temporal accumulation model, the first normal dimension features and the first depth dimension features may be convolved to obtain the second image embedding features corresponding to the target image; then, the first image embedding features and the second image embedding features may be processed by pixel multiplication operations to obtain the first mixed embedding features for the target image.

[0145] Similarly, the normal dimension feature of the historical image can be called the second normal dimension feature, and the depth dimension feature of the historical image can be called the second depth dimension feature. Then the historical image associated features can include the second normal dimension feature and the second depth dimension feature corresponding to the historical image. The specific implementation method for generating the second hybrid embedding feature for the historical image based on the target image associated features and the historical image associated features is as follows: the second normal dimension feature and the second depth dimension feature can be affine transformed respectively to obtain the normal transformation feature corresponding to the second normal dimension feature and the depth transformation feature corresponding to the second depth dimension feature; then, the first normal dimension feature, the first depth dimension feature, the normal transformation feature and the depth transformation feature can be input into the temporal accumulation model; through the first convolutional network layer of the temporal accumulation model, the first normal dimension feature and the first depth dimension feature can be convolved to obtain the target image embedding feature corresponding to the target image (in fact, the target image embedding feature and the above-mentioned first image embedding feature can be the same feature); through the second convolutional network layer of the temporal accumulation model, the normal transformation feature and the depth transformation feature can be convolved to obtain the historical image embedding feature corresponding to the historical image; then, the target image embedding feature and the historical image embedding feature can be pixel multiplied to obtain the second hybrid embedding feature for the historical image.

[0146] It can be understood that the present application can calculate the mixed embedding features corresponding to the current frame image (i.e., the target image) and the previous frame image (i.e., the historical image) based on the normal dimension features and depth dimension features corresponding to the two images respectively, and accumulate features of the two images based on these two mixed embedding features.

[0147] Among them, for accumulating features of the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature, the specific implementation method to obtain the target accumulated feature corresponding to the target image can be: a logistic regression function can be obtained, and the first feature fusion coefficient for the target image can be determined through the logistic regression function and the first hybrid embedding feature; the second feature fusion coefficient for the historical image can be determined through the logistic regression function and the second hybrid embedding feature; according to the first feature fusion coefficient and the second feature fusion coefficient, the target image and the historical image are feature accumulated to obtain the target accumulated feature corresponding to the target image.

[0148] Among them, according to the first feature fusion coefficient and the second feature fusion coefficient, the target image and the historical image can be feature accumulated to obtain the target accumulated features corresponding to the target image. The specific implementation method can be as follows: the target noise image features, target shadow features and target albedo features corresponding to the target image can be obtained, and the historical accumulated noise image features, historical accumulated shadow features and historical accumulated albedo features corresponding to the historical image can be obtained; according to the first feature fusion coefficient and the second feature fusion coefficient, the target shadow features can be accumulated with the historical accumulated shadow features to obtain the target accumulated shadow features corresponding to the target image; according to the first feature fusion coefficient and the second feature fusion coefficient, the target noise image features can be accumulated with the historical noise image features to obtain the target accumulated noise image features corresponding to the target acoustic image; according to the first feature fusion coefficient and the second feature fusion coefficient, the target albedo features can be accumulated with the historical accumulated albedo features to obtain the target accumulated albedo features corresponding to the target image; then, the target accumulated shadow features, the target accumulated noise image features and the target accumulated albedo features can be determined as the target accumulated features corresponding to the target image. That is, the target accumulated features can include the target accumulated shadow features, the target accumulated noise image features and the target accumulated albedo features.

[0149] It can be understood that the temporal accumulation model in the present application may refer to a temporal accumulator (TemporalAccumulator), in which two neural networks may be included, each of which may include two layers of convolutional networks (Convolutional Neural Network, CNN), the above-mentioned first convolutional network layer may refer to one of the neural networks in the temporal accumulator, and the second convolutional network layer may refer to another neural network in the temporal accumulator. After acquiring the target image associated features and the historical image associated features, the target image associated features (i.e., the first normal dimension features and the first depth dimension features) may be input into a neural network (such as the first convolutional network layer) in the temporal accumulator, and the first convolutional network layer may output embedding features for the target image associated features. For ease of distinction, the embedding features output by the first convolutional network layer may be referred to as the first image embedding features corresponding to the target image. As for the other neural network (the second convolutional network layer) of the time series accumulator, the second normal dimension feature and the second depth dimension feature of the historical image can first be subjected to image affine transformation processing (warp) to obtain the normal transformation feature and the depth transformation feature; then, the first normal dimension feature, the first depth dimension feature, the normal transformation feature and the depth transformation feature can be input into another neural network (i.e., the second convolutional network layer), and the embedding features corresponding to the first normal dimension feature and the first depth dimension feature can be output through the second convolutional network layer (it should be noted that, due to the different network structures of the two neural networks, although the two neural networks have the same input, the output embedding features may still be different; the embedding features can be referred to as the second image embedding features corresponding to the target image), and the embedding features corresponding to the normal transformation features and the depth transformation features (which can be referred to as the historical image embedding features of the historical image) can also be output through the second convolutional network layer.

[0150] Furthermore, the first image embedding feature and the second image embedding feature can be multiplied at the pixel level (i.e., pixel multiplication operation processing, which can be understood as pixel-to-pixel multiplication operation processing to mix the two image embedding features), and the obtained operation result can be used as the new mixed embedding feature corresponding to the target image (i.e., the first mixed embedding feature). That is to say, the two network convolutional layers of the temporal accumulation model can respectively output two different (or the same) embedding features (embedding features, which can be referred to as image embedding features in this application) for the associated features of the target image, and the two embedding features can be mixed by pixel-level calculation to obtain a new mixed embedding feature. Since the mixed embedding feature is the result of mixing, it can be referred to as the first mixed embedding feature, that is, the first mixed embedding feature refers to the mixed embedding feature obtained by mixing the first image embedding feature (generated based on the associated features of the target image) and the second image embedding feature (generated based on the associated features of the target image); similarly, the first image embedding feature and the historical image embedding feature can also be multiplied at the pixel level (i.e., pixel multiplication operation Processing), the operation result obtained can be used as a new mixed embedding feature corresponding to the historical image (i.e., the second mixed embedding feature), that is, through one of the network convolutional layers of the temporal accumulation model, two embedding features for the target image associated features and the historical image associated features can be output respectively, and the two embedding features can be mixed by pixel-level calculation to obtain a new mixed embedding feature. Since the mixed embedding feature is a mixed result, it can be called a second mixed embedding feature, that is, the second mixed embedding feature refers to the mixed embedding feature obtained by mixing the target image embedding feature (generated based on the target image associated features, the target image embedding feature and the above-mentioned first image embedding feature can be the same embedding feature) with the historical image embedding feature. It should be understood that when accumulating features of the target image and the historical image in the future, a mixing factor is needed for accumulation, and the above-mentioned first mixed embedding feature and the second mixed embedding feature are mainly used to determine the mixing factor when accumulating features. For example, a logistic regression function (such as a softmax function) can be called, and the first mixed embedded feature and the second mixed embedded feature can be respectively subjected to logistic regression through the logistic regression function. The result obtained after performing logistic regression on the first mixed embedded feature can be used as the mixing factor corresponding to the target image (which can be called the first feature fusion coefficient); the result obtained after performing logistic regression on the second mixed embedded feature can be used as the mixing factor corresponding to the historical image (which can be called the second feature fusion coefficient).According to the first feature fusion coefficient and the second feature fusion coefficient, features of the target image and the historical images may be accumulated to obtain target accumulated features corresponding to the target image.

[0151] When accumulating features of the target image and the historical image, shadow features, noisy images, and albedo may be accumulated. The specific implementation of feature accumulation based on the first feature fusion coefficient and the second feature fusion coefficient may be as shown in formula (1):

[0152]

[0153] Among them, as shown in formula (1) It can be used to characterize the cumulative features corresponding to the current frame image (target image, i.e., the tth frame) (it can refer to the target cumulative shadow features; it can also refer to the target cumulative noise image features; it can also refer to the target cumulative albedo features); It can refer to the cumulative features up to the previous frame image (i.e. historical image, frame t-1) (when When accumulating shadow features for a target, the It can refer to the historical accumulated shadow features up to frame t-1; when When accumulating noise image features for the target, the It can refer to the historical accumulated noise image features up to frame t-1; when When the target accumulates albedo characteristics, the It can refer to the historical accumulated albedo characteristics up to frame t-1); f s Can be used to characterize the relevant features of the current frame image (when When accumulating shadow features for the target, the f s It can refer to the target shadow feature of the tth frame; when When the noise image features are accumulated for the target, the f s It can refer to the target noise image feature of the tth frame; when When the target cumulative albedo characteristic is s May refer to the target albedo feature of the tth frame). α in formula (1) may refer to the above-mentioned first feature fusion coefficient, β may refer to the above-mentioned second feature fusion coefficient; W() may be a warping operator, which represents the use of motion vectors to reproject the previous frame image to the current frame image. Among them, since the image auxiliary features of each image can be obtained during the image rendering process, the target shadow features, target albedo features, and target noise image features can all be directly obtained from the image auxiliary features of the target image. It should be noted that when the target image is an image containing noise, the target noise image feature may refer to the target image itself. And when the target image is the first image in the target image sequence, the That is, the accumulated feature is its own feature, such as the target accumulated shadow feature is the target shadow feature of the target image.

[0154] It can be understood that, through the above formula (1), the target cumulative albedo feature, the target cumulative shadow feature and the target cumulative noise image feature corresponding to the target image can be obtained.

[0155] Step S104: determining a reconstruction result image corresponding to the target image according to the target image associated features, the target accumulated features and the historical images.

[0156] In the present application, determining the reconstruction result image corresponding to the target image according to the target image association features, target cumulative features and historical images can be understood as: performing image reconstruction processing on the target image according to the target image association features, target cumulative features and historical images to obtain a reconstruction result image (that is, the reconstruction result image is actually the image obtained after the target image is reconstructed). That is to say, in the process of image reconstruction processing on the target image, the present application requires not only the target image association features and target cumulative features of the target image itself, but also the relevant information of the historical image. For determining the reconstruction result image corresponding to the target image according to the target image association features, target cumulative features and historical images, the specific implementation method can be: the target fusion cascade features corresponding to the target image can be determined according to the target image association features, target cumulative features and historical images; then, the target fusion cascade features can be input into the target image reconstruction model, and in the target image reconstruction model, the reconstruction result image corresponding to the target image can be output according to the target fusion cascade features.

[0157] Among them, the specific implementation method for determining the target fusion cascade features corresponding to the target image according to the target image association features, target cumulative features and historical images may be as follows: the auxiliary features corresponding to the target image may be obtained; wherein the auxiliary features include the first normal dimension features and the first depth dimension features; then, the dimensional features other than the first normal dimension features and the first depth dimension features in the auxiliary features may be determined as the remaining dimensional features; then, the target cumulative features, the first normal dimension features, the first depth dimension features and the remaining dimensional features may be fused to obtain the image fusion features corresponding to the target image; then, the second normal dimension features and the second depth dimension features may be affine transformed respectively to obtain the normal transformation features corresponding to the second normal dimension features, and the depth transformation features corresponding to the second depth dimension features; the image fusion features, the normal transformation features and the depth transformation features may be cascaded to obtain the target fusion cascade features corresponding to the target image.

[0158] It is understandable that after accumulating image features containing noise (noise image features), shadow features and albedo, the accumulated target accumulated noise image features, target accumulated shadow features and target accumulated albedo features can be combined with the normal dimension features, depth dimension features, transparency dimension features, metalness dimension features and roughness dimension features of the target image through a feature fusion device (Feature Fusion) to perform feature fusion to obtain the image fusion features corresponding to the target image. It should be understood that when the number of samples per target pixel is small, the sampling rendering of the target scene can actually be understood as sparse sampling rendering, so the obtained target image and various features (such as target accumulated shadow features, etc.) are also sparse. Through the feature fusion device, the features of each dimension can be fused, and the image fusion features obtained can perceive information in different spaces. Among them, the method for feature fusion includes but is not limited to splicing on the channel dimension. Furthermore, the image fusion features corresponding to the target image can be cascaded with the normal dimension features (i.e., normal transformation features) of the historical image after warping, and the depth dimension features (depth transformation features), thereby obtaining cascade features (which can be called target fusion cascade features).

[0159] Furthermore, the target fusion cascade feature can be input into the target image reconstruction model, and the target image reconstruction model can output the reconstruction result image corresponding to the target image. The specific implementation method can be as follows: through the target image reconstruction model, the predicted high-quality image, the predicted low-quality image, the first image reconstruction parameter and the second image reconstruction parameter corresponding to the target fusion cascade feature can be determined; wherein, the predicted high-quality image does not contain noise, and the resolution of the predicted high-quality image is greater than the resolution of the predicted low-quality image; then, in the target image reconstruction model, the predicted high-quality image and the predicted low-quality image can be image-fused through the first image reconstruction parameter to obtain the initial reconstructed image corresponding to the target image; then, the historical reconstructed image corresponding to the historical image can be obtained, and the initial reconstructed image and the historical reconstructed image can be image-fused through the second image reconstruction parameter to obtain the reconstruction result image corresponding to the target image.

[0160] It can be understood that the specific implementation method for determining the initial reconstructed image of the target image through the target image reconstruction model can be as shown in formula (2):

[0161]

[0162] Among them, as shown in formula (2), O pCan be used to characterize the initial reconstructed image corresponding to the target image; d f It can be used to characterize the predicted high-quality image (high-resolution image) predicted by the target image reconstruction model; d c It can be used to characterize the predicted low-quality image (low-resolution image) predicted by the target image reconstruction model; D and U can be used to characterize the 2X times downsampling and nearest neighbor upsampling operations respectively; α s It can be used to characterize the reconstruction parameters of the first image. The subscript p of each parameter in formula (2) can be used to characterize a pixel (i.e., the operation is performed in pixels). The method shown in formula (2) mainly adopts a multi-scale reconstruction method to mix high and low resolution images to obtain an initial reconstructed image corresponding to the target image.

[0163] It can be understood that the target image reconstruction model can predict the target image corresponding to the image with higher resolution after noise removal, and this image can be used as the predicted high-quality image; the target image reconstruction model can also output two channel parameters α s With α t For image reconstruction, the parameter α s It can be called the first image reconstruction parameter, parameter α t It can be called the second image reconstruction parameter; through the target image reconstruction model, it is also possible to predict a 2X downsampled image with a lower resolution from the last layer (such as from the last layer of the predicted high-quality image), and this image can be used as the predicted low-quality image. s As well as the above formula (2), the high-resolution and low-resolution images can be mixed, and the obtained mixed result can be used as the initial reconstructed image of the target image.

[0164] Furthermore, based on another parameter α O (i.e., the second image reconstruction parameter), the final reconstruction result image of the target image can be determined. The specific implementation method can be shown in formula (3):

[0165] O t =α t O p +(1.0-α t ) t-1 Formula (3)

[0166] Among them, as shown in formula (3), O t It can be used to represent the reconstructed image corresponding to the target image; α t Can be used to characterize the second image reconstruction parameter; t-1 The reconstructed image corresponding to the historical image (i.e., the historical reconstructed image) can be used to represent the current output image. p Reconstruct the image with history at αt Linear mixture of factors.

[0167] It should be understood that the embodiment of the present application can calculate the correlation of each pixel between the normal feature and the depth feature of the current frame and the previous frame, and the weight factor (the first feature fusion coefficient and the second feature fusion coefficient) can be calculated based on the correlation between the two frames before and after. Based on the weight factor, the features of the two images before and after can be fused and accumulated, so that the features finally input to the reconstruction model include the common features of the two frames before and after, and the feature dimensions are richer and the content is deeper; based on the fusion features and the accumulated features, an interpolated multi-scale image (initial reconstructed image) can be obtained, and the reconstructed image of the current frame can be reconstructed by interpolating the multi-scale image (initial reconstructed image) and the historical frame image, and the information of the historical image can be further utilized. Even if the target unit pixel sampling number is small, the application can still make full use of the time domain information under the low SPP sampling image, and reconstruct the image obtained by sparse sampling in real time. Thus, when outputting the image, the output is a high-quality reconstruction result image (that is, the reconstruction result image is the output image). For example, when an image of a target scene (such as a game scene) is output in the human-computer interaction interface of a terminal device, a high-quality reconstruction result image that has been processed by image reconstruction is output.

[0168] In an embodiment of the present application, the hybrid embedded features corresponding to the two images can be calculated based on the associated features of the current image and the previous frame image, and the features of the two images can be accumulated based on the hybrid embedded features. The obtained target accumulated features include the features of the previous and next frames after fusion. The fused features can characterize the correlation between the previous and next frames, and the features include a wider dimension. At the same time, when determining the reconstructed result image of the target image, it is also determined based on the target image associated features of the target image, the target accumulated features and the previous frame image (historical image). When determining the reconstructed result image of the target image, not only the features of the current frame are taken into account, but also the features of the previous historical frames are taken into account. The time domain information of the current frame can be fully utilized, and the obtained reconstructed result image has higher time domain stability and higher image quality. In summary, regardless of whether the image is obtained with a high pixel sampling number or a low pixel sampling number, the present application can accumulate features based on the image correlation features of the two frames before and after in the target image sequence, and determine the reconstructed result image of the target image together with the accumulated features and the historical images, which has universal applicability; by calculating the accumulated features of the two frames before and after, and then reconstructing the image based on the accumulated features, the method can make full use of the time domain information of the image, thereby greatly improving the image quality of the reconstructed image.

[0169] From the above, it can be known that after determining the target fusion cascade features of the target image, the target fusion cascade features can be input into the target image reconstruction model, and the predicted reconstructed image corresponding to the target fusion cascade features (i.e., the reconstructed result image corresponding to the target image) can be determined through the target image reconstruction model. In order to optimize the reconstruction effect after the target image reconstruction model performs image reconstruction processing (i.e., improve the image quality of the model output image), the initial image reconstruction model can be trained and adjusted so that the Mubai image reconstruction model obtained after the training and adjustment reaches the optimal value. Based on the trained target image reconstruction model, image reconstruction processing can be performed. For ease of understanding, please refer to Figure 4 , Figure 4 1 is a flow chart of a model training process provided by an embodiment of the present application. The process may be a specific process for training and adjusting an image reconstruction model to obtain a target image reconstruction model. Figure 4 As shown, the process may at least include the following steps S301 to S307:

[0170] Step S301, obtaining a target sample noise image and a historical sample noise image in a sample image sequence; the sample image sequence is an image sequence obtained after image acquisition of a sample scene based on a first unit pixel sampling number; the historical sample noise image refers to a sample noise image previous to the target sample noise image in the sample image sequence; the first unit pixel sampling number is less than a unit pixel sampling number threshold.

[0171] Specifically, the sample scene may refer to a virtual scene, for example, a virtual scene of a game application may be used as a sample scene. Data may be collected for the sample scene, and the collected sample data to be rendered may be rendered. The sample image sequence in the present application may refer to a rendered image sequence containing noise (which may be referred to as a sample noise image sequence) obtained by rendering the collected sample data to be rendered. The first unit pixel sampling number here may refer to the number of samples per pixel (Sample Per Pixel, SPP). The first unit pixel sampling number in the present application may refer to a value lower than a unit pixel sampling number threshold, wherein the unit pixel sampling number threshold may be an artificially specified value (such as 0.3, 0.5...), and when the value of the unit pixel sampling number threshold is set to be small (such as less than 1), the image acquisition and rendering of the sample scene based on the first unit pixel sampling number can actually be understood as sparse sampling. That is, the sample image sequence in the present application may refer to a rendered image sequence containing noise obtained after sampling and rendering the sample scene using sparse sampling. When the value of the unit pixel sampling number threshold is set to be small (such as less than 1), the first unit pixel sampling number may be a value less than 1. It should be noted that when the first unit pixel sampling number is less than 1, it can be represented that for the sample scene, some pixels will not be collected. In a feasible embodiment, the first unit pixel sampling number in the present application and the above-mentioned target unit pixel sampling number may be the same value.

[0172] The target sample noise image may refer to any sample noise image in the sample image sequence, and the historical sample noise image may refer to the previous sample noise image of the target sample noise image in the sample image sequence.

[0173] Step S302, obtaining target sample image correlation features corresponding to the target sample noise image and historical sample image correlation features corresponding to the historical sample noise image.

[0174] Specifically, like the above-mentioned target image association features and historical image association features, the target sample image association features may include the normal dimension features (which may be referred to as the first sample normal dimension features) and the depth dimension features (which may be referred to as the first sample depth dimension features) corresponding to the target sample noise image; the historical sample image association features may include the normal dimension features (which may be referred to as the second sample normal dimension features) and the depth dimension features (which may be referred to as the second sample depth dimension features) corresponding to the historical sample noise image.

[0175] For the specific method of obtaining the associated features of the target sample image and the associated features of the historical sample image, please refer to the above Figure 3 In the corresponding embodiment, the description of obtaining the target image correlation features and the historical image correlation features will not be repeated here.

[0176] Step S303, generating a first sample mixed embedding feature for the target sample noise image according to the target sample image associated feature, generating a second sample mixed embedding feature for the historical sample noise image according to the target sample image associated feature and the historical sample image associated feature, and accumulating features of the target sample noise image and the historical sample noise image based on the first sample mixed embedding feature and the second sample mixed embedding feature to obtain a target sample accumulated feature corresponding to the target sample noise image.

[0177] Specifically, the second sample normal dimension feature and the second sample depth dimension feature can be subjected to image affine transformation processing to obtain the sample normal transformation feature and the sample depth transformation feature; the first sample normal dimension feature and the first sample depth dimension feature can be input into the temporal accumulation model, and an embedding feature (which can be called the first sample image embedding feature) can be output through the first convolutional network layer of the temporal accumulation model; the sample normal transformation feature, the sample depth transformation feature, the first sample normal dimension feature and the first sample depth dimension feature can also be input into the temporal accumulation model, and an embedding feature (which can be called the historical sample image embedding feature) corresponding to the sample normal transformation feature and the sample depth transformation feature can be output through the second convolutional network layer of the temporal accumulation model, and an embedding feature (which can be called the second sample image embedding feature) corresponding to the first sample normal dimension feature and the first sample depth dimension feature can also be output; similarly, based on the first sample image embedding feature and the second sample image embedding feature, the first sample mixed embedding feature of the target sample noise image can be determined; based on the first sample image embedding feature and the historical sample image embedding feature, the second sample mixed embedding feature of the historical sample noise image can also be determined. Based on the first sample mixed embedding feature and the second sample mixed embedding feature, the target sample noise image and the historical sample noise image may be feature accumulated to obtain the target sample accumulated feature corresponding to the target sample noise image.

[0178] For the specific process of determining the mixed embedding feature of the first sample, please refer to the above Figure 3 For the description of determining the first mixed embedding feature in the corresponding embodiment; for the specific process of determining the second sample mixed embedding feature, please refer to the above Figure 3 In the corresponding embodiment, for the description of determining the second mixed embedding feature; for the specific method of accumulating the target sample noise image and the historical sample noise image based on the first sample mixed embedding feature and the second sample mixed embedding feature to obtain the target sample accumulated feature corresponding to the target sample noise image, please refer to the above Figure 3In the corresponding embodiment, based on the first hybrid embedding feature and the second hybrid embedding feature, the target image and the historical image are subjected to feature accumulation to obtain a description of the target accumulated feature, which will not be described in detail here.

[0179] Step S304, determining the target sample fusion cascade feature corresponding to the target sample noise image according to the target sample image association feature, the target sample accumulation feature and the historical sample noise image.

[0180] Specifically, the target sample cumulative feature may also include the target sample cumulative shadow feature, the target sample cumulative noise image feature and the target sample cumulative albedo feature. The specific method of determining the target sample fusion cascade feature corresponding to the target sample noise image according to the target sample image association feature, the target sample cumulative feature and the historical sample noise image can be referred to above. Figure 3 In the corresponding embodiment, the description of determining the target fusion cascade feature according to the target image association feature, the target cumulative feature and the historical image will not be repeated here.

[0181] Step S305 , inputting the target sample fusion cascade features into the image reconstruction model, and in the image reconstruction model, outputting the target sample reconstructed image corresponding to the target sample noise image according to the target sample fusion cascade features.

[0182] Specifically, the image reconstruction model may refer to a model to be trained, in which a target sample reconstructed image corresponding to the target sample noise image may be output based on the model parameters to be optimized. For a specific method of outputting a target sample reconstructed image corresponding to the target sample noise image according to the target sample fusion cascade feature in the image reconstruction model, please refer to the above Figure 3 In the corresponding embodiment, the description of outputting a reconstructed result image corresponding to the target image according to the target fusion cascade features in the target image reconstruction model will not be repeated here.

[0183] Step S306, obtaining a target label sampling image corresponding to the target sample noise image; the target label sampling image refers to an image obtained after image acquisition of the sample scene based on the second unit pixel sampling number; the first unit pixel sampling number is less than the second unit pixel sampling number.

[0184] Specifically, the target label sampling image may refer to an image obtained after image acquisition of the sample scene based on the second unit pixel sampling number, wherein the second unit pixel sampling number may refer to a value lower than the unit pixel sampling number threshold, wherein the unit pixel sampling number threshold may be an artificially specified value (such as 0.3, 0.5...), and when the value set for the unit pixel sampling number threshold is small (such as less than 1), the image acquisition and rendering of the sample scene based on the first unit pixel sampling number can actually be understood as sparse sampling. The image acquisition and rendering of the sample scene based on the second unit pixel sampling number can be understood as ordinary sampling (or traditional sampling). It should be understood that sparse sampling may refer to sampling each pixel once (some pixels are not even sampled), and ordinary sampling may refer to sampling each pixel at least once (some pixels are even sampled multiple times). The high-resolution image without noise obtained by ordinary sampling rendering can be used as the label sampling image (i.e., reference image) of this application; the target label sampling image may refer to the reference image corresponding to the target sample noise image, which is a high SPP image (a high-resolution image without noise).

[0185] Step S307 , training an image reconstruction model according to the target label sampling image and the target sample reconstructed image, to obtain a target image reconstruction model for performing image reconstruction processing on the target image in the target image sequence.

[0186] Specifically, the image reconstruction model can be trained based on the target label sampling image and the target sample reconstructed image output by the image reconstruction model. The specific method can be as follows: the sample reconstructed image corresponding to the remaining sample noise image in the sample image sequence can be determined as the reconstructed image to be calculated, and the label sampling image corresponding to the remaining sample noise image can be determined as the label sampling image to be calculated; wherein the remaining sample noise image refers to the sample noise image other than the target sample noise image in the sample image sequence; then, according to the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image, the target loss value for the image reconstruction model can be determined; according to the target loss value, the image reconstruction model can be trained to obtain the target image reconstruction model.

[0187] Among them, the specific implementation method of determining the target loss value for the image reconstruction model based on the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image can be as follows: based on the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image, the spatial loss value for the image reconstruction model can be determined; based on the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image, the temporal loss value for the image reconstruction model can be determined; based on the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image, the relative edge loss value for the image reconstruction model can be determined; based on the spatial loss value, the temporal loss value and the relative edge loss value, the target loss value for the image reconstruction model can be determined.

[0188] Among them, the specific implementation method of determining the spatial loss value for the image reconstruction model based on the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image can be as follows: a spatial loss function can be obtained, and a first spatial sub-loss value for the remaining sample noise image can be determined based on the spatial loss function, the reconstructed image to be calculated and the label sampling image to be calculated; a second spatial sub-loss value for the target sample noise image can be determined based on the spatial loss function, the target label sampling image and the target sample reconstructed image; then, the first spatial sub-loss value can be fused with the second spatial sub-loss value to obtain the spatial loss value of the image reconstruction model. As for fusing the first spatial sub-loss value with the second spatial sub-loss value to obtain the spatial loss value of the image reconstruction model, a specific implementation method may be as follows: a first training weight for the residual sample noise image and a second training weight for the target sample noise image may be obtained; the first training weight and the first spatial sub-loss value may be operated on to obtain a first operation spatial sub-loss value; the second training weight and the second spatial sub-loss value may be operated on to obtain a second operation spatial sub-loss value; and then, the first operation spatial sub-loss value and the second operation spatial sub-loss value may be added to obtain the spatial loss value of the image reconstruction model.

[0189] For ease of understanding, please refer to formula (4), which can be a specific method for determining the space loss value:

[0190]

[0191] Among them, the spatial loss value, temporal loss value and relative edge loss value can all be calculated on a sample image sequence of N consecutive frames (N is a positive integer). The above formula (4) can be a specific method for determining the spatial sub-loss value of a single frame in the sample image sequence. As shown in formula (4), l(r, d) can be used to characterize the spatial sub-loss value (such as the first spatial sub-loss value) corresponding to a single frame (such as a target sample noise image); p can be used to characterize pixels, and N can be the total number of image pixels; c can be used to characterize the number of image color channels (such as for RGB images, the image color channel can contain three channels); d can be used to characterize the sample reconstructed image output by the image reconstruction model (such as the target sample reconstructed image); r can be used to characterize the label sampling image (such as the target label sampling image). ε can be used to characterize the loss parameter, which can be 10 -2 . It should be understood that after obtaining the spatial sub-loss value corresponding to each sample noise image in the sample image sequence, these spatial sub-loss values ​​can be added and summed, and the result obtained can be used as the final spatial loss value for training the model. Optionally, when adding and summing these spatial sub-loss values, different weights can be assigned to different sample noise images in the sample image sequence in advance, and these spatial sub-loss values ​​can be weighted and summed based on these different weights. For example, taking the sample image sequence as 5 frames of images as an example, the allocation method of [0.05, 0.25, 0.5, 0.75, 1] ​​can be used to assign weights to these 5 frames of sample noise images (i.e., as their training weights). After obtaining the spatial sub-loss values ​​corresponding to the 5 frames of sample images, their training weights can be multiplied by the spatial sub-loss values ​​to obtain the computational spatial sub-loss value corresponding to the frame; then, these computational spatial sub-loss values ​​can be added and summed to obtain the final spatial loss value. It should be understood that the training weight corresponding to the target sample noise image may be referred to as the first training weight, and the training weight corresponding to the remaining sample noise image may be referred to as the second training weight.

[0192] The same as the spatial loss value, the same is true for the temporal loss value and the relative edge loss value. The temporal sub-loss value (or relative edge sub-loss value) corresponding to each sample noise image in the sample image sequence can be calculated in turn, and then the temporal sub-loss values ​​(or relative edge sub-loss values) of each sample image are weighted and summed (the training weights of each sample noise image can be pre-assigned) to obtain the final temporal loss value (or relative edge loss value).

[0193] For ease of understanding, please refer to formula (5), which can be a specific method for determining the timing loss value:

[0194]

[0195] Among them, l(△r,△d) as shown in formula (5) can be used to represent the temporal sub-loss value (such as the first temporal sub-loss value) corresponding to a single frame (such as the target sample noise image); the meanings represented by r, d, p, c, and ε can all be referred to in the above formula (4). △ can be used to represent the temporal gradient calculated between two consecutive frames (that is, △r can be used to represent the temporal gradient calculated between the sample reconstructed image of the current frame image and the sample reconstructed image of the previous frame image; △d can be used to represent the temporal gradient calculated between the two previous and next label sampling images).

[0196] For ease of understanding, please refer to formula (6), which can be a specific method for determining the relative edge loss value:

[0197]

[0198] Among them, as shown in formula (6) It can be used to characterize the relative edge sub-loss value (such as the first relative edge sub-loss value) corresponding to a single frame (such as a target sample noise image); the meanings of r, d, p, c, and ε can all be referred to in the above formula (4). It can be used to characterize the gradient calculated using the High Frequency Error Norm (HFEN) The gradient calculated by representing the sample reconstructed image of the current frame image and the sample reconstructed image of the previous frame image; It can be used to characterize the gradient calculated from the two labeled sampling images before and after.

[0199] The above-mentioned spatial loss value, temporal loss value and relative edge loss value can be part of the loss value for training the image reconstruction model. For the image reconstruction model, another part of the loss value can also be included, and the two parts of the loss value are used together to train the image reconstruction model. Another part of the loss value can include an affine transformation loss value and an albedo loss value, that is, the target loss value can be determined based on the spatial loss value, the temporal loss value, the relative edge loss value, the affine transformation loss value and the albedo loss value. Among them, the affine transformation loss value is only calculated on the last two frames in the sample image sequence, and the albedo loss value is only calculated on the last image in the sample image sequence. Here we take the target sample noise image as the last sample noise image in the sample image sequence (then the historical sample noise image and the target sample noise image are the last two images in the sample image sequence) as an example. The specific implementation method for determining the target loss value can be as follows: the historical sample reconstructed image and the historical label sampling image corresponding to the historical sample noise image can be obtained. According to the historical sample reconstructed image, the target sample reconstructed image, the historical label sampling image and the target label sampling image, the affine transformation loss value for the image reconstruction model can be determined; then, the target sample albedo feature corresponding to the target sample noise image can be obtained, and in the target sample cumulative feature, the target sample cumulative albedo feature corresponding to the target sample noise image is obtained. According to the albedo loss function, the target sample albedo feature and the target sample cumulative albedo feature, the albedo loss value for the image reconstruction model can be determined; the spatial loss value, the temporal loss value, the relative edge loss value, the affine transformation loss value and the albedo loss value are fused to obtain the target loss value of the image reconstruction model.

[0200] For ease of understanding, please refer to formula (7), which can be a specific way to determine the affine transformation loss value:

[0201]

[0202] Among them, as in formula (7), l(ωr,ωd) can be used to represent the affine transformation loss value; the meanings of r, d, p, c, and ε can all be referred to in the above formula (4); ωr can be used to represent r 4 -W(r 3 ), where r 4 is the last frame image (target sample noise image), r 3 It can be used to characterize the previous frame image (historical sample noise image). W is a warping operator, which means reprojecting the previous frame to the current frame.

[0203] For ease of understanding, please refer to formula (8), which can be a specific way to determine the affine transformation loss value:

[0204]

[0205] Among them, as shown in formula (8), l(a acc ,a r ) can be used to characterize the albedo loss value; the meanings of r, d, p, c, and ε can all be referred to in the above formula (4); a acc Can be used to characterize the accumulated albedo characteristics (accumulated albedo characteristics of target samples); a r Can be used to characterize the albedo characteristics of the final image.

[0206] Furthermore, we can perform a weighted summation on the above loss values ​​to obtain the target loss value used to train the image reconstruction model. The method for determining the target loss value is shown in formula (9):

[0207] l=0.7l s +0.1l t +0.2l e +0.4l wt +5.0l a Formula (9)

[0208] Among them, l s Can be used to characterize the space loss value; l t Can be used to characterize the timing loss value; l e Can be used to characterize the relative edge loss value; l wt Can have a value that represents the affine transformation loss; l a It can be used to characterize the albedo loss value. It should be noted that the weights of the above-mentioned loss values ​​(such as 0.7, 0.1, 0.2, 0.4, 5.0) are preset values, which are not limited thereto, and can also be other weight values ​​determined by experimental tests, which are not limited thereto.

[0209] In summary, the loss value l can be used to measure the difference between the sample reconstructed image and its reference image. The model parameters of the image reconstruction model can be adjusted by the target loss value l to obtain the optimal estimated parameters. The specific implementation method can be shown in formula (10):

[0210]

[0211] Among them, as in formula (10) It can be used to characterize neural network functions, mainly used to reconstruct images without noise; can be used to characterize the optimal estimation parameters; N can be used to characterize the total number of images in the sample image sequence; c i Can be used to characterize a sample noise image; fi iCan be used to characterize auxiliary features obtained during the rendering process; r i It can be used to characterize high SPP reference images. l can be used to characterize the target loss value. We can use the gradient descent algorithm on the sample image sequence to minimize the loss value l to obtain the optimal estimated parameters, thereby training a deep neural network. The deep neural network It can be deployed in the target image reconstruction model for image reconstruction processing (equivalent to being the target image reconstruction model).

[0212] In an embodiment of the present application, the hybrid embedded features corresponding to the two images can be calculated based on the associated features of the current image and the previous frame image, and the features of the two images can be accumulated based on the hybrid embedded features. The obtained target accumulated features include the features of the previous and next frames after fusion. The fused features can characterize the correlation between the previous and next frames, and the features include a wider dimension. At the same time, when determining the reconstructed result image of the target image, it is also determined based on the target image associated features of the target image, the target accumulated features and the previous frame image (historical image). When determining the reconstructed result image of the target image, not only the features of the current frame are taken into account, but also the features of the previous historical frames are taken into account. The time domain information of the current frame can be fully utilized, and the obtained reconstructed result image has higher time domain stability and higher image quality. In summary, regardless of whether the image is obtained with a high pixel sampling number or a low pixel sampling number, the present application can accumulate features based on the image correlation features of the two frames before and after in the target image sequence, and determine the reconstructed result image of the target image together with the accumulated features and the historical images, which has universal applicability; by calculating the accumulated features of the two frames before and after, and then reconstructing the image based on the accumulated features, the method can make full use of the time domain information of the image, thereby greatly improving the image quality of the reconstructed image.

[0213] For further information, see Figure 5 , Figure 5 This is a system architecture diagram provided by an embodiment of the present application. Figure 5 As shown, the system architecture may include a time series accumulator (also referred to as a time series accumulation model), a feature fusion device, and a target image reconstruction model (also referred to as a reconstruction network). The time series accumulator may be composed of a first convolutional network layer and a second convolutional network layer, and the first convolutional network layer and the second convolutional network layer may each include two convolutional neural networks (CNNs). The feature fusion device may also include multiple convolutional neural networks (CNNs). The target image reconstruction model may be designed as a U-Net structure including a residual module, that is, a codec module is obtained by stacking multiple convolutional layers (convlayer), and a network structure of a residual module is interspersed in the codec module.

[0214] It should be understood that the image-related features (including normal features and depth features) of the current frame and the image-related features (including normal features and depth features) of the historical frame can be input into the time series accumulator, and the noisy image features, shadow features and albedo features of the two frames can be accumulated based on the time domain through the time series accumulator to obtain the accumulated features corresponding to each feature. Each accumulated feature accumulated based on the time series accumulator is input into the feature fusion device, and the accumulated features can be fused with other auxiliary features (such as normal features, transparency dimension features, roughness dimension features, etc.) through the feature fusion device to obtain the image fusion features of the current frame. Subsequently, the fusion features of the current frame and the normal features and depth features of the historical frame after warp denoising can be cascaded to obtain the fused cascade features; the fused cascade features can be input into the reconstruction network (i.e., the target image reconstruction model), and the reconstruction network can output the reconstructed result image of the current frame.

[0215] It should be understood that the present application can render the target scene with a lower number of pixel samplings, and use deep learning methods to reduce the large amount of computing overhead required to render high-pixel sampling images. The present application can still reconstruct sparsely sampled images in real time at high resolutions. Therefore, while ensuring good resolution and frame rate, the target scene can be rendered using sparse sampling rendering, which can greatly reduce computing costs.

[0216] For further information, see Figure 6 , Figure 6 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may be a computer program (including program code) running in a computer device, for example, the image processing device is an application software; the image processing device may be used to execute Figure 3 As shown in the method. Figure 6 As shown, the image processing device 1 may include: an image acquisition module 11 , a feature acquisition module 12 , a feature generation module 13 , a feature accumulation module 14 and an image reconstruction module 15 .

[0217] The image acquisition module 11 is used to acquire a target image and a historical image in a target image sequence; the target image sequence is an image sequence obtained after image acquisition of a target scene based on the target unit pixel sampling number; the historical image refers to the previous image of the target image in the target image sequence;

[0218] A feature acquisition module 12 is used to acquire target image associated features corresponding to the target image and historical image associated features corresponding to the historical image;

[0219] A feature generation module 13, used to generate a first hybrid embedding feature for the target image according to the target image associated feature;

[0220] The feature generation module 13 is further used to generate a second hybrid embedding feature for the historical image according to the target image association feature and the historical image association feature;

[0221] A feature accumulation module 14 is used to accumulate features of the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature to obtain a target accumulated feature corresponding to the target image;

[0222] The image reconstruction module 15 is used to determine the reconstruction result image corresponding to the target image according to the target image association features, target cumulative features and historical images.

[0223] The specific implementation of the image acquisition module 11, the feature acquisition module 12, the feature generation module 13, the feature accumulation module 14 and the image reconstruction module 15 can be found in the above Figure 3 The description of step S101 to step S104 in the corresponding embodiment will not be repeated here.

[0224] In one embodiment, the target image associated features include a first normal dimension feature and a first depth dimension feature corresponding to the target image;

[0225] The feature generation module 13 may include: a first feature input unit 131 , a first feature convolution unit 132 and a first feature operation unit 133 .

[0226] A first feature input unit 131 is used to input the first normal dimension feature and the first depth dimension feature into the temporal accumulation model;

[0227] A first feature convolution unit 132, configured to perform convolution processing on the first normal dimension feature and the first depth dimension feature through a first convolutional network layer of a temporal accumulation model to obtain a first image embedding feature corresponding to the target image;

[0228] The first feature convolution unit 132 is further used to perform convolution processing on the first normal dimension feature and the first depth dimension feature through the second convolution network layer of the temporal accumulation model to obtain a second image embedding feature corresponding to the target image;

[0229] The first feature operation unit 133 is used to perform pixel multiplication operation on the first image embedding feature and the second image embedding feature to obtain a first mixed embedding feature for the target image.

[0230] The specific implementation of the first feature input unit 131, the first feature convolution unit 132 and the first feature operation unit 133 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0231] In one embodiment, the target image associated features include a first normal dimension feature and a first depth dimension feature corresponding to the target image, and the historical image associated features include a second normal dimension feature and a second depth dimension feature corresponding to the historical image;

[0232] The feature generation module 13 may include: a feature transformation unit 134 , a second feature input unit 135 , a second feature convolution unit 136 and a second feature operation unit 137 .

[0233] A feature transformation unit 134 is used to perform affine transformation processing on the second normal dimension feature and the second depth dimension feature respectively to obtain a normal transformation feature corresponding to the second normal dimension feature and a depth transformation feature corresponding to the second depth dimension feature;

[0234] A second feature input unit 135 is used to input the first normal dimension feature, the first depth dimension feature, the normal transformation feature and the depth transformation feature into the temporal accumulation model;

[0235] A second feature convolution unit 136 is used to perform convolution processing on the first normal dimension feature and the first depth dimension feature through the first convolution network layer of the temporal accumulation model to obtain a target image embedding feature corresponding to the target image;

[0236] The second feature convolution unit 136 is further used to perform convolution processing on the normal transformation feature and the depth transformation feature through the second convolution network layer of the temporal accumulation model to obtain the historical image embedding feature corresponding to the historical image;

[0237] The second feature operation unit 137 is used to perform pixel multiplication operation on the target image embedding feature and the historical image embedding feature to obtain a second mixed embedding feature for the historical image.

[0238] The specific implementation of the feature conversion unit 134, the second feature input unit 135, the second feature convolution unit 136 and the second feature operation unit 137 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0239] In one embodiment, the feature accumulation module 14 may include: a fusion coefficient determination unit 141 and a feature accumulation unit 142 .

[0240] A fusion coefficient determination unit 141 is used to obtain a logistic regression function, and determine a first feature fusion coefficient for a target image through the logistic regression function and the first hybrid embedding feature;

[0241] The fusion coefficient determination unit 141 is further used to determine the second feature fusion coefficient for the historical image through the logistic regression function and the second hybrid embedding feature;

[0242] The feature accumulation unit 142 is used to accumulate features of the target image and the historical images according to the first feature fusion coefficient and the second feature fusion coefficient to obtain a target accumulated feature corresponding to the target image.

[0243] The specific implementation of the fusion coefficient determination unit 141 and the feature accumulation unit 142 can be found in the above Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0244] In one embodiment, the feature accumulation unit 142 may include: an image feature acquisition subunit 1421 , a shadow accumulation subunit 1422 , a noise accumulation subunit 1423 , an albedo accumulation subunit 1424 , and a cumulative feature determination subunit 1425 .

[0245] The image feature acquisition subunit 1421 is used to acquire the target noise image feature, the target shadow feature and the target albedo feature corresponding to the target image, and acquire the historical accumulated noise image feature, the historical accumulated shadow feature and the historical accumulated albedo feature corresponding to the historical image;

[0246] The shadow accumulation subunit 1422 is used to accumulate the target shadow feature and the historical accumulated shadow feature according to the first feature fusion coefficient and the second feature fusion coefficient to obtain the target accumulated shadow feature corresponding to the target image;

[0247] The noise accumulation subunit 1423 is used to accumulate the target noise image feature and the historical noise image feature according to the first feature fusion coefficient and the second feature fusion coefficient to obtain the target accumulated noise image feature corresponding to the target acoustic image;

[0248] The albedo accumulation subunit 1424 is used to accumulate the target albedo feature and the historical accumulated albedo feature according to the first feature fusion coefficient and the second feature fusion coefficient to obtain the target accumulated albedo feature corresponding to the target image;

[0249] The cumulative feature determination subunit 1425 is used to determine the target cumulative shadow feature, the target cumulative noise image feature and the target cumulative albedo feature as the target cumulative feature corresponding to the target image.

[0250] The specific implementation of the image feature acquisition subunit 1421, the shadow accumulation subunit 1422, the noise accumulation subunit 1423, the albedo accumulation subunit 1424 and the accumulation feature determination subunit 1425 can be found in the above Figure 3The description of step S103 in the corresponding embodiment will not be repeated here.

[0251] In one embodiment, the image reconstruction module 15 may include: a cascade feature determination unit 151 and a reconstructed image determination unit 152 .

[0252] The cascade feature determination unit 151 is used to determine the target fusion cascade feature corresponding to the target image according to the target image association feature, the target accumulation feature and the historical image;

[0253] The reconstructed image determination unit 152 is used to input the target fusion cascade features into the target image reconstruction model, and output the reconstructed result image corresponding to the target image in the target image reconstruction model according to the target fusion cascade features.

[0254] The specific implementation of the cascade feature determination unit 151 and the reconstructed image determination unit 152 can be found in the above Figure 3 The description of step S104 in the corresponding embodiment will not be repeated here.

[0255] In one embodiment, the target image associated features include a first normal dimension feature and a first depth dimension feature; the historical image associated features include a second normal dimension feature and a second depth dimension feature corresponding to the historical image;

[0256] The cascade feature determination unit 151 may include: a feature fusion subunit 1511 , a feature transformation processing subunit 1512 , and a feature cascade subunit 1513 .

[0257] The feature fusion subunit 1511 is used to obtain auxiliary features corresponding to the target image; the auxiliary features include a first normal dimension feature and a first depth dimension feature;

[0258] The feature fusion subunit 1511 is further used to determine the dimensional features other than the first normal dimensional feature and the first depth dimensional feature in the auxiliary features as the remaining dimensional features;

[0259] The feature fusion subunit 1511 is further used to fuse the target accumulated features, the first normal dimension features, the first depth dimension features and the remaining dimension features to obtain an image fusion feature corresponding to the target image;

[0260] The feature transformation processing subunit 1512 is used to perform affine transformation processing on the second normal dimension feature and the second depth dimension feature respectively to obtain a normal transformation feature corresponding to the second normal dimension feature and a depth transformation feature corresponding to the second depth dimension feature;

[0261] The feature cascade subunit 1513 is used to perform cascade processing on the image fusion feature, the normal transformation feature and the depth transformation feature to obtain a target fusion cascade feature corresponding to the target image.

[0262] The specific implementation of the feature fusion subunit 1511, the feature transformation processing subunit 1512 and the feature cascade subunit 1513 can be found in the above Figure 3 The description of step S104 in the corresponding embodiment will not be repeated here.

[0263] In one embodiment, the reconstructed image determination unit 152 may include: a parameter output subunit 1521 and an image fusion subunit 1522 .

[0264] The parameter output subunit 1521 is used to determine the predicted high-quality image, the predicted low-quality image, the first image reconstruction parameter and the second image reconstruction parameter corresponding to the target fusion cascade feature through the target image reconstruction model; the predicted high-quality image does not contain noise, and the resolution of the predicted high-quality image is greater than the resolution of the predicted low-quality image;

[0265] An image fusion subunit 1522 is used to fuse the predicted high-quality image with the predicted low-quality image in the target image reconstruction model by using the first image reconstruction parameter to obtain an initial reconstructed image corresponding to the target image;

[0266] The image fusion subunit 1522 is further used to obtain a historical reconstructed image corresponding to the historical image, and to fuse the initial reconstructed image with the historical reconstructed image through the second image reconstruction parameter to obtain a reconstructed result image corresponding to the target image.

[0267] The specific implementation of the parameter output subunit 1521 and the image fusion subunit 1522 can be found in the above Figure 3 The description of step S104 in the corresponding embodiment will not be repeated here.

[0268] In an embodiment of the present application, the hybrid embedded features corresponding to the two images can be calculated based on the associated features of the current image and the previous frame image, and the features of the two images can be accumulated based on the hybrid embedded features. The obtained target accumulated features include the features of the previous and next frames after fusion. The fused features can characterize the correlation between the previous and next frames, and the features include a wider dimension. At the same time, when determining the reconstructed result image of the target image, it is also determined based on the target image associated features of the target image, the target accumulated features and the previous frame image (historical image). When determining the reconstructed result image of the target image, not only the features of the current frame are taken into account, but also the features of the previous historical frames are taken into account. The time domain information of the current frame can be fully utilized, and the obtained reconstructed result image has higher time domain stability and higher image quality. In summary, regardless of whether the image is obtained with a high pixel sampling number or a low pixel sampling number, the present application can accumulate features based on the image correlation features of the two frames before and after in the target image sequence, and determine the reconstructed result image of the target image together with the accumulated features and the historical images, which has universal applicability; by calculating the accumulated features of the two frames before and after, and then reconstructing the image based on the accumulated features, the method can make full use of the time domain information of the image, thereby greatly improving the image quality of the reconstructed image.

[0269] For further information, see Figure 7 , Figure 7 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may be a computer program (including program code) running in a computer device, for example, the image processing device is an application software; the image processing device may be used to execute Figure 4 As shown in the method. Figure 7 As shown, the image processing device 2 may include: a sample image acquisition module 21, a sample feature acquisition module 22, a sample feature generation module 23, a sample feature accumulation module 24, a sample feature cascade module 25, a sample reconstruction image output module 26, a label image acquisition module 27 and a model training module 28.

[0270] The sample image acquisition module 21 is used to acquire a target sample noise image and a historical sample noise image in a sample image sequence; the sample image sequence is an image sequence obtained after image acquisition of a sample scene based on a first unit pixel sampling number; the historical sample noise image refers to a sample noise image previous to the target sample noise image in the sample image sequence; the first unit pixel sampling number is less than a unit pixel sampling number threshold;

[0271] A sample feature acquisition module 22, used to acquire target sample image correlation features corresponding to the target sample noise image, and historical sample image correlation features corresponding to the historical sample noise image;

[0272] A sample feature generation module 23, used to generate a first sample mixed embedding feature for a target sample noise image according to a target sample image associated feature;

[0273] The sample feature generation module 23 is further used to generate a second sample mixed embedding feature for the historical sample noise image according to the target sample image association feature and the historical sample image association feature;

[0274] A sample feature accumulation module 24 is used to accumulate features of the target sample noise image and the historical sample noise image based on the first sample mixed embedding feature and the second sample mixed embedding feature to obtain a target sample accumulated feature corresponding to the target sample noise image;

[0275] The sample feature cascade module 25 is used to determine the target sample fusion cascade feature corresponding to the target sample noise image according to the target sample image association feature, the target sample accumulation feature and the historical sample noise image;

[0276] The sample reconstructed image output module 26 is used to input the target sample fusion cascade feature into the image reconstruction model, and in the image reconstruction model, output the target sample reconstructed image corresponding to the target sample noise image according to the target sample fusion cascade feature;

[0277] The label image acquisition module 27 is used to acquire a target label sampling image corresponding to the target sample noise image; the target label sampling image refers to an image obtained after image acquisition of the sample scene based on the second unit pixel sampling number; the first unit pixel sampling number is less than the second unit pixel sampling number;

[0278] The model training module 28 is used to train the image reconstruction model according to the target label sampling image and the target sample reconstruction image, so as to obtain a target image reconstruction model for performing image reconstruction processing on the target image in the target image sequence.

[0279] The specific implementation of the sample image acquisition module 21, the sample feature acquisition module 22, the sample feature generation module 23, the sample feature accumulation module 24, the sample feature cascade module 25, the sample reconstruction image output module 26, the label image acquisition module 27 and the model training module 28 can be seen in the above Figure 4 The description of steps S301 to S307 in the corresponding embodiment will not be repeated here.

[0280] In one embodiment, the model training module 28 may include: an image to be operated determination unit 281, a loss value determination unit 282 and a model training unit 283.

[0281] The image to be calculated determination unit 281 is used to determine the sample reconstructed image corresponding to the remaining sample noise image in the sample image sequence as the reconstructed image to be calculated, and determine the label sampled image corresponding to the remaining sample noise image as the label sampled image to be calculated; the remaining sample noise image refers to the sample noise image other than the target sample noise image in the sample image sequence;

[0282] A loss value determining unit 282, configured to determine a target loss value for the image reconstruction model according to the reconstructed image to be calculated, the label sampled image to be calculated, the target label sampled image, and the target sample reconstructed image;

[0283] The model training unit 283 is used to train the image reconstruction model according to the target loss value to obtain a target image reconstruction model.

[0284] The specific implementation of the image determination unit 281, the loss value determination unit 282 and the model training unit 283 can be found in the above Figure 4 The description of step S307 in the corresponding embodiment will not be repeated here.

[0285] In one embodiment, the loss value determining unit 282 may include: a loss value determining subunit 2821 and a target loss value determining subunit 2822 .

[0286] The loss value determination subunit 2821 is used to determine the spatial loss value for the image reconstruction model according to the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image;

[0287] The loss value determination subunit 2821 is further used to determine the temporal loss value for the image reconstruction model according to the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image;

[0288] The loss value determination subunit 2821 is further used to determine the relative edge loss value for the image reconstruction model according to the reconstructed image to be calculated, the label sampling image to be calculated, the target label sampling image and the target sample reconstructed image;

[0289] The target loss value determination subunit 2822 is used to determine the target loss value for the image reconstruction model according to the spatial loss value, the temporal loss value and the relative edge loss value.

[0290] The specific implementation of the loss value determination subunit 2821 and the target loss value determination subunit 2822 can be found in the above Figure 4 The description of step S307 in the corresponding embodiment will not be repeated here.

[0291] In one embodiment, the loss value determination subunit 2821 is further specifically used to obtain a spatial loss function, and determine a first spatial sub-loss value for the remaining sample noise image according to the spatial loss function, the reconstructed image to be calculated, and the label sampling image to be calculated;

[0292] The loss value determination subunit 2821 is further specifically used to determine a second spatial sub-loss value for the target sample noise image according to the spatial loss function, the target label sampling image, and the target sample reconstructed image;

[0293] The loss value determination subunit 2821 is further specifically used to fuse the first spatial sub-loss value with the second spatial sub-loss value to obtain the spatial loss value of the image reconstruction model.

[0294] In one embodiment, the loss value determination subunit 2821 is further specifically used to obtain a first training weight for the residual sample noise image and a second training weight for the target sample noise image;

[0295] The loss value determination subunit 2821 is further specifically used to perform operation processing on the first training weight and the first spatial sub-loss value to obtain the first operation spatial sub-loss value;

[0296] The loss value determination subunit 2821 is further specifically used to perform operation processing on the second training weight and the second space sub-loss value to obtain the second operation space sub-loss value;

[0297] The loss value determination subunit 2821 is further specifically used to add the first operation space sub-loss value and the second operation space sub-loss value to obtain the spatial loss value of the image reconstruction model.

[0298] In one embodiment, the target sample noise image is the last sample noise image in the sample image sequence;

[0299] The target loss value determination subunit 2822 is further specifically used to obtain a historical sample reconstructed image and a historical label sampling image corresponding to the historical sample noise image, and determine an affine transformation loss value for the image reconstruction model according to the historical sample reconstructed image, the target sample reconstructed image, the historical label sampling image and the target label sampling image;

[0300] The target loss value determination subunit 2822 is further specifically used to obtain the target sample albedo feature corresponding to the target sample noise image, obtain the target sample cumulative albedo feature corresponding to the target sample noise image in the target sample cumulative feature, and determine the albedo loss value for the image reconstruction model according to the albedo loss function, the target sample albedo feature and the target sample cumulative albedo feature;

[0301] The target loss value determination subunit 2822 is also specifically used to fuse the spatial loss value, the temporal loss value, the relative edge loss value, the affine transformation loss value and the albedo loss value to obtain the target loss value of the image reconstruction model.

[0302] In an embodiment of the present application, the hybrid embedded features corresponding to the two images can be calculated based on the associated features of the current image and the previous frame image, and the features of the two images can be accumulated based on the hybrid embedded features. The obtained target accumulated features include the features of the previous and next frames after fusion. The fused features can characterize the correlation between the previous and next frames, and the features include a wider dimension. At the same time, when determining the reconstructed result image of the target image, it is also determined based on the target image associated features of the target image, the target accumulated features and the previous frame image (historical image). When determining the reconstructed result image of the target image, not only the features of the current frame are taken into account, but also the features of the previous historical frames are taken into account. The time domain information of the current frame can be fully utilized, and the obtained reconstructed result image has higher time domain stability and higher image quality. In summary, regardless of whether the image is obtained with a high pixel sampling number or a low pixel sampling number, the present application can accumulate features based on the image correlation features of the two frames before and after in the target image sequence, and determine the reconstructed result image of the target image together with the accumulated features and the historical images, which has universal applicability; by calculating the accumulated features of the two frames before and after, and then reconstructing the image based on the accumulated features, the method can make full use of the time domain information of the image, thereby greatly improving the image quality of the reconstructed image.

[0303] For further information, see Figure 8 , Figure 8 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 8 As shown above Figure 7The image processing device 1 in the corresponding embodiment can be applied to the above-mentioned computer device 8000, and the above-mentioned computer device 8000 may include: a processor 8001, a network interface 8004 and a memory 8005. In addition, the above-mentioned computer device 8000 also includes: a user interface 8003, and at least one communication bus 8002. Among them, the communication bus 8002 is used to realize the connection and communication between these components. Among them, the user interface 8003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 8003 may optionally include a standard wired interface and a wireless interface. The network interface 8004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 8005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 8005 may optionally also be at least one storage device located away from the aforementioned processor 8001. Figure 8 As shown, the memory 8005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0304] exist Figure 8 In the computer device 8000 shown, the network interface 8004 can provide a network communication function; the user interface 8003 is mainly used to provide an input interface for the user; and the processor 8001 can be used to call the device control application stored in the memory 8005 to achieve:

[0305] A target image and a historical image are obtained in a target image sequence; the target image sequence is an image sequence obtained after image acquisition of a target scene based on the target unit pixel sampling number; the historical image refers to the previous image of the target image in the target image sequence;

[0306] Obtaining target image associated features corresponding to the target image and historical image associated features corresponding to the historical image;

[0307] Generate a first hybrid embedding feature for the target image according to the target image association feature, generate a second hybrid embedding feature for the historical image according to the target image association feature and the historical image association feature, and perform feature accumulation of the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature to obtain a target accumulated feature corresponding to the target image;

[0308] According to the target image associated features, target accumulated features and historical images, a reconstructed result image corresponding to the target image is determined.

[0309] Or implement:

[0310] A target sample noise image and a historical sample noise image are obtained in a sample image sequence; the sample image sequence is an image sequence obtained after image acquisition of a sample scene based on a first unit pixel sampling number; the historical sample noise image refers to a previous sample noise image of the target sample noise image in the sample image sequence; the first unit pixel sampling number is less than a unit pixel sampling number threshold;

[0311] Obtaining target sample image correlation features corresponding to the target sample noise image and historical sample image correlation features corresponding to the historical sample noise image;

[0312] Generate a first sample mixed embedding feature for the target sample noise image according to the target sample image association feature, generate a second sample mixed embedding feature for the historical sample noise image according to the target sample image association feature and the historical sample image association feature, and perform feature accumulation on the target sample noise image and the historical sample noise image based on the first sample mixed embedding feature and the second sample mixed embedding feature to obtain a target sample accumulated feature corresponding to the target sample noise image;

[0313] Determine the target sample fusion cascade feature corresponding to the target sample noise image according to the target sample image association feature, the target sample accumulation feature and the historical sample noise image;

[0314] The target sample fusion cascade feature is input into the image reconstruction model, and in the image reconstruction model, a target sample reconstructed image corresponding to the target sample noise image is output according to the target sample fusion cascade feature;

[0315] Obtaining a target label sampling image corresponding to the target sample noise image; the target label sampling image refers to an image obtained after image acquisition of the sample scene based on the second unit pixel sampling number; the first unit pixel sampling number is less than the second unit pixel sampling number;

[0316] The image reconstruction model is trained according to the target label sampling image and the target sample reconstructed image to obtain a target image reconstruction model for performing image reconstruction processing on the target image in the target image sequence.

[0317] It should be understood that the computer device 8000 described in the embodiment of the present application can execute the above Figure 3 or Figure 4 The description of the image processing method in the corresponding embodiment can also be performed as described above. Figure 6 In the corresponding embodiment, the image processing device 1, or Figure 7 The description of the image processing device 2 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0318] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the computer device 1000 for data processing mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, the computer program can execute the above-mentioned data processing. Figure 3 or Figure 4 The description of the above-mentioned image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0319] The computer-readable storage medium may be an internal storage unit of the image processing apparatus or the computer device provided in any of the aforementioned embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0320] In one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in one aspect of the embodiments of the present application.

[0321] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0322] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0323] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0324] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a target image and a historical image in a target image sequence; the target image sequence is an image sequence obtained after image acquisition of a target scene based on a target unit pixel sampling number; the historical image refers to an image previous to the target image in the target image sequence; Acquire target image associated features corresponding to the target image and historical image associated features corresponding to the historical image; the target image associated features include a first normal dimension feature and a first depth dimension feature corresponding to the target image; Inputting the first normal dimension feature and the first depth dimension feature into a temporal accumulation model; performing convolution processing on the first normal dimension feature and the first depth dimension feature through a first convolutional network layer of the temporal accumulation model to obtain a first image embedding feature corresponding to the target image; Performing convolution processing on the first normal dimension feature and the first depth dimension feature through the second convolutional network layer of the temporal accumulation model to obtain a second image embedding feature corresponding to the target image; Performing pixel multiplication processing on the first image embedding feature and the second image embedding feature to obtain a first mixed embedding feature for the target image; Generate a second hybrid embedding feature for the historical image according to the target image association feature and the historical image association feature, and perform feature accumulation on the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature to obtain a target accumulation feature corresponding to the target image; A reconstruction result image corresponding to the target image is determined according to the target image associated features, the target accumulated features and the historical images.

2. The method according to claim 1, characterized in that The historical image associated features include a second normal dimension feature and a second depth dimension feature corresponding to the historical image; The step of generating a second hybrid embedding feature for the historical image according to the target image association feature and the historical image association feature comprises: Performing affine transformation processing on the second normal dimension feature and the second depth dimension feature respectively to obtain a normal transformation feature corresponding to the second normal dimension feature and a depth transformation feature corresponding to the second depth dimension feature; Inputting the first normal dimension feature, the first depth dimension feature, the normal transformation feature, and the depth transformation feature into a temporal accumulation model; Performing convolution processing on the first normal dimension feature and the first depth dimension feature through the first convolutional network layer of the temporal accumulation model to obtain a target image embedding feature corresponding to the target image; The normal transformation feature and the depth transformation feature are convolved by the second convolutional network layer of the temporal accumulation model to obtain a historical image embedding feature corresponding to the historical image; The target image embedding feature and the historical image embedding feature are subjected to pixel multiplication operation to obtain a second mixed embedding feature for the historical image.

3. The method according to claim 1, characterized in that: The step of accumulating features of the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature to obtain a target accumulated feature corresponding to the target image includes: Obtaining a logistic regression function, and determining a first feature fusion coefficient for the target image through the logistic regression function and the first hybrid embedding feature; Determining a second feature fusion coefficient for the historical image by using the logistic regression function and the second hybrid embedding feature; According to the first feature fusion coefficient and the second feature fusion coefficient, features of the target image and the historical image are accumulated to obtain target accumulated features corresponding to the target image.

4. The method according to claim 3, characterized in that The step of accumulating features of the target image and the historical image according to the first feature fusion coefficient and the second feature fusion coefficient to obtain a target accumulation feature corresponding to the target image includes: Obtain target noise image features, target shadow features, and target albedo features corresponding to the target image, and obtain historical accumulated noise image features, historical accumulated shadow features, and historical accumulated albedo features corresponding to the historical image; Accumulating the target shadow feature and the historical accumulated shadow feature according to the first feature fusion coefficient and the second feature fusion coefficient to obtain a target accumulated shadow feature corresponding to the target image; Accumulating the target noise image feature and the historical accumulated noise image feature according to the first feature fusion coefficient and the second feature fusion coefficient to obtain a target accumulated noise image feature corresponding to the target image; Accumulating the target albedo feature and the historical accumulated albedo feature according to the first feature fusion coefficient and the second feature fusion coefficient to obtain a target accumulated albedo feature corresponding to the target image; The target cumulative shadow feature, the target cumulative noise image feature and the target cumulative albedo feature are determined as target cumulative features corresponding to the target image.

5. The method according to claim 1, characterized in that The step of determining the reconstructed result image corresponding to the target image according to the target image associated features, the target accumulated features and the historical images includes: Determining a target fusion cascade feature corresponding to the target image according to the target image association feature, the target cumulative feature and the historical image; The target fusion cascade features are input into a target image reconstruction model, and in the target image reconstruction model, a reconstruction result image corresponding to the target image is output according to the target fusion cascade features.

6. The method according to claim 5, characterized in that The target image associated features include a first normal dimension feature and a first depth dimension feature; the historical image associated features include a second normal dimension feature and a second depth dimension feature corresponding to the historical image; The step of determining the target fusion cascade feature corresponding to the target image according to the target image association feature, the target accumulation feature and the historical image includes: Acquire auxiliary features corresponding to the target image; the auxiliary features include the first normal dimension feature and the first depth dimension feature; Determine, among the auxiliary features, dimensional features other than the first normal dimensional feature and the first depth dimensional feature as remaining dimensional features; Performing feature fusion on the target accumulated features, the first normal dimension features, the first depth dimension features, and the remaining dimension features to obtain an image fusion feature corresponding to the target image; Performing affine transformation processing on the second normal dimension feature and the second depth dimension feature respectively to obtain a normal transformation feature corresponding to the second normal dimension feature and a depth transformation feature corresponding to the second depth dimension feature; The image fusion feature, the normal transformation feature and the depth transformation feature are cascaded to obtain a target fusion cascade feature corresponding to the target image.

7. The method according to claim 5, characterized in that In the target image reconstruction model, outputting a reconstruction result image corresponding to the target image according to the target fusion cascade feature includes: Determining, by means of the target image reconstruction model, a predicted high-quality image, a predicted low-quality image, a first image reconstruction parameter, and a second image reconstruction parameter corresponding to the target fusion cascade feature; the predicted high-quality image does not contain noise, and a resolution of the predicted high-quality image is greater than a resolution of the predicted low-quality image; In the target image reconstruction model, the predicted high-quality image and the predicted low-quality image are fused by using the first image reconstruction parameter to obtain an initial reconstructed image corresponding to the target image; A historical reconstructed image corresponding to the historical image is obtained, and the initial reconstructed image and the historical reconstructed image are fused using the second image reconstruction parameter to obtain a reconstructed result image corresponding to the target image.

8. An image processing method, characterized in that: include: A target sample noise image and a historical sample noise image are obtained in a sample image sequence; the sample image sequence is an image sequence obtained after image acquisition of a sample scene based on a first unit pixel sampling number; the historical sample noise image refers to a previous sample noise image of the target sample noise image in the sample image sequence; the first unit pixel sampling number is less than a unit pixel sampling number threshold; Acquire target sample image correlation features corresponding to the target sample noise image, and historical sample image correlation features corresponding to the historical sample noise image; The target sample image associated features include a first sample normal dimension feature and a first sample depth dimension feature corresponding to the target sample noise image; Inputting the first sample normal dimension feature and the first sample depth dimension feature into a temporal accumulation model; performing convolution processing on the first sample normal dimension feature and the first sample depth dimension feature through a first convolutional network layer of the temporal accumulation model to obtain a first sample image embedding feature corresponding to the target sample noise image; performing convolution processing on the first sample normal dimension feature and the first sample depth dimension feature through a second convolutional network layer of the temporal accumulation model to obtain a second sample image embedding feature corresponding to the target sample noise image; Performing pixel multiplication processing on the first sample image embedding feature and the second sample image embedding feature to obtain a first sample mixed embedding feature for the target sample noise image; Generate a second sample mixed embedding feature for the historical sample noise image according to the target sample image association feature and the historical sample image association feature, and perform feature accumulation on the target sample noise image and the historical sample noise image based on the first sample mixed embedding feature and the second sample mixed embedding feature to obtain a target sample accumulated feature corresponding to the target sample noise image; Determine, according to the target sample image association feature, the target sample accumulation feature and the historical sample noise image, the target sample fusion cascade feature corresponding to the target sample noise image; Inputting the target sample fusion cascade feature into an image reconstruction model, and in the image reconstruction model, outputting a target sample reconstructed image corresponding to the target sample noise image according to the target sample fusion cascade feature; Acquire a target label sampling image corresponding to the target sample noise image; the target label sampling image refers to an image obtained after image acquisition of the sample scene based on a second unit pixel sampling number; the first unit pixel sampling number is less than the second unit pixel sampling number; The image reconstruction model is trained according to the target label sampling image and the target sample reconstructed image to obtain a target image reconstruction model for performing image reconstruction processing on the target image in the target image sequence.

9. The method according to claim 8, characterized in that The step of training the image reconstruction model according to the target label sampling image and the target sample reconstructed image to obtain a target image reconstruction model for performing image reconstruction processing on the target image in the target image sequence includes: Determine the sample reconstructed image corresponding to the remaining sample noise image in the sample image sequence as the reconstructed image to be calculated, and determine the label sampled image corresponding to the remaining sample noise image as the label sampled image to be calculated; the remaining sample noise image refers to the sample noise image in the sample image sequence other than the target sample noise image; Determining a target loss value for the image reconstruction model according to the to-be-calculated reconstructed image, the to-be-calculated label sampled image, the target label sampled image, and the target sample reconstructed image; The image reconstruction model is trained according to the target loss value to obtain the target image reconstruction model.

10. The method according to claim 9, characterized in that The step of determining a target loss value for the image reconstruction model according to the to-be-calculated reconstructed image, the to-be-calculated label sampled image, the target label sampled image, and the target sample reconstructed image comprises: Determine a spatial loss value for the image reconstruction model according to the to-be-calculated reconstructed image, the to-be-calculated label sampled image, the target label sampled image, and the target sample reconstructed image; Determine a temporal loss value for the image reconstruction model according to the to-be-calculated reconstructed image, the to-be-calculated label sampled image, the target label sampled image, and the target sample reconstructed image; Determining a relative edge loss value for the image reconstruction model according to the to-be-calculated reconstructed image, the to-be-calculated label sampled image, the target label sampled image, and the target sample reconstructed image; A target loss value for the image reconstruction model is determined according to the spatial loss value, the temporal loss value, and the relative edge loss value.

11. The method according to claim 10, characterized in that The target sample noise image is the last sample noise image in the sample image sequence; Determining a target loss value for the image reconstruction model according to the spatial loss value, the temporal loss value, and the relative edge loss value includes: Obtain a historical sample reconstructed image and a historical label sampling image corresponding to the historical sample noise image, and determine an affine transformation loss value for the image reconstruction model according to the historical sample reconstructed image, the target sample reconstructed image, the historical label sampling image, and the target label sampling image; Obtaining a target sample albedo feature corresponding to the target sample noise image, obtaining a target sample cumulative albedo feature corresponding to the target sample noise image from the target sample cumulative features, and determining an albedo loss value for the image reconstruction model according to an albedo loss function, the target sample albedo feature, and the target sample cumulative albedo feature; The spatial loss value, the temporal loss value, the relative edge loss value, the affine transformation loss value and the albedo loss value are fused to obtain a target loss value of the image reconstruction model.

12. An image processing device, characterized in that: include: An image acquisition module is used to acquire a target image and a historical image in a target image sequence; the target image sequence is an image sequence obtained after image acquisition of a target scene based on a target unit pixel sampling number; the historical image refers to an image previous to the target image in the target image sequence; A feature acquisition module, used to acquire target image associated features corresponding to the target image and historical image associated features corresponding to the historical image; the target image associated features include a first normal dimension feature and a first depth dimension feature corresponding to the target image; A feature generation module, used for inputting the first normal dimension feature and the first depth dimension feature into a temporal accumulation model; performing convolution processing on the first normal dimension feature and the first depth dimension feature through a first convolutional network layer of the temporal accumulation model to obtain a first image embedding feature corresponding to the target image; Performing convolution processing on the first normal dimension feature and the first depth dimension feature through the second convolutional network layer of the temporal accumulation model to obtain a second image embedding feature corresponding to the target image; Performing pixel multiplication processing on the first image embedding feature and the second image embedding feature to obtain a first mixed embedding feature for the target image; The feature generation module is further used to generate a second hybrid embedding feature for the historical image according to the target image association feature and the historical image association feature; A feature accumulation module, configured to accumulate features of the target image and the historical image based on the first hybrid embedding feature and the second hybrid embedding feature to obtain a target accumulated feature corresponding to the target image; An image reconstruction module is used to determine a reconstruction result image corresponding to the target image according to the target image association features, the target cumulative features and the historical images.

13. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a network communication function, the memory is used to store program code, and the processor is used to call the program code so that the computer device executes the method described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 11.

15. A computer program product or a computer program, characterized in that The computer program product or computer program comprises computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 11.