Image processing method and device, electronic equipment, medium and program product

Abstract

The invention provides an image processing method and device, electronic equipment, a medium and a program product, and can be applied to the technical field of artificial intelligence, the technical field of biological recognition and the field of financial science and technology. The method comprises the following steps: acquiring original image sequence data, and constructing an image frame sequence tensor based on the original image sequence data; performing multi-scale sampling on the image frame sequence tensor to generate a plurality of target image frame sequence tensors with different resolutions; the target image frame sequence tensor is input into a feature extraction and reconstruction network in parallel, a multi-scale feature map is obtained, and the feature extraction and reconstruction network comprises an attention calculation structure used for modeling the spatial-temporal correlation of the target image frame sequence tensor; and performing scale reduction on the multi-scale feature map, and performing feature fusion on the multi-scale feature map after scale reduction and the original image sequence data to generate a reconstructed image frame sequence.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, biometrics, and fintech, and more specifically to an image processing method, apparatus, device, medium, and program product. Background Technology

[0002] With the widespread application of video capture and image computing in fields such as financial risk control, data preprocessing and quality enhancement for image sequences have become crucial for improving downstream perception and decision-making performance. However, the deployment environment of financial terminals is complex, often accompanied by problems such as low illumination, limited viewing angles, and dynamic interference, resulting in low resolution and blurred structures in the captured images. To address this issue, image super-resolution (SR) technology has been introduced into the preprocessing workflow of image recognition to improve image clarity and structural fidelity.

[0003] Despite the achievements of SR technology in improving image quality, it still faces challenges in practical financial applications: existing image-level evaluation metrics are difficult to measure the actual improvement in recognition performance; some models may introduce artifacts and false details, affecting the reliability and security of the recognition system; in addition, some high-precision models are complex in structure and computationally intensive, making it difficult to meet the low-latency and real-time processing requirements of financial terminal devices, thus limiting their deployment and promotion in real-world scenarios. Summary of the Invention

[0004] In view of the above problems, this application provides image processing methods, apparatus, devices, media, and program products.

[0005] According to a first aspect of this application, an image processing method is provided, the method comprising: acquiring original image sequence data; constructing an image frame sequence tensor based on the original image sequence data; performing multi-scale sampling on the image frame sequence tensor to generate multiple target image frame sequence tensors with different resolutions; inputting the target image frame sequence tensors in parallel into a feature extraction and reconstruction network to obtain a multi-scale feature map, wherein the feature extraction and reconstruction network includes an attention computation structure for modeling the spatiotemporal correlation of the target image frame sequence tensors; and performing scale restoration on the multi-scale feature map, fusing the scale-restored multi-scale feature map with the original image sequence data to generate a reconstructed image frame sequence.

[0006] According to an embodiment of this application, modeling the spatiotemporal correlation of the target image frame sequence tensor includes: dividing the target image frame sequence tensor into windows along the spatial dimension, performing local self-attention calculation within each window to obtain spatially dependent features; and offsetting the window division position between adjacent processing layers of the attention calculation structure so that the spatially dependent features interact between different windows to model the spatiotemporal correlation of the target image frame sequence tensor.

[0007] According to an embodiment of this application, the feature extraction and reconstruction network further includes a residual feature extraction structure, which is used to extract local spatial features from the target image frame sequence tensor, and the local spatial features are used to characterize at least the detailed texture information in the target image frame sequence tensor.

[0008] According to an embodiment of this application, the method further includes: calculating optical flow information based on adjacent time frames in the image frame sequence tensor; and fusing the optical flow information into the processing flow of the residual feature extraction structure and the attention calculation structure, respectively.

[0009] According to an embodiment of this application, the step of fusing the scale-restored multi-scale feature map with the original image sequence data to generate a reconstructed image frame sequence includes: performing spatial alignment and channel mapping processing on the scale-restored multi-scale feature map; fusing the processed multi-scale feature map with the original image sequence data of the corresponding time frame; and inputting the fusion result into the convolutional reconstruction sub-network of the feature extraction and reconstruction network to generate the reconstructed image frame sequence.

[0010] According to an embodiment of this application, the original image sequence data includes at least original face image sequence data, and the method further includes: obtaining an identity feature vector corresponding to the original face image sequence data; and adjusting the attention weights of the attention calculation structure based on the identity feature vector.

[0011] According to an embodiment of this application, the method further includes: assembling the reconstructed image frame sequence in chronological order to form a dynamic face image frame sequence; and converting the dynamic face image frame sequence to a type, outputting face image data in a target format.

[0012] According to an embodiment of this application, the multi-scale sampling of the image frame sequence tensor includes: calculating the content complexity index of each frame image based on the image frame sequence tensor; and determining the scale level and sampling strategy information of the image frame sequence tensor in multi-scale sampling based on the content complexity index.

[0013] A second aspect of this application provides an image processing apparatus, comprising: a data acquisition module for: acquiring original image sequence data and constructing an image frame sequence tensor based on the original image sequence data; a multi-scale sampling module for: performing multi-scale sampling on the image frame sequence tensor to generate multiple target image frame sequence tensors with different resolutions; a feature extraction module for: inputting the target image frame sequence tensors in parallel into a feature extraction and reconstruction network to obtain a multi-scale feature map, wherein the feature extraction and reconstruction network includes an attention computation structure for modeling the spatiotemporal correlation of the target image frame sequence tensors; and an image reconstruction module for: performing scale restoration on the multi-scale feature map and fusing the scale-restored multi-scale feature map with the original image sequence data to generate a reconstructed image frame sequence.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] According to embodiments of this application, by constructing an image frame sequence tensor from the original image sequence and introducing a multi-scale sampling mechanism, the structural information of the image at different spatial resolutions can be fully extracted, improving the model's ability to perceive image details and contextual structure. At the same time, by inputting the multi-scale target image frame sequence into a feature extraction and reconstruction network containing an attention computation structure in parallel and modeling the spatiotemporal correlation of the image frame sequence, the dynamic changes between frames and the global dependencies within the image can be effectively captured, thereby improving the consistency and accuracy of image reconstruction. Attached Figure Description

[0018] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustrations depict application scenarios of image processing methods, apparatus, devices, media, and program products according to embodiments of this application.

[0020] Figure 2A flowchart illustrating an image processing method according to an embodiment of this application is shown schematically.

[0021] Figure 3 The flowchart schematically illustrates a method for modeling the spatiotemporal correlation of a target image frame sequence tensor according to some exemplary embodiments of this application;

[0022] Figure 4 The illustration shows a schematic diagram of a method for feature fusion according to some exemplary embodiments of this application;

[0023] Figure 5 A schematic block diagram of an image processing apparatus according to an embodiment of this application is shown; and

[0024] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing an image processing method according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] First, the technical terms used in this article are explained and clarified as follows.

[0030] A residual block is a structural unit used to improve the stability and performance of deep neural network training. Its basic idea is to introduce "residual connections" into each convolutional or nonlinear transformation module, where the input data bypasses intermediate processing modules and is directly added to the output. Through this skip connection method, residual blocks can effectively alleviate the vanishing gradient problem in deep networks, enhancing feature transfer and information preservation capabilities. In image processing tasks, residual blocks are often used to extract detailed features, preserve image edge information, and help the network focus on regions of structural difference during high-resolution reconstruction.

[0031] Optical flow refers to the pixel-level displacement vectors formed by the movement of objects between consecutive frames in an image sequence. It is commonly used to characterize the motion changes between video frames. In image processing and video analysis, calculating optical flow information can reveal the motion trend of each pixel in an image over time, thereby aiding in modeling temporal consistency and dynamic features. Introducing optical flow information can effectively improve the accuracy of inter-frame alignment and dynamic region recognition in dynamic image sequence processing, and reduce the impact of motion blur on image reconstruction quality.

[0032] Encoders and decoders are used to construct feature extraction and reconstruction modules in neural networks. The encoder receives the original image or image sequence as input and extracts high-level abstract feature representations through layer-by-layer convolution or attention mechanisms, gradually reducing spatial resolution to compress information. The decoder then upsamples and reconstructs the extracted features, restoring an image output with the same or higher resolution as the original input. Together, they form a typical encoder-decoder structure, facilitating image compression, restoration, and enhancement, and are widely used in tasks such as image reconstruction, super-resolution, and semantic segmentation.

[0033] With the continuous evolution of biometric technology, especially its widespread application in artificial intelligence and computer vision, facial recognition, as an efficient and contactless authentication method, has been widely deployed in various scenarios within the financial industry, such as remote account opening, smart teller machines, facial recognition payment, cardless withdrawals, and risk control audits. In the process of identity verification based on remote images, the quality of the facial images provided by the user or collected by the system directly affects the accuracy and security of the recognition system.

[0034] However, in practical applications, the complex and varied imaging environment, such as financial service terminals deployed in low-light, limited-viewpoint, or dynamic scenes, can lead to problems such as insufficient resolution and blurred structure in the acquired facial images. To enhance the clarity and structural information of facial images and restore more realistic facial features, SR technology has been introduced into the facial recognition preprocessing workflow as a key means to improve recognition performance.

[0035] Currently, commonly used SR (Self-Resolution Image) technologies mainly fall into the following categories: The first category is end-to-end learning methods based on Convolutional Neural Networks (CNNs), such as Super-Resolution Convolutional Neural Networks (SRCNN) and Enhanced Deep Super-Resolution Networks (EDSR), which use deep networks to extract features and restore details from low-resolution images. The second category is SR methods based on Generative Adversarial Networks (GANs), such as Super-Resolution Generative Adversarial Networks (SRGAN) and Enhanced Super-Resolution Generative Adversarial Networks (ESRGAN), which improve the perceptual quality of generated images and enhance the ability to restore details by introducing adversarial training mechanisms. The third category is style-based generative adversarial network generator architectures (StyleGAN), which integrate face priors during image generation to guide the generation process to converge towards the distribution of real faces.

[0036] While the aforementioned SR (Self-Recognition) technologies have improved image quality to some extent, they still face numerous challenges in practical applications within the financial sector. First, existing SR algorithms primarily rely on image-level metrics for performance evaluation. While these metrics reflect image clarity or realism, they cannot accurately assess the actual improvement SR technology brings to facial recognition results, making it difficult to provide effective feedback on performance improvements in financial identity verification systems. Second, although GAN-based SR models can generate images with higher visual quality, they are also prone to artifacts and false details inconsistent with the original face, such as unnatural textures or distorted facial structures. This can mislead the recognition system and affect the credibility and attack resistance of the financial system. Furthermore, some high-performance SR models have complex network structures and consume significant computational resources, making it difficult to meet the real-time processing requirements of financial edge devices (e.g., video stream processing speeds exceeding camera frame rates), thus hindering their promotion and application in terminal devices and large-scale real-time scenarios.

[0037] Based on this, embodiments of this application provide an image processing method, the method comprising: acquiring original image sequence data; constructing an image frame sequence tensor based on the original image sequence data; performing multi-scale sampling on the image frame sequence tensor to generate multiple target image frame sequence tensors with different resolutions; inputting the target image frame sequence tensors in parallel into a feature extraction and reconstruction network to obtain a multi-scale feature map, wherein the feature extraction and reconstruction network includes an attention computation structure for modeling the spatiotemporal correlation of the target image frame sequence tensor; and performing scale restoration on the multi-scale feature map, fusing the scale-restored multi-scale feature map with the original image sequence data to generate a reconstructed image frame sequence. According to embodiments of this application, by constructing an image frame sequence tensor from the original image sequence and introducing a multi-scale sampling mechanism, the structural information of the image at different spatial resolutions can be fully extracted, improving the model's ability to perceive image details and contextual structure. At the same time, by inputting the multi-scale target image frame sequence into a feature extraction and reconstruction network containing an attention computation structure in parallel and modeling the spatiotemporal correlation of the image frame sequence, the dynamic changes between frames and the global dependencies within the image can be effectively captured, thereby improving the consistency and accuracy of image reconstruction.

[0038] It should be noted that the image processing methods, apparatus, devices, media, and program products defined in this application can be used in the fields of artificial intelligence, biometrics, and fintech, and can also be used in various other fields besides artificial intelligence, biometrics, and fintech. The application fields of the image processing methods, apparatus, devices, media, and program products provided in the embodiments of this application are not limited.

[0039] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0040] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0041] Figure 1 The illustrations depict application scenarios of image processing methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0042] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0043] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0044] In the embodiments of this application, the first terminal device 101 can be an example of the first device, and the second terminal device 102 and / or the third terminal device 103 can be an example of at least one second device. The first device and the second device can communicate collaboratively through an internal client mechanism to implement the data distribution and rendering logic described in the image processing method.

[0045] In some embodiments, the first device and at least one second device may be different display modules, windows or screens on the same computing terminal (such as a host), or multiple physical devices that work together through a network, such as different client instances deployed on a desktop computer, tablet terminal or mobile device respectively.

[0046] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smart mobile terminals, tablet computers, laptop computers, and desktop computers.

[0047] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0048] It should be noted that the image processing method provided in this application embodiment can generally be executed by server 105. Correspondingly, the image processing apparatus provided in this application embodiment can generally be located in server 105. The image processing method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the image processing apparatus provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0049] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0050] The following will be based on Figure 1 The described scene, through Figures 2-4 The image processing method of the disclosed embodiments will be described in detail.

[0051] Figure 2 A flowchart illustrating an image processing method according to an embodiment of this application is shown schematically.

[0052] like Figure 2 As shown, the image processing method 200 of this embodiment includes operations S210 to S240.

[0053] In operation S210, the original image sequence data is acquired, and an image frame sequence tensor is constructed based on the original image sequence data.

[0054] In embodiments of this application, acquiring raw image sequence data may include obtaining a continuous sequence of image frames from a video data source to be processed. The raw image sequence data may originate from various image acquisition terminals such as surveillance cameras, smartphones, drones, or industrial inspection equipment. The system can employ differentiated image preprocessing strategies for different types of data sources to improve the efficiency and accuracy of model building.

[0055] In some embodiments, the original image sequence data may contain a video stream with a fixed frame rate, such as video clips at 25 or 30 frames per second. The system can extract corresponding image frame subsequences according to a set processing window (e.g., 5 consecutive frames, 7 frames, or more frames) and organize them into tensor representations with uniform size, color space, and data type. To ensure the robustness of tensor construction, the image frames can be further normalized, including size normalization, pixel value normalization (e.g., scaling pixel values ​​to the [0, 1] interval or a standard Gaussian distribution), image alignment, or background removal.

[0056] In the embodiments of this application, if the resolution or format of the original image sequence data is inconsistent, the system can introduce an image resampling mechanism to unify the scale of the original frames through bilinear interpolation, cubic interpolation, or reconstruction methods based on super-resolution networks, thereby constructing an image frame sequence tensor with a consistent shape. Furthermore, to enhance the system's ability to perceive the temporal characteristics of the video, temporal coding information can be introduced into the channel dimension when constructing the tensor. For example, timestamps, frame numbers, or inter-frame position information can be encoded and injected as additional channels, further improving the subsequent network's ability to model temporal dependencies.

[0057] In other embodiments, for high frame rate video data, to reduce system processing overhead and suppress inter-frame redundancy, the system can perform frame selection operations on the original image sequence, such as using uniform sampling, motion activation sampling, or keyframe extraction algorithms, to select a representative set of image frames to construct tensor input. This operation not only helps compress the data size but also improves the focus and generalization ability of subsequent feature extraction and reconstruction models.

[0058] It should be noted that the construction method of image frame sequence tensors is flexible and can be adjusted according to different task objectives and model structures. For example, it can support grayscale or color image processing and can adopt a tensor layout structure of (N,T,C,H,W) or (N,C,T,H,W), where N represents the batch size, T represents the number of time frames, C represents the number of channels, and H and W represent the height and width of the image, respectively. The embodiments of this application are not limited to a specific tensor format, but allow for the customization of tensor structures according to application scenarios to adapt to different neural network computing needs.

[0059] In operation S220, the image frame sequence tensor is sampled at multiple scales to generate multiple target image frame sequence tensors with different resolutions.

[0060] In the embodiments of this application, multi-scale sampling can be implemented by downsampling, upsampling or adaptive reconstruction, and the specific method can be flexibly configured according to the processing target and computing resources.

[0061] For example, multi-scale sampling can be implemented based on an image pyramid structure, that is, from the original image frame sequence tensor, several image frame sequences with progressively decreasing resolutions are constructed through bilinear interpolation or average pooling. For instance, while keeping the time dimension unchanged, the system can construct three sets of image frame tensors at the original resolution, half resolution, and quarter resolution, thereby achieving the ability to extract spatial features at different scales. These tensors at different resolutions can be input in parallel into subsequent feature extraction and reconstruction networks, enhancing the joint modeling capability of detail and structural information.

[0062] For example, multi-scale sampling is reflected not only in changes in spatial resolution but also in changes in temporal resolution. For instance, the system can sparsely sample or resample image frame sequences along the time axis to construct image subsequence tensors representing different temporal window granularities. This processing helps improve the system's ability to model dynamic scene changes and is suitable for video content recognition and reconstruction tasks in fast-moving scenes or with periodic changes.

[0063] It should be noted that the unified resolution processing in the embodiment of S210 is mainly to solve the problem of inconsistent input size of the original image sequence data, while the multi-scale sampling in S220 is a scale transformation performed inside the network to enhance feature diversity and spatial awareness.

[0064] In some embodiments, to enhance the expressive power of the sampling results, the system may employ learnable sampling modules, such as spatial transformation networks, content-aware resampling networks, or super-resolution networks, so that the generation of multi-scale image tensors no longer depends on fixed interpolation rules, but dynamically determines the sampling position and scaling strategy according to the image content, thereby improving the overall sampling quality and downstream modeling efficiency.

[0065] According to embodiments of this application, the multi-scale sampling process can also be combined with image processing strategies such as edge-preserving filtering and region attention enhancement to preserve key details of the target region as much as possible during downsampling. For example, for image frame sequences with complex textures or fine edges, the system can introduce a structure-sensitive weighting mechanism to give higher sampling density to key regions during sampling, thereby mitigating the information loss problem that may occur during scale transformation.

[0066] It should be noted that the embodiments of this application are not limited to a fixed sampling level or resolution combination in multi-scale sampling operations, but can be flexibly configured according to the perception requirements of the actual application scenario, custom scale combination strategies, and the input requirements of specific models. Through the multi-scale sampling mechanism, the system can fully integrate the low-level details and high-level semantics of image sequences, providing a richer structural information foundation for subsequent feature extraction and image reconstruction.

[0067] In operation S230, the target image frame sequence tensor is input in parallel to the feature extraction and reconstruction network to obtain a multi-scale feature map. The feature extraction and reconstruction network includes an attention computation structure for modeling the spatiotemporal correlation of the target image frame sequence tensor.

[0068] In the embodiments of this application, the feature extraction and reconstruction network can be a type of deep convolutional neural network structure, used for joint modeling and enhancement of spatiotemporal features of the input multi-scale target image frame sequence tensor. The system can input multiple image frame sequence tensors of different resolutions in parallel into different branches of the network or a shared weight structure, enabling the network to independently or collaboratively extract representative spatial and temporal features at each scale level, ultimately forming a multi-scale feature map for subsequent fusion and reconstruction. This network structure supports both hierarchical encoding and decoding frameworks and distributed multi-branch modeling architectures to adapt to image restoration requirements under complex spatiotemporal patterns.

[0069] In some embodiments, multiple sets of image frame sequence tensors with different resolutions obtained through multi-scale sampling can be input in parallel into multiple encoder-decoder structures with the same network structure and shared weights for feature extraction and refinement. Since the tensors of the multiple input image frame sequences have different resolutions, the network can independently model feature patterns at each scale while maintaining consistent weights, thereby fully extracting the feature representations of the target image frame sequence at different spatial scales. Shared weights ensure the consistency and comparability of feature extraction across scales, facilitating alignment and integration in subsequent scale restoration and fusion operations, while avoiding model parameter redundancy and training instability.

[0070] In the embodiments of this application, a multi-layer attention computation structure can be integrated into the feature extraction and reconstruction network to explicitly model the dependencies between image frames in the time series and inter-frame redundancy information. The attention computation structure can be a temporal attention mechanism, a spatial attention mechanism, or a combination thereof, which dynamically allocates weights to highlight image regions and keyframes with significant semantic or motion features, thereby improving the recognition and representation capabilities of target regions. The attention computation structure can be embedded in various scale pathways of the network, such as between encoder-decoder structures, or it can serve as a bridge for cross-scale information integration, enhancing information interaction and joint modeling between feature maps.

[0071] In other embodiments, to further enhance the modeling capability of dynamic image sequences, the system can employ a 3D convolutional network or a temporal convolutional network structure to process multi-scale image tensors. By utilizing the temporal expansion of the convolutional kernel, motion trajectory information can be extracted, enhancing the system's temporal sensitivity to target change patterns. Furthermore, the system can introduce recurrent neural network modules such as gated recurrent units and long short-term memory networks to perform long-term modeling of semantic continuity between frames, thereby effectively handling gradual changes or occlusion issues in the scene.

[0072] In some specific applications, the system can be configured with a U-shaped network or a multi-path Transformer structure with skip connections to perform contextual alignment and fusion of feature maps at different scales, while suppressing redundant or irrelevant regions through an attention mechanism. This design not only improves the model's ability to reconstruct fine structures, but also enhances the model's performance in texture restoration and edge sharpness preservation in complex scenes.

[0073] It should be noted that the feature extraction and reconstruction network in the embodiments of this application is not limited to a specific network depth or module combination, but can be flexibly tailored and expanded according to the hardware environment, target task, and model training strategy. Regarding the integration method of the attention computation structure, lightweight attention or multi-head self-attention mechanisms can be selected according to performance requirements, thereby achieving a trade-off between model accuracy and inference efficiency.

[0074] In operation S240, the scale of the multi-scale feature map is restored, and the scale-restored multi-scale feature map is fused with the original image sequence data to generate a reconstructed image frame sequence.

[0075] In the embodiments of this application, the multi-scale feature maps have different resolutions due to their origin from different sampling scales. Therefore, they need to be restored to a unified scale through upsampling or decoding modules before subsequent fusion operations can be performed. Scale restoration can be achieved based on deconvolution (transposed convolution), bilinear interpolation, content-aware upsampling, or pixel shifting, etc. The specific method can be flexibly selected according to the system performance requirements and reconstruction accuracy requirements.

[0076] In some embodiments, to avoid detail blurring or artifacts caused by simple upsampling, the system can introduce a scale alignment module to coordinate the spatial and semantic domain adjustments of feature maps at each scale, ensuring that the restored feature maps are structurally aligned with the original image content. For example, a cross-scale residual enhancement mechanism can be introduced during the restoration stage to supplement high-semantic information from low-scale features into high-scale features, thereby improving the expressive power of the final fused image. Furthermore, the scale restoration process can also incorporate an attention mechanism to guide the restoration process to focus on regions with rich textures or prominent edges in the image, thereby improving the restoration quality of key details.

[0077] In the feature fusion stage, the system can integrate the scale-restored multi-scale feature maps with the original image sequence data using methods such as channel concatenation, element-weighted fusion, gated fusion, and dynamic convolution. This fusion process not only restores the original structure of the image but also enhances the coherence and realism of the content by utilizing the contextual semantics extracted by the network. For example, in element-level fusion, the system can assign dynamic fusion weights to feature maps from different sources to adapt to the differences in feature reliability under different scenarios, thereby improving the adaptability and robustness of the fusion result.

[0078] In other embodiments, the system can also be designed with a multi-stage fusion strategy, that is, first fusing multi-scale reconstruction results within the same scale, and then fusing them with the original image tensor, or constructing local and global reconstruction paths in stages, thereby modeling local texture details and the relationship between the overall structure in greater detail. This design is suitable for scenarios requiring high-precision restoration, such as medical image enhancement, remote sensing image restoration, or video surveillance restoration.

[0079] It should be noted that the scale restoration and feature fusion process in this application can form a closed-loop feedback with the prior feature extraction stage. For example, the original image can be used as a guiding signal to be fed back into the restoration network through residual connection or deep supervision strategy to improve the convergence speed of the training stage and the accuracy and stability of the reconstruction stage.

[0080] According to embodiments of this application, high-quality reconstruction of image frame sequences can be achieved by constructing tensor representations from the original image sequence data, introducing a multi-scale sampling mechanism to generate image frame sequences of different resolutions, and utilizing an attention feature extraction and reconstruction network with spatiotemporal correlation modeling capabilities to extract multi-scale features in parallel. The method provided by the embodiments of this application can take into account both the global structure and local details of the image, effectively improving the clarity, temporal consistency, and semantic integrity of image reconstruction, and is applicable to video enhancement and image restoration tasks in various dynamic scenarios.

[0081] The image processing method of this application will be specifically described below by way of preferred embodiments.

[0082] In the embodiments of this application, the process of multi-scale sampling of the image frame sequence tensor may further include: calculating the content complexity index of each frame image based on the content of each frame image in the image frame sequence tensor, and dynamically determining the scale level division method and sampling strategy information of each frame image in the multi-scale sampling process based on this, thereby realizing more refined and adaptive multi-scale feature modeling.

[0083] Specifically, content complexity metrics can include image edge density, texture gradient intensity, color variation amplitude, structural entropy, local frequency features, or activation intensity statistics based on convolutional feature maps. For each image frame, the system can evaluate its perceptual structural richness and detail complexity by analyzing its distribution characteristics in the spatial and frequency domains. For image frames with higher complexity (e.g., images containing fine textures, sharp edges, or multiple objects), the system can allocate a higher resolution sampling scale and retain more fine-grained image feature information; while for image frames with lower complexity (e.g., background areas, low-texture areas, or dynamically blurred areas), a lower resolution sampling scale can be used to reduce redundant computation and improve overall processing efficiency.

[0084] In some embodiments, an adaptive sampling strategy generation module may be introduced to further generate sampling strategy information for each frame of image based on content complexity metrics. Examples include the number of sampling levels, scale factor, whether to retain high-scale paths, and the sparsity density of the sampling region. The sampling strategy information can be implemented through rule setting, model prediction, or an end-to-end learnable module, and used as a control parameter in the sampling process to dynamically adjust the distribution of input image frames in multi-scale paths.

[0085] Furthermore, global content complexity modeling can be performed on the entire image frame sequence. Considering the complexity of a single frame, inter-frame variation trends and redundancy information within the time domain are integrated to achieve joint scale control at the frame and sequence levels. For example, if a segment of an image sequence is found to have slow changes and high structural repetition, a unified low-scale strategy can be applied to that segment to reduce unnecessary redundant computation. During periods of rapid motion, occlusion changes, or the introduction of new objects, more scale branches can be automatically activated to enhance feature extraction capabilities.

[0086] The content complexity-driven multi-scale sampling strategy not only improves the system's adaptability and expressive power in complex scenarios, but also significantly optimizes resource allocation efficiency, enabling the feature extraction and reconstruction network to maintain high-quality reconstruction performance while having better computational controllability.

[0087] In the embodiments of this application, in order to effectively model the spatiotemporal correlation in the tensor of the target image frame sequence, a local self-attention mechanism based on window partitioning and cross-layer offset can be adopted, thereby integrating the local sensitivity of spatial structure modeling with the global expressive ability of cross-region feature interaction.

[0088] Figure 3 The flowchart illustrates a method for modeling the spatiotemporal correlation of a target image frame sequence tensor according to some exemplary embodiments of this application.

[0089] like Figure 3 As shown, the method for modeling the spatiotemporal correlation of the target image frame sequence tensor includes operations S310 to S320.

[0090] In operation S310, the target image frame sequence tensor is divided into windows along the spatial dimension, and local self-attention calculation is performed in each window to obtain spatially dependent features.

[0091] Specifically, the input target image frame sequence tensor can be windowed along the spatial dimension at each time frame. For example, each frame can be divided into several local windows of a fixed size (such as 4×4 or 8×8), with each window containing image blocks or feature blocks of a continuous spatial region. Within each window, self-attention computation can be performed to capture the dependencies between pixels or features within that region.

[0092] According to embodiments of this application, the local self-attention mechanism can significantly reduce computational complexity, avoid the resource consumption problem of global self-attention in high-resolution image scenes, and at the same time maintain the sensitivity of local details, making it suitable for image processing tasks with complex textures and dense targets.

[0093] In operation S320, the window partitioning position is offset between adjacent processing layers of the attention calculation structure, so that the spatially dependent features interact between different windows to model the spatiotemporal correlation of the target image frame sequence tensor.

[0094] In the embodiments of this application, to compensate for the lack of cross-window information caused by fixed windows, a window offset strategy can be further introduced between adjacent attention processing layers. This involves offsetting the window division position by a fixed step size in the horizontal or vertical direction, so that pixels in the same region may be assigned to different windows in the later layer. Through this "staggered" or "sliding" window mechanism, feature interaction between different windows in the preceding and following processing layers can be achieved, thereby establishing cross-window spatial connections and enhancing the information flow between local attention.

[0095] Furthermore, considering that the input is an image frame sequence tensor, the aforementioned attention calculation structure can be kept synchronized in the temporal dimension. This ensures the consistency of window structures at the same spatial location across different time frames, thereby improving the model's ability to track moving targets and reducing jitter effects during the temporal propagation of attention weights, thus enhancing the stability of object features in the video sequence. In addition, a cross-time frame attention mechanism can be introduced, where spatial windows between adjacent frames are selected in a specific layer for joint modeling. This allows for further modeling of temporal dependencies based on spatial attention, improving the modeling ability for complex dynamic phenomena such as action changes, object movement, and occlusion recovery.

[0096] In some embodiments, a multi-scale window partitioning strategy can be combined to perform window attention and offset modeling at different resolutions, thereby adapting to feature changes at different levels in the image. For example, a large-scale window can capture the overall motion trend of larger structures in the image, while a small-scale window focuses on texture and edge completion at the detail level; by jointly modeling across scales, the hierarchical expressive power of the model can be enhanced.

[0097] By employing a local attention computation structure based on window partitioning and cross-layer offset, efficient and in-depth spatiotemporal correlation modeling can be achieved in the spatial dimension. Furthermore, through multi-level, multi-scale, and cross-frame fusion mechanisms, the model's ability to understand temporal consistency and structural integrity in image sequences is enhanced.

[0098] In embodiments of this application, the feature extraction and reconstruction network may further include a residual feature extraction structure, which is used to extract local spatial features from the target image frame sequence tensor, wherein the local spatial features are at least used to characterize the detailed texture information in the target image frame sequence tensor.

[0099] Specifically, the residual feature extraction structure can be a residual block. Residual blocks can capture edge structures, texture variations, or subtle local contrast features in an image, thereby further enhancing the modeling ability for local details while maintaining global consistency. The extracted local spatial features can at least characterize the detailed texture information in the image frame sequence and are fused with other scale features in the subsequent reconstruction process to improve image restoration quality, enhance structural restoration accuracy, and improve subjective perceived clarity. The introduction of this structure allows the model to more fully utilize the rich low-level features in the input image frame sequence, thereby enhancing the overall reconstruction ability and detail recovery effect of the network.

[0100] Optionally, local spatial features may also include edge structure information for accurately characterizing the boundary contours and structural transition regions of objects in the image, abnormal or abrupt region features or context compensation information representing local abnormal regions such as occlusion, noise, and compression artifacts in the image, and information on small targets or fine-grained objects.

[0101] In the embodiments of this application, in order to more effectively model the temporal dynamic characteristics in the image frame sequence tensor, it is also possible to introduce: based on adjacent time frames in the image frame sequence tensor, calculate optical flow information, and fuse the optical flow information into the processing flow of the residual feature extraction structure and attention calculation structure in the feature extraction and reconstruction network, respectively, so as to improve the ability to model the spatiotemporal coupling relationship in dynamic image sequences.

[0102] Specifically, classical or deep learning-driven optical flow estimation algorithms can be used to estimate the pixel-level displacement field between any two or more adjacent frames in the image frame sequence tensor, thereby obtaining an optical flow vector map. The optical flow vector map not only contains the direction and velocity of the object's motion in space, but also contains temporal consistency information between frames, serving as an intermediate feature to characterize the spatiotemporal changes of the image.

[0103] Since residual structures focus on minute differences in an image, a lack of motion compensation can easily lead to temporal misalignment. Therefore, introducing optical flow as a guiding feature helps align locally moving regions between frames, improving extraction accuracy. The system can use the optical flow field as a guiding feature, pairing it with local image patches in the current frame to dynamically align texture regions that have moved or deformed in adjacent frames. For example, guided by optical flow, the network can identify and enhance feature extraction at moving edges, thereby better recovering motion blur, occlusion reproduction, or dynamic texture details.

[0104] In the embodiments of this application, optical flow information can also be used to construct motion-guided residual paths, enabling the residual learning module to focus on the changes in motion regions between frames, thereby improving the modeling accuracy of dynamic details and the ability to restore structures.

[0105] Integrating optical flow information into the attention computation structure can further enhance the selectivity and sensitivity of the attention mechanism in the spatiotemporal dimensions. The optical flow field can serve as a weight modulation signal in the calculation of attention weights, guiding the attention mechanism to focus more on regions with active inter-frame motion or window segments with prominent motion changes. For example, in local window attention computation, the system can use the amplitude or direction of optical flow as part of the spatial location encoding, guiding the network to emphasize feature aggregation in dynamic regions during attention allocation. Under the window offset strategy, optical flow can also assist in determining the dynamic relative position of windows, enhancing the feature linkage and structural alignment capabilities between windows.

[0106] Furthermore, optical flow information can also be used to enhance the spatiotemporal consistency modeling between multi-scale paths. For example, during multi-scale sampling, image frames at different resolutions may experience loss or weakening of motion information. The system can inject downsampled optical flow maps into the feature maps at each scale as dynamic priors to supplement the missing temporal structure, thereby improving the matching quality and reconstruction stability during the scale restoration and fusion stages.

[0107] By integrating optical flow information calculated based on image frame sequences into the processing flow of residual feature extraction structure and attention calculation structure, the embodiments of this application can not only improve the ability to express details in the spatial dimension, but also enhance the dynamic consistency expression between frames in the temporal dimension, effectively supporting high-quality, strongly temporally stable image frame sequence reconstruction tasks, and further expanding its application in complex tasks such as dynamic image restoration, multi-frame super-resolution and video reconstruction.

[0108] Figure 4 The illustration shows a schematic diagram of a method for feature fusion according to some exemplary embodiments of the present application.

[0109] like Figure 4 As shown, the method for performing feature fusion may include operations S410 to S430.

[0110] In operation S410, spatial alignment and channel mapping are performed on the scaled multi-scale feature maps.

[0111] Specifically, feature preprocessing, including spatial alignment and channel mapping, can be performed on multi-scale feature maps that have undergone scale restoration. This ensures that features from different scales and sources have a uniform size distribution in the spatial dimension and matches the structural requirements of the fusion network in the channel dimension. Spatial alignment can be achieved through upsampling, interpolation, or cropping, while channel mapping can be accomplished using 1×1 convolutions, fully connected layers, or learnable linear mapping modules to unify the number of channels and semantic distribution of different features.

[0112] In operation S420, the processed multi-scale feature map is fused with the original image sequence data of the corresponding time frame.

[0113] After feature preprocessing, the processed multi-scale feature maps can be fused with the corresponding time frames in the original image sequence data at the feature level. The fusion process can be implemented in various ways, such as channel stitching, element-wise addition, weighted fusion, attention-guided fusion, or dynamic convolutional fusion. Specific fusion strategies can be flexibly selected based on task complexity, computational resources, and model design. In some embodiments, an explicit fusion module can be constructed, introducing a spatiotemporal attention mechanism or a channel selection module to assign different fusion weights to features from different sources, enhancing the fusion strength of dynamic regions, edge regions, or texture regions, thereby improving the overall reconstruction quality and visual consistency.

[0114] In operation S430, the fusion result is input into the convolutional reconstruction subnetwork of the feature extraction and reconstruction network to generate the reconstructed image frame sequence.

[0115] In the embodiments of this application, the fusion result can be input into the convolutional reconstruction sub-network in the feature extraction and reconstruction network to generate the final reconstructed image frame sequence. The convolutional reconstruction sub-network can be a shallow or deep convolutional structure, capable of progressively restoring image content. In practical implementation, the convolutional reconstruction sub-network can adopt a residual network structure or other residual-enhanced encoder-decoder structure, combined with an upsampling module and a skip connection strategy, to achieve the restoration of fused features to the original image frame space.

[0116] By using the above method, not only are the low-level feature information from the perceptual source data in the original image sequence preserved, but also the multi-scale semantic and context-enhanced features obtained through the feature extraction network are superimposed, thereby achieving a combination between data-driven foundation and semantic enhancement modeling.

[0117] In other embodiments, to enhance the temporal consistency of reconstructed images across different time frames, the convolutional reconstruction subnetwork may also introduce inter-frame residual supervision or temporal consistency loss functions to constrain the dynamic change trends between adjacent output images and avoid phenomena such as frame skipping and texture drift caused by fusion errors or scale conversion.

[0118] Preferably, the original image sequence data may include at least the original face image sequence data. To improve the ability to maintain facial identity consistency during reconstruction, embodiments of this application may further include: obtaining an identity feature vector corresponding to the original face image sequence data, and dynamically adjusting the attention weight distribution in the attention calculation structure based on the identity feature vector.

[0119] Specifically, the system can use a pre-trained face recognition model to extract features from the original face image sequence to obtain an identity feature vector representing an individual's identity. The identity feature vector can be regarded as an embedded expression of a person's identity information, which has strong discriminativeness and stability, and can remain consistent under different expressions, angles, and lighting conditions.

[0120] In the attention calculation structure of the feature extraction and reconstruction network, an identity-guided attention control mechanism can be introduced. This means that when performing spatiotemporal attention allocation, not only are the semantic and structural relationships of the input features considered, but the calculation process of attention weights is also guided based on the aforementioned identity feature vectors. Specifically, the system can map the identity feature vectors and interact with each attention window or attention head to form identity-related weight modulation factors, thereby enhancing the model's ability to focus on key facial regions (such as eyes, corners of the mouth, and facial contours) that have identity-discriminating value.

[0121] In some embodiments, the identity guidance mechanism can also be extended to a cross-frame consistency enhancement module, that is, by utilizing the consistency of facial identity features in different frames within the same sequence, the attention mechanism is guided to suppress inter-frame drift or accumulation of reconstruction errors during reconstruction. For example, for frames in the input facial image sequence where some regions are missing due to pose or occlusion, the system can refer to the stable identity feature information extracted from other frames to redirect the attention focus, thereby completing the structural completion and identity consistency restoration of the missing regions.

[0122] By introducing an identity feature vector corresponding to the original face image sequence data and using this vector to dynamically adjust the weight distribution of the attention calculation structure, the system can effectively maintain the consistency and stability of the person's identity features during the reconstruction process, significantly improve the quality of detail reconstruction and semantic expression accuracy of the face region, and lay a reliable structural foundation for subsequent face recognition, comparison, editing and other tasks.

[0123] Furthermore, in scenarios where the original image sequence data includes the original face image sequence data, after the reconstructed image frame sequence is generated, the reconstructed image frame sequence can be assembled in chronological order to form a dynamic face image frame sequence; and the dynamic face image frame sequence can be converted to a target format face image data to meet the needs of subsequent storage, display or downstream processing tasks.

[0124] Specifically, after reconstructing the image frame sequence, the system can sort and integrate the images in an orderly manner based on the timestamps or the original input order to construct a dynamic face image frame sequence that conforms to temporal logic. During this process, inter-frame interpolation, redundant frame removal, or time alignment operations can be introduced to further improve the dynamic consistency and visual coherence of the output sequence. For example, when some frames have temporal breaks due to missing original input or reconstruction failure, the system can automatically fill in the missing frame content through interpolation prediction algorithms. Simultaneously, redundant frames that repeat within a short period or show no effective change can be merged or compressed to improve output efficiency and bandwidth adaptability.

[0125] After generating a complete sequence of dynamic image frames, the system can perform type conversion according to the target application requirements, outputting face image data in a specified format. Type conversion can cover multiple dimensions, such as data encoding format conversion, temporal encapsulation format conversion, and multi-channel permutation format conversion. Furthermore, the system can also support outputting data in a format compatible with neural network inference interfaces, allowing for direct input to downstream processing such as face recognition, expression analysis, and pose estimation.

[0126] In some application scenarios, the type conversion process can also embed privacy protection strategies or image encryption mechanisms. For example, by occluding or blurring the output face data, or by combining watermark embedding and hash fingerprint encryption technologies, compliance control and security assurance can be achieved in the data output process.

[0127] Through the above frame order assembly and type conversion operations, the system not only ensures the continuity and expressive integrity of the reconstructed image frames in the time dimension, but also achieves seamless integration with various terminal devices, application systems or network transmission protocols, greatly improving the applicability and deployment efficiency of the system in various practical scenarios such as face video enhancement, identity authentication, facial animation generation, remote collaboration and communication.

[0128] According to embodiments of this application, an image frame reconstruction method integrating multi-scale perception and spatiotemporal modeling is constructed. By combining multi-scale sampling, self-attention mechanisms, residual feature extraction, and optical flow guidance, the detail fidelity and temporal consistency of image sequence reconstruction are effectively improved. A sampling strategy driven by dynamic content complexity achieves synergistic optimization of accuracy and efficiency; an attention control mechanism guided by identity features enhances identity stability during face reconstruction; and the ability to integrate the ordered assembly and format conversion of reconstructed image frames further improves the system's adaptability and practicality in real-world scenarios.

[0129] Corresponding to the above image processing method, embodiments of this application also provide an image processing apparatus.

[0130] Figure 5A schematic block diagram of an image processing apparatus according to an embodiment of this application is shown.

[0131] like Figure 5 As shown, the image processing apparatus 500 of this embodiment includes a data acquisition module 510, a multi-scale sampling module 520, a feature extraction module 530, and an image reconstruction module 540.

[0132] The data acquisition module 510 can be used to acquire raw image sequence data and construct an image frame sequence tensor based on the raw image sequence data. In one embodiment, the data acquisition module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0133] The multi-scale sampling module 520 can be used to perform multi-scale sampling on the image frame sequence tensor to generate multiple target image frame sequence tensors with different resolutions. In one embodiment, the multi-scale sampling module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0134] The feature extraction module 530 can be used to input the target image frame sequence tensor into the feature extraction and reconstruction network in parallel to obtain a multi-scale feature map. The feature extraction and reconstruction network includes an attention computation structure for modeling the spatiotemporal correlation of the target image frame sequence tensor. In one embodiment, the feature extraction module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0135] The image reconstruction module 540 can be used to scale-restore the multi-scale feature map, and then fuse the scale-restored multi-scale feature map with the original image sequence data to generate a reconstructed image frame sequence. In one embodiment, the image reconstruction module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0136] According to an embodiment of this application, the multi-scale sampling module 520 can also be used to calculate the content complexity index of each frame image based on the image frame sequence tensor; and to determine the scale level and sampling strategy information of the image frame sequence tensor in multi-scale sampling based on the content complexity index.

[0137] According to an embodiment of this application, the feature extraction module 530 can also be used to divide the target image frame sequence tensor into windows along the spatial dimension, perform local self-attention calculation in each window to obtain spatially dependent features; and offset the window division position between adjacent processing layers of the attention calculation structure so that the spatially dependent features can interact between different windows to model the spatiotemporal correlation of the target image frame sequence tensor.

[0138] According to an embodiment of this application, the feature extraction module 530 further includes a residual feature extraction module. The residual feature extraction module can be used to extract local spatial features from the target image frame sequence tensor, wherein the local spatial features are at least used to characterize the detailed texture information in the target image frame sequence tensor.

[0139] According to an embodiment of this application, the image processing apparatus 500 further includes an optical flow information processing module. The optical flow information processing module can be used to calculate optical flow information based on adjacent time frames in the image frame sequence tensor; and to fuse the optical flow information into the processing flow of the residual feature extraction structure and the attention calculation structure, respectively.

[0140] According to an embodiment of this application, the image reconstruction module 540 can also be used to perform spatial alignment and channel mapping processing on the scale-restored multi-scale feature map; fuse the processed multi-scale feature map with the original image sequence data of the corresponding time frame; and input the fusion result into the convolutional reconstruction sub-network of the feature extraction and reconstruction network to generate the reconstructed image frame sequence.

[0141] According to an embodiment of this application, the image processing apparatus 500 further includes a face image processing module. The face image processing module can be used to obtain an identity feature vector corresponding to the original face image sequence data; and to adjust the attention weights of the attention calculation structure based on the identity feature vector.

[0142] According to an embodiment of this application, the face image processing module can also be used to assemble the reconstructed image frame sequence in chronological order to form a face dynamic image frame sequence; and to perform type conversion on the face dynamic image frame sequence to output face image data in the target format.

[0143] According to embodiments of this application, any multiple modules among the data acquisition module 510, multi-scale sampling module 520, feature extraction module 530, and image reconstruction module 540 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the data acquisition module 510, multi-scale sampling module 520, feature extraction module 530, and image reconstruction module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the data acquisition module 510, the multi-scale sampling module 520, the feature extraction module 530, and the image reconstruction module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0144] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing an image processing method according to an embodiment of this application.

[0145] like Figure 6 As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0146] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0147] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0148] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0149] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0150] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the image processing method provided in the embodiments of this application.

[0151] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0152] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0153] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0154] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0156] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0157] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the original image sequence data, and construct an image frame sequence tensor based on the original image sequence data; Multi-scale sampling is performed on the image frame sequence tensor to generate multiple target image frame sequence tensors with different resolutions; The target image frame sequence tensor is input in parallel into a feature extraction and reconstruction network to obtain a multi-scale feature map. The feature extraction and reconstruction network includes an attention computation structure for modeling the spatiotemporal correlation of the target image frame sequence tensor; and The multi-scale feature map is scaled back, and the scaled-backed multi-scale feature map is fused with the original image sequence data to generate a reconstructed image frame sequence.

2. The method according to claim 1, characterized in that, The modeling of the spatiotemporal correlation of the target image frame sequence tensor includes: The target image frame sequence tensor is divided into windows along the spatial dimension, and local self-attention computation is performed within each window to obtain spatial dependency features; and The window partitioning position is offset between adjacent processing layers of the attention calculation structure, so that the spatially dependent features can interact between different windows to model the spatiotemporal correlation of the target image frame sequence tensor.

3. The method according to claim 1 or 2, characterized in that, The feature extraction and reconstruction network further includes a residual feature extraction structure, which is used to extract local spatial features from the target image frame sequence tensor. The local spatial features are used to characterize at least the detailed texture information in the target image frame sequence tensor.

4. The method according to claim 3, characterized in that, The method further includes: Based on adjacent time frames in the image frame sequence tensor, optical flow information is calculated; and The optical flow information is fused into the processing flow of the residual feature extraction structure and the attention calculation structure, respectively.

5. The method according to claim 1, characterized in that, The step of fusing the scale-restored multi-scale feature maps with the original image sequence data to generate a reconstructed image frame sequence includes: Spatial alignment and channel mapping are performed on the scaled-restored multi-scale feature maps. The processed multi-scale feature maps are fused with the original image sequence data of the corresponding time frames; and The fusion result is input into the convolutional reconstruction subnetwork of the feature extraction and reconstruction network to generate the reconstructed image frame sequence.

6. The method according to claim 1, characterized in that, The original image sequence data includes at least original face image sequence data, and the method further includes: Obtain the identity feature vector corresponding to the original face image sequence data; and Based on the identity feature vector, the attention weights of the attention calculation structure are adjusted.

7. The method according to claim 6, characterized in that, The method further includes: The reconstructed image frame sequence is assembled in chronological order to form a dynamic face image frame sequence; and The sequence of dynamic face images is converted to a different type, and the output is face image data in the target format.

8. The method according to claim 1, characterized in that, The multi-scale sampling of the image frame sequence tensor includes: Based on the image frame sequence tensor, calculate the content complexity index of each image frame; and Based on the content complexity index, the scale level and sampling strategy information of the image frame sequence tensor in multi-scale sampling are determined.

9. An image processing apparatus, characterized in that, The device includes: The data acquisition module is used to: acquire raw image sequence data and construct an image frame sequence tensor based on the raw image sequence data; The multi-scale sampling module is used to: perform multi-scale sampling on the image frame sequence tensor to generate multiple target image frame sequence tensors with different resolutions; The feature extraction module is used to: input the target image frame sequence tensor in parallel into a feature extraction and reconstruction network to obtain a multi-scale feature map, wherein the feature extraction and reconstruction network includes an attention computation structure for modeling the spatiotemporal correlation of the target image frame sequence tensor; and The image reconstruction module is used to: scale the multi-scale feature map, fuse the scale-restored multi-scale feature map with the original image sequence data, and generate a reconstructed image frame sequence.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images