A damaged image repairing method and device, electronic equipment and storage medium

Abstract

The application discloses a damaged image repairing method and device, electronic equipment and storage medium, and relates to the technical field of image repairing, which mainly comprises the following steps: inputting a target image sequence containing stains into a damaged image repairing model of a robot binocular vision system based on space-time information fusion to obtain a repaired target image sequence; the model comprises a hierarchical encoder module, a decoupling space-time domain Transformer module connected with the output end of the hierarchical encoder module, an adjacent reference perspective downsampling module, and a multi-frame decoder connected with the output end of the decoupling space-time domain Transformer module and the output end of the adjacent reference perspective downsampling module; and the application makes the final repaired image sequence have single-frame complete content, reasonable structure and no time flicker phenomenon between continuous frames.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image inpainting, in particular to a damaged image inpainting method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the continuous development of autonomous robots, it plays an increasingly important role in the fields of industrial automation, service industry, intelligent security and national security, especially in some extreme environments, autonomous robots are expected to replace humans to do dangerous work. In order to make autonomous robots have autonomous ability, first of all, autonomous robots should be given good environmental perception ability. Visual sensor is widely used in the perception field of autonomous robots because of its advantages of obtaining rich image information and low energy consumption. Due to the complex and unstructured working environment of autonomous robots, the visual sensor is often affected by the external environment or unexpected events, resulting in the attachment of dirt, water droplets and other debris to the visual sensor, so that the obtained image is polluted, and then the autonomous perception and decision operation and other autonomous operation processes of the autonomous robot are affected. Therefore, for autonomous robots working in harsh working conditions, it is extremely important to be able to repair the obtained polluted image online in real time and ensure the integrity and accuracy of the obtained visual image information.

[0003] Image inpainting refers to the reconstruction of the damaged part of the image. The traditional image inpainting method uses the surrounding similar pixels of the damaged area of the image to obtain reference information, and directly copies and edge repair fusion, which makes the repair effect obvious on the image with small damaged area and simple texture, but when dealing with images with large damaged area and complex texture, the repair result of the traditional method often appears semantic structure confusion. For autonomous robots, the images collected have time sequence characteristics, which can be approximated as video information, which puts higher requirements on image inpainting. In addition, most of the existing image inpainting schemes only focus on the intuitive logic consistent with the human eye, without introducing the actual situation of the damaged area for repair, and the final repaired image may be far from the real situation of the environment where the autonomous robot is located, thereby causing the autonomous robot to make wrong decisions and causing serious hidden dangers. Therefore, a kind of image inpainting method is needed which can realize the spatial consistency of the image in real time and ensure the time continuity between video frames SUMMARY

[0004] The purpose of the present application is to provide a damaged image inpainting method, device, electronic equipment and storage medium, so that the final repaired image sequence has single-frame content integrity, reasonable structure and no time flicker phenomenon between consecutive frames.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] In a first aspect, the present application provides a damaged image restoration method, comprising:

[0007] obtaining a target image sequence containing damage; the target image sequence comprises a plurality of continuous target images;

[0008] inputting the target image sequence containing damage into a robot binocular vision system damage image restoration model based on spatio-temporal information fusion to obtain a restored target image sequence;

[0009] The robot binocular vision system damage image restoration model based on spatio-temporal information fusion comprises a hierarchical encoder module, a decoupled spatio-temporal domain Transformer module connected to the output end of the hierarchical encoder module, an adjacent reference view downsampling module, and a multi-frame decoder connected to the output end of the decoupled spatio-temporal domain Transformer module and the output end of the adjacent reference view downsampling module.

[0010] The hierarchical encoder module is configured to:

[0011] extract and fuse the features of each target image to obtain a fused feature map, and group and convolve the fused feature maps of each layer to obtain a target feature map;

[0012] The decoupled spatio-temporal domain Transformer module is configured to process the target feature map to obtain the temporal information and spatial information of the target image, and obtain a temporal information feature map under the current view based on the temporal information and spatial information of the target image.

[0013] The adjacent reference view downsampling module is configured to obtain a feature map under an adjacent reference view.

[0014] The multi-frame decoder is configured to perform correlation calculation and decoding on the feature map under the adjacent reference view and the corresponding feature block in the temporal information feature map under the current view to obtain the restored target image sequence.

[0015] In a second aspect, the present application provides a damaged image restoration device, comprising:

[0016] A data collector is configured to obtain a target image sequence containing damage; the target image sequence comprises a plurality of continuous target images.

[0017] An image restorer is configured to input the target image sequence containing damage into a robot binocular vision system damage image restoration model based on spatio-temporal information fusion to obtain a restored target image sequence.

[0018] The robot binocular vision system damage image repair model based on spatio-temporal information fusion comprises a hierarchical encoder module, a decoupling spatio-temporal domain Transformer module connected with the output end of the hierarchical encoder module, a neighboring reference view downsampling module, and a multi-frame decoder connected with the output end of the decoupling spatio-temporal domain Transformer module and the output end of the neighboring reference view downsampling module.

[0019] The hierarchical encoder module is used for:

[0020] The feature of each target image is extracted and fused to obtain a fused feature map, and the fused feature map of each layer is convolved to obtain a target feature map.

[0021] The decoupling spatio-temporal domain Transformer module is used for processing the target feature map to obtain the time sequence information and the spatial information of the target image, and obtaining a time sequence information feature map in the current view based on the time sequence information and the spatial information of the target image.

[0022] The neighboring reference view downsampling module is used for obtaining a feature map in a neighboring reference view.

[0023] The multi-frame decoder is used for performing correlation calculation and decoding on the feature map in the neighboring reference view and a corresponding feature block in the time sequence information feature map in the current view to obtain a repaired target image sequence.

[0024] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory is used for storing a computer program, and the processor is used for running the computer program to make the electronic device execute the damaged image repair method according to the first aspect.

[0025] In a fourth aspect, a computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the damaged image repair method according to the first aspect.

[0026] According to the embodiments of the present application, the following technical effects are provided:

[0027] The application provides a damaged image repairing method and device, electronic equipment and a storage medium. Mainly including: inputting a target image sequence containing stains into a robot binocular vision system stain image repairing model based on space-time information fusion to obtain a repaired target image sequence. The repaired target image sequence has single-frame complete content, reasonable structure and no time flicker phenomenon between consecutive frames. The robot binocular vision system stain image repairing model based on space-time information fusion comprises a hierarchical encoder module, a decoupled space-time domain Transformer module, an adjacent reference perspective downsampling module and a multi-frame decoder. The hierarchical encoder module is used for extracting the features of each frame of image, fusing and splicing the features between different levels after convolution, and performing group convolution on each layer feature map to finely extract the features of each level. Then the obtained feature map is split into feature blocks and input into the decoupled space-time domain Transformer module, and the decoupled time domain module and the decoupled space module are used to respectively acquire the time sequence information of multiple frames of image and the space information of the current frame of image, so that the model can acquire the texture information required for repairing the stained area in two scales and realize image repairing. Then the adjacent reference perspective downsampling module introduces the space information of the adjacent reference perspective feature image, guarantees the authenticity and robustness of the repair result and improves the repair performance of the model. Finally, the multi-frame decoder module decodes and reconstructs the obtained feature map into multiple frames of image data to complete the repair of the stained image. The application realizes the repair function of the stained image in the binocular autonomous robot application scene, can guarantee that the autonomous robot working in a complex environment is not disturbed by the stained image when perceiving the external environment, and opens up a new way for subsequent research on the repair model. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 The flowchart of the damaged image repairing method of the present application;

[0030] Figure 2 The structure diagram of the robot binocular vision system stain image repairing model based on space-time information fusion of the present application;

[0031] Figure 3 The feature fusion schematic diagram of the hierarchical encoder module of the present application;

[0032] Figure 4 The calculation flowchart of the decoupled space-time domain Transformer module of the present application;

[0033] Figure 5 This is a schematic diagram of the time-domain Transformer module of the present invention;

[0034] Figure 6 This is a schematic diagram of the spatial Transformer module of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] In view of the shortcomings of existing technologies and considering that autonomous robots often use binocular or multi-view vision systems to better perceive the external environment, this invention proposes a method, device, electronic device and storage medium for repairing damaged images based on the spatiotemporal information fusion of spatial information of adjacent reference view images and temporal information of the current view image in a robot binocular vision system.

[0037] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Example 1

[0039] like Figure 1 As shown, the damaged image restoration method provided in this embodiment of the invention includes:

[0040] Step 100: Obtain a sequence of target images containing contamination; the target image sequence includes multiple consecutive frames of target images.

[0041] Step 200: Input the target image sequence containing dirt into the dirt image restoration model of the robot binocular vision system based on spatiotemporal information fusion to obtain the restored target image sequence. The restored target image sequence has complete content in each frame, reasonable structure, and no temporal flickering between consecutive frames.

[0042] like Figure 2As shown in the embodiment of the present invention, the robot binocular vision system image restoration model based on spatiotemporal information fusion includes a hierarchical encoder module, a decoupled spatiotemporal domain Transformer module connected to the output of the hierarchical encoder module, an adjacent reference view downsampling module, and a multi-frame decoder connected to the output of both the decoupled spatiotemporal domain Transformer module and the adjacent reference view downsampling module.

[0043] The hierarchical encoder module is used for:

[0044] Features of each frame of the target image are extracted and fused to obtain a fused feature map, and the fused feature map is then convolved in each layer to obtain the target feature map.

[0045] The decoupled spatiotemporal domain Transformer module is used to: process the target feature map to obtain the temporal and spatial information of the target image, and obtain the temporal information feature map under the current viewpoint based on the temporal and spatial information of the target image.

[0046] The adjacent reference view downsampling module is used to acquire feature maps under adjacent reference views.

[0047] The multi-frame decoder is used to perform correlation calculation and decoding on the corresponding feature blocks in the feature maps of adjacent reference viewpoints and the temporal information feature maps of the current viewpoint, so as to obtain the repaired target image sequence.

[0048] During the training process of the above model, various forms of loss functions are used to constrain the training process of the model, so that the final repaired image sequence has complete content in each frame, reasonable structure, and no temporal flickering between consecutive frames.

[0049] like Figure 3 As shown, the hierarchical encoder module provided in this embodiment of the invention mainly includes pre-processing multiple frames of target images, initially extracting features of each frame of target image, fusing features between different levels to extract more feature information from multiple frames of target images, and then convolving the feature maps of each layer to refine the extraction of features at each level.

[0050] In terms of extracting and fusing features from each frame of the target image to obtain a fused feature map, the hierarchical encoder module is further configured to:

[0051] Pre-extract features from multiple frames of target images;

[0052] The features from different levels are fused to obtain a fused feature map.

[0053] Preferably, the process of pre-extracting features from multiple frames of target images is as follows:

[0054] (1) The scale of the input multi-frame target image data is transformed into target data, namely [b×t,c,h,w], where b, t, c, h, and w represent the batch size, number of frames, channel degree, image height, and image width, respectively.

[0055] (2) The target data is subjected to a preliminary feature extraction operation through a downsampling layer to obtain the first-level feature map.

[0056] Preferably, the feature fusion operation between different levels is as follows:

[0057] (1) The intermediate-level feature map with the same size and wider receptive field obtained after convolving each level feature map is spliced ​​and fused with the first-level feature map (i.e. the feature map obtained by downsampling) along the channel dimension to obtain the next-level feature map that combines the spatial structure of both.

[0058] Regarding the fusion of feature maps in each of the group of convolutional layers to obtain the target feature map, the hierarchical encoder module is further configured to:

[0059] The feature map after feature splicing, i.e. the fused feature map, is unfolded along each splicing channel, and group convolution is performed on each unfolded layer feature map to make the features of each layer more refined. At the same time, it can avoid feature fusion of intermediate layers and ensure that the original local feature information of different layers is introduced into the final feature map to form the final target feature map.

[0060] One example is:

[0061] First, the multi-frame target images containing contamination are scaled, that is, the scale of the input multi-frame target images is transformed into the form of [b×t,c,h,w]. Then, the target image data of each frame is pre-extracted through a downsampling layer to obtain the first-level feature map.

[0062] Secondly, the first-level feature map is passed through a 3×3 convolutional layer to obtain an intermediate-level feature map of the same size but with a wider receptive field. This intermediate-level feature map and the first-level feature map are then concatenated along the channel dimension to obtain a second-level feature map with the spatial structure of both the first-level and intermediate-level feature maps. Next, the intermediate-level feature map resulting from the convolution of each level's output feature map is concatenated and fused with the first-level feature map to obtain the next-level feature map, such as... Figure 3 As shown, the final output of the feature concatenation is the fourth-level feature map.

[0063] Then, the spliced ​​feature map is unfolded along the four splicing channels during fusion, and group convolution is performed on each unfolded feature map to make the features of each level more refined. At the same time, the fusion of features of intermediate levels can be avoided, ensuring that the original local feature information of different levels is introduced into the final feature map to form the final target feature map.

[0064] like Figure 4 As shown, the decoupled spatiotemporal domain Transformer module provided in this embodiment of the invention mainly includes:

[0065] The splitting module is used to divide the target feature map, i.e. the high-dimensional feature map, into multiple feature blocks along the two scales of height and width, and to encode the multiple feature blocks into query vector Q, key vector K and value vector V.

[0066] The temporal Transformer module is used to combine feature blocks at the same location in multiple frames of target images according to the time dimension, and input the feature library combined by the time dimension into the multi-head attention model according to the query vector Q, the key value vector K and the value vector V, so that the model can continuously search for objects in a complex small area and determine the temporal information of the target image.

[0067] The Spatial Transformer module is used to spatially combine the feature blocks corresponding to each frame of the target image according to the original target image, and input the spatially combined feature blocks into the multi-head attention model, so that the model can find similar texture backgrounds in the spatial scale of the dirty area to complete the repair and determine the spatial information of the target image.

[0068] The merging module is used to obtain a temporal information feature map from the current viewpoint based on the temporal and spatial information of the target image.

[0069] One example is:

[0070] To reduce the computational complexity of the multi-head attention module, the operations on the temporal and spatial relationships of the processed multi-frame target image data are decoupled and processed separately by two Transformer modules, as follows: Figure 2 As shown, the two Transformer modules, namely the temporal Transformer module and the spatial Transformer module, are arranged alternately in temporal and spatial order and cycled 4 times to obtain the preliminary repaired feature image.

[0071] In the time-domain Transformer module, such as Figure 5 As shown, feature blocks at the same location in multiple target images are combined according to the time dimension, and then input into a multi-head attention model according to the query vector Q, key vector K, and value vector V. This enables the model to continuously search for objects in a complex small area. Finally, the input is fed forward to obtain the feature image after temporal restoration.

[0072] In the spatial Transformer module, such as Figure 6As shown, the feature blocks of each frame of the target image are spatially combined according to the original target image. Then, the feature blocks of each frame of the target image are input into the multi-head attention model, so that the model can find similar texture backgrounds in the spatial scale of the contaminated area to complete the repair. Similarly, the feature blocks are input into the feedforward network to obtain the feature image after spatial repair.

[0073] In the spatial domain, this invention employs reconstruction loss and perceptual loss functions to constrain the model's training process, emphasizing the restoration quality of single-frame images. In the temporal domain, it uses an adversarial loss function based on temporal PatchGAN to constrain the model's training process, thereby enhancing the realism and temporal consistency of the restored image sequence.

[0074] The adjacent reference view downsampling module provided in this embodiment of the invention is used for:

[0075] Align image data from adjacent reference viewpoints frame by frame;

[0076] The high-level semantic features of the aligned image data are extracted, which are the feature maps under the adjacent reference viewpoints.

[0077] Preferably, the frame-by-frame alignment of image data from adjacent reference viewpoints includes: using a spatial transformation network to align multiple frames of image data from adjacent reference viewpoints frame by frame.

[0078] Preferably, the high-level semantic feature extraction operation of the aligned image data includes: merging the frame number dimension and batch dimension of the aligned image data, and inputting the dimension-transformed data frame by frame into the downsampling module for feature extraction to obtain high-level semantic features, that is, obtaining the feature map under the adjacent reference view.

[0079] The multi-frame decoder provided in this embodiment of the invention mainly includes: performing correlation calculation on the feature maps of adjacent reference viewpoints and the corresponding feature blocks in the temporal information feature map of the current viewpoint, fusing them with the feature map output by the decoupled spatiotemporal domain Transformer module of the current viewpoint, and then decoding the feature maps to reconstruct multi-frame image data to obtain the final repair result.

[0080] The technical solution of the present invention has the following advantages:

[0081] 1. The repair algorithm model of the present invention is a single-stage repair algorithm, which enables the repair network to achieve end-to-end training.

[0082] 2. The method of combining the spatial information of adjacent reference viewpoint images with the temporal information of the current viewpoint image sequence introduced in this invention fully utilizes the characteristics and motion properties of the robot's binocular vision system.

[0083] 3. The hierarchical encoder module designed in this invention ensures that the semantic information of multiple frames of data will not be mixed during the feature extraction process of the model, and facilitates the modeling of the temporal information of the current viewpoint image sequence, as well as the search for similar texture information in the current image frame.

[0084] 4. This invention obtains the real information of the damaged area by using adjacent reference viewpoint images, making the repaired image more realistic. It also ensures the continuity between repaired image frames by using the temporal information of the current viewpoint image sequence, making the model have more accurate and robust repair performance.

[0085] Example 2

[0086] In order to perform the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a damaged image restoration device is provided below, specifically including:

[0087] A data acquisition device is used to acquire a sequence of target images containing contamination; the target image sequence includes multiple consecutive frames of target images.

[0088] An image inpainting tool is used to input the target image sequence containing dirt into a dirt image inpainting model of a robot binocular vision system based on spatiotemporal information fusion, and obtain the repaired target image sequence.

[0089] The robot binocular vision system image restoration model based on spatiotemporal information fusion includes a hierarchical encoder module, a decoupled spatiotemporal domain Transformer module connected to the output of the hierarchical encoder module, an adjacent reference view downsampling module, and a multi-frame decoder connected to the output of both the decoupled spatiotemporal domain Transformer module and the adjacent reference view downsampling module.

[0090] The hierarchical encoder module is used for:

[0091] Features of each frame of the target image are extracted and fused to obtain a fused feature map, and the fused feature map is then convolved in each layer to obtain the target feature map.

[0092] The decoupled spatiotemporal domain Transformer module is used to: process the target feature map to obtain the temporal and spatial information of the target image, and obtain the temporal information feature map under the current viewpoint based on the temporal and spatial information of the target image.

[0093] The adjacent reference view downsampling module is used to acquire feature maps under adjacent reference views.

[0094] The multi-frame decoder is used to perform correlation calculation and decoding on the corresponding feature blocks in the feature maps of adjacent reference viewpoints and the temporal information feature maps of the current viewpoint, so as to obtain the repaired target image sequence.

[0095] Example 3

[0096] This invention provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the damaged image restoration method of Embodiment 1.

[0097] Alternatively, the aforementioned electronic device may be a server.

[0098] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the damaged image restoration method of Embodiment 1.

[0099] This invention comprehensively considers the temporal and spatial information of feature images from the current viewpoint and fully utilizes the characteristics of the robot's binocular vision system. By introducing spatial information from adjacent reference viewpoint images, it ensures the authenticity and robustness of the restoration results and improves the model's restoration performance. Furthermore, the restoration model proposed in this invention features a single-stage design, which improves inference speed and facilitates its deployment and application on binocular autonomous robots.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0101] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for restoring damaged images, characterized in that, include: Obtain a sequence of target images containing contamination; The target image sequence includes multiple consecutive target images; The target image sequence containing dirt is input into the dirt image restoration model of the robot binocular vision system based on spatiotemporal information fusion to obtain the restored target image sequence. The robot binocular vision system dirt image restoration model based on spatiotemporal information fusion includes a hierarchical encoder module, a decoupled spatiotemporal domain Transformer module connected to the output of the hierarchical encoder module, an adjacent reference view downsampling module, and a multi-frame decoder connected to the output of both the decoupled spatiotemporal domain Transformer module and the adjacent reference view downsampling module. The hierarchical encoder module is used for: Features of each frame of the target image are extracted and fused to obtain a fused feature map, and the fused feature maps of each layer are convolved to obtain the target feature map; The decoupled spatiotemporal domain Transformer module is used to: process the target feature map to obtain the temporal and spatial information of the target image, and obtain the temporal information feature map under the current viewpoint based on the temporal and spatial information of the target image; The adjacent reference view downsampling module is used to acquire feature maps under adjacent reference views; The multi-frame decoder is used to perform correlation calculation and decoding on the corresponding feature blocks in the feature maps of adjacent reference viewpoints and the temporal information feature maps of the current viewpoint, so as to obtain the repaired target image sequence.

2. The method for repairing damaged images according to claim 1, characterized in that, Regarding the fusion of feature maps in each of the group of convolutional layers to obtain the target feature map, the hierarchical encoder module specifically includes: The fused feature map is unfolded along each splicing channel, and group convolution is performed on each unfolded layer feature map to obtain the target feature map.

3. The method for repairing damaged images according to claim 1, characterized in that, The decoupled spatiotemporal domain Transformer module specifically includes: The splitting module is used to divide the target feature map into multiple feature blocks along the two scales of height and width, and to encode the multiple feature blocks into a query vector Q, a key vector K, and a value vector V; The temporal Transformer module is used to combine feature blocks at the same position in multiple frames of target images according to the time dimension, and input the feature library combined by the time dimension into the multi-head attention model according to the query vector Q, the key value vector K and the value vector V to determine the temporal information of the target image; The spatial Transformer module is used to spatially combine the feature blocks corresponding to each frame of the target image according to the original target image, and input the spatially combined feature blocks into the multi-head attention model to determine the spatial information of the target image. The merging module is used to obtain a temporal information feature map from the current viewpoint based on the temporal and spatial information of the target image.

4. The method for repairing damaged images according to claim 3, characterized in that, The temporal Transformer module and the spatial Transformer module are arranged alternately in temporal and spatial order.

5. The method for repairing damaged images according to claim 3, characterized in that, In the spatial domain, reconstruction loss function and perception loss function are used to constrain the training process of the model, focusing on the restoration quality of single-frame images.

6. The method for repairing damaged images according to claim 3, characterized in that, In the time domain, an adversarial loss function based on temporal PatchGAN is used to constrain the training process of the model.

7. The method for repairing damaged images according to claim 3, characterized in that, The adjacent reference view downsampling module is used for: Align image data from adjacent reference viewpoints frame by frame; High-level semantic features are extracted from the aligned image data; the high-level semantic features are feature maps from adjacent reference viewpoints.

8. A damaged image restoration device, characterized in that, include: A data acquisition device is used to acquire a sequence of target images containing contamination. The target image sequence includes multiple consecutive target images; An image restoration tool is used to input the target image sequence containing dirt into a dirt image restoration model of a robot binocular vision system based on spatiotemporal information fusion, and obtain a restored target image sequence. The robot binocular vision system dirt image restoration model based on spatiotemporal information fusion includes a hierarchical encoder module, a decoupled spatiotemporal domain Transformer module connected to the output of the hierarchical encoder module, an adjacent reference view downsampling module, and a multi-frame decoder connected to the output of both the decoupled spatiotemporal domain Transformer module and the adjacent reference view downsampling module. The hierarchical encoder module is used for: Features of each frame of the target image are extracted and fused to obtain a fused feature map, and the fused feature maps of each layer are convolved to obtain the target feature map; The decoupled spatiotemporal domain Transformer module is used to: process the target feature map to obtain the temporal and spatial information of the target image, and obtain the temporal information feature map under the current viewpoint based on the temporal and spatial information of the target image; The adjacent reference view downsampling module is used to acquire feature maps under adjacent reference views; The multi-frame decoder is used to perform correlation calculation and decoding on the corresponding feature blocks in the feature maps of adjacent reference viewpoints and the temporal information feature maps of the current viewpoint, so as to obtain the repaired target image sequence.

9. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the damaged image restoration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the damaged image restoration method as described in any one of claims 1 to 7.

Images