Methods, apparatus, electronic devices and storage media for image restoration on open-pit mine vehicles
By using a deep learning network model to perform unified restoration of multiple degradations and suppress inter-frame interference in open-pit mine vehicle images, the image stability problem under complex degradation scenarios in open-pit mines is solved, image clarity and temporal stability are improved, and the safety of unmanned operations is ensured.
Patent Information
- Application Number
- CN202611085222.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing open-pit mine vehicle-mounted image restoration methods perform poorly in complex degradation scenarios, and inter-frame temporal interference is difficult to eliminate, resulting in insufficient image stability, which affects the accuracy of downstream perception algorithms and the safety of mine operations.
A deep learning network model is used to perform unified restoration of multiple degradations and suppression of inter-frame interference in consecutive degraded images. Through spatial feature extraction, feature enhancement and fusion, and image reconstruction modules, clear restored images are generated, and instantaneous interference is eliminated by utilizing inter-frame information.
It improves the clarity and temporal stability of image restoration under complex working conditions in open-pit mines, ensures the recognition accuracy of downstream visual perception algorithms, and supports the safe and stable operation of unmanned open-pit mines.
Smart Images

Figure CN123118384A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, electronic device and storage medium for image restoration on an open-pit mine vehicle. Background Technology
[0002] Open-pit mines, as a core scenario for intelligent and unmanned operations, utilize vehicle-mounted visual perception systems for critical tasks such as target detection, semantic segmentation, and autonomous driving. Among related technologies, a technical system has been constructed, evolving from single degradation processing to composite degradation restoration, through the synergistic evolution of traditional image algorithms and deep learning algorithms.
[0003] Existing image restoration methods often employ a single algorithm for a single degradation, which leads to a sharp decline in restoration performance under complex degradation scenarios. The inability to eliminate temporal interference results in insufficient image stability, thereby affecting the accuracy of downstream perception algorithms and the safety of mining operations. Summary of the Invention
[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for restoring images on open-pit mine vehicles.
[0005] According to a first aspect of this disclosure, a method for restoring images from an open-pit mine vehicle is provided, comprising:
[0006] Acquire consecutive degraded images from multiple frames; The consecutive multi-frame degraded images are input into a deep learning network model so that the deep learning network model can perform unified restoration of multiple degradations and suppression of inter-frame interference on the consecutive multi-frame degraded images. Output a single clear restored image after processing by the deep learning network model.
[0007] Optionally, the deep learning network model includes a spatial feature extraction module, a feature enhancement and fusion module, and an image reconstruction module; The spatial feature extraction module is used to extract the spatial features of each frame of the degraded image; The feature enhancement and fusion module is used to enhance the spatial features in a targeted manner and fuse inter-frame temporal information; The image reconstruction module is used to generate the single clear restored image based on the features after fusing temporal information.
[0008] Optionally, the feature enhancement and fusion module includes an attention enhancement unit and a temporal modeling unit; The attention enhancement unit is used to weighted enhance the degenerate regions in the spatial features; The temporal modeling unit is used to perform temporal modeling on the spatial features of each frame after enhancement processing, so as to eliminate instantaneous interference by utilizing inter-frame information.
[0009] Optionally, the attention enhancement unit includes a channel attention branch and a spatial attention branch; The channel attention branch is used to generate attention weights in the channel dimension, and the spatial attention branch is used to generate attention weights in the spatial dimension. The attention enhancement unit enhances the features by sequentially weighting and fusing the channel attention weights and spatial attention weights with the features to be processed.
[0010] Optionally, the timing modeling unit is a timing model based on a gated loop structure or a self-attention mechanism; The timing model includes at least one timing processing layer, and each timing processing layer has a hidden state for memorizing and transmitting inter-frame information.
[0011] Optionally, the consecutive multi-frame degraded images are image sequences extracted from the vehicle video stream and containing multiple environmental degradation types.
[0012] Optionally, the deep learning network model is obtained through supervised learning, including the following steps: Construct a training dataset that includes real paired image data collected from open-pit mine sites and enhanced paired image data generated through simulation; The continuous multi-frame degraded images are used as model input, and a single clear image paired with them is used as the supervision target; The model parameters are optimized using a joint loss, which includes a first loss for constraining the overall structural similarity of the image, a second loss for constraining the similarity of high-level semantic features, and a third loss for constraining key edge details.
[0013] According to a second aspect of this disclosure, an open-pit mine vehicle-mounted image restoration device is provided, comprising: The acquisition unit is used to acquire multiple consecutive frames of degraded images. The processing unit is also configured to input the consecutive multi-frame degraded images into a deep learning network model, so as to perform multi-degradation unified restoration and inter-frame interference suppression on the consecutive multi-frame degraded images through the deep learning network model; The output unit is also used to output a single clear restored image after being processed by the deep learning network model.
[0014] Optionally, the processing unit is further configured to: Extract spatial features from each frame of the degraded image; The spatial features are enhanced in a targeted manner and inter-frame temporal information is fused. The single clear restored image is generated based on the features after fusing temporal information.
[0015] Optionally, the processing unit is further configured to: Weighted enhancement is applied to the degraded regions in the aforementioned spatial features; Temporal modeling is performed on the spatial features of each frame after enhancement processing to eliminate transient interference using inter-frame information.
[0016] Optionally, the processing unit is further configured to: Generate attention weights in the channel dimension and attention weights in the spatial dimension; By sequentially weighting and fusing the channel attention weights and spatial attention weights with the features to be processed, feature enhancement is achieved.
[0017] Optionally, the processing unit is further configured to: Temporal modeling is performed using a temporal model based on a gated loop structure or a self-attention mechanism. The temporal model contains at least one temporal processing layer, and each temporal processing layer has a hidden state for memorizing and transmitting inter-frame information.
[0018] Optionally, the consecutive multi-frame degraded images are image sequences extracted from the vehicle video stream and containing multiple environmental degradation types.
[0019] Optional, also includes: The building unit is used to construct a training dataset containing real paired image data collected from open-pit mine sites and enhanced paired image data generated through simulation. The optimization unit is further configured to use the consecutive multi-frame degraded images as model input and a single clear image paired with them as the supervision target, and to optimize the parameters of the deep learning network model using joint loss. The joint loss includes a first loss for constraining the overall structural similarity of the images, a second loss for constraining the similarity of high-level semantic features, and a third loss for constraining key edge details.
[0020] According to a third aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect above.
[0021] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first aspect above.
[0022] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect above.
[0023] The open-pit mine vehicle-mounted image restoration method, apparatus, electronic device, and storage medium disclosed herein, through the use of a unified deep learning network to synchronously process various composite degradation problems of multiple frames of images and suppress inter-frame temporal interference, eliminate the need to configure corresponding algorithms separately for different image degradation types. It can simultaneously take into account the restoration quality of single-frame images and the temporal stability of multiple frames of images. Therefore, it can solve the technical problems of existing image restoration methods that require separate algorithms to process single degradation types, resulting in poor restoration effects, difficulty in eliminating inter-frame temporal interference, and insufficient image output stability in open-pit mine composite degradation scenarios. It achieves the technical effects of improving the clarity and temporal stability of image restoration under complex mining conditions, ensuring the recognition accuracy of downstream visual perception algorithms, and supporting the safe and stable operation of unmanned open-pit mines.
[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description
[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 A schematic flowchart illustrating an open-pit mine vehicle-mounted image restoration method provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of an open-pit mine vehicle-mounted image restoration device provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of another open-pit mine vehicle-mounted image restoration device provided in an embodiment of this disclosure; Figure 4 A schematic block diagram of an example electronic device provided for embodiments of this disclosure. Detailed Implementation
[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0027] The following description, with reference to the accompanying drawings, outlines an open-pit mine vehicle-mounted image restoration method, apparatus, electronic device, and storage medium according to embodiments of the present disclosure.
[0028] Figure 1 This is a schematic flowchart of an open-pit mine vehicle-mounted image restoration method provided in an embodiment of this disclosure.
[0029] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain multiple consecutive frames of degraded images; In extreme industrial scenarios such as open-pit mines, vehicle-mounted cameras are constantly subjected to multiple environmental factors during operation, including dust, fog, rain, snow, strong backlight, and vehicle vibrations. This results in significant degradation of the acquired images, such as blurriness, noise, overexposure or underexposure, reduced contrast, and loss of detail. Acquiring consecutive multiple frames rather than isolated single frames aims to preserve the temporal correlation information between adjacent frames due to differences in the type and degree of degradation. This lays the data foundation for subsequent network models to utilize inter-frame redundancy to eliminate transient interference and improve restoration stability.
[0030] The acquired frame sequence exhibits continuity and correlation across frames over time. Its degradation characteristics encompass both common environmental degradation components and transient interference components that randomly occur between frames. This provides effective supervisory signals for the model to learn degradation patterns and construct restoration maps. For example, in a specific application scenario, the number of input frames can be selected from three to five to balance temporal modeling depth and inference computation efficiency, while the image resolution is adapted to the standard output specifications of automotive cameras.
[0031] Step 102: Input the consecutive multi-frame degraded images into a deep learning network model, so as to perform multi-degradation unified restoration and inter-frame interference suppression on the consecutive multi-frame degraded images through the deep learning network model; During the feature extraction stage, the model performs joint representation learning on multiple input frames, enabling it to automatically identify and distinguish various overlapping degradation types, such as dust, fog, rain, snow, strong light, blur, and noise. This allows for a unified network structure to restore the overall composite degradation in a single operation, eliminating the need for separate processing models or step-by-step sequential processing for different degradation types. Simultaneously, during inference, the model fully leverages the temporal correlation features of the input frame sequence. Utilizing the differences in degradation components between adjacent frames and the redundancy of effective information, it identifies and suppresses inter-frame interference such as transient dust, random snowflakes, and sudden glare that only appear in a few frames, ensuring the output image remains stable and consistent in the temporal dimension.
[0032] In one possible implementation, the network model can be composed of two cascaded parts: a spatial feature extraction pathway and a temporal feature modeling pathway. The former is responsible for extracting spatial feature representations related to degradation type and scene content from each frame of images, while the latter is responsible for modeling the temporal dependencies between spatial features across multiple frames. Together, they complete the full mapping process from multi-frame degraded input to a single-frame sharp output. During the training phase, the network uses consecutive degraded frames as input and the corresponding single sharp image as supervision signal. It optimizes the model parameters by minimizing the difference between the restored output and the sharp target, enabling it to learn the ability to perform multi-degradation inverse mapping applicable to the target scene.
[0033] Step 103: Output a single clear restored image after processing by the deep learning network model.
[0034] After processing by the deep learning network model, the final output is a clear restored image. This image eliminates the multiple degradation components and inter-frame random interference contained in the input frame sequence, exhibiting high clarity, contrast, and color fidelity. Although the model's input consists of consecutive degraded images across multiple frames, its output is a single clear image. This many-to-one mapping structure allows the model to integrate complementary information and temporal redundancy contained in multiple input frames, averaging and canceling out interference in the temporal dimension, thus achieving a restoration quality superior to that achievable through any single-frame independent processing. The output clear restored image preserves the structural contours and texture details of key targets in the original scene, such as vehicles, road surfaces, and slopes, effectively reducing detail loss and color distortion caused by degradation.
[0035] The restored images can be directly used as high-quality input for downstream perception algorithms such as object detection, semantic segmentation, or autonomous driving. This improves upon issues like missed detections, false detections, and decision biases caused by image degradation at the data source, providing stable and reliable image support for visual perception systems in complex and harsh environments. For example, in a specific application scenario, the output image resolution is consistent with the input frame, allowing seamless integration into the perception algorithm pipeline already deployed on an automotive embedded platform without additional format conversion or resolution adaptation.
[0036] In some embodiments, the deep learning network model includes a spatial feature extraction module, a feature enhancement and fusion module, and an image reconstruction module; The spatial feature extraction module is used to extract the spatial features of each frame of the degraded image; The feature enhancement and fusion module is used to enhance the spatial features in a targeted manner and fuse inter-frame temporal information; The image reconstruction module is used to generate the single clear restored image based on the features after fusing temporal information.
[0037] The spatial feature extraction module, acting as the network's front end, receives multiple consecutive frames of degraded images and encodes spatial features frame by frame. This module consists of cascaded convolutional operations, with each convolutional layer progressively reducing spatial resolution and increasing channel dimensions, thereby extracting high-dimensional feature representations rich in semantics while preserving the image's structural information. In one possible implementation, this module employs a four-layer convolutional coding structure, with each convolutional layer followed by batch normalization and a non-linear activation function. The first convolutional layer uses the original pixel values of the degraded image as input, extracting features at different levels, from low-level texture to high-level semantics, layer by layer. Each frame of image is independently encoded using the same set of convolutional layer parameters, ensuring the comparability of features extracted between different frames and laying the foundation for subsequent inter-frame temporal fusion. The multi-scale feature maps output during the encoding process are passed to subsequent modules through skip connections to preserve spatial detail information that may be lost due to downsampling.
[0038] The feature enhancement and fusion module, located in the middle layer of the network, performs dual functions: first, it selectively enhances the features output by the spatial feature extraction module for each frame, focusing on the feature responses of degraded regions; second, it performs temporal fusion modeling on the enhanced features from multiple frames. For the enhancement function, this module introduces a residual attention substructure, which includes two parallel paths: a channel attention branch and a spatial attention branch. The channel attention branch compresses the spatial dimension of features through global pooling, and then generates weight coefficients for each channel through dimensionality reduction and expansion convolutions, selectively weighting the feature channels to strengthen effective channels related to degradation recovery and suppress redundant channels. The spatial attention branch, on the other hand, assigns weights to the spatial location of the feature maps, concentrating the model's computational resources on significantly degraded areas such as dust-covered areas, fog-shrouded areas, or areas of intense light overexposure.
[0039] The outputs of the two paths are residually fused with the original features, and then refined by convolution to obtain enhanced features. For the temporal fusion function, this module takes the enhanced multi-frame feature sequence as input and uses a ConvLSTM structure to model the inter-frame temporal dependencies. ConvLSTM replaces fully connected operations with convolutional operations on the gating mechanism of traditional LSTM, enabling it to both model temporal dynamics and retain the ability to process spatial structural information. This structure processes the enhanced features of each frame sequentially, selectively retaining effective scene information consistent across frames through the synergistic effect of forget gates, input gates, and output gates, and suppressing inter-frame interference such as instantaneous dust, random snowflakes, or sudden glare that only appear in individual frames, ultimately outputting a feature representation that fuses the temporal information of multiple frames.
[0040] The image reconstruction module, acting as the network's end point, receives the temporal fusion features output by the feature enhancement and fusion module. Through layer-by-layer upsampling and convolution operations, it restores these features to the same spatial resolution as the input image, generating a single, clear, restored image. The upsampling process in this module can employ transposed convolution or subpixel convolution to progressively restore spatial details. Features from each upsampling layer and the corresponding layer in the encoding stage are concatenated and fused via skip connections to compensate for the texture and edge information lost during the encoding downsampling process. Finally, a convolutional layer maps the high-dimensional features to a three-channel pixel space, outputting a clear image containing the complete scene content and eliminating degradation effects.
[0041] In some embodiments, the feature enhancement and fusion module includes an attention enhancement unit and a temporal modeling unit; The attention enhancement unit is used to weighted enhance the degenerate regions in the spatial features; The temporal modeling unit is used to perform temporal modeling on the spatial features of each frame after enhancement processing, so as to eliminate instantaneous interference by utilizing inter-frame information.
[0042] The attention enhancement unit and the modeling unit are connected in series in sequence within the feature enhancement and fusion module. The former is responsible for selectively weighting features in the spatial dimension, while the latter is responsible for dynamically modeling the weighted multi-frame features in the temporal dimension. The two work in sequence to complete the entire process from spatial enhancement to temporal fusion.
[0043] The attention enhancement unit employs a dual-branch parallel structure of channel attention and spatial attention. The feature maps output from the spatial feature extraction module first enter the channel attention branch. This branch performs global average pooling and global max pooling operations along the spatial dimension on the feature maps, compressing each frame's feature map from high-dimensional space into a set of channel description vectors. Both sets of description vectors are then passed through a shared dimensionality-reducing convolutional layer to compress the number of channels to a lower dimension, such as from 128 channels to 64 channels. Subsequently, after passing through a non-linear activation function, they are restored to the original number of channels via an dimensionality-increasing convolutional layer. The two results are then element-wise summed and activated by a sigmoid function to generate a channel-dimensional weight coefficient matrix. This weight matrix is multiplied channel-wise by the input feature map, recalibrating each feature channel. This enhances the channel responses related to degradation recovery while suppressing the contributions of redundant or interfering channels.
[0044] The spatial attention branch performs global average pooling and global max pooling on the feature maps along the channel dimension. The results of these two pooling operations are concatenated along the channel axis and then passed through a convolutional layer and a sigmoid activation function to generate a weight map in the spatial dimension. This weight map identifies significantly degraded spatial locations in the feature maps, such as dust-covered areas, foggy areas, or overexposed areas, allowing the model's subsequent computational resources to focus on these key regions. The outputs of the two attention branches are weighted and then element-wise superimposed with the original features through residual connections. A further convolutional layer refines the fused result and adjusts the channels, yielding the attention-enhanced feature representation. This residual structure ensures that while enhancing the feature responses in degraded areas, the effective basic information in the original image is preserved, avoiding detail loss or color shift due to over-enhancement.
[0045] The modeling unit employs a multi-layer ConvLSTM structure, receiving the enhanced feature sequences from each frame output by the attention enhancement unit and recursively updating the temporal state frame by frame. The ConvLSTM unit contains three gate structures: a forget gate, an input gate, and an output gate. Each gate replaces the fully connected operations in traditional LSTM with convolutional operations, allowing it to maintain spatial structure awareness while processing temporal information. The forget gate determines the retention and discard ratio of historical temporal information based on the current frame input and the hidden state of the previous frame. The input gate controls the intensity of newly extracted feature information in the current frame being written into the memory unit. The output gate selects temporal features relevant to the reconstruction task from the updated memory unit as the hidden state output for the current frame. In one possible implementation, the modeling unit consists of two cascaded ConvLSTM layers, with each layer having 128 hidden state channels, a 3×3 kernel size, a stride, and padding of 1. A random deactivation operation is introduced between layers with a deactivation probability of 0.2 to prevent overfitting during training. It should be noted that this description is merely an illustrative example and is not intended to limit specific model parameters. The embodiments of this application do not impose any limitations on this.
[0046] The first ConvLSTM layer receives the attention-enhanced multi-frame feature sequence and outputs preliminary temporal fusion features. The second ConvLSTM layer further performs deep dynamic modeling of the temporal fusion features, recursively integrating complementary information and consistency features between adjacent frames frame by frame. Since transient dust, random snowflakes, or sudden glare that only appear in individual frames lack stable temporal consistency between consecutive frames, the modeling unit naturally attenuates these transient interferences as anomalous signals during the recursive process. Meanwhile, the stable scene structure and effective information between frames are accumulated and enhanced in the temporal dimension. Finally, the enhanced features after temporal cleansing and information fusion are output and passed to the image reconstruction module to generate a clear image.
[0047] In some embodiments, the attention enhancement unit includes a channel attention branch and a spatial attention branch; The channel attention branch is used to generate attention weights in the channel dimension, and the spatial attention branch is used to generate attention weights in the spatial dimension. The attention enhancement unit enhances the features by sequentially weighting and fusing the channel attention weights and spatial attention weights with the features to be processed.
[0048] The channel attention branch and the spatial attention branch are connected sequentially within the attention enhancement unit. The channel attention weight is applied to the input features first, and its output is then used as the input of the spatial attention branch for secondary weighting, forming a layer-by-layer focusing mechanism from coarse to fine and from global to local.
[0049] The channel attention branch takes the original feature map output by the spatial feature extraction module as input and performs two aggregation operations—global average pooling and global max pooling—along the height and width directions of the spatial dimension, respectively. Global average pooling extracts the average response of each channel across the entire spatial range, reflecting the overall activation level of the channels; global max pooling captures the peak response of each channel within the spatial range, highlighting local salient features. Each pooling operation outputs a one-dimensional vector of the same length as the number of channels, and the two vectors are then fed into a multilayer perceptron with shared weights. This multilayer perceptron consists of a dimensionality-reducing fully connected layer, a non-linear activation layer, and an dimensionality-increasing fully connected layer connected in series. The dimensionality-reducing layer compresses the channel dimension to a lower intermediate dimension, such as compressing 128 channels to 64 channels, and the dimensionality-increasing layer restores it to the original number of channels.
[0050] The outputs of the two paths processed by the multilayer perceptron are summed element-wise, and then the values are normalized to the range of zero to one using the sigmoid activation function, generating an attention weight vector along the channel dimension. Each component of this vector represents the importance of the corresponding channel to the degradation restoration task. This vector is multiplied channel-wise with the input feature map to recalibrate the feature channels—channels closely associated with degradation components such as dust, fog, and strong light receive higher weight responses, while channels carrying redundant information or noise interference are suppressed accordingly. The channel-calibrated feature map is then used as input to the spatial attention branch for further processing.
[0051] The spatial attention branch receives the channel-weighted feature map and performs global average pooling and global max pooling on each channel. The results of these two pooling operations are concatenated along the channel axis to obtain a dual-channel feature map. This map is then fused spatially through a 7x7 convolutional layer. The convolutional output is normalized using a sigmoid activation function to generate a spatial attention weight map. This weight map has the same spatial resolution as the input feature map. The weight value at each location in the map represents the degree of attention given to the restoration task at that spatial location—significantly degraded areas, such as dust-covered areas, fog-filled areas, and overexposed areas, receive higher weight values, while areas with slight or no degradation receive lower weight values. Multiplying this weight map position-by-position with the channel-weighted feature map in the spatial dimension allows the model's computational resources and feature representation capabilities to focus on severely degraded areas, enhancing these areas while preserving the original features of less degraded areas, thus avoiding overprocessing and introducing new distortions.
[0052] After sequential weighting by channel attention and spatial attention, the output feature map of the enhancement unit selectively enhances the effective feature channels in the channel dimension and accurately focuses on the degraded regions in the spatial dimension. This double-weighted feature is then element-wise superimposed with the original unweighted feature through residual connections, ensuring that the enhancement process highlights the feature responses of the degraded regions while preserving the basic structural information and texture details of the original image that are unaffected by degradation. Subsequently, a 3x3 convolutional layer refines the fusion result, and the output is passed to the modeling unit for temporal processing.
[0053] In some embodiments, the timing modeling unit is a timing model based on a gated loop structure or a self-attention mechanism; The timing model includes at least one timing processing layer, and each timing processing layer has a hidden state for memorizing and transmitting inter-frame information.
[0054] The core task of the temporal modeling unit is to establish inter-frame dependencies in the temporal dimension of the multi-frame feature sequence output after attention enhancement. Internally, it consists of at least one temporal processing layer, and each layer maintains a set of hidden state vectors as the carrier of inter-frame information. At each time step, the hidden state records the scene information and degradation mode features of the current frame and previous frames that have been selectively retained by the model. It is continuously updated as the frame sequence progresses, so that the processing of subsequent frames can refer to the effective information accumulated in historical frames.
[0055] When a gated recurrent structure is used as the temporal model, each temporal processing layer contains three gating mechanisms: a forget gate, an input gate, and an output gate. The computation of each gate uses the input features of the current frame and the hidden state of the previous frame as joint inputs, generating gating signals through a learnable parameter matrix. The forget gate uses continuous values in the range of zero to one to determine the retention and rejection ratio of information in each dimension of the hidden state of the previous frame, enabling the model to actively forget transient interference components that only appeared in the previous frame, such as sudden glare or single-frame snow noise. The input gate controls the intensity of writing newly extracted features of the current frame into the memory unit, continuously accumulating effective scene information consistent across frames into the memory unit. The output gate selects the temporal features most relevant to the current frame's reconstruction task from the updated memory unit, encodes them as the hidden state output of the current frame, and passes them to the next time step.
[0056] In one possible implementation, the gated recurrent structure can be implemented using ConvLSTM, which replaces the fully connected operations in the traditional gated recurrent unit with convolutional operations, enabling the gated computation to complete temporal dynamic modeling while preserving spatial structure information. In this structure, the convolutional kernel size is set to, for example, 3x3, the number of channels in the hidden state is set to, for example, 128, and the number of layers is set to, for example, two. A random deactivation operation is introduced between the two layers to suppress overfitting, with the deactivation probability set to, for example, 0.2. The first ConvLSTM layer receives the attention-enhanced multi-frame feature sequence, initially establishes inter-frame temporal dependencies, and outputs temporal fusion features. The second ConvLSTM layer further deepens the temporal modeling based on this, recursively integrating complementary information between adjacent frames frame by frame.
[0057] When using a self-attention mechanism as a temporal model, each temporal processing layer calculates attention weights between frames in the temporal dimension for the input multi-frame feature sequence. Specifically, each frame's features undergo three different linear transformations to generate a query vector, a key vector, and a value vector. The query vector is then multiplied by the key vectors of all frames, and the result is scaled and normalized using Softmax to generate an attention weight distribution. This weight distribution is then used to weight and sum the value vectors of all frames to obtain the output representation of the current frame that incorporates global temporal context information. The multi-head self-attention mechanism can further divide the feature dimension into multiple subspaces, independently calculating attention weights within each subspace before concatenation and linear transformation to capture temporal correlation patterns across different dimensions between frames. To enable the model to perceive the temporal positional order of each frame, a positional encoding signal is superimposed on the input features before they enter the self-attention layer.
[0058] Regardless of whether a gated loop structure or a self-attention mechanism is used, the core principle of the temporal modeling unit lies in selectively memorizing and transmitting inter-frame information through hidden states or attention weights. Since interference components such as instantaneous dust, random snowflakes, and sudden glare lack stable temporal consistency between consecutive frames, the temporal processing layer naturally identifies these abnormal signals as low-consistency features and attenuates them during frame-by-frame recursion or global attention calculation. Meanwhile, stable scene structures, target contours, and texture details between frames are cumulatively enhanced in the temporal dimension. Finally, the enhanced feature representation after temporal purification and information fusion is output and passed to the image reconstruction module to complete the generation of a clear image.
[0059] In some embodiments, the consecutive multi-frame degraded images are image sequences extracted from an in-vehicle video stream that contain multiple types of environmental degradation.
[0060] The vehicle-mounted vision system is deployed on the body of open-pit mining vehicles, such as the front, rear, and sides of mining trucks, continuously acquiring video stream data covering the vehicle's driving path, the surrounding work area, and the surrounding environment. This video stream records the complete scene changes experienced by the vehicle during operation in the form of continuous frames. Each frame is the optical imaging result of the scene by the vehicle-mounted camera at a specific moment. Adjacent frames are temporally continuous, and the frame interval is usually matched to the camera's acquisition frame rate, such as 25 or 30 frames per second. Based on this continuous video stream, a sequence of consecutive degraded images is extracted according to a preset frame interval or frame number window. The number of frames extracted is, for example, three to five frames, to retain sufficient temporal information while controlling the computational load of subsequent network models.
[0061] The captured image sequence contains various types of environmental degradation, stemming from the harsh working conditions unique to open-pit mines. Dust and sand degradation, caused by blasting operations, dust from moving mine carts, and natural windblown sand, manifests as large areas of diffused gray-yellow particles covering the image, severely obscuring distant targets such as mine carts, slopes, and roads. Fog degradation originates from water vapor condensation in the early morning or in low-lying areas of the mine, leading to a decrease in overall image contrast and blurring or even disappearing the outlines of distant targets. Rain and snow degradation is caused by precipitation; raindrops or snowflakes accumulate on the lens surface or form randomly distributed streaks and spots in the field of view, interfering with normal image formation.
[0062] Strong light and backlight degradation occur under direct sunlight or reflected light conditions, causing localized or overall overexposure of the image and complete loss of detail in bright areas. Furthermore, the bumps and vibrations generated by vehicles traveling on unpaved roads in mining areas can cause motion blur, while onboard electronic devices operating in high-dust, strong electromagnetic environments can introduce random noise. These various degradation types do not occur in isolation during open-pit mining operations but frequently overlap and coexist, such as the simultaneous presence of sandstorms and fog, alternating rain and snow with strong light, and the simultaneous existence of motion blur and noise, forming typical complex degradation scenarios.
[0063] In a specific application scenario, the starting frame for image sequence cropping can be dynamically determined based on the degradation detection module's judgment results. Cropping is triggered when consecutive frames in the video stream exhibit degradation levels exceeding a preset threshold, ensuring that the image sequence input to the network model contains degraded content with practical restoration value. The cropped image sequence is input into the subsequent network model at its original resolution, with image sizes adapted to the standard output specifications of automotive cameras, such as 640x640 pixels or 1280x720 pixels. While each frame in this image sequence shares the main content of the same scene, there are inter-frame differences in the location distribution of degradation areas, the degree of degradation, and the presence or absence of transient interference. These differences provide an effective signal source for subsequent temporal modeling units to distinguish between stable degradation and transient interference.
[0064] In some embodiments, the deep learning network model is obtained through supervised learning, including the following steps: Construct a training dataset that includes real paired image data collected from open-pit mine sites and enhanced paired image data generated through simulation; The continuous multi-frame degraded images are used as model input, and a single clear image paired with them is used as the supervision target; The model parameters are optimized using a joint loss, which includes a first loss for constraining the overall structural similarity of the image, a second loss for constraining the similarity of high-level semantic features, and a third loss for constraining key edge details.
[0065] The training dataset was constructed using a strategy combining real and simulated data to obtain sufficient training samples covering various degradation scenarios in open-pit mines. Real-world paired image data was collected from actual open-pit mine operations. During the acquisition process, degraded images and corresponding clear reference images were obtained in both degraded and favorable environments under similar time windows and shooting angles, forming one-to-one paired sample groups. The collected scenarios cover typical mining areas, such as working faces, transport roads, spoil heaps, and slope areas. Degradation types cover common mining degradation conditions such as dust, fog, rain, snow, strong backlighting, and motion blur caused by vehicle bumps.
[0066] In a specific dataset, there are no fewer than 10,000 real-world paired data sets. Simulated enhanced paired image data is generated by applying digital degradation simulations to clear images. Specifically, high-quality clear images are used as a baseline. Using physical degradation models or data-driven style transfer methods, different levels and combinations of dust particle distributions, fog concentration gradients, rain and snow streaks and spots, overexposed areas of strong light, and Gaussian or shot noise are superimposed on the images to generate corresponding synthetic degradation images. The clear images themselves serve as the supervised targets for pairing. In a specific dataset, there are no fewer than 50,000 simulated enhanced data sets. The simulated degradation types cover all common mine degradation types, including dust, fog, rain, snow, and strong light, and are randomly combined to simulate complex degradation scenarios. Real and simulated data together constitute a complete training dataset. Real data provides in-domain distribution priors consistent with actual deployment scenarios, while simulated data expands the sample space through large-scale and diverse combinations, improving the model's generalization ability to unseen degradation patterns.
[0067] During training, the organization of input and supervised targets remains consistent with the inference phase. Each training sample extracts multiple consecutive frames of degraded images from a paired degraded image sequence as network input, and a single clear image as the supervised target. The degraded image sequence is extracted in the same way as in the inference phase, i.e., several frames are selected from consecutive degraded frames at fixed intervals, such as three to five frames, to ensure that the input data maintains a consistent distribution between training and inference. The model receives multiple frames of degraded input in an end-to-end manner and outputs a single frame restoration result. The goal of training optimization is to make the model output as close as possible to the paired clear image supervised target.
[0068] The optimization of model parameters is supervised by a joint loss function, which consists of a weighted sum of three loss components with different emphases. The first loss constrains the similarity between the model output image and the sharp target image at the overall structural level. It measures the numerical deviation between the two images at each spatial location by calculating the absolute difference pixel-by-pixel and taking the mean. This loss constrains the global brightness, contrast, and low-frequency structural information of the image, ensuring that the restored image maintains overall visual perception consistency with the sharp target. The weight coefficient of this component is set to, for example, 1.0, serving as the dominant supervisory signal in the joint loss. The second loss constrains the similarity between the model output image and the sharp target image at the high-level semantic feature level. Specifically, the two images are input into a pre-trained feature extraction network, such as a VGG network pre-trained on the ImageNet dataset, to extract feature maps from several layers in the middle of the network. The mean of the element-wise absolute differences between the two sets of feature maps is then calculated.
[0069] The first loss operates on the feature space rather than the pixel space, focusing on high-level information such as the semantic structure and texture patterns of objects in the image. This guides the model to maintain the integrity of the scene's semantic content during degradation restoration, such as the structure of the mining truck, the slope texture, and the material features of the road surface. The weight coefficient for this component is set to, for example, 0.3. The third loss constrains the similarity between the model's output image and the clear target image at the level of key edge details. Specifically, edge detection operations are performed on both images separately, such as using the Sobel operator to extract the gradient responses of the images along the horizontal and vertical directions and synthesizing edge maps. Then, the mean of the element-wise absolute differences between the two edge maps is calculated. This loss focuses on high-frequency information such as the contour boundaries and texture edges of objects in the image. It imposes key constraints on key structural information such as the contour of the mining truck, the top edge of the slope, road markings, and the geometric edges of the ore pile in the open-pit mine scene, preventing edge blurring or loss of detail during degradation restoration. The weight coefficient for this component is set to, for example, 0.5. The three loss components are linearly weighted and summed according to their respective weight coefficients to form the joint loss. During model training, the gradient of the joint loss with respect to the parameters of each layer of the model is calculated through the backpropagation algorithm, and the parameters are updated along the gradient direction using stochastic gradient descent or optimizers such as Adam. The optimization is iterative until the joint loss converges to a stable level.
[0070] Corresponding to the above-described open-pit mine vehicle-mounted image restoration method, this invention also proposes an open-pit mine vehicle-mounted image restoration device. Since the device embodiments of this invention correspond to the above-described method embodiments, details not disclosed in the device embodiments can be referred to the above-described method embodiments, and will not be repeated here.
[0071] Figure 2 This is a schematic diagram of the structure of an open-pit mine vehicle-mounted image restoration device provided in an embodiment of this disclosure, as shown below. Figure 2 As shown, it includes: Acquisition unit 21 is used to acquire multiple consecutive frames of degraded images; Processing unit 22 is further configured to input the continuous multi-frame degraded images into a deep learning network model, so as to perform multi-degradation unified restoration and inter-frame interference suppression on the continuous multi-frame degraded images through the deep learning network model; The output unit 23 is also used to output a single clear restored image after being processed by the deep learning network model.
[0072] Furthermore, in one possible implementation of this disclosure, the processing unit 22 is further configured to: Extract spatial features from each frame of the degraded image; The spatial features are enhanced in a targeted manner and inter-frame temporal information is fused. The single clear restored image is generated based on the features after fusing temporal information.
[0073] Furthermore, in one possible implementation of this disclosure, the processing unit 22 is further configured to: Weighted enhancement is applied to the degraded regions in the aforementioned spatial features; Temporal modeling is performed on the spatial features of each frame after enhancement processing to eliminate transient interference using inter-frame information.
[0074] Furthermore, in one possible implementation of this disclosure, the processing unit 22 is further configured to: Generate attention weights in the channel dimension and attention weights in the spatial dimension; By sequentially weighting and fusing the channel attention weights and spatial attention weights with the features to be processed, feature enhancement is achieved.
[0075] Furthermore, in one possible implementation of this disclosure, the processing unit 22 is further configured to: Temporal modeling is performed using a temporal model based on a gated loop structure or a self-attention mechanism. The temporal model contains at least one temporal processing layer, and each temporal processing layer has a hidden state for memorizing and transmitting inter-frame information.
[0076] Furthermore, in one possible implementation of this disclosure, the consecutive multi-frame degraded images are an image sequence containing multiple environmental degradation types, captured from an in-vehicle video stream.
[0077] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 3 As shown, it also includes: Building unit 24 is used to build a training dataset containing real paired image data collected from the open-pit mine site and enhanced paired image data generated through simulation. The optimization unit 25 is further configured to use the continuous multi-frame degraded images as model input and a single clear image paired with them as the supervision target, and to optimize the parameters of the deep learning network model using joint loss. The joint loss includes a first loss for constraining the overall structural similarity of the images, a second loss for constraining the similarity of high-level semantic features, and a third loss for constraining key edge details.
[0078] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of the embodiments of this disclosure, and the principle is the same. Therefore, the embodiments of this disclosure are not limited thereto.
[0079] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0080] Figure 4A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0081] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 402 or a computer program loaded from storage unit 408 into RAM (Random Access Memory) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O (Input / Output) interface 405 is also connected to bus 404.
[0082] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0083] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the open-pit mine vehicle image restoration method. For example, in some embodiments, the open-pit mine vehicle image restoration method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the aforementioned open-pit mine vehicle image restoration method by any other suitable means (e.g., by means of firmware).
[0084] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0085] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0086] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0088] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.
[0089] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.
[0090] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.
[0091] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0092] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A method for image restoration on an open-pit mine vehicle, characterized in that, include: Acquire consecutive degraded images from multiple frames; The consecutive multi-frame degraded images are input into a deep learning network model so that the deep learning network model can perform unified restoration of multiple degradations and suppression of inter-frame interference on the consecutive multi-frame degraded images. Output a single clear restored image after processing by the deep learning network model.
2. The method according to claim 1, characterized in that, The deep learning network model includes a spatial feature extraction module, a feature enhancement and fusion module, and an image reconstruction module; The spatial feature extraction module is used to extract the spatial features of each frame of the degraded image; The feature enhancement and fusion module is used to enhance the spatial features in a targeted manner and fuse inter-frame temporal information; The image reconstruction module is used to generate the single clear restored image based on the features after fusing temporal information.
3. The method according to claim 2, characterized in that, The feature enhancement and fusion module includes an attention enhancement unit and a temporal modeling unit; The attention enhancement unit is used to weighted enhance the degenerate regions in the spatial features; The temporal modeling unit is used to perform temporal modeling on the spatial features of each frame after enhancement processing, so as to eliminate instantaneous interference by utilizing inter-frame information.
4. The method according to claim 3, characterized in that, The attention enhancement unit includes a channel attention branch and a spatial attention branch; The channel attention branch is used to generate attention weights in the channel dimension, and the spatial attention branch is used to generate attention weights in the spatial dimension. The attention enhancement unit enhances the features by sequentially weighting and fusing the channel attention weights and spatial attention weights with the features to be processed.
5. The method according to claim 3, characterized in that, The timing modeling unit is a timing model based on a gated loop structure or a self-attention mechanism; The timing model includes at least one timing processing layer, and each timing processing layer has a hidden state for memorizing and transmitting inter-frame information.
6. The method according to claim 1, characterized in that, The continuous multi-frame degraded images are image sequences extracted from the vehicle video stream and contain various environmental degradation types.
7. The method according to claim 1, characterized in that, The deep learning network model is obtained through supervised learning, including the following steps: Construct a training dataset that includes real paired image data collected from open-pit mine sites and enhanced paired image data generated through simulation; The continuous multi-frame degraded images are used as model input, and a single clear image paired with them is used as the supervision target; The model parameters are optimized using a joint loss, which includes a first loss for constraining the overall structural similarity of the image, a second loss for constraining the similarity of high-level semantic features, and a third loss for constraining key edge details.
8. An image restoration device for open-pit mine vehicles, characterized in that, include: The acquisition unit is used to acquire multiple consecutive frames of degraded images. The processing unit is also configured to input the consecutive multi-frame degraded images into a deep learning network model, so as to perform multi-degradation unified restoration and inter-frame interference suppression on the consecutive multi-frame degraded images through the deep learning network model; The output unit is also used to output a single clear restored image after being processed by the deep learning network model.
9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.
