Image restoration method and system

Abstract

The present invention discloses an image restoration method and system, which specifically relates to the field of image restoration technology, and is used to solve the existing problem that when processing defective areas involving the linkage of multiple modal information, the generated filling content is often unable to be consistent with other modal information; the temporal dynamic interaction of multimodal images is feature extracted through a spatiotemporal graph convolutional network, and an interaction weight matrix is ​​generated to identify the spatiotemporal relationship between multimodal images; then the spatial features are analyzed to determine whether there is overlap or interaction of objects in the scene; if there is no overlap, the information mismatch is evaluated using the interaction weight matrix and repaired; if there is overlap, the dimensionality reduction processing is performed through principal component analysis to determine whether the change in semantic alignment rate is normal, and the occlusion relationship and conflict intensity are analyzed in combination with a hidden Markov model; finally, based on a comprehensive analysis, potential hidden dangers in the restoration process are warned in advance, and the restoration strategy is dynamically adjusted to improve the restoration quality of multimodal images.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and more specifically, to an image restoration method and system. Background Art

[0002] Image restoration technology aims to fill in the defective areas in an image to restore its integrity and visual quality. Existing technologies mainly focus on the restoration of single-modality images, such as RGB images, and have achieved certain results in filling in details and textures using methods such as generative adversarial networks (GANs). However, with the increasing application of multimodal images, such as combining depth images and infrared images for comprehensive analysis, single-modality restoration methods face significant challenges in multimodal scenarios.

[0003] When existing technologies process defective areas involving the linkage of multiple modal information, the generated filling content is often inconsistent with other modal information, resulting in distortion and misalignment of the repaired image in multimodal applications. This information mismatch not only affects the visual quality of the image, but also weakens the accuracy and reliability of multimodal data fusion and analysis.

[0004] In order to solve the above problems, a technical solution is now provided. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an image restoration method and system to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An image restoration method comprises the following steps:

[0008] Based on the spatiotemporal graph convolutional network, the temporal dynamic interaction of multimodal images is feature extracted to generate the interaction weight matrix of each modality to identify the spatiotemporal relationship between multimodal images;

[0009] By analyzing the spatial features in multimodal images, the feature vectors of each modality are spatially analyzed to identify whether there are overlaps or interactions between multiple objects in the image scene;

[0010] If there is no overlap or interaction between multiple objects, the generated interaction weight matrix is ​​used to evaluate the information mismatch between multiple modalities, and image restoration is performed based on the evaluation results;

[0011] If there are multiple objects overlapping or interacting with each other, the context-dependent features in the multimodal image are reduced in dimension through principal component analysis to determine whether the changes in the semantic alignment rate under different modalities are normal; the occlusion relationship in the multimodal image is analyzed through the hidden Markov model to evaluate the degree of change in the intensity of occlusion conflict under different modalities;

[0012] Based on a comprehensive analysis of the changes in semantic alignment rates and occlusion conflict intensity under different modalities, potential risks in the repair process can be warned in advance.

[0013] In a preferred embodiment, the temporal dynamic interaction of multimodal images is feature extracted based on the spatiotemporal graph convolutional network to generate an interaction weight matrix of each modality to identify the spatiotemporal relationship between multimodal images, specifically:

[0014] Organize multimodal image data in time series and construct a spatiotemporal graph model, where each modality is used as a node and the time frame is used as the temporal dimension of the graph;

[0015] The spatiotemporal graph convolutional network is used to extract features from the constructed spatiotemporal graph model, capture the dynamic features of different modalities in the time dimension, and obtain the spatiotemporal feature representation of each modality;

[0016] Introducing the attention mechanism into the spatiotemporal graph convolutional network to weigh the influence of each modality in different time frames and generate a dynamic modality interaction weight matrix;

[0017] Based on the generated modality interaction weight matrix, the spatiotemporal relationships between multimodal images are identified.

[0018] In a preferred embodiment, by analyzing the spatial features in the multimodal image, the feature vector of each modality is spatially analyzed to identify whether there is overlap or interaction of multiple objects in the image scene, specifically:

[0019] Extract the spatial features of the multimodal image and generate a corresponding spatial feature vector for each modality;

[0020] Using the extracted spatial feature vectors, construct the feature map of each modality, and capture the potential overlapping information between objects through the spatial dimension relationship in the feature map;

[0021] Compare and analyze the feature maps of each modality, and identify potential object overlap areas by cross-comparing feature points of different modalities;

[0022] In the identified overlapping area, fine-grained analysis of spatial features is further performed to identify whether there is overlap or interaction between multiple objects in the image scene.

[0023] In a preferred embodiment, the generated interaction weight matrix is ​​used to evaluate the information mismatch between multiple modalities, and image restoration is performed according to the evaluation results, specifically:

[0024] If there is no object overlap or interaction, the information transfer and matching between the modes are evaluated based on the generated dynamic modal interaction weight matrix;

[0025] By calculating the feature similarity between modalities, the information consistency between different modalities is evaluated to determine whether there is information mismatch;

[0026] The assessed information mismatch area is marked, and the repair method is dynamically adjusted according to the degree of information mismatch.

[0027] In a preferred embodiment, the context-dependent features in the multimodal image are subjected to dimensionality reduction processing by principal component analysis to determine whether the change in semantic alignment rate under different modalities is normal, specifically including:

[0028] Extract context-dependent features from multimodal images and generate corresponding feature vectors for each modality to capture the semantic information of the image;

[0029] The eigenvectors of each mode are constructed into a feature matrix to ensure that the data between different modes correspond in space and time;

[0030] Apply principal component analysis to the constructed feature matrix, reduce the dimension of the feature data, and extract the main semantic features;

[0031] Calculate the semantic alignment rate of each modality to measure the proportion of variance explained by the principal components in different modalities;

[0032] Analyze the changes in semantic alignment rates under different modalities and determine whether the changes in semantic alignment rates under different modalities are normal.

[0033] In a preferred embodiment, the occlusion relationship in the multimodal image is analyzed by a hidden Markov model to evaluate the degree of change of the occlusion conflict intensity under different modes, specifically including:

[0034] Discretize the multimodal image data into multiple observation sequences and extract occlusion-related feature vectors for each modality;

[0035] According to the time series data of multimodal images, the state set of the hidden Markov model is defined, and the initial state probability and state transfer matrix of each mode are set;

[0036] The forward algorithm of the hidden Markov model is used to estimate the occlusion state transition probability of each mode in different time periods;

[0037] The occlusion conflict strength between modes is calculated based on the transition probability of the occlusion state, and the conflict change is judged by the difference in conflict strength;

[0038] The calculation results of the occlusion conflict strength are normalized, and the changes in the occlusion conflict strength under different modes are compared to determine the degree of conflict change: when the normalized occlusion conflict strength value is greater than the corresponding preset threshold, it is judged that the degree of change of the occlusion conflict strength under different modes is large; when the normalized occlusion conflict strength value is less than or equal to the corresponding preset threshold, it is judged that the degree of change of the occlusion conflict strength under different modes is normal.

[0039] In a preferred embodiment, based on a comprehensive analysis of the changes in semantic alignment rates and occlusion conflict strengths under different modalities, potential risks in the repair process are warned in advance, specifically:

[0040] When the degree of change of the occlusion conflict intensity under different modes is normal, and the change of the semantic alignment rate under different modes is normal, the image restoration process is judged to be normal; otherwise, it is judged that there are potential risks in the image restoration process;

[0041] When it is determined that there are potential risks in the image restoration process, the image restoration strategy is automatically adjusted.

[0042] On the other hand, the present invention provides an image restoration system, including a dynamic feature extraction module, a feature overlap analysis module, an interaction matrix generation module, a semantic alignment analysis module, an occlusion conflict assessment module, and a comprehensive analysis and early warning module;

[0043] Dynamic feature extraction module: Based on the spatiotemporal graph convolutional network, it extracts features of the temporal dynamic interaction of multimodal images and generates the interaction weight matrix of each modality to identify the spatiotemporal relationship between multimodal images;

[0044] Feature overlap analysis module: By analyzing the spatial features in multimodal images, the feature vectors of each modality are spatially analyzed to identify whether there is overlap or interaction between multiple objects in the image scene;

[0045] If there is no overlap or interaction between multiple objects, the interaction matrix generation module uses the generated interaction weight matrix to evaluate the information mismatch between multiple modalities and performs image restoration based on the evaluation results;

[0046] If there are overlaps or interactions between multiple objects, the semantic alignment analysis module performs dimensionality reduction on the context-dependent features in the multimodal images through principal component analysis to determine whether the changes in the semantic alignment rates under different modalities are normal;

[0047] If there are multiple objects overlapping or interacting with each other, the occlusion conflict assessment module analyzes the occlusion relationship in the multimodal image through the hidden Markov model to assess the degree of change in the intensity of occlusion conflicts under different modalities;

[0048] Comprehensive analysis and warning module: Based on the comprehensive analysis of the changes in semantic alignment rate and occlusion conflict intensity under different modalities, potential risks in the repair process can be warned in advance.

[0049] The technical effects and advantages of an image restoration method and system of the present invention are as follows:

[0050] 1. Accurately analyze the information interaction of different modalities in different time dimensions through the multimodal interaction weight matrix, avoiding the problem of inconsistent information among different modalities in traditional restoration methods, thereby improving the quality and consistency of the restored image. In the absence of overlapping or interacting objects, the interaction weight matrix can be used to evaluate the information mismatch between multiple modalities, ensuring that the restored image maintains high-precision matching in a multimodal environment and avoiding information distortion and dislocation problems.

[0051] 2. When multiple objects overlap or interact with each other in the image scene, the combined analysis method of principal component analysis and hidden Markov model is introduced to effectively deal with complex occlusion and semantic inconsistency problems. The principal component analysis is used to reduce the dimension of the context-dependent features of multimodal images to ensure that the semantic alignment rate between different modalities is normal; at the same time, the hidden Markov model is used to evaluate the changes in the intensity of occlusion conflicts, which can provide early warning of potential risks in the image restoration process and ensure that the restoration strategy is automatically optimized and adjusted according to the actual scene. This comprehensive analysis method improves the ability to handle complex scenes during the restoration process, making the restoration results more natural and accurate, and meeting the high-demand restoration needs in multimodal scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of an image restoration method of the present invention;

[0053] Figure 2 The figure is a schematic diagram of the structure of an image restoration system of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example

[0055] Figure 1The present invention provides an image restoration method, which comprises the following steps:

[0056] Based on the spatiotemporal graph convolutional network, the temporal dynamic interaction of multimodal images is feature extracted, and the interaction weight matrix of each modality is generated to identify the spatiotemporal relationship between multimodal images.

[0057] By parsing the spatial features in multimodal images, the feature vectors of each modality are spatially analyzed to identify whether there is overlap or interaction between multiple objects in the image scene.

[0058] If there is no overlap or interaction between multiple objects, the generated interaction weight matrix is ​​used to evaluate the information mismatch between multiple modalities, and image restoration is performed based on the evaluation results.

[0059] If there is overlap or interaction between multiple objects, the context-dependent features in the multimodal image are reduced in dimension through principal component analysis to determine whether the changes in the semantic alignment rate under different modalities are normal; the occlusion relationship in the multimodal image is analyzed through the hidden Markov model to evaluate the degree of change in the occlusion conflict intensity under different modalities.

[0060] Based on a comprehensive analysis of the changes in semantic alignment rates and occlusion conflict intensity under different modalities, potential risks in the repair process can be warned in advance.

[0061] Based on the spatiotemporal graph convolutional network, the temporal dynamic interaction of multimodal images is feature extracted to generate the interaction weight matrix of each modality to identify the spatiotemporal relationship between multimodal images. Specifically:

[0062] The multimodal image data is organized in time series to construct a spatiotemporal graph model, where each modality is used as a node and the time frame is used as the temporal dimension of the graph:

[0063] Multimodal image data are organized according to their time series characteristics; image data includes multiple modalities, such as RGB images, depth images, and infrared images. These multimodal data are constructed into a spatiotemporal graph model to characterize the interactive relationship between each modality image in the time dimension. Each modality data can be regarded as a node at different time points, and the nodes are connected by edges. The edges represent the relationship between different modalities in the same time frame, as well as the association in different time frames.

[0064] When constructing a spatiotemporal graph, each node represents the characteristics of a specific modality in a specific time frame. The characteristics are represented in the form of vectors, and the characteristic vector of a modality is composed of multiple characteristic values. The edge weights between nodes are used to describe the correlation between modalities. The edge weights not only consider the modal correlation of the current time frame, but also the relationship between modalities across time frames. Based on this construction method, the spatiotemporal graph model fully expresses the interaction between time and modality.

[0065] The node feature vector is expressed as: ;in, is modal In time frame The characteristic vector of is the characteristic of the mode on this time frame, Represents a modality number in a multimodal image. Indicates the number of features contained in the feature vector.

[0066] The edge weight is expressed as: ;in, Representing modality and modal In time frame Similarity on is modal In time frame The eigenvectors on ; Representation and modality The corresponding other modality number is used to indicate the interaction relationship between different modalities.

[0067] The spatiotemporal graph convolutional network is used to extract features from the constructed spatiotemporal graph model, capture the dynamic features of different modalities in the time dimension, and obtain the spatiotemporal feature representation of each modality:

[0068] After constructing the spatiotemporal graph model, the spatiotemporal graph convolutional network is used to extract features from the nodes in the model. The convolution operation is divided into two parts: the first part is the convolution operation on the spatial dimension, and the second part is the convolution operation on the temporal dimension. The convolution operation on the spatial dimension captures the interactive features between different modalities in the same time frame; the convolution operation on the temporal dimension captures the changes of the same modality between different time frames.

[0069] The size and shape of the convolution kernel are set according to the different dimensions of space and time. The spatial convolution kernel acts on adjacent modal nodes, while the temporal convolution kernel acts on the same modal nodes in adjacent time frames.

[0070] The spatial convolution operation is implemented by the following formula: ;in, Represents the mode after spatial convolution In time frame New features on Representing modality The adjacent nodes of is the spatial convolution kernel.

[0071] The temporal convolution operation is implemented by the following formula: ;in, The new features after time convolution, represents adjacent time frames in the time dimension, Representing modality In time frame The adjacent time frame of Representing modality In time frame The eigenvectors on is the temporal convolution kernel.

[0072] Through spatial convolution operations and temporal convolution operations, the node feature representation in the spatiotemporal graph is obtained.

[0073] The attention mechanism is introduced into the spatiotemporal graph convolutional network to weigh the influence of each modality in different time frames and generate a dynamic modality interaction weight matrix:

[0074] On the basis of the spatiotemporal graph convolution operation, in order to dynamically adjust the influence of each modality in different time frames, an attention mechanism is added. The attention mechanism can dynamically assign weights according to the importance of each modality in different time frames, thereby more accurately capturing the interaction between modalities.

[0075] Calculate the attention weight, the expression is: ;in, Representing modality In time frame The attention weight on is modal In time frame The attention score on represents the set of all modes, Representing modality In time frame Attention score on .

[0076] Among them, the attention score is calculated by the similarity between the modal features and the features of the current time frame. The calculation process uses the attention vector and the feature transformation matrix, and normalization is performed after the weight is obtained.

[0077] Through convolution operation and attention mechanism, a dynamic modal interaction weight matrix is ​​generated, specifically: ;in, Representing modality With modal The interaction weights over the entire time series, Indicates the total number of time frames included in the time series.

[0078] Based on the generated modality interaction weight matrix, the spatiotemporal relationship between multimodal images is identified:

[0079] If the modal With modal The interaction weight over the entire time series is high, indicating that the modality With modal There is a strong correlation or interaction in multiple time frames. This indicates that the two modalities may capture similar scene changes, object movements, or interactions between objects in the same time period. With modal The interaction weights over the entire time series are low, indicating that the modality With modal The interaction between them is weak, perhaps because the features of these modes in the time series are quite different, or because the objects in the scene do not interact significantly.

[0080] By analyzing the spatial features in multimodal images, the feature vectors of each modality are spatially analyzed to identify whether there are overlaps or interactions between multiple objects in the image scene. Specifically:

[0081] The spatial features of the multimodal image are extracted, and the corresponding spatial feature vector is generated for each modality:

[0082] For each modality image (such as RGB image, depth image and infrared image), its spatial features are extracted through convolutional neural network (CNN) or other existing feature extraction algorithms. Each modality image generates a feature map through multi-layer convolution operation. These feature maps describe the edge, texture, shape and other information in the image. The output of the convolution operation is a spatial feature vector, which represents the main feature information of each pixel or region in the image.

[0083] Through the above process, a spatial feature vector is generated for each mode, providing data for subsequent comparison and analysis.

[0084] Using the extracted spatial feature vectors, we construct a feature map for each modality and capture the potential overlapping information between objects through the spatial dimension relationship in the feature map:

[0085] The generated spatial feature vectors are used to construct the feature graph of each modality. The feature graph is constructed by treating each pixel (or region) in the image as a node of the graph, and the edges between the nodes represent the spatial adjacency relationship. The feature graph describes the spatial structure of objects in the image. In the feature graph, if two objects overlap in space, their corresponding feature graph nodes will show similar characteristics. Therefore, by analyzing the nodes and edges in the feature graph, the potential overlapping information between objects can be preliminarily captured.

[0086] Compare and analyze the feature graphs of each modality, and identify potential object overlap areas by cross-comparing feature points of different modalities:

[0087] Align and cross-compare feature maps of different modalities.

[0088] Spatial alignment: Use existing image registration techniques, such as feature point-based registration methods such as scale-invariant feature transform (SIFT), to spatially align images of different modalities to ensure that the same position in each modality can be correctly compared.

[0089] Feature point comparison: By comparing the feature vectors of different modalities at the same location, it is determined whether the location has similar features in multiple modalities. For example, if a region exhibits similar spatial features in the RGB image and the depth image, it can be inferred that the region may contain the same object and there may be overlap.

[0090] By calculating the similarity between feature vectors of different modalities, the potential overlapping areas of multiple objects in space can be identified.

[0091] In the identified overlapping area, we further analyze the spatial features in a fine-grained manner to identify whether there are multiple objects overlapping or interacting in the image scene:

[0092] After the potential overlapping areas are identified, these areas are further subjected to fine-grained spatial feature analysis to accurately determine whether there is overlap or interaction between multiple objects. The specific methods include:

[0093] The overlapping areas are analyzed by edge detection technology (such as Canny edge detection) to try to separate the boundaries of different objects. By detecting the edges in the image, the boundary lines between objects can be clearly identified and whether there is any overlap between objects.

[0094] The contour detection algorithm is further applied to separate the objects in the overlapping areas. By performing a detailed analysis of the local areas in the image, it is possible to determine whether multiple objects are physically occluded or overlapped.

[0095] Through geometric spatial analysis, the relative position, direction, and distance between objects are modeled. After identifying the edges and contours of the objects, the relative positions of these objects in space are further analyzed to determine whether they are blocked or interacting on the same plane. For example, when the contour of an object is partially within the contour of another object and its position conforms to the occlusion law, it can be determined that there is overlap or occlusion between the objects.

[0096] For multiple objects, analyze their dynamic behaviors to determine whether there is physical interaction. You can determine whether there is interaction between these objects by tracking the objects' motion trajectories, deformations, and relative motion directions. For example, if the motion directions of two objects converge or collide, it can be inferred that there is interaction between them.

[0097] Among them, image scene refers to a comprehensive spatial representation of objects, background and their relative positions in a multimodal image, which is used to describe the physical layout and interaction relationship of objects at a specific moment.

[0098] The generated interaction weight matrix is ​​used to evaluate the information mismatch between multiple modalities, and image restoration is performed based on the evaluation results, specifically:

[0099] If there is no object overlap or interaction, the information transfer and matching between the modes are evaluated based on the generated dynamic modal interaction weight matrix:

[0100] Assume that there are multiple modes, respectively , as well as Letters like denoted different modes. For each pair of modes and , their information transfer is evaluated based on the interaction weight matrix between them. Each element of this matrix can represent the modality and modal The interaction strength at different time frames. The interaction weight is defined as , Indicated in time frame Next, modal With modal The intensity of interaction between them.

[0101] Calculate the feature similarity of the two modalities in a certain time frame. Feature similarity is the core indicator to measure the consistency of the information of the two modalities. This evaluation is performed by calculating the similarity between the feature vectors of the two modalities.

[0102] By calculating the feature similarity between modalities, the information consistency between different modalities is evaluated to determine whether there is information mismatch:

[0103] Assuming that in the time frame Next, modal The eigenvector of , modal The eigenvector of In order to evaluate the consistency of information, it is necessary to calculate and Similarity can be calculated using a variety of metrics. Here, the commonly used cosine similarity is used to measure the similarity between two modal feature vectors.

[0104] The calculation formula for cosine similarity is: ;in, Representing modality and modal In time frame The feature similarity under Represents the feature vector and The inner product of and They represent the modulus of the eigenvectors respectively.

[0105] By calculating the feature similarity at each time frame, the average similarity of the two modalities over the entire time series can be obtained.

[0106] If the average similarity of the two modalities over the entire time series is high, it means that the information between the two modalities matches well in each time frame; if the average similarity of the two modalities over the entire time series is low, it indicates that there may be information mismatch between the two modalities.

[0107] The assessed information mismatch areas are marked, and the repair method is dynamically adjusted according to the degree of information mismatch:

[0108] Setting Thresholds , if the average similarity of two modalities over the entire time series is lower than , then we can identify the mode and modal There is a high degree of information mismatch between them. It is set based on experience or specific scenarios. Generally speaking, The value is between 0.7 and 0.9.

[0109] When information mismatch is detected, the specific area of ​​the mismatch needs to be marked.

[0110] For each time frame ,if Below threshold , then mark the time frame The mismatch area is marked by the position information of the feature points. Specifically, for the time frame , modal and modal When the similarity between the feature vectors is low, the specific position of the feature point in the image can be tracked and marked as a mismatch area.

[0111] For example, assuming the modal The characteristic points and modes The characteristic points represent the time frame A specific spatial position under Below threshold , the position can be marked as a mismatch area, indicating that at this position, the information of the two modes is inconsistent.

[0112] After marking the information mismatch areas, these areas are repaired. The specific method of repair depends on the severity of the mismatch. The repair method can be dynamically adjusted according to the degree of mismatch.

[0113] The degree of mismatch is measured based on If The value of ), it means that the mismatch is high and a more complex repair algorithm is required; if The value is slightly below the threshold , it means that the mismatch is relatively minor and a simpler repair method can be used.

[0114] The specific repair algorithm can be a generative adversarial network (GAN) based on deep learning. The generative adversarial network can infer the lost or distorted parts based on the non-mismatched areas, thereby completing the repair. The repair process includes the following steps:

[0115] Region inference: Generate possible image completion content based on the image information around the mismatched area.

[0116] Repair generation: Repair the mismatched area through the generative network. The input of the generative network is the feature vector of the mismatched area, and the output is the repaired image of the area.

[0117] Adversarial training: Through adversarial training, a discriminator is used to evaluate the authenticity of the repaired image and the generated network is continuously optimized until the generated repaired image is well integrated with the original image.

[0118] The context-dependent features in multimodal images are reduced in dimension through principal component analysis to determine whether the changes in semantic alignment rates under different modalities are normal, including:

[0119] Extract context-dependent features from multimodal images and generate corresponding feature vectors for each modality to capture the semantic information of the image:

[0120] Context-dependent features are extracted from images of each modality (such as RGB images, infrared images, depth images, etc.). In order to capture the semantic information of the image, existing methods such as Convolutional Neural Network (CNN) are used to extract features from the image.

[0121] For each modality image, after preprocessing, it is input into the feature extraction model to obtain the corresponding feature vector. The feature vector is set as: ;in, Indicates The first mode The feature vector of the image; represents the first element in the feature vector, and so on to elements; Number the modalities to distinguish images of different modalities, such as the first modality, the second modality, etc. is the image number, indicating the image number of each modality; is the dimension of the feature vector.

[0122] By performing feature extraction on all images of each modality, a feature vector set of each modality is obtained.

[0123] The eigenvectors of each mode are constructed into a feature matrix to ensure that the data between different modes correspond in space and time:

[0124] The eigenvector of each mode is constructed according to the correspondence between space and time to construct the feature matrix, which is defined as: ;in, For the The characteristic matrix of each mode; is the total number of images, indicating the number of images under the modality; is the element of the feature matrix, indicating the first The first feature of the image.

[0125] By constructing a feature matrix, it is ensured that the feature data of different modalities remain consistent in space and time.

[0126] Apply principal component analysis to the constructed feature matrix, reduce the dimension of the feature data, and extract the main semantic features:

[0127] Compute the mean vector: ;in, For the The mean vector of the modes.

[0128] Subtract the mean vector from each eigenvector in the feature matrix to obtain the centered data matrix: ;in, is the centralized data matrix, for A column vector of all ones.

[0129] Compute the covariance matrix: ;in, For the The covariance matrix of the modes is, for The transposed matrix of .

[0130] Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors: ;in, For the The first mode eigenvalues, is the corresponding eigenvector, Indicates the number of the current principal component.

[0131] According to the size of the eigenvalue, select the The eigenvectors corresponding to the largest eigenvalues ​​constitute the projection matrix: ;in, is the projection matrix.

[0132] Then, the centralized data matrix is ​​projected into the principal component space to obtain the reduced-dimensional data matrix: ;in, is the feature matrix after dimensionality reduction.

[0133] Calculate the semantic alignment rate of each modality and measure the proportion of variance explained by the principal components in different modalities:

[0134] For modes, calculate the variance contribution of each principal component: ;in, For the The modal The variance contribution of the principal components is The number used to represent the principal component.

[0135] in, Usually used to refer to the number of the specific principal component currently being processed or concerned. It usually represents the number range of all principal components generated during the entire principal component analysis process.

[0136] Before calculation The cumulative variance contribution of the principal components is used as semantic alignment: ;in, Indicates Semantic alignment rate of modalities; It is the number of principal components selected in principal component analysis, which is used to indicate the number of principal components retained during dimensionality reduction. The top principal components are usually selected by the size of the eigenvalue. The most important principal component.

[0137] Analyze the changes in semantic alignment rates under different modalities and determine whether the changes in semantic alignment rates under different modalities are normal:

[0138] Set a threshold , represents the normal variation range of the semantic alignment rate.

[0139] For any two modes and modal , calculate the semantic alignment rate difference: ;in, For modal and The difference in semantic alignment rate.

[0140] If the semantic alignment rate differences between all modalities satisfy , then the variation of semantic alignment rate under different modalities is judged to be normal; otherwise, the variation of semantic alignment rate under different modalities is judged to be abnormal.

[0141] The hidden Markov model is used to analyze the occlusion relationship in multimodal images and evaluate the degree of change in the intensity of occlusion conflicts under different modes, including:

[0142] Discretize the multimodal image data into multiple observation sequences and extract the occlusion-related feature vectors for each modality:

[0143] First, the image data of different modalities are processed and converted into observation sequences suitable for hidden Markov model processing. Since the hidden Markov model needs to process discrete input data, it is necessary to extract key occlusion-related features from continuous multimodal images. The specific approach is to extract features related to occlusion phenomena for each modal image sequence. For example, RGB modal images can identify object boundaries through edge detection algorithms, depth modal images can analyze the distance changes of objects in the scene, and infrared images can capture occlusion through changes in thermal signals.

[0144] The image of each modality forms an occlusion feature vector in the time series, which represents the occlusion feature of the modality at each time point. At this point, the image sequence has been converted into multiple observation sequences, each of which represents the time frame occlusion feature of a different modality.

[0145] According to the time series data of multimodal images, the state set of the hidden Markov model is defined, and the initial state probability and state transfer matrix of each mode are set:

[0146] After extracting the occlusion features of the multimodal image, it is necessary to define the possible occlusion state set for each modality. The state set represents the occlusion status of the object at a certain moment, for example, the object may be in a state of "no occlusion", "partial occlusion" or "full occlusion". By analyzing the image, these occlusion phenomena can be classified into states.

[0147] For example, for the occlusion state of a multimodal image, a common state set may include three occlusion cases:

[0148] State 1: The object is not occluded.

[0149] State 2: The object is partially occluded.

[0150] State 3: The object is completely occluded.

[0151] After defining the state set, we need to set the probability distribution for the initial state of each mode. That is, at a certain point in time, the system may start with an object being unobstructed, partially obstructed, or fully obstructed, and these states have certain initial probabilities. For example, the initial state of a mode may have an 80% probability of being "unobstructed", a 15% probability of being "partially obstructed", and a 5% probability of being "fully obstructed".

[0152] In addition to the initial state setting, a state transition matrix is ​​defined for each mode. The state transition matrix describes the probability of a mode transitioning from one occlusion state to another occlusion state at different time points. For example, if a mode is in the "partially occluded" state at a certain moment, then at the next moment, it may transition to the "unoccluded" state with a probability of 70%, to the "fully occluded" state with a probability of 20%, and to continue to remain in the "partially occluded" state with a probability of 10%.

[0153] Through the forward algorithm of the hidden Markov model, the probability of occlusion state transition of each mode in different time periods is estimated:

[0154] After defining the state set and state transition matrix, the forward algorithm of the hidden Markov model is used to estimate the occlusion state of each modality at each time period. The forward algorithm is a recursive calculation method of the hidden Markov model, which is used to estimate the correlation between the observed data of the current time frame and different states.

[0155] Specifically, the forward algorithm estimates the probability of each occlusion state at the current moment based on the occlusion state of the previous time frame and the transition probability from that state to the occlusion state of the current time frame. The probability of the occlusion state in each time period is calculated by accumulating the transition probability of the state in the previous time period and combining it with the occlusion features in the current observation sequence. For example, if an object of a certain modality is in a "partially occluded" state at a certain moment, the forward algorithm will combine the observation sequence at the current time to calculate the probability that the modality may transition to other occlusion states at the next moment.

[0156] Through the recursive calculation of the forward algorithm, the occlusion state transition probability of each mode in each time period can be estimated, and the probability distribution of each mode being "unoccluded", "partially occluded" or "fully occluded" in different time periods can be obtained.

[0157] The occlusion conflict strength between modes is calculated based on the transition probability of the occlusion state, and the conflict change is determined by the difference in conflict strength:

[0158] To calculate the occlusion conflict strength between modalities, we first need to compare the occlusion state probabilities of the two modalities in the same time period. Assuming that at a certain moment, the occlusion state of one modality is "partially occluded" and the occlusion state of the other is "unoccluded", we can calculate the conflict strength of the two modalities at that moment. The specific method is to calculate the probability difference of the two modalities in each occlusion state, accumulate these differences, and obtain the final conflict strength value.

[0159] For example, if one modality has an 80% probability of being in the "partially occluded" state and the other has a 90% probability of being in the "unoccluded" state, the conflict strength between the two will reflect how different the two states are. The final occlusion conflict strength can be obtained by accumulating the probability differences of all states.

[0160] The occlusion conflict intensity reflects the degree of difference in the occlusion status of different modes in the same time period. The greater the conflict intensity, the more obvious the difference in occlusion between the modes.

[0161] The calculation results of the occlusion conflict strength are normalized, and the changes in the occlusion conflict strength under different modes are compared to determine the degree of conflict change:

[0162] After obtaining the occlusion conflict strength between different modes in each time period, the next step is to normalize these calculation results. The main purpose of normalization is to compress the occlusion conflict strength of different modes in different time periods into a standardized numerical range, usually [0,1]. Through this process, the occlusion conflict strength between different modes in different time periods can be more clearly compared, and the degree of change in conflict strength can be determined.

[0163] After normalization, the change in occlusion conflict strength needs to be determined based on the value. To this end, a reasonable preset threshold can be set to classify the occlusion conflict strength as "large" or "small". For example, 0.5 is set as the preset threshold for determining the degree of change in occlusion conflict strength under different modes:

[0164] When the normalized occlusion conflict intensity value is greater than its corresponding preset threshold, it is determined that the occlusion conflict intensity under different modes varies greatly, indicating that the occlusion conflict intensity in this time period is large, indicating that the occlusion conflict between different modes in this time period is more obvious and there is a large difference in occlusion status.

[0165] When the normalized occlusion conflict intensity value is less than or equal to its corresponding preset threshold, it is determined that the degree of change in the occlusion conflict intensity under different modes is normal, indicating that the occlusion conflict intensity in this time period is small, indicating that the occlusion status of different modes changes little in this time period and the conflict is not obvious.

[0166] Based on the comprehensive analysis of the changes in semantic alignment rate and occlusion conflict intensity under different modalities, potential risks in the repair process are warned in advance, specifically:

[0167] The occlusion conflict intensity reflects the mutual consistency of different modal images when dealing with occlusion phenomena. If the conflict intensity is too large, it means that some modalities fail to correctly handle occlusion information during restoration, which may cause unnatural overlap or misalignment in parts of the image. The semantic alignment rate reflects whether the multimodal image maintains consistency at the semantic level during the restoration process. An abnormal semantic alignment rate indicates that the restoration results of different modalities do not match, which may cause the overall semantic inconsistency of the image. Therefore, when both are normal, the restoration process is considered stable and reliable; if any one is abnormal, the image restoration result may not meet expectations, and the restoration strategy must be adjusted to ensure the quality and consistency of the final image.

[0168] Therefore, when the degree of change in the occlusion conflict intensity under different modalities is normal and the change in the semantic alignment rate under different modalities is normal, the image restoration process is judged to be normal; otherwise, it is judged that there are potential risks in the image restoration process.

[0169] When it is determined that there are potential risks in the image restoration process, the image restoration strategy is automatically adjusted, specifically:

[0170] Adjustment strategy for abnormal changes in occlusion conflict intensity:

[0171] Enhanced occlusion processing: When abnormal changes in the intensity of occlusion conflicts are detected, the system needs to further optimize the processing of occluded areas. For example, this can be done by introducing more occlusion detection models, increasing the focus on occluded areas, or using more sophisticated occlusion processing algorithms. This includes inferring the shape and position of occluded objects based on physical models, and providing consistent processing methods for different modalities to reduce conflicts between modalities.

[0172] Local enhancement model: Refine local areas with high conflict intensity, increase image resolution or feature extraction depth to more accurately solve occlusion-related problems and reduce information inconsistency between modalities.

[0173] Adjustment strategy for abnormal changes in semantic alignment rate:

[0174] Realigning semantic information: When the semantic alignment rate changes abnormally, it means that the images of different modalities are not properly aligned at the semantic level, resulting in semantic conflicts in the restored images. At this time, the system can adjust the semantic alignment strategy between modalities and re-evaluate the semantic consistency between multiple modalities. More complex multimodal alignment algorithms can be introduced, such as a semantic matching model based on an attention mechanism, to ensure that multimodal data maintains consistent semantic expression during the restoration process.

[0175] Multi-level semantic analysis: Add semantic hierarchical analysis to jointly evaluate high-level (such as object level) and low-level (such as pixel level) semantics to ensure that semantic information at all levels can be effectively aligned under different modalities.

[0176] Joint adjustments based on holistic analysis:

[0177] Dynamic adjustment of multi-modal weights: If the system detects that the influence of a certain modality in the repair process is too large or too small, it may cause other modalities to be unable to effectively align with it. At this time, the system can dynamically adjust the weights between multi-modalities and perform adaptive weighting processing based on the importance and conflict intensity of the modalities to reduce the adverse impact of a single modality on the overall repair result.

[0178] Introducing Generative Adversarial Networks (GANs): In more complex scenarios, if both the occlusion conflict strength and semantic alignment rate are abnormal, the system can generate new image completion content by introducing generative adversarial networks. GANs can generate more natural restoration results based on existing multimodal images, ensuring the coherence and consistency of the overall image.

[0179] By introducing a real-time feedback mechanism, the repair algorithm can be gradually optimized according to the conflicts or inconsistencies detected during the repair process. Through feedback iteration, the system can gradually adjust the repair strategy until the conflict intensity and semantic alignment rate between the modalities reach the normal range. Example

[0180] The difference between Example 2 of the present invention and Example 1 is that this example introduces an image restoration system.

[0181] Figure 2 A structural schematic diagram of an image restoration system of the present invention is given, which includes a dynamic feature extraction module, a feature overlap analysis module, an interaction matrix generation module, a semantic alignment analysis module, an occlusion conflict assessment module and a comprehensive analysis and early warning module.

[0182] Dynamic feature extraction module: Based on the spatiotemporal graph convolutional network, it extracts features of the temporal dynamic interactions of multimodal images and generates the interaction weight matrix of each modality to identify the spatiotemporal relationship between multimodal images.

[0183] Feature overlap analysis module: By analyzing the spatial features in multimodal images, the feature vectors of each modality are spatially analyzed to identify whether there is overlap or interaction between multiple objects in the image scene.

[0184] If there is no overlap or interaction between multiple objects, the interaction matrix generation module uses the generated interaction weight matrix to evaluate the information mismatch between multiple modalities and performs image restoration based on the evaluation results.

[0185] If there is overlap or interaction between multiple objects, the semantic alignment analysis module performs dimensionality reduction on the context-dependent features in the multimodal images through principal component analysis to determine whether the changes in the semantic alignment rates under different modalities are normal.

[0186] If there is overlap or interaction between multiple objects, the occlusion conflict assessment module analyzes the occlusion relationship in the multimodal image through the hidden Markov model to evaluate the degree of change in the occlusion conflict intensity under different modes.

[0187] Comprehensive analysis and warning module: Based on the comprehensive analysis of the changes in semantic alignment rate and occlusion conflict intensity under different modalities, potential risks in the repair process can be warned in advance.

[0188] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0189] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or may be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0190] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0191] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0192] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0193] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0195] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0196] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0197] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image restoration method, characterized in that: The steps include: Based on the spatiotemporal graph convolutional network, the temporal dynamic interaction of multimodal images is feature extracted to generate the interaction weight matrix of each modality to identify the spatiotemporal relationship between multimodal images; By analyzing the spatial features in multimodal images, the feature vectors of each modality are spatially analyzed to identify whether there are overlaps or interactions between multiple objects in the image scene; If there is no overlap or interaction between multiple objects, the generated interaction weight matrix is ​​used to evaluate the information mismatch between multiple modalities, and image restoration is performed based on the evaluation results; If there is overlap or interaction between multiple objects, the context-dependent features in the multimodal image are reduced in dimension through principal component analysis to determine whether the changes in the semantic alignment rate under different modalities are normal; The occlusion relationship in multimodal images is analyzed by using a hidden Markov model to evaluate the degree of change in occlusion conflict intensity under different modalities. Based on the comprehensive analysis of the changes in semantic alignment rate and occlusion conflict intensity under different modalities, potential risks in the repair process can be warned in advance; Based on the spatiotemporal graph convolutional network, the temporal dynamic interaction of multimodal images is feature extracted to generate the interaction weight matrix of each modality to identify the spatiotemporal relationship between multimodal images. Specifically: Organize multimodal image data in time series and construct a spatiotemporal graph model, where each modality is used as a node and the time frame is used as the temporal dimension of the graph; The spatiotemporal graph convolutional network is used to extract features from the constructed spatiotemporal graph model, capture the dynamic features of different modalities in the time dimension, and obtain the spatiotemporal feature representation of each modality; Introducing the attention mechanism into the spatiotemporal graph convolutional network to weigh the influence of each modality in different time frames and generate a dynamic modality interaction weight matrix; Based on the generated modality interaction weight matrix, the spatiotemporal relationships between multimodal images are identified.

2. The image restoration method according to claim 1, characterized in that: By analyzing the spatial features in multimodal images, the feature vectors of each modality are spatially analyzed to identify whether there are overlaps or interactions between multiple objects in the image scene. Specifically: Extract the spatial features of the multimodal image and generate a corresponding spatial feature vector for each modality; Using the extracted spatial feature vectors, construct the feature map of each modality, and capture the potential overlapping information between objects through the spatial dimension relationship in the feature map; Compare and analyze the feature maps of each modality, and identify potential object overlap areas by cross-comparing feature points of different modalities; In the identified overlapping area, fine-grained analysis of spatial features is further performed to identify whether there is overlap or interaction between multiple objects in the image scene.

3. An image restoration method according to claim 2, characterized in that: The generated interaction weight matrix is ​​used to evaluate the information mismatch between multiple modalities, and image restoration is performed based on the evaluation results, specifically: If there is no object overlap or interaction, the information transfer and matching between the modes are evaluated based on the generated dynamic modal interaction weight matrix; By calculating the feature similarity between modalities, the information consistency between different modalities is evaluated to determine whether there is information mismatch; The assessed information mismatch area is marked, and the repair method is dynamically adjusted according to the degree of information mismatch.

4. The image restoration method according to claim 3, characterized in that: The context-dependent features in multimodal images are reduced in dimension through principal component analysis to determine whether the changes in semantic alignment rates under different modalities are normal, including: Extract context-dependent features from multimodal images and generate corresponding feature vectors for each modality to capture the semantic information of the image; The eigenvectors of each mode are constructed into a feature matrix to ensure that the data between different modes correspond in space and time; Apply principal component analysis to the constructed feature matrix, reduce the dimension of the feature data, and extract the main semantic features; Calculate the semantic alignment rate of each modality to measure the proportion of variance explained by the principal components in different modalities; Analyze the changes in semantic alignment rates under different modalities and determine whether the changes in semantic alignment rates under different modalities are normal.

5. The image restoration method according to claim 4, characterized in that: The hidden Markov model is used to analyze the occlusion relationship in multimodal images and evaluate the degree of change in the intensity of occlusion conflicts under different modes, including: Discretize the multimodal image data into multiple observation sequences and extract occlusion-related feature vectors for each modality; According to the time series data of multimodal images, the state set of the hidden Markov model is defined, and the initial state probability and state transfer matrix of each mode are set; The forward algorithm of the hidden Markov model is used to estimate the occlusion state transition probability of each mode in different time periods; The occlusion conflict strength between modes is calculated based on the transition probability of the occlusion state, and the conflict change is judged by the difference in conflict strength; The calculation results of the occlusion conflict strength are normalized, and the changes in the occlusion conflict strength under different modes are compared to determine the degree of conflict change: when the normalized occlusion conflict strength value is greater than the corresponding preset threshold, it is judged that the degree of change in the occlusion conflict strength under different modes is large; when the normalized occlusion conflict strength value is less than or equal to the corresponding preset threshold, it is judged that the degree of change in the occlusion conflict strength under different modes is normal.

6. The image restoration method according to claim 5, characterized in that: Based on the comprehensive analysis of the changes in semantic alignment rate and occlusion conflict intensity under different modalities, potential risks in the repair process are warned in advance, specifically: When the occlusion conflict intensity changes under different modalities are normal, and the semantic alignment rate changes under different modalities are normal, the image restoration process is judged to be normal; Otherwise, it is determined that there are potential risks in the image restoration process; When it is determined that there are potential risks in the image restoration process, the image restoration strategy is automatically adjusted.

7. An image restoration system, used to implement an image restoration method according to any one of claims 1 to 6, characterized in that: It includes dynamic feature extraction module, feature overlap analysis module, interaction matrix generation module, semantic alignment analysis module, occlusion conflict assessment module and comprehensive analysis and early warning module; Dynamic feature extraction module: Based on the spatiotemporal graph convolutional network, it extracts features of the temporal dynamic interaction of multimodal images and generates the interaction weight matrix of each modality to identify the spatiotemporal relationship between multimodal images; Feature overlap analysis module: By analyzing the spatial features in multimodal images, the feature vectors of each modality are spatially analyzed to identify whether there is overlap or interaction between multiple objects in the image scene; If there is no overlap or interaction between multiple objects, the interaction matrix generation module uses the generated interaction weight matrix to evaluate the information mismatch between multiple modalities and performs image restoration based on the evaluation results; If there are overlaps or interactions between multiple objects, the semantic alignment analysis module performs dimensionality reduction on the context-dependent features in the multimodal images through principal component analysis to determine whether the changes in the semantic alignment rates under different modalities are normal; If there are multiple objects overlapping or interacting with each other, the occlusion conflict assessment module analyzes the occlusion relationship in the multimodal image through the hidden Markov model to assess the degree of change in the intensity of occlusion conflicts under different modalities; Comprehensive analysis and warning module: Based on the comprehensive analysis of the changes in semantic alignment rate and occlusion conflict intensity under different modalities, potential risks in the repair process can be warned in advance.

Images