A cross-medium operation visual image processing method and device

Abstract

The application provides a cross-medium operation visual image processing method and device, and relates to the technical field of image processing. The method comprises the following steps: acquiring visual image sequences of air medium and water medium in a cross-medium operation scene; for each frame of image, based on a physical degradation model corresponding to the environment medium, physical priori knowledge graphs corresponding to the air medium and the water medium are respectively generated; key feature parameters of each frame of image are taken as state vectors, and theoretical feature change rates calculated by the physical priori knowledge graphs are taken as control inputs, and a state transition equation is used to establish a state association between adjacent frames; based on the state transition equation, each frame of image is corrected by using an unscented Kalman filtering method, and a final visual image sequence is output. The application can effectively suppress the mutation of adjacent frame images in the cross-medium operation, eliminate the inter-frame flicker and jitter, improve the time domain stability of the image sequence, and guarantee the continuity and safety of the cross-medium operation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a cross-media visual image processing method and apparatus. Background Technology

[0002] In cross-media operation scenarios, visual image acquisition involves both air and water media. The operation process itself introduces time-varying disturbances, such as sudden changes in water quality caused by underwater excavation stirring up sediment, and water mist interference from unloading on the surface. These disturbances lead to image quality degradation under strong cross-media impacts, particularly manifested as abrupt changes in brightness, contrast, and color characteristics between adjacent frames, resulting in temporal instability phenomena such as inter-frame flickering and jitter. These abrupt changes make it difficult for operators or automated systems to accurately track the position of the excavation face, the posture of the unloading vessel, and other operational targets, thus affecting the continuity and safety of the operation. Summary of the Invention

[0003] The present invention provides a method and apparatus for cross-media operation visual image processing, which aims to solve the problem in the prior art that the time-varying disturbances introduced during the operation process cause abrupt changes in the image features of adjacent frames, thereby affecting the continuity and safety of the operation.

[0004] To achieve the above objectives, in a first aspect, the present invention provides a cross-media visual image processing method, comprising the following steps: Acquire visual image sequences of various environmental media in a cross-media operation scenario; the environmental media include air and water. For each frame of the visual image sequence, physical prior knowledge graphs corresponding to the air medium and the water medium are generated based on the physical degradation model corresponding to the environmental medium. The key feature parameters of each frame in the visual image sequence are used as state vectors, and the theoretical feature change rate of the key feature parameters calculated by the physical prior knowledge graph is used as control input. The state association between adjacent frames is established through the state transition equation. The key feature parameters include edge intensity, local contrast and color consistency. Based on the state transition equation, each frame of the image is corrected by the unscented Kalman filter method to output the final visual image sequence.

[0005] Furthermore, a random sampling consensus algorithm and constrained least squares filtering are used to perform spatiotemporal registration of visual images from different environmental media.

[0006] Furthermore, the physical prior knowledge graph includes: For the air medium, the visual image is decomposed into clear scene, atmospheric light, and transmittance, and a rain line mask and an illumination non-uniformity model are introduced. By analyzing the gradient distribution and color constancy after image decomposition, the fog concentration, rain line density, and illumination intensity of the current frame are estimated, and a water physics prior knowledge graph containing a transmittance map, atmospheric light, rain line distribution probability, and illumination non-uniformity map is generated. For the water medium, a water attenuation coefficient and an equivalent backscattering coefficient are defined, and a time-varying sediment concentration function caused by underwater operations is introduced. By calculating the local contrast and dark channel change rate of its visual image, the sediment concentration and water turbidity of the current frame are estimated, and an underwater physical prior knowledge graph containing a transmittance map and a background light vector is generated.

[0007] Furthermore, the visual image is enhanced using a predetermined image enhancement network, including the following steps: The visual image is decomposed into a low-frequency sub-band and a high-frequency sub-band using discrete wavelet transform; Adaptively fuse the transmittance map in the physical prior knowledge graph with the low-frequency sub-band to correct global illumination and color; The high-frequency subband is filtered using noise or texture prior information in the physical prior knowledge graph to suppress disturbance noise; Intermediate features are obtained by reconstructing the low-frequency and high-frequency subbands using inverse wavelet transform; Dynamic detail reconstruction and dynamic color correction are performed in parallel on the intermediate features; The dynamic detail reconstruction employs deformable convolution, which dynamically adjusts the sampling position based on local contrast or edge information in the physical prior knowledge graph to enhance the texture and edges in the image. The dynamic color correction is based on the color prior information in the physical prior knowledge graph and performs an affine transformation on the color distribution. The results of dynamic detail reconstruction and dynamic color correction are fused through a gated cross-attention mechanism to output a preliminary enhanced image.

[0008] Furthermore, the image enhancement network updates its parameters using a joint loss function, which is calculated using the following formula: , In the formula, The total loss of the joint function; Pixel reconstruction loss measures the absolute difference between the generated image and the reference image at the pixel level. These are the corresponding pixel reconstruction loss weight coefficients; To measure the perceptual loss of a VGG-19 pre-trained network, the difference between the generated image and the reference image in the intermediate layer feature maps of the network is calculated to measure their similarity at a high semantic level. For the corresponding sensing loss weighting coefficient; This is the physical prior consistency loss, used to ensure that the physical parameters of the generated image remain consistent with those of the reference image. This refers to the weighting coefficient of the corresponding physical prior consistency loss; This is a color consistency loss used to suppress color bias and chromaticity shift in the generated image. This represents the corresponding color consistency loss weighting coefficient.

[0009] Furthermore, the correction of each frame image using the unscented Kalman filtering method includes: Initialize the state vector and covariance; The Sigma point set is generated by unscented transformation, and prediction is performed according to the state transition equation. The mean and covariance of the predicted state are calculated. Calculate the Kalman gain and update the state estimate based on the residuals between the observed and predicted observed values; The displacement of feature points and the change in contrast in the updated state vector are mapped to the entire image plane through interpolation. This performs pixel-level spatial transformation and contrast modulation on the current frame image, and outputs the corrected image.

[0010] Furthermore, the cross-media operation scenario is configured as a backhoe dredger excavation and unloading operation scenario.

[0011] Furthermore, the visual image is configured as a binocular image.

[0012] In a second aspect, the present invention provides a cross-media operation visual image processing apparatus, including a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the cross-media operation visual image processing method as described above.

[0013] The above technical solution has the following technical effects: This invention acquires visual image sequences of air and water media during cross-media operations and generates corresponding physical prior knowledge graphs for each medium to characterize the perturbation distribution characteristics under different media. Based on this, key feature parameters of each frame in the image sequence are used as state vectors, and the theoretical feature change rate calculated from the physical prior knowledge graph is used as control input. State relationships between adjacent frames are established through state transition equations, thus fully exploring the temporal correlation of the cross-media image sequence. Furthermore, unscented Kalman filtering is used to correct each frame, adaptively correcting abrupt changes in adjacent frame images caused by time-varying and non-stationary perturbations during the operation. This effectively suppresses inter-frame flicker and jitter, significantly improving the temporal stability and coherence of the image sequence, ensuring that operators or automated systems can accurately track the operation target, and guaranteeing the continuity and safety of cross-media operations. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a cross-media visual image processing method according to an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of the structure of a cross-media visual image processing device according to an embodiment of the present invention. Detailed Implementation

[0016] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments and to explain the operating principles of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. Components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0017] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0018] Example 1: This embodiment uses a general cross-media operation scenario as the application object to describe the specific implementation of the present invention.

[0019] Figure 1 This is a schematic flowchart of a cross-media visual image processing method according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Acquire visual image sequences of various environmental media in a cross-media operation scenario. These environmental media include air and water. The visual image sequences can be acquired using image acquisition devices deployed in different media; for example, a visible light camera can be deployed in the air, and an underwater camera in the water. LED lighting and shock-resistant bases can be configured. The acquired image sequences are hardware-synchronized using a GPS / IMU integrated navigation system as the time reference.

[0020] Optionally, a random sample consensus algorithm and constrained least squares filtering are used to perform spatiotemporal registration of visual images from different environmental media. Specifically, the random sample consensus algorithm is used to remove anomalous matching feature point pairs, and constrained least squares filtering is used to enhance the spatial domain of the image pairs, smooth out minor perturbations, and strengthen edges, resulting in a registered visual image sequence.

[0021] For each frame in the visual image sequence, physical prior knowledge graphs corresponding to the air medium and the water medium are generated based on the physical degradation model corresponding to the environmental medium.

[0022] For the air medium, the visual image is decomposed into sharp scene, atmospheric light, and transmittance, and rain line masking and illumination inhomogeneity models are introduced. Specifically, this is based on an atmospheric scattering model. Construct a joint degradation model of rain, fog, and light over water, in which... Degraded images on water. A clear scene on the water awaiting restoration. This is a water transmittance diagram. For atmospheric light. Introduce a rain line mask. and uneven lighting model By analyzing the gradient distribution and color constancy of the decomposed image, the fog density, rain line density, and illumination intensity of the current frame are estimated. Specifically, the fog density is estimated by analyzing the image gradient distribution and color constancy; the main direction of the rain lines is detected using a Gabor filter to generate the rain line distribution probability; the low-frequency components of the illumination components are extracted and combined with an illumination non-uniformity model to generate an illumination non-uniformity map. Finally, a priori knowledge graph of water physics is generated. It includes a transmittance map Atmospheric light Rain line distribution probability and light unevenness map .

[0023] For the aquatic medium, a water attenuation coefficient and an equivalent backscattering coefficient are defined, and a time-varying sediment concentration function caused by underwater operations is introduced. Specifically, an underwater imaging physical model is established. Considering backscattering and forward scattering. In the formula, This is an underwater degraded image; A clear underwater scene awaiting restoration; This is an underwater transmittance diagram; The background light vector is defined. The water attenuation coefficient is defined. and equivalent backscattering coefficient And introduce a time-varying sediment concentration function caused by underwater operations. Its modulation relationship with attenuation and scattering is modeled as follows: , , In the formula, Indicates the time elapsed during the underwater operation. This represents the water decay coefficient at time t. Let represent the equivalent backscattering coefficient at time t. Time-varying volumetric sediment concentration caused by underwater operations The baseline total attenuation coefficient (m) of the water body during operational disturbance. -1 ), The corresponding reference equivalent backscattering coefficient, This is the linear modulation coefficient of the attenuation coefficient with respect to sediment concentration. This is the square modulation coefficient of the backscattering coefficient with respect to the sediment concentration.

[0024] The sediment concentration and water turbidity of the current frame are estimated by calculating the local contrast and dark channel variation rate of the visual image. Specifically, the sediment concentration and water turbidity of the current frame are estimated by calculating the standard deviation variation rate and dark channel variation rate within the local contrast window of the image. Based on the estimated... Calculate the transmittance map: , And estimate the background light vector d(x) represents the scene depth or medium propagation distance corresponding to pixel position x, and z represents the depth direction coordinate or propagation path coordinate in the image coordinate system. The final result is an underwater physics prior knowledge graph. It includes underwater transmittance maps. and background light vector .

[0025] Optionally, the visual image is enhanced using a predetermined image enhancement network. This image enhancement network includes the following steps: decomposing the visual image into low-frequency and high-frequency sub-bands using discrete wavelet transform; adaptively fusing the transmittance map from the physical prior knowledge graph with the low-frequency sub-band to correct global illumination and color; filtering the high-frequency sub-band to suppress perturbation noise using noise or texture prior information from the physical prior knowledge graph; reconstructing intermediate features from the low-frequency and high-frequency sub-bands using inverse wavelet transform; performing dynamic detail reconstruction and dynamic color correction in parallel on the intermediate features; wherein, dynamic detail reconstruction employs deformable convolution, dynamically adjusting the sampling position based on local contrast or edge information from the physical prior knowledge graph to enhance texture and edges in the image; dynamic color correction performs affine transformation on the color distribution based on color prior information from the physical prior knowledge graph; and fusing the results of dynamic detail reconstruction and dynamic color correction through a gated cross-attention mechanism to output a preliminary enhanced image.

[0026] Image enhancement networks use a joint loss function to update network parameters. The joint loss function is calculated using the following formula: , In the formula, The total loss of the joint function; Pixel reconstruction loss measures the absolute difference between the generated image and the reference image at the pixel level. These are the corresponding pixel reconstruction loss weight coefficients; To measure the perceptual loss of a VGG-19 pre-trained network, the difference between the generated image and the reference image in the intermediate layer feature maps of the network is calculated to measure their similarity at a high semantic level. For the corresponding sensing loss weighting coefficient; This is the physical prior consistency loss, used to ensure that the physical parameters of the generated image remain consistent with those of the reference image. This refers to the weighting coefficient of the corresponding physical prior consistency loss; This is a color consistency loss used to suppress color bias and chromaticity shift in the generated image. This represents the corresponding color consistency loss weighting coefficient.

[0027] The key feature parameters of each frame in the visual image sequence are used as state vectors, and the theoretical feature change rate of the key feature parameters calculated from the physical prior knowledge graph is used as control input. State relationships between adjacent frames are established through state transition equations. In one specific implementation, the key feature parameters include edge intensity, local contrast, and color consistency.

[0028] Specifically, the state vector is constructed using the key feature parameters of the k-th frame image. The control input is composed of the rate of change of theoretical characteristics calculated from the physical prior knowledge graph. The system state transition equation and observation equation are respectively , , in, Let k be the state vector of the k-th frame. For the first The state vector of a frame. This represents the state transition function, which indicates the mapping relationship from the state of the previous frame to the state of the current frame. This represents the observation value in the k-th frame; and These are process noise and observation noise, respectively, and their covariance is dynamically determined by the intensity of the disturbance in the physical prior. The observation function is used to represent the state vector of the k-th frame. In observation noise The effect is mapped to the predicted observation.

[0029] Based on the state transition equation, each frame of the image is corrected using an unscented Kalman filter to output the final visual image sequence. The specific steps include: Initialize the state vector and covariance; The Sigma point set is generated through unscented transformation, and each state vector generates a total of [number missing]. There are Sigma points, where L is the dimension of the state vector; Prediction is performed based on the state transition equation, and the predicted state mean and predicted covariance are calculated. Calculate the Kalman gain and update the state estimate based on the residuals between the observed and predicted observed values; The displacement of feature points and the change in contrast in the updated state vector are mapped to the entire image plane through interpolation. This performs pixel-level spatial transformation and contrast modulation on the current frame image, and outputs the corrected image.

[0030] Example 2 This embodiment uses the cross-media operation of a backhoe dredger as a specific application scenario to further describe the invention. During underwater dredging and unloading of silt onto side or stern barges, the working environment of a backhoe dredger is extremely complex: the underwater working surface is subject to strong disturbances such as selective absorption and scattering of light by the water, and rapid deterioration of water quality caused by siltation; the surface portion is affected by rain, fog, changes in lighting, and image jitter caused by hull vibration. In one specific implementation, the visual image is configured as a binocular image; the cross-media operation scenario is configured as a backhoe dredger dredging and unloading operation scenario.

[0031] A surface-mounted binocular camera with a resolution of 1920×1080, a focal length of 12mm, and calibrated extrinsic parameters is installed on the top of the dredger's cab and mounted on a stabilized gimbal. This camera is used to acquire images of the dredger's hatches and overall pose. An underwater binocular camera with a resolution of 1920×1080, equipped with an LED light source, an IP68 protection rating, and calibrated extrinsic parameters is installed near the bucket of the underwater excavator arm. This camera is used to acquire images of the excavation face. All images are hardware-synchronized using a GPS / IMU integrated navigation system as the time reference.

[0032] A random sample consensus algorithm and constrained least squares filtering were used to perform spatiotemporal registration of visual images from different environmental media. Specifically, the random sample consensus algorithm was used for coarse feature point matching to eliminate mismatches; constrained least squares filtering was used for image enhancement to adjust the dynamic range, resulting in registered surface and underwater stereo images.

[0033] For each frame in a visual image sequence, physical prior knowledge graphs corresponding to air and water media are generated based on the physical degradation model corresponding to the environmental medium.

[0034] For the air medium, the visual image is decomposed into clear scene, atmospheric light, and transmittance, and a rain line mask and an illumination non-uniformity model are introduced. Specifically, a joint degradation model of rain, fog, and illumination on water is constructed based on an atmospheric scattering model. Fog concentration is determined by analyzing the local entropy of the image, a rain line mask is generated by detecting the main direction of rain lines using a Gabor filter, and an illumination non-uniformity map is generated by extracting the low-frequency components of the illumination component. Finally, a priori knowledge graph of water physics is generated. It includes a transmittance map Atmospheric light Rain line distribution probability and light unevenness map .

[0035] For the aquatic medium, a water attenuation coefficient and an equivalent backscattering coefficient are defined, and a time-varying sediment concentration function caused by underwater operations is introduced. Specifically, an underwater imaging physical model is established, considering both backscattering and forward scattering. By analyzing the variance of grayscale value changes across multiple consecutive frames, sediment concentration is estimated in real time. If the pixel intensity in a local dark area of ​​an image frame fluctuates rapidly over time, it is determined to be due to an increase in sediment concentration, and the rate of change is calculated. Based on underwater optical characteristics, the attenuation coefficient for each wavelength is calculated using formulas. and scattering coefficient Based on the prior knowledge of the dark passage, and utilizing the estimated... By resizing the local window, we obtain the corrected transmittance map and background light. This information is then integrated into an underwater physics prior knowledge graph. The data in the image is a multi-channel feature map, which includes a transmittance map and a background light vector.

[0036] Image enhancement is performed on the visual image using a predetermined image enhancement network. The underwater image and its physical prior knowledge graph are input into a physical prior-guided binocular image enhancement network. Specifically, this includes the following steps: decomposing the visual image into low-frequency and high-frequency sub-bands using discrete wavelet transform; performing a second-level discrete wavelet decomposition on the input image, such as using a Haar wavelet basis; and adaptively fusing the transmittance map from the physical prior knowledge graph with the low-frequency sub-band to correct global illumination and color. Specifically, the transmittance map from the physical prior is downsampled and then combined with the low-frequency sub-band... Perform weighted fusion: , This refers to the low-frequency subband of the original image; This is the corrected low-frequency sub-band; For element-wise multiplication, After downsampling, the transmittance map in the physical prior knowledge graph is compared with the low-frequency subband. Transmittance map results with the same spatial dimensions. Noise or texture prior information from the physical prior knowledge graph is used to filter the high-frequency subbands to suppress perturbation noise. Specifically, for the high-frequency subbands, a noise confidence mask is calculated. Then we can get , This represents the high-frequency subband image obtained from wavelet decomposition. This represents the updated high-frequency subband image obtained after suppressing perturbations using a noise confidence mask. The noise prior value corresponding to the high-frequency sub-band in the physical prior knowledge graph is represented by this value, which is usually normalized to [0, 1]. A larger value indicates stronger noise or disturbance at that location, while a smaller value indicates a cleaner signal. Intermediate features are obtained by reconstructing the low-frequency and high-frequency sub-bands using inverse wavelet transform. .

[0037] Dynamic detail reconstruction and dynamic color correction are performed in parallel on intermediate features. The dynamic detail reconstruction employs deformable convolution, dynamically adjusting the sampling position based on local contrast or edge information in the physical prior knowledge graph to enhance texture and edges in the image. Specifically, a local contrast gradient map is calculated from the physical prior. The offset prediction network consists of two convolutional layers (3×3, outputting 2N channels, N=9). The output of the deformable convolution is passed through residual blocks (two 3×3 convolutions) to obtain the dynamic detail reconstruction result. The dynamic color correction is based on color prior information in the physical prior knowledge graph, performing an affine transformation on the color distribution. Specifically, the color affine matrix... The basis vectors are derived from intermediate features and illumination unevenness maps by a small network (two 1×1 convolutions). The regression was obtained, and a pixel-by-pixel transformation was applied. In the formula, This represents the color vector of the output pixel. This represents the color vector of the input pixel. The color affine transformation matrix is ​​derived from the physical priors of atmospheric light. Uneven illumination diagram Common modulation, output Number of feature channels and Maintain consistency.

[0038] The results of dynamic detail reconstruction and dynamic color correction are fused using a gated cross-attention mechanism. Specifically, global average pooling, max pooling, and standard deviation pooling of the two branch features are calculated, concatenated, and fed into an MLP (32-dimensional hidden layer, 32 output channels). A sigmoid function is then used to obtain the gated signal, which is then processed according to... To merge, The output feature map after fusion. To reconstruct the feature map output by the branch for dynamic details, The feature map output by the dynamic color correction branch. It is the Sigmoid activation function. The gated signal calculated from the color correction branch features is used as the fusion weight for the detail branch features after passing through a sigmoid function. The gate signal calculated from the branch features reconstructed from details is used as the fusion weight for the color branch features after passing through a sigmoid function. Finally, a preliminary enhanced image is output after a 3×3 convolution and residual connection. .

[0039] The image augmentation network uses a joint loss function to update network parameters. A paired dataset of 5000 frames of actual cross-media video was used, and clear reference images were obtained through a combination of traditional augmentation and manual correction. The Adam optimizer was used with an initial learning rate of 1e-4, decaying by 0.5 times every 50 epochs, a batch size of 8, and a total training duration of 200 epochs. The weight coefficients were set based on the importance of each task and experience. , .

[0040] The key feature parameters of each frame in the visual image sequence are used as state vectors, and the theoretical feature change rate of the key feature parameters calculated by the physical prior knowledge graph is used as control input. A state transition equation is used to establish the state association between adjacent frames. The key feature parameters include edge intensity, local contrast, and color consistency.

[0041] Specifically, the state vector is constructed using the key feature parameters of the k-th frame image. The control input is composed of the rate of change of theoretical characteristics calculated from the physical prior knowledge graph. The system state transition equation and observation equation are respectively , , in, and These are process noise and observation noise, respectively, and their covariance is dynamically determined by the intensity of the disturbance in the physical prior.

[0042] Based on the state transition equation, each frame of the image is corrected using an unscented Kalman filter to output the final visual image sequence. This initially enhanced image sequence... Input an unscented Kalman filter. Select N=100 feature points and use Shi-Tomasi corner detection. The state vector dimension... Specifically, it includes the following steps: Initialize state vector Covariance ; Generate Sigma point sets through unscented transformation, with each state vector generating a total of Sigma points; predict based on the state transition equation, and calculate the mean of the predicted state. and predicted covariance ; Calculate Kalman gain The state estimate is updated based on the residuals between the observed and predicted observed values. In the formula, This is the state estimate after the update in the k-th frame; The mean of the predicted state in the k-th frame; The Kalman gain for the k-th frame; Let be the predicted observation value for the k-th frame; The observed values ​​are for the k-th frame; the displacement and contrast change of the feature points in the updated state vector are mapped to the entire image plane through thin-plate spline interpolation, for the current frame image. Perform pixel-level spatial transformation and contrast modulation to output the corrected image. .

[0043] The parameters for the unscented Kalman filter are configured as follows: Unscented transform parameters are set to... , , ; Principal scale parameter For scale parameters, Subscale parameter; process noise covariance Set as a diagonal matrix, its diagonal elements and and Proportional; This represents the rate of change in sediment concentration. The rate of change of light intensity; observation noise covariance. Let I represent the identity matrix that matches the dimension of the state vector. The state transition model is as follows: , Indicates the first i The predicted state of the nth Sigma point, based on the previous frame. i Sigma point status and control input get; It is the identity matrix. The gain matrix controls the input. .

[0044] Finally, the underwater image, after being corrected by unscented Kalman filtering, is displayed in real time on the monitor of the control console in the cab and input into the downstream dredging trajectory planning system; the above-water enhanced image is input into the mud barge pose estimation system for automatic unloading positioning.

[0045] Experimental results show that this method improves the PSNR and SSIM of enhanced images by an average of more than 15% and the feature point matching success rate by more than 30% under conditions such as turbid water, strong light interference, and rainy / foggy weather.

[0046] Example 3: Figure 2 This is a schematic diagram of the structure of a cross-media visual image processing device according to an embodiment of the present invention, as shown below. Figure 2 As shown, the device includes a processor 201, a memory 202, a bus 203, and a computer program stored in the memory 202 and executable on the processor 201. The processor 201 includes one or more processing cores. The memory 202 is connected to the processor 201 via the bus 203. The memory 202 is used to store program instructions. When the processor executes the computer program, it implements the steps in the above-described method embodiment of Embodiment 1 of the present invention.

[0047] Furthermore, as an executable solution, the cross-media visual image processing device can be a computer unit, which can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer unit may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above-described computer unit structure is merely an example and does not constitute a limitation on the computer unit. It may include more or fewer components, or combine certain components, or use different components. For example, the computer unit may also include input / output devices, network access devices, buses, etc., and this embodiment of the invention does not limit this.

[0048] Furthermore, as an executable solution, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, connecting various parts of the entire computer unit via various interfaces and lines.

[0049] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0050] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A cross-media visual image processing method, characterized in that, Includes the following steps: Acquire visual image sequences of various environmental media in a cross-media operation scenario; the environmental media include air and water. For each frame of the visual image sequence, physical prior knowledge graphs corresponding to the air medium and the water medium are generated based on the physical degradation model corresponding to the environmental medium. The key feature parameters of each frame in the visual image sequence are used as state vectors, and the theoretical feature change rate of the key feature parameters calculated by the physical prior knowledge graph is used as control input. The state association between adjacent frames is established through the state transition equation. The key feature parameters include edge intensity, local contrast and color consistency. Based on the state transition equation, each frame of the image is corrected using an unscented Kalman filter to output the final visual image sequence.

2. The cross-media visual image processing method according to claim 1, characterized in that, The random sampling consensus algorithm and constrained least squares filtering were also used to perform spatiotemporal registration of visual images in different environmental media.

3. The cross-media visual image processing method according to claim 1, characterized in that, The physical prior knowledge graph includes: For the air medium, the visual image is decomposed into clear scene, atmospheric light, and transmittance, and a rain line mask and an illumination non-uniformity model are introduced. By analyzing the gradient distribution and color constancy after image decomposition, the fog concentration, rain line density, and illumination intensity of the current frame are estimated, and a water physics prior knowledge graph containing a transmittance map, atmospheric light, rain line distribution probability, and illumination non-uniformity map is generated. For the water medium, a water attenuation coefficient and an equivalent backscattering coefficient are defined, and a time-varying sediment concentration function caused by underwater operations is introduced. By calculating the local contrast and dark channel change rate of its visual image, the sediment concentration and water turbidity of the current frame are estimated, and an underwater physical prior knowledge graph containing a transmittance map and a background light vector is generated.

4. The cross-media visual image processing method according to claim 3, characterized in that, The visual images are also enhanced using a predetermined image enhancement network, including the following steps: The visual image is decomposed into a low-frequency sub-band and a high-frequency sub-band using discrete wavelet transform; Adaptively fuse the transmittance map in the physical prior knowledge graph with the low-frequency sub-band to correct global illumination and color; The high-frequency subband is filtered using noise or texture prior information in the physical prior knowledge graph to suppress disturbance noise; Intermediate features are obtained by reconstructing the low-frequency and high-frequency subbands using inverse wavelet transform; Dynamic detail reconstruction and dynamic color correction are performed in parallel on the intermediate features; The dynamic detail reconstruction employs deformable convolution, which dynamically adjusts the sampling position based on local contrast or edge information in the physical prior knowledge graph to enhance the texture and edges in the image. The dynamic color correction is based on the color prior information in the physical prior knowledge graph and performs an affine transformation on the color distribution. The results of dynamic detail reconstruction and dynamic color correction are fused through a gated cross-attention mechanism to output a preliminary enhanced image.

5. The cross-media visual image processing method according to claim 4, characterized in that, The image enhancement network updates its parameters using a joint loss function, which is calculated using the following formula: ， In the formula, The total loss of the joint function; Pixel reconstruction loss measures the absolute difference between the generated image and the reference image at the pixel level. These are the corresponding pixel reconstruction loss weight coefficients; To measure the perceptual loss of a VGG-19 pre-trained network, the difference between the generated image and the reference image in the intermediate layer feature maps of the network is calculated to measure their similarity at a high semantic level. For the corresponding sensing loss weighting coefficient; This is the physical prior consistency loss, used to ensure that the physical parameters of the generated image remain consistent with those of the reference image. This refers to the weighting coefficient of the corresponding physical prior consistency loss; This is a color consistency loss used to suppress color bias and chromaticity shift in the generated image. This represents the corresponding color consistency loss weighting coefficient.

6. The cross-media visual image processing method according to claim 1, characterized in that, The correction of each frame of the image using the unscented Kalman filtering method includes: Initialize the state vector and covariance; The Sigma point set is generated by unscented transformation, and prediction is performed according to the state transition equation. The mean and covariance of the predicted state are calculated. Calculate the Kalman gain and update the state estimate based on the residuals between the observed and predicted observed values; The displacement of feature points and the change in contrast in the updated state vector are mapped to the entire image plane through interpolation. Pixel-level spatial transformation and contrast modulation are then performed on the current frame image, and the corrected image is output.

7. The cross-media visual image processing method according to claim 1, characterized in that, The cross-media operation scenario refers to the excavation and unloading operation scenario of a backhoe dredger.

8. The cross-media visual image processing method according to claim 1, characterized in that, The visual image is configured as a binocular image.

9. A cross-media visual image processing device, characterized in that, It includes a memory and a processor, the memory storing at least one program, the at least one program being executed by the processor to implement the cross-media operation visual image processing method as described in any one of claims 1-8.

Images