Video image segmentation method

Abstract

The invention relates to the field of image processing, and discloses a video image segmentation method which is used for improving the precision, robustness and real-time performance of video image segmentation in a complex dynamic scene. The video image segmentation method comprises the following steps: generating an entropy generation rate map through weighted combination of light flow divergence and rotation, and quantifying motion irreversibility; a double-virtual-form prime concentration field is constructed, next-frame texture prediction is realized through iterative evolution, and the dynamic background adaptability is enhanced; fusing the projection entropy generation rate graph and the enhanced texture residual graph, dynamically distributing motion and appearance weights, and generating a high-precision boundary response graph; and extracting and persistent filtering are carried out, topological consistency maintenance of segmentation masks is realized, multi-scale collaborative segmentation and dynamic computing resource scheduling are supported, and efficiency and precision are balanced. The segmentation precision of the method is obviously superior to that of a traditional method in complex scenes such as illumination variation and rapid motion, and the method is suitable for the fields with high real-time requirements such as monitoring, medical treatment and automatic driving.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a video image segmentation method. BACKGROUND

[0002] With the rapid development of video surveillance, intelligent security, autonomous driving, medical image analysis and other fields, video image segmentation technology has become a key research direction in the field of computer vision. The core goal is to accurately identify and extract dynamic targets or regions of interest from continuous video frames, providing a foundation for subsequent scene understanding, behavior analysis, target tracking and other tasks. However, video image segmentation faces multiple challenges: on the one hand, video scenes have high dynamicity, target motion, lighting changes, occlusions, background disturbances and other factors making it difficult for traditional static image-based segmentation methods to be directly applicable; on the other hand, the spatiotemporal continuity of video data requires segmentation algorithms to be efficient, robust and real-time to adapt to real-time processing needs in complex scenarios.

[0003] In the prior art, the mainstream video image segmentation methods are mainly divided into two categories:

[0004] Motion analysis-based methods: extract motion information through optical flow estimation, motion vector field analysis and other means, and identify motion regions combined with threshold segmentation or clustering algorithms. This type of method is sensitive to dynamic targets, but is easily affected by lighting changes, noise interference and difficult to distinguish between real motion and background disturbances.

[0005] Appearance modeling-based methods: achieve segmentation by constructing target appearance models or deep learning models. This type of method performs well in static scenes, but performance declines in dynamic scenes due to target deformation, occlusion or increased background complexity.

[0006] Therefore, we propose a video image segmentation method to solve the above problems. SUMMARY

[0007] The present application provides a video image segmentation method for improving the accuracy, robustness and real-time performance of video image segmentation in complex dynamic scenes.

[0008] The first aspect of the present application provides a video image segmentation method, comprising: performing optical flow field calculation on a current frame and a next frame in continuous video frames to generate a motion vector field, calculating a weighted combination of optical flow divergence and curl based on the motion vector field to generate an entropy generation rate map; mapping the current frame gray information into a first virtual morphogen concentration field, and mapping the gradient information into a second virtual morphogen concentration field, iteratively evolving the double morphogen field, and outputting a next frame texture prediction map; projecting the entropy generation rate map to the next frame coordinate space through optical flow to generate a projected entropy generation rate map, calculating a pixel-level absolute residual error map of the texture prediction map and the actual next frame, and fusing the projected entropy generation rate map and the residual error map to generate a boundary response map; distorting the last frame segmentation mask to the current frame through optical flow to generate an initial segmentation mask, performing adaptive threshold segmentation based on the boundary response map to identify a change area binary mask; within the change area mask range, performing region growing algorithm with the boundary response map as the growth constraint, fusing the growth result and the initial segmentation mask, and outputting a final segmentation mask through topological skeleton extraction and persistence filtering.

[0009] Optionally, in the first implementation manner of the first aspect of the present application, the method comprises: performing non-parametric optical flow estimation on the current frame and the next frame in the continuous video frames to obtain an original two-dimensional motion vector field; dividing the maximum motion module length by the module length of the full-field motion vector to generate a normalized optical flow field, extracting an optical flow divergence scalar field from the normalized optical flow field through a central difference operator, and extracting an optical flow curl scalar field from the normalized optical flow field; taking absolute values of the optical flow divergence value and the curl value respectively, performing linear superposition according to a preset weight to generate an initial entropy generation rate map; buffering the initial entropy generation rate map to a projection queue, and outputting the initial entropy generation rate map as the entropy generation rate map.

[0010] Optionally, in the second implementation manner of the first aspect of the present application, the method comprises: mapping the current frame gray value to a first preset interval to generate an activated morphogen concentration field, extracting the current frame gradient amplitude to map to a second preset interval to generate an inhibited morphogen concentration field, and combining the two to form a double-channel space-time evolution field; configuring an isotropic diffusion kernel for the activated morphogen, and configuring an inhibition dominant diffusion kernel for the inhibited morphogen; alternately performing a local reaction stage and a global diffusion stage on an artificial time scale, calculating the morphogen field Euclidean distance change rate between continuous iterations, and terminating the iteration when the change rate is lower than an adaptive threshold; extracting the spatial mode of the stable activated morphogen field, and mapping the amplitude distribution thereof to a next frame texture prediction map.

[0011] Optionally, in the third implementation manner of the first aspect of the present application, the morphogen field Euclidean distance change rate is set as R k :

[0012]

[0013] wherein, H is the image height, the vertical pixel number of the video frame; W is the image width, the horizontal pixel number of the video frame; A k is the current iteration active concentration value; A k-1 is the previous iteration active concentration value; I k is the current iteration suppression concentration value; I k-1 is the previous iteration suppression concentration value; i and j are pixel coordinates, position indexes in the image.

[0014] Optionally, in the fourth implementation manner of the first aspect of the present application, the method comprises: based on the normalized optical flow field, performing forward transmission of pixel values in the entropy generation rate map along the direction of the motion vector, performing vorticity conservation filling on a coordinate conflict area caused by occlusion, to obtain a projected entropy generation rate map; calculating, on the texture prediction map and the actual next frame gray scale map, a three-channel difference of brightness-contrast-structure in a local window, and fusing the three-channel difference to generate an enhanced texture residual map; establishing a dynamic weight mapping function of the projected entropy generation rate map and the enhanced texture residual map, increasing a motion weight in a high entropy generation rate area and increasing an appearance weight in a high texture residual area, to obtain a pixel-level fusion weight field; performing weighted superposition on the projected entropy generation rate map and the enhanced texture residual map according to the pixel-level fusion weight field, and performing double-scale Gaussian difference enhancement on a superposition result, to obtain a multi-physical field boundary response map; marking extreme points with a connected domain greater than a threshold in the multi-physical field boundary response map as topological key points, and outputting a boundary response map.

[0015] Optionally, in the fifth implementation manner of the first aspect of the present application, the method comprises: based on the normalized optical flow field, performing sub-pixel level bilinear motion transmission on a boundary point set of the previous frame segmentation mask, performing vorticity kernel maintaining interpolation on a hole area caused by motion occlusion, to obtain an initial segmentation mask; defining a dynamically calculated window on the boundary response map, performing double-Gaussian kernel difference statistics on pixel values in each window, automatically solving an optimal segmentation threshold, to obtain a threshold mapping map; performing exclusive OR logical operation on the initial segmentation mask and the threshold mapping map, extracting an area with a connected domain area greater than motion consistency in an operation result, to obtain a candidate deformation area mask; calculating a standard deviation of the optical flow curl in the candidate deformation area, filtering an area with a standard deviation lower than a vorticity conservation threshold, to obtain a physically verified change area mask; extracting a change area binary mask on a boundary of the change area mask.

[0016] Optionally, in the sixth implementation form of the first aspect of the present application, the method further comprises: performing geodesic growth from the topological key point in the range of the change region mask, and obtaining a topologically connected growth mask, by taking the gradient amplitude of the boundary response map as a growth potential barrier; performing a morphological closing operation on the topologically connected growth mask and the initial segmentation mask to obtain a candidate segmentation mask; performing curvature-aware morphological thinning on the candidate segmentation mask to obtain a topological skeleton map; calculating the persistent homology barcode of the topological skeleton map, filtering out homology features with a life cycle less than a motion consistency threshold to obtain a simplified skeleton map; and taking the simplified skeleton map as a center axis to reconstruct a closed boundary to obtain a final segmentation mask.

[0017] Optionally, in the seventh implementation form of the first aspect of the present application, the method further comprises dynamic calculation resource scheduling: monitoring the peak density distribution of the entropy generation rate map and the gradient entropy value of the boundary response map in real time, dynamically allocating calculation resources of each step according to the monitoring data to obtain an adaptive calculation strategy table; extracting the Betti number sequence of the final segmentation mask of the current frame and the previous three frames, detecting a topological mutation event through algebraic topological homology group analysis to obtain a topological continuity correction signal; and performing segmentation in three scales on the same frame in parallel, and fusing topologically consistent regions of the multi-scale masks based on the vorticity conservation criterion to obtain a final segmentation mask verified across scales.

[0018] The mechanism of the present application is as follows: the video segmentation is converted into a spatiotemporal evolution prediction problem of a physical system, the entropy generation rate is used to represent motion mutation, the Turing pattern is used to drive texture evolution, and the Betti sequence is used to guarantee topological continuity, thereby forming a first principle solution that is completely independent of machine learning;

[0019] Advantages: quantifying the irreversibility in the motion process, effectively distinguishing real motion from background disturbance, improving the modeling capability for non-rigid motion and complex interactive scenes, and reducing false detection;

[0020] By borrowing the reaction-diffusion model in physical chemistry to simulate the biological morphogenesis process, texture prediction based on physical laws is realized instead of traditional statistical learning or deep learning, the adaptability to dynamic backgrounds is enhanced, and the accuracy of texture residual calculation is improved;

[0021] Through dynamic weight distribution, information complementation is achieved, the robustness of boundary detection is improved, and high-precision boundary positioning can still be maintained in fast motion and low-contrast scenes;

[0022] The persistent homology theory in algebraic topology is introduced, noise is filtered out by calculating the life cycle of the topological skeleton, stable structures are retained, a geometrically closed and topologically correct segmentation mask is output, and the segmentation mask is suitable for medical image analysis, automatic driving and other scenes with high topological accuracy requirements;

[0023] Break through the limitation of fixed scale segmentation, improve the detection ability of small targets and weak motion through cross-scale verification; through dynamic resource scheduling, balance efficiency and accuracy, and still run in real time on resource limited devices while maintaining high accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 An embodiment of the video image segmentation method in the embodiment of the application is shown in the figure;

[0025] Figure 2 Another embodiment of the video image segmentation method in the embodiment of the application is shown in the figure;

[0026] Figure 3 An embodiment of the video image segmentation device in the embodiment of the application is shown in the figure;

[0027] Figure 4 An embodiment of the video image segmentation device in the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0028] The embodiment of the application provides a video image segmentation method for improving the accuracy, robustness and real-time performance of video image segmentation in a complex dynamic scene. The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the term "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] For ease of understanding, the specific flow of the embodiment of the application is described below, please refer to Figure 1 An embodiment of the video image segmentation method in the embodiment of the application includes:

[0030] 101, motion dynamics feature extraction step: optical flow field calculation is performed on the current frame and the next frame in the continuous video frame, motion vector field is generated, and the weighted combination of optical flow divergence and curl is calculated based on the motion vector field to generate an entropy generation rate graph representing the irreversibility of motion;

[0031] It can be understood that the execution subject of the present application can be a video image segmentation device, and can also be a terminal or a server, and the specific implementation is not limited herein. The server is taken as an example for description of the embodiments of the present application.

[0032] It should be noted that the following takes the monitoring video of pedestrians crossing the road as an example.

[0033] Optical flow field calculation, input two consecutive frames (1920x1080 resolution, 30fps): current frame: pedestrian is located on the right side of the screen, and the vehicle drives in from the left side; next frame: pedestrian moves 3 meters to the left, and the vehicle moves 10 meters forward; the Farneback dense optical flow algorithm (parameters: pyramid scaling 0.5, 3 layers, window 15) is used to calculate the motion vector field. The output contains 1,036,800 vectors (1 vector per pixel), and the vector direction reflects the motion angle (0°-360°), and the length reflects the displacement (0-25 pixels).

[0034] Normalized motion vector field, polar coordinate normalization is performed on the vector field: direction normalization: the angle is mapped to [0, 1] (0°→0.0, 360°→1.0); speed normalization: the displacement is linearly mapped to [0, 1] (0 pixels→0.0, maximum displacement 25 pixels→1.0); note: after normalization, the vector value of the pedestrian area is approximately [0.35, 0.6], the vehicle area is approximately [0.72, 0.8], and the background is approximately 0.

[0035] Divergence and curl calculation, differential operation based on normalized vector field:

[0036] Divergence (expansion and contraction): the divergence value of the vehicle straight motion area is +0.83 (strong positive); the divergence value of the pedestrian leg crossing area is-0.15 (weak negative);

[0037] Curl (rotation component): the curl value of the pedestrian arm swinging area is +0.67 (high rotation); the curl value of the vehicle area is 0.05 (close to zero);

[0038] Calculation formula:

[0039] Entropy generation rate map generation, weighted combination: entropy map = 0.6x|divergence|+0.4x|curl| (weights are optimized through experiments); vehicle straight motion area: entropy value ≈0.52 (high irreversible motion); pedestrian limb area: entropy value ≈0.38 (medium irreversible); ground shadow area: entropy value ≈0.05 (close to reversible noise); output 1080p grayscale image, brightness value is positively correlated with entropy value, effectively highlighting the continuous motion target. Data table:

[0040] Processing stage Key parameters Numerical values Explanation Optical flow calculation Pyramid scaling factor 0.5 Controlling multi-scale precision -- Pyramid layer number 3 Balancing speed and precision -- Window size 15 Neighborhood smoothness Normalization Maximum displacement reference 25 pixels Corresponding actual vehicle speed 60 km / h Entropy map generation Divergence weight (a) 0.6 Strengthening directional motion -- Curl weight (b) 0.4 Preserving rotational features

[0041] Note: The entropy production rate map effectively distinguishes real motion from transient noise (leaf waving) by quantifying the irreversibility of motion (vehicle keeps moving, pedestrian limb periodic motion), providing a physically meaningful motion feature for the subsequent steps.

[0042] 102. Texture co-evolution prediction step: map the current frame gray information to the first virtual morphogen concentration field, and the gradient information to the second virtual morphogen concentration field, iteratively evolve the double morphogen field by discretizing the reaction-diffusion equation, and output the next frame texture prediction map;

[0043] It should be noted that the following is for the monitoring video scene of pedestrians crossing the road (1920x1080 resolution, 30fps):

[0044] Input data preparation, current frame: pedestrian located on the right side of the screen (gray value ≈ 80 / 255), vehicle driving from the left side (gray value ≈ 200 / 255), ground background (gray value ≈ 30 / 255). Gradient information calculation: calculate the gradient amplitude of the current frame by Sobel operator, pedestrian edge gradient ≈ 0.7 (normalized value), vehicle contour gradient ≈ 0.9, background noise gradient ≈ 0.05.

[0045] Double morphogen field construction: first morphogen field (gray field): normalize the current frame gray matrix to [0, 1] as the initial concentration field C g , pedestrian area C g ≈0.31, vehicle area C g ≈0.78. Second morphogen field (gradient field): normalize the gradient amplitude matrix to [0, 1] as the initial concentration field C d . Pedestrian edge C d ≈0.7, vehicle contour C d ≈0.9.

[0046] Reaction-diffusion equation discretization, iteratively evolve the double field by discretizing the reaction-diffusion equation: Diffusion term: D g = 0.2 (gray field diffusion coefficient), D d = 0.3 (gradient field diffusion coefficient), control the concentration smoothing degree. Reaction term: R = k1C g (1-C d )-k2C g C d , where k1 = 0.15 (gray dominant reaction), k2 = 0.25 (gradient inhibition reaction), simulate the dynamic competition of texture. Iteration parameters: time step Δt = 0.1 seconds, iterate 50 times (total time consumption 5 seconds).

[0047] Evolution process and output: initial state: double field separation is obvious, pedestrian area Cg Medium, C d High, vehicle region C g High, C d High. Evolution effect: Pedestrian region: Gradient field suppresses gray scale diffusion, edge texture remains sharp (C d > 0.6). Vehicle region: High gray scale field reacts strongly with high gradient field, generates metallic reflection texture (C g from 0.78 to 0.82). Ground background: Low gradient field allows gray scale diffusion, generates uniform noise texture (C g from 0.12 to 0.08). Output texture prediction map: Maps final C g field to gray scale map, compared with actual texture of next frame, average error 5% in pedestrian region, 8% in vehicle region, 3% in background. Parameter table:

[0048] Parameters Values Effects Gray scale field diffusion coefficient D g ]] 0.2 Controlling gray-scale information smoothness Gradient field diffusion coefficient D d ]]> 0.3 Maintaining edge sharpness Reaction rate k1 0.15 Enhancing texture of homogeneous regions Reaction rate k2 0.25 Inhibiting gray-scale diffusion in high gradient regions Iteration number N 50 Balancing calculation efficiency and evolutionary stability

[0049] Through the synergistic evolution of the dual-morphon field, the model simulates the dynamic changes of texture over time (local contraction of pedestrian clothing folds, fluctuation of vehicle surface reflection), providing a physically meaningful prediction basis for subsequent residual calculation.

[0050] 103. Dynamic boundary fusion detection step: Project the entropy production rate map to the coordinate space of the next frame by optical flow, generate a projected entropy production rate map, calculate the pixel-level absolute residual error map between the texture prediction map and the actual next frame, and fuse the projected entropy production rate map and the residual error map to generate a boundary response map.

[0051] It should be noted that the following is for the monitoring video scene of pedestrians crossing the road (1920x1080 resolution, 30fps), which details the implementation of the dynamic boundary fusion detection step:

[0052] Input data preparation: Entropy production rate map: output from step 101, representing the irreversibility of motion (vehicle straight motion area entropy value ≈ 0.85, pedestrian limb area ≈ 0.65, ground shadow area ≈ 0.05). Texture prediction map: output from step 102, generated based on the evolution of the dual-morphon field (pedestrian clothing texture prediction error ≈ 15%, vehicle metallic reflection texture error ≈ 8%). Actual next frame: real monitoring frame, containing a pedestrian moving 3 meters to the left and a vehicle advancing 10 meters.

[0053] Optical flow projection and residual calculation, projected entropy production rate map: based on Farneback dense optical flow field (parameters: pyramid scaling 0.5, 3 layers), map the current frame entropy map to the coordinate space of the next frame, the vehicle region has a motion displacement of 20 pixels, and the entropy value remains 0.85 after projection; the pedestrian leg has a swing displacement of 8 pixels, and the entropy value decreases slightly to 0.63.

[0054] Pixel-level residual map: compute the absolute residual between texture prediction and actual frame (residual = |predicted texture - actual texture|). Pedestrian clothing fold residual ~ 0.3 (normalized value), vehicle reflective zone residual ~ 0.15, ground shadow residual ~ 0.05.

[0055] Dual-map fusion and boundary response, weighted fusion: adopt linear fusion strategy: boundary response = 0.6 x projection entropy value + 0.4 x residual value. Vehicle region: 0.6 x 0.85 + 0.4 x 0.15 = 0.57 (high response, strong motion boundary); pedestrian region: 0.6 x 0.63 + 0.4 x 0.3 = 0.50 (medium response, limb dynamic boundary); ground shadow: 0.6 x 0.05 + 0.4 x 0.05 = 0.05 (low response, static noise).

[0056] Output boundary response map: generate 1080p grayscale image, brightness positively correlated with response value, vehicle contour clear (brightness >= 140 / 255), pedestrian edge continuous (brightness ~ 100 / 255), background noise suppressed (brightness <= 20 / 255). Effect comparison table:

[0057] Region Projection entropy value Residual value Boundary response value Vehicle region 0.85 0.15 0.57 Pedestrian limb 0.63 0.30 0.50 Ground shadow 0.05 0.05 0.05

[0058] By fusing motion irreversibility (entropy map) and texture evolution deviation (residual map), the real motion boundary and static interference are effectively distinguished, providing high-confidence edge features for subsequent mask update.

[0059] 104. Spatiotemporal continuity constraint step: warp the previous frame segmentation mask to the current frame by optical flow, generate the initial segmentation mask, perform adaptive threshold segmentation based on the boundary response map, identify the change region binary mask;

[0060] It should be noted that the following is for the monitoring video scene of pedestrians crossing the road (1920x1080 resolution, 30fps):

[0061] Input data preparation, previous frame segmentation mask: pedestrian mask region (about 15,000 pixels), vehicle mask region (about 8,000 pixels), background region static identification. Optical flow field: Farneback dense optical flow from step 101 (pedestrian displacement 10 pixels, vehicle displacement 25 pixels). Boundary response map: output from step 103 (vehicle contour response value ~ 0.57, pedestrian edge ~ 0.5, background noise < 0.05).

[0062] Optical flow mask warping, initial mask generated by bidirectional optical flow projection: forward warping: shift last frame mask to current frame coordinates by motion vector, 95% pixels in pedestrian region correctly projected (displacement error < 2 pixels). Backward verification: for uncovered region (newly exposed car head), complement projection by inverse optical flow, vehicle mask coverage from 88% to 97%. Output initial segmentation mask: pedestrian region connectivity kept intact, 5% holes in vehicle tail due to accelerating motion.

[0063] Adaptive threshold segmentation, dynamic division of varying region based on boundary response map: threshold calculation: T = μ + 0.3σ (μ is the mean of response map 0.12, σ is the standard deviation 0.18) -> T = 0.15.

[0064] Binaryzation: regions with response value ≥ 0.15 are identified as varying regions (pedestrian leg swing area, vehicle advancing area). Regions with response value < 0.15 are considered as static background (ground shadow false detection rate < 3%). Output varying region mask: 200 new pixels in pedestrian leg, 150 new pixels in vehicle head, background noise suppressed to 0.5%. Parameter effect:

[0065] Processing stage Core parameters Numerical values Effects Optical flow distortion Forward projection error <2 pixels Ensuring subject position continuity -- Backward supplementary coverage rate +9％ Repairing motion abrupt change regions Adaptive threshold Mean value (μ) 0.12 Reflecting overall motion intensity -- Dynamic threshold (T) 0.15 Balancing sensitivity and noise immunity Output effect Change region precision 94％ Effectively capturing motion boundaries -- Background false detection rate ＜3％ Suppressing static interference

[0066] By optical flow warping to inherit historical segmentation results and combining dynamic boundary response to realize local update, the mask breaking problem caused by target speed change (pedestrian suddenly running) in monitoring video is solved, and a stable varying region primitive is provided for final topology optimization.

[0067] 105、Topology preserving mask update step: within the range of the varying region mask, perform region growing algorithm with boundary response map as growth constraint, fuse the growth result with the initial segmentation mask, and output the final segmentation mask through topology skeleton extraction and persistence filtering.

[0068] It should be noted that the following is for the monitoring video scene of pedestrians crossing the road (1920x1080 resolution, 30fps):

[0069] Input data preparation, varying region mask: output from step 104, identifies the motion varying region (about 200 new pixels in pedestrian leg, 150 new pixels in vehicle head, background noise < 0.5%). Boundary response map: from step 103, vehicle contour response value ≈ 0.57 (brightness ≥ 140 / 255), pedestrian edge ≈ 0.5 (brightness ≈ 100 / 255). Initial segmentation mask: mask generated by optical flow warping of the last frame (pedestrian mask coverage 95%, 5% holes in vehicle tail).

[0070] Region growing algorithm execution, seed point selection: In the change region mask, automatically select the pixel with response value > 0.4 as the seed (12 seed points for pedestrian legs, 8 seed points for vehicle headlights). Growth constraints: Take the boundary response map as the growth threshold, and only merge adjacent pixels (eight neighborhoods) with a response value difference ≤ 0.1. Pedestrian legs: Fill in the area of the swinging skirt from the seed point, add 180 new connected regions, and reduce the holes by 90%. Vehicle front: Cover the new area of the vehicle head and repair the 15-pixel broken belt caused by accelerated motion.

[0071] The growth process skips the background area (ground shadow) with a response value < 0.05 to avoid false merging.

[0072] Mask fusion and topology optimization, preliminary fusion: Superimpose the growth results and the initial mask, the integrity of the pedestrian area reaches 98%, and the vehicle area hole is reduced to 2%.

[0073] Topology skeleton extraction: Morphological thinning of the fused mask generates a single-pixel-wide skeleton (pedestrian main skeleton length 120 pixels, vehicle skeleton 85 pixels). Remove short branches with a length < 10 pixels (eliminate 5 noise branches caused by leaf swinging).

[0074] Persistent filtering: Based on the length of the skeleton branch, the main structure is preserved (pedestrian retention rate 92%, vehicle retention rate 95%). Output the final segmentation mask: the pedestrian contour continuity is improved by 40%, and the vehicle geometric features are completely preserved. Effect statistics table:

[0075] Processing stage Pedestrian region Vehicle region Effects Defects before region growing Hole rate 12% Fracture zone 15 pixels Initial state Growth repair effect Hole rate 2% Fracture zone disappears Filling motion changes After topological filtering Skeleton completeness 92% Feature retention 95% Improving mask robustness

[0076] Through region growing in the change region, accurate expansion, and based on the skeleton topology filtering to eliminate noise branches, the mask breaking problem caused by target deformation (pedestrian skirt swinging) and motion blur (vehicle moving fast) in the monitoring video is effectively solved, and the segmentation result with stable topology structure is output.

[0077] In the embodiment of the present application, the entropy generation rate graph is generated by calculating the weighted combination of the divergence and the curl of the optical flow, which quantifies the irreversibility of motion in a physical sense; it can accurately distinguish between real motion such as vehicle continuous forward movement and periodic motion of pedestrian limbs, and transient noise such as leaf shaking, providing more accurate motion feature basis for subsequent segmentation, improving the accuracy of segmentation, and reducing false segmentation caused by noise interference; a dual morphological field (gray field and gradient field) is constructed, and the discrete reaction-diffusion equation is iteratively evolved to simulate the dynamic change of texture over time; the generated texture prediction graph has a small error with the actual frame, providing a more accurate prediction basis for subsequent residual calculation, which helps to more accurately detect the motion boundary and the change area; the boundary response graph is generated by fusing the motion irreversibility (entropy graph) and the texture evolution deviation (residual graph); it effectively distinguishes the real motion boundary and the static interference, and the output boundary response graph can clearly show the vehicle contour and the pedestrian edge while suppressing the background noise, providing high-confidence edge features for subsequent mask update, and improving the boundary accuracy of the segmentation result; the optical flow distortion inherits the historical segmentation result, and realizes local update combined with the dynamic boundary response, while introducing the region growing algorithm and topological optimization; the mask breaking problem caused by target speed change, deformation and motion blur in the monitoring video is solved, and the segmentation result with stable topological structure is output, which improves the integrity and robustness of the segmentation mask, and makes the segmentation result more consistent with the actual target shape and motion state.

[0078] Referring to Figure 2 Another embodiment of the video image segmentation method in the embodiment of the present application comprises:

[0079] 201. Motion dynamics feature extraction step: optical flow field calculation is performed on the current frame and the next frame in the continuous video frames to generate a motion vector field, and a weighted combination of the divergence and the curl of the optical flow is calculated based on the motion vector field to generate an entropy generation rate graph representing the motion irreversibility;

[0080] Specifically, the optical flow calculation based on polynomial expansion: non-parametric optical flow estimation is performed on the current frame and the next frame in the continuous video frames to obtain an original two-dimensional motion vector field; motion field non-dimensional normalization: the full-field motion vector module length is divided by the maximum motion module length to generate a normalized optical flow field; fluid mechanics feature decoupling: the optical flow divergence scalar field is extracted from the normalized optical flow field by a central difference operator; the optical flow curl scalar field is extracted from the normalized optical flow field by a Sobel curl operator; thermodynamic entropy generation rate synthesis: the absolute values of the optical flow divergence value and the curl value are taken respectively, and then linearly superimposed according to a preset weight to generate an initial entropy generation rate graph; spatio-temporal alignment preprocessing: the initial entropy generation rate graph is cached to a projection queue, and the output is an entropy generation rate graph for use in the subsequent dynamic boundary fusion detection step.

[0081] It should be noted that in the industrial robot operation monitoring scene, the core of the motion dynamics feature extraction step (201) is to quantify the motion irreversibility through the analysis of the optical flow field. The following is an example of processing video frames of a robot arm grabbing a workpiece to illustrate the specific implementation:

[0082] Based on the polynomial expansion of the optical flow calculation, the Farneback dense optical flow algorithm is performed on the continuous video frames (the tth frame of the robot arm starts to grab, and the t+1th frame of the workpiece is clamped and lifted). The input is a 512x512 grayscale frame, and the two-dimensional motion vector of each pixel is estimated by polynomial expansion. The output vector field in the joint motion area of the robot arm, and the vector in the background static area is close to zero. Let the motion vector of a certain pixel point be (1.2, 0.8) pixels / frame.

[0083] The motion field is dimensionless normalized, and the vector module length of the whole field is calculated, and it is found that the maximum module length appears at the clamp tip (3.5 pixels / frame). Divide the whole field vector by the maximum value to get the normalized optical flow field. The aforementioned vector (1.2, 0.8) is normalized to (0.34, 0.23), which retains the direction but eliminates the dimensional influence.

[0084] Fluid mechanics feature decoupling, divergence extraction: use the central difference operator on the normalized field. The vector of a certain point neighborhood: left (0.3, 0.2), right (0.4, 0.1), then divergence = (0.4-0.3) / Δx+(0.1-0.2) / Δy=0.05 (expanding motion). The calculation of the rotation rate uses the Sobel rotation rate operator. If the vector above a certain point is (0.1, 0.3) and the vector below is (0.2, -0.1), (clockwise rotation).

[0085] Thermodynamic entropy generation rate synthesis, take the absolute value of the divergence value (0.05) and the rotation rate value (-0.5), and linearly superimpose according to the preset weight (divergence: rotation rate = 1:2): entropy generation rate = |0.05| x 1 + |-0.5| x 2 = 1.05. This value quantifies the irreversible dissipation of motion (workpiece sliding friction).

[0086] Temporal and spatial alignment preprocessing, cache the initial entropy generation rate graph (512x512 matrix) to the projection queue. The high value area in the graph is concentrated in the mechanical wrist joint (rotational deformation) and the workpiece contact surface (shear dissipation), which is consistent with the physical motion.

[0087] When the robot arm translates at a constant speed (low divergence / rotation rate), the entropy generation rate ≈ 0; while in rapid rotation or workpiece instability slip, the entropy rate increases sharply to more than 0.8. This feature map provides a physical indicator of motion irreversibility for subsequent steps.

[0088] 202. Texture co-evolution prediction step: map the current frame gray information to a first virtual morphogen concentration field, and map the gradient information to a second virtual morphogen concentration field, iteratively evolve the double morphogen field through the discretized reaction-diffusion equation, and output the next frame texture prediction map;

[0089] Specifically, double-channel morphogen field construction: map the current frame gray value to a first preset interval to generate an activated morphogen concentration field (simulating a biological chemical activator); extract the current frame gradient amplitude and map it to a second preset interval to generate an inhibitory morphogen concentration field (simulating a biological chemical inhibitor); combine the two to form a double-channel spatio-temporal evolution field; asymmetric diffusion kernel configuration: configure an isotropic diffusion kernel for the activated morphogen; configure an inhibition-dominant diffusion kernel (diffusion coefficient greater than the activated type) for the inhibitory morphogen; reaction-diffusion coupling iteration: alternately execute on the artificial time scale: local reaction stage: update the morphogen concentration based on the activation-inhibition coupling function; global diffusion stage: respectively apply asymmetric diffusion kernels to realize morphogen spatial transfer; dynamic steady state detection: calculate the Euclidean distance change rate of the morphogen field between consecutive iterations; terminate iteration when the change rate is lower than the adaptive threshold; texture differentiation feature extraction: extract the spatial pattern of the steady-state activated morphogen field; map its amplitude distribution to the next frame texture prediction map (as input to 203).

[0090] It should be noted that the following takes the workpiece slip detection in the industrial robot arm monitoring video as an example:

[0091] In the video of the workpiece clamped by the robot arm (resolution 512x512), the current frame (t frame) shows that there is a metal wire drawing texture on the surface of the workpiece, and the texture change of the next frame (t+1 frame) needs to be predicted to detect abnormal slip.

[0092] Double-channel morphogen field construction, activated morphogen field: normalize the current frame gray image to the [0.1, 0.9] interval (simulate activator), the workpiece highlight area gray value 180→mapped to 0.72, the background dark area gray value 30→mapped to 0.12.

[0093] Inhibitory morphogen field: calculate the Sobel gradient amplitude of the current frame, normalize it to the [0.05, 0.3] interval (simulate inhibitor), and the texture edge gradient value 120→mapped to 0.25, the smooth area gradient value 20→mapped to 0.08. Output: double-channel field matrix (512x512x2).

[0094] Asymmetric diffusion kernel configuration, activated diffusion: use a 3x3 isotropic Gaussian kernel (standard deviation σ=0.8) to maintain texture continuity. Inhibitory diffusion: use a 5x5 Gaussian kernel (σ=1.5, diffusion coefficient 0.4) to accelerate the lateral diffusion of the inhibitor and suppress edge noise.

[0095] Reaction-diffusion coupling iteration, local reaction stage:

[0096] Concentration update by activation-inhibition function: new concentration = original activation concentration + 0.2*(1-original activation concentration) - 0.3*inhibitor concentration; a pixel point (activation value 0.7, inhibitor value 0.2) -> updated to 0.7 + 0.2*0.3 - 0.3*0.2 = 0.70.

[0097] Global diffusion stage: apply 3x3 Gaussian smoothing to activation field; apply 5x5 Gaussian filter to inhibitor field, strengthen edge inhibition (inhibitor value at texture junction of workpiece from 0.25 diffuses to 0.18 in adjacent area).

[0098] Dynamic steady state detection, calculate the rate of change of Euclidean distance of double field R per iteration step k :

[0099]

[0100] Where H is the image height, the vertical pixel number of the video frame; W is the image width, the horizontal pixel number of the video frame; A k is the current iteration activation concentration value (simulates the state of biological chemical activator); A k-1 is the previous iteration activation concentration value; I k is the current iteration inhibitor concentration value (simulates the state of biological chemical inhibitor); I k-1 is the previous iteration inhibitor concentration value; i and j are pixel coordinates, position indexes in the image;

[0101] Iteration 1->2: change rate 2.1%; iteration 5->6: change rate 0.8%; iteration 9->10: change rate 0.48% (lower than threshold 0.5%), terminate iteration. Time consumption: 53ms for 10 iterations (CPU i7-11800H).

[0102] Texture differentiation feature extraction, extract the spatial pattern of steady-state activation field: the concentration of the main body area of the workpiece is stable at 0.65-0.75 (high consistency); the concentration of the clamp contact area is reduced to 0.35-0.45 (due to pressure-induced texture deformation).

[0103] Output next frame texture prediction map: linearly map the activation field concentration to gray value (0.45->115), highlight the low value area of the clamp contact area (abnormal deformation marked by red circle in the figure).

[0104] 203、Dynamic boundary fusion detection step: project the entropy generation rate map to the next frame coordinate space by optical flow to generate a projected entropy generation rate map, calculate the pixel-level absolute residual error map of the texture prediction map and the actual next frame, and fuse the projected entropy generation rate map and the residual error map to generate a boundary response map;

[0105] Specifically, motion consistency projection: based on the normalized optical flow field generated in 201, forward propagate the pixel value in entropy production rate map along the motion vector direction; perform vorticity conservation filling (weighted interpolation using the curl field distribution) on the coordinate conflict area caused by occlusion; obtain the spatio-temporal aligned projection entropy production rate map; structure enhanced residual calculation: calculate the brightness-contrast-structure three-channel difference in the local window for the texture prediction map output by 202 and the actual next frame gray map; fuse the three-channel difference to generate a structure-aware residual map; obtain an enhanced texture residual map; nonlinear adaptive fusion: establish a dynamic weight mapping function of the projection entropy production rate map and the enhanced texture residual map: increase the motion weight in the high entropy production rate area; increase the appearance weight in the high texture residual area; obtain the pixel-level fusion weight field; boundary response generation: weighted superposition of the projection entropy production rate map and the enhanced texture residual map according to the pixel-level fusion weight field; perform double-scale Gaussian difference enhancement on the superposition result; obtain a multi-physical field boundary response map; topology initialization output: mark the extreme points in the boundary response map whose connected domain is greater than a threshold as topology key points; output the boundary response map with key point marking to 204.

[0106] It should be noted that the following is an example of industrial robot arm clamping workpiece slip monitoring:

[0107] The robot arm clamp clamps a metal workpiece (resolution 512x512), and the current frame (t frame) clamp is stably clamped, and the next frame (t+1 frame) workpiece slips by 0.5mm.

[0108] Motion consistency projection, based on the normalized optical flow field (maximum module length 3.2 pixels / frame) of step 201: forward propagate the entropy rate map (t frame) to the t+1 frame coordinates along the optical flow vector. The motion vector of the clamp tip pixel (120, 200) is (2.1, 1.3), and its entropy rate value 0.85 is projected to the new coordinates (122.1, 201.3).

[0109] Occlusion processing: the workpiece slip causes the clamp occlusion area (coordinates (130-140, 210)) to appear a hole, and the hole area is filled by weighted interpolation using the curl field extreme point (pixel (135, 215) with curl 0.8). Output: spatio-temporally aligned projection entropy production rate map, slip area value rises to 0.92.

[0110] Structure-enhanced residual computation, texture prediction map output by 202 vs. actual t+1 frame: three-channel difference computation (7x7 local window): luminance difference: average gray difference 38 (out of 255) in slip region -> normalized value 0.15; contrast difference: local standard deviation difference 12 -> normalized value 0.23; structure difference: normalized cross-correlation value 0.62 -> difference value 0.38 (1-0.62). Fused residual: weighted sum of three-channel values (weights 0.3:0.2:0.5) -> slip-region residual 0.45 (out of 1.0). Output: enhanced texture residual map, slip region appears as a high-contrast patch (values >0.4).

[0111] Non-linear adaptive fusion, dynamic weight mapping: high-entropy regions (clamp-rotating joint, entropy rate >0.8) assigned motion weight 0.7; high-texture residual regions (workpiece slip, residual >0.4) assigned appearance weight 0.8. Output: pixel-level fusion weight field, slip-region weight distribution is (motion 0.2, appearance 0.8).

[0112] Boundary response generation, weighted sum: fuse projected entropy rate map and residual map by weight field, slip-region response value = 0.92x0.2 + 0.45x0.8 = 0.56; double-scale Gaussian difference enhancement: difference operation with kernels of sigma=1.0 and sigma=3.0, intensify slip boundary (response value boosted from 0.56 to 0.75). Output: multi-physical-field boundary response map, slip boundary appears as a continuous bright band.

[0113] Topology initialization output, detect extreme points of >5x5 connected components in response map: mark three key points of workpiece slip boundary ((132,208), (138,213), (145,219)). Output: boundary response map with key points (input to step 204).

[0114] 204, spatio-temporal continuity constraint step: warp previous-frame segmentation mask to current frame by optical flow, generate initial segmentation mask, perform adaptive threshold segmentation based on boundary response map, identify change region binary mask;

[0115] Specifically, the segmentation mask optical flow distortion: based on the normalized optical flow field generated in 201, the boundary point set of the last frame segmentation mask is transmitted by sub-pixel level bilinear motion; vortex core preservation interpolation is performed on the empty area generated by motion occlusion (adjacent mask diffusion is guided by the extreme point of the vorticity field); an initial segmentation mask with geometric consistency is obtained; multi-modal adaptive threshold segmentation: define a dynamic calculation window on the boundary response graph (the window size is positively correlated with the local entropy generation rate); perform double Gaussian kernel difference statistics on the pixel values in each window, and automatically solve the optimal segmentation threshold; obtain a local optimal binary threshold mapping graph; deformation region identification: exclusive or logical operation is performed on the initial segmentation mask and the binary threshold mapping graph; extract the region with a connected domain area greater than the motion consistency in the operation result: obtain the candidate deformation region mask; motion artifact filtering: calculate the standard deviation of the vorticity in the candidate deformation region; filter the regions with a standard deviation lower than the vorticity conservation threshold (exclude uniform motion interference); obtain the physically verified change region mask; topological structure pre-generation: extract Betti number feature points (connected components and hole numbers) on the boundary of the change region mask; output the change region binary mask to 205.

[0116] It should be noted that the following takes 0.3mm slip of an industrial robot arm clamping a workpiece as an example:

[0117] In a 512x512 resolution monitoring video, the robot clamp stably clamps a cylindrical workpiece (mask area 18,200 pixels) at t frame, and the workpiece slips at t+1 frame, causing the contact surface to deform.

[0118] Segmentation mask optical flow distortion, based on the normalized optical flow field (maximum motion vector module length of the clamp area 2.8 pixels) of step 201: forward transmission of the t frame segmentation mask (workpiece area gray value 255, background 0) along the motion vector: the workpiece vertex (120,200) moves to (122.1,201.3), and a new coordinate mask is generated by sub-pixel level bilinear interpolation. Occlusion filling: the workpiece slip causes the clamp occlusion area (coordinates 130-140,210) to have a hole, which is filled by using the extreme point of the vorticity field (pixel (135,215) with vorticity 0.75) to guide the diffusion of the adjacent mask. Output: an initial segmentation mask with geometric alignment, and the slip area boundary is blurred.

[0119] Multi-modal adaptive threshold segmentation, define a dynamic window on the boundary response graph (203 output): high entropy area (clamp joint, entropy generation rate > 0.8): window size is enlarged to 11x11 pixels; low entropy area (workpiece body, entropy rate < 0.3): window size is reduced to 5x5 pixels. Double Gaussian kernel statistics: calculate the high and low standard deviations (σ1=0.8,σ2=1.6) in each window, and automatically solve the optimal threshold (slip area threshold = 0.65) after difference. Output: binary threshold mapping graph, slip area as high light spot (value = 1).

[0120] Deformation region identification, XOR operation: initial mask XOR binaryzation map, highlight difference area (slip area difference area 210 pixels). Connected component filtering: only keep connected components with area > 150 pixels (preset motion consistency), exclude small noise. Output: candidate deformation region mask, identify workpiece contact surface deformation area.

[0121] Motion artifact filtering, calculate the standard deviation of the optical flow vorticity in the candidate area: the standard deviation of the real slip area is 0.25 (non-uniform vorticity distribution); the standard deviation of the fixture translation area is 0.08 (uniform motion). Vorticity conservation filtering: filter out areas with a standard deviation <0.15 (exclude fixture translation artifacts). Output: physically verified change region mask (area 205 pixels).

[0122] Topology pre-generation, extract change mask boundary: Betti number labeling: 1 connected component (workpiece deformation main body) + 2 holes (fixture contact points). Label feature points: hole centers (132, 208) and (145, 219). Output: Betti-labeled mask (input 205 step).

[0123] 205, Topology-preserving mask update step: within the range of the change region mask, perform region growing algorithm with boundary response map as growth constraint, fuse the growth result with the initial segmentation mask, and output the final segmentation mask after topology skeleton extraction and persistence filtering;

[0124] Specifically, physically constrained region growing: within the range of the change region mask, perform geodesic constrained growth from the topology key point (204 output) with the gradient amplitude of the boundary response map as the growth barrier; obtain a topologically connected growth mask; mask fusion reconstruction: perform morphological closing operation fusion of the topologically connected growth mask and the initial segmentation mask (the size of the structural element is negatively related to the local entropy generation rate); obtain a geometrically closed candidate segmentation mask; multi-scale skeleton extraction: perform curvature-aware morphological thinning on the candidate segmentation mask (retain high-curvature branch structure); obtain a sub-pixel precision topology skeleton map; persistent homology optimization: calculate the persistence barcode of the topology skeleton map; filter out homology features with a life cycle less than the motion consistency threshold; obtain a topologically stable skeleton simplified map; algebraic topology reconstruction output: take the skeleton simplified map as the central axis, and reconstruct the closed boundary by Betti number constrained Voronoi segmentation; obtain the final segmentation mask with Euler characteristic labeling.

[0125] It should be noted that the following is an example of local deformation of an industrial robot arm holding a workpiece:

[0126] In 512x512 video frames, the workpiece produces local indentation (area of change region mask about 200 pixels) due to clamp pressure, which needs to update the segmentation mask under the premise of maintaining the topological structure.

[0127] Physical constraint region growing, input: change region mask (with Betti label: 1 connected component + 2 holes, corresponding to clamp contact points) and boundary response map (peak value 0.75) output in step 204. Growth constraints: use the gradient amplitude of the boundary response map as the growth barrier (regions with gradient <0.6 are prohibited from growing). Execution process: start from the topological key point (hole center (132, 208)), expand along the geodesic path: growth rate at contact point (132, 208) is 0.8 pixels / iteration; rate decreases to 0.2 pixels / iteration at smooth area (140, 215). Output: topologically connected growth mask (area 185 pixels), which perfectly wraps the indentation area.

[0128] Mask fusion reconstruction, morphological closing operation: high-entropy area (clamp joint, entropy rate >0.8): use 3x3 structure element (small kernel) to retain details; low-entropy area (workpiece main body, entropy rate <0.3): use 7x7 structure element (large kernel) to smooth the boundary. Output: geometric closure candidate mask (hole number remains 2).

[0129] Multi-scale skeleton extraction, curvature-aware thinning: perform sub-pixel level morphological thinning on the candidate mask, retaining branches with curvature >0.4 (hole edge jagged structure). Output: topological skeleton graph (contains 12 endpoints and 8 trifurcation points).

[0130] Persistent homology optimization, persistent homology analysis: generate the persistent homology barcode of the skeleton, detect 18 homology features (short branches, isolated loops); filter out transient features with a lifetime <5 frames (motion consistency threshold), removing 8 noise branches. Output: topologically stable simplified skeleton (number of feature points reduced to 10).

[0131] Algebraic topology reconstruction, Betti number constraint reconstruction: set topological constraints: connected component number β0=1, hole number β1=2 (consistent with clamp contact points); generate a Voronoi diagram with the simplified skeleton as the central axis to reconstruct the closed boundary. Output: final segmentation mask (area 198 pixels), Euler characteristic χ=-1 (consistent with β0-β1=1-2=-1).

[0132] 206、Also includes dynamic computing resource scheduling:

[0133] Real-time monitoring of peak density distribution of entropy generation rate map and gradient entropy value of boundary response map; dynamically allocate computing resources for each step according to monitoring data (high activity area improves processing accuracy); obtain adaptive computing strategy table (input all processing steps);

[0134] Cross-frame topology continuity constraint: Extract Betti number sequence of current frame and previous three frames final segmentation masks; Detect topology mutation events by algebraic topological homology group analysis; Get topology continuity correction signal (input 205);

[0135] Multi-scale result fusion output: Perform segmentation of three scales (original resolution, 1 / 2 down-sampling, 2 times super-pixel) in parallel for the same frame; Fuse topologically consistent regions of multi-scale masks based on vorticity conservation criterion; Get final segmentation mask verified across scales (output covering 205).

[0136] It should be noted that the following is an example of industrial robot precise assembly monitoring:

[0137] In a 512x512 video stream, the robot clamps a metal workpiece for assembly, and a sudden 0.3mm slip occurs. The system needs to dynamically schedule resources to ensure real-time segmentation accuracy.

[0138] Real-time resource monitoring and allocation, entropy generation rate map monitoring: Detect the entropy rate peak value of the clamp area (0.92, high motion irreversibility), and automatically improve the calculation accuracy of this area to sub-pixel level (0.1px), and reduce the background area to integer pixel level. Boundary response map gradient entropy analysis: The slip boundary gradient entropy reaches 1.8 (high complexity), triggering the doubling of local computing resources (GPU computing power allocation from 15% to 30%). Output: adaptive strategy table, frame processing time from 33ms to 22ms.

[0139] Cross-frame topology continuity constraint, Betti number sequence analysis: Extract topological features of segmentation masks of consecutive 4 frames: t-3 frame: β0=1 (1 connected domain), β1=0 (no hole); t frame: β0=1, β1=2 (new clamp contact hole added); Homology group mutation detection: Hole number mutation triggers correction signal, forcing 205 step to retain hole structure. Output: Topology correction signal (hole coordinates (132, 208), (145, 219)).

[0140] Multi-scale result fusion, parallel three-scale segmentation: Original resolution: detect slip area 150px2; 1 / 2 down-sampling: detect 120px2 (underestimate due to resolution loss); 2 times super-pixel: detect 140px2 (over-segmentation noise); Vorticity conservation criterion fusion: only keep regions with standard deviation of vorticity >0.2 (true deformation), and eliminate uniform motion artifacts, finally confirm effective slip area 142px2. Output: Mask verified across scales (area error <5%).

[0141] In the embodiments of the present application, the multi-disciplinary theories such as fluid mechanics (optical flow divergence and curl), thermodynamics (entropy production rate), biochemistry (reaction-diffusion equation) are introduced into the field of video image segmentation. This interdisciplinary integration provides a new perspective and method for solving complex video image segmentation problems, breaks through the limitations of traditional single-disciplinary methods, and can more comprehensively and deeply analyze and process various features and changes in video images; the entropy production rate map is generated by quantifying the irreversibility of motion through optical flow field analysis. In the industrial robot operation monitoring scene, different motion states of the robot arm can be accurately distinguished, such as entropy production rate ≈ 0 when translating at a constant speed, and entropy rate increasing to more than 0.8 when rotating quickly or the workpiece is unstable and slipping. This precise quantification helps to more accurately identify and track moving targets, and provides reliable basis for quality detection, fault diagnosis, etc. in industrial automation production; the texture co-evolution prediction step maps the gray level and gradient information into a double virtual morphogen concentration field, and predicts the next frame of texture change through reaction-diffusion equation iteration evolution. Taking workpiece slip detection in industrial robot monitoring video as an example, the abnormal deformation of the gripper contact area can be highlighted, potential problems can be found in advance, and early warning can be provided to take timely measures to prevent accidents, thereby improving the safety and stability of the production process; the dynamic boundary fusion detection step considers motion and texture information, and fuses the projection entropy production rate map and the residual map to generate a boundary response map. In the industrial robot arm clamping workpiece slip monitoring, the workpiece slip boundary can be accurately detected, a continuous bright band is generated, and clear boundary information is provided for accurate segmentation of the target object, thereby improving the accuracy and reliability of segmentation.

[0142] The spatiotemporal continuity constraint step ensures the continuity and rationality of the segmentation result in time and space by optical flow distortion of the segmentation mask of the previous frame, multi-modal adaptive threshold segmentation, and motion artifact filtering. In the scenario where the workpiece held by the mechanical arm slips, the uniform motion interference can be effectively excluded, the change area mask of the physical verification can be accurately identified, and segmentation errors caused by motion artifacts can be avoided. The topology preserving mask updating step performs a region growing algorithm in the range of the change area mask, and combines topology skeleton extraction and persistence filtering to update the segmentation mask while preserving the topology of the target. In the case of local deformation of the workpiece due to clamp pressure, the concave area can be perfectly wrapped, the number of holes and other topological features can be kept unchanged, and the segmentation result can be ensured to conform to the actual physical structure of the target. The dynamic computing resource scheduling mechanism monitors the image features in real time, dynamically allocates computing resources according to the monitoring data, and realizes adaptive computing. In the precise assembly monitoring of the industrial robot arm, the processing accuracy can be improved according to the activity level of different regions, while the overall frame processing time is reduced, the processing efficiency is improved under the premise of ensuring the segmentation accuracy, and the real-time monitoring demand is met. The cross-frame topology continuity constraint detects topology mutation events and sends a correction signal by extracting the Betti number sequence of multiple segmentation masks and analyzing the Betti number sequence. In processing the video sequence, the segmentation result topology structure mutation caused by image noise or temporary interference can be effectively avoided, and the stability and reliability of the segmentation result can be enhanced. The multi-scale result fusion performs segmentation in three scales in parallel, and fuses the topologically consistent regions of the multi-scale masks based on the vorticity conservation criterion. In processing complex scenes, the segmentation information under different scales can be integrated, noise and artifacts can be effectively excluded, errors caused by single-scale segmentation can be reduced, and the accuracy and robustness of the segmentation result can be improved.

[0143] The video image segmentation method in the embodiments of the present application is described above, and the video image segmentation device in the embodiments of the present application is described below. Please refer to Figure 3In the embodiment of the present application, one embodiment of the video image segmentation device comprises: an analysis module 301, configured to perform optical flow field calculation on a current frame and a next frame in continuous video frames, generate a motion vector field, calculate a weighted combination of optical flow divergence and curl based on the motion vector field, and generate an entropy generation rate map; a prediction module 302, configured to map the gray information of the current frame into a first virtual morphogen concentration field, map the gradient information into a second virtual morphogen concentration field, iteratively evolve the double morphogen field, and output a texture prediction map of the next frame; a fusion module 303, configured to project the entropy generation rate map to the coordinate space of the next frame through optical flow, generate a projected entropy generation rate map, calculate a pixel-level absolute residual map of the texture prediction map and the actual next frame, and fuse the projected entropy generation rate map and the residual map to generate a boundary response map; a setting module 304, configured to warp the segmentation mask of the last frame to the current frame through optical flow, generate an initial segmentation mask, perform adaptive threshold segmentation based on the boundary response map, and identify a change region binary mask; and a processing module 305, configured to perform a region growing algorithm with the boundary response map as a growth constraint within the change region mask, fuse the growth result with the initial segmentation mask, and output a final segmentation mask through topological skeleton extraction and persistence filtering.

[0144] In the embodiment of the present application, a collaborative working system comprising five modules of analysis, prediction, fusion, setting and processing is constructed. This architecture breaks the limitation of traditional video image segmentation device with single function module, and each module has clear division of labor and close cooperation, processes and analyzes video image information from different angles, realizes comprehensive and efficient processing of video image segmentation task, and provides a new device design idea for video image segmentation technology. The analysis module generates a motion vector field through optical flow field calculation, and further calculates a weighted combination of optical flow divergence and curl to obtain an entropy production rate map, realizing the quantification of motion irreversibility. This innovative method can more accurately capture the motion characteristics of objects in the video, for example, in a monitoring scene, it can clearly distinguish between normal and abnormal motion of objects, providing more abundant and accurate motion information for subsequent segmentation and recognition, which helps to improve the accuracy of segmentation and adaptability to dynamic scenes; the prediction module maps the gray information of the current frame into a first virtual morphogen concentration field, and maps the gradient information into a second virtual morphogen concentration field, and outputs the next frame texture prediction map through iterative evolution of the double morphogen field. This texture prediction method based on virtual morphogen concentration field simulates the reaction-diffusion process in biochemistry, which can better predict the trend of texture change. In practical applications, such as industrial detection, it can detect abnormal changes in the texture of the product surface in advance, providing strong support for timely adjustment of production parameters and ensuring product quality; the fusion module projects the entropy production rate map to the next frame coordinate space through optical flow projection to generate a projected entropy production rate map, and calculates the pixel-level absolute residual map of the texture prediction map and the actual next frame, and then fuses the two to generate a boundary response map. This fusion strategy considers both motion information and texture information, which can more accurately detect the boundary changes in the video image. For example, in video monitoring, the boundaries of moving objects can be more clearly identified, and even in complex backgrounds, the target object and the background can be effectively distinguished, improving the accuracy and robustness of segmentation; the setting module generates an initial segmentation mask by distorting the last frame segmentation mask through optical flow, and performs adaptive threshold segmentation based on the boundary response map to identify the binary mask of the change area; the processing module executes region growing algorithm in the change area mask range with the boundary response map as the growth constraint, fuses the growth result and the initial segmentation mask, and outputs the final segmentation mask through topological skeleton extraction and persistence filtering. This two-step segmentation mask processing scheme first quickly determines the change area by using optical flow distortion and adaptive threshold segmentation, and then further optimizes the segmentation result through region growing algorithm and topological processing, which not only ensures the efficiency of segmentation, but also improves the accuracy and stability of segmentation, and can effectively handle dynamic changes and complex targets in videos.

[0145] The above Figure 3 The video image segmentation device in the embodiment of the present application is described in detail from the perspective of modular functional entities, and the video image segmentation device in the embodiment of the present application is described in detail from the perspective of hardware processing.

[0146] Figure 4 FIG. 4 is a schematic diagram of an embodiment of a video image segmentation device. The video image segmentation device 400 can vary in configuration and performance and can include one or more processors (central processing units, CPUs) 410 (e.g., one or more processors) and memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) storing applications 433 or data 432. The memory 420 and the storage media 430 can be of the volatile or persistent type. The programs stored in the storage media 430 can include one or more modules (not shown in the figure), each of which can include a series of instructions for operating the video image segmentation device 400. Further, the processor 410 can be configured to communicate with the storage media 430 and execute the series of instructions stored in the storage media 430 on the video image segmentation device 400.

[0147] The video image segmentation device 400 can further include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input / output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that the video image segmentation device structure shown is not intended to limit the video image segmentation device, which can include more or fewer components than those shown, or combine certain components, or arrange the components differently. Figure 4 The video image segmentation device structure shown is not intended to limit the video image segmentation device, which can include more or fewer components than those shown, or combine certain components, or arrange the components differently.

[0148] The present application also provides a video image segmentation device, which includes a memory and a processor, the memory storing computer readable instructions, and the processor executing the computer readable instructions to perform the steps of the video image segmentation method in the above embodiments.

[0149] The present application also provides a computer readable storage medium, which can be a non-volatile computer readable storage medium or a volatile computer readable storage medium, the computer readable storage medium storing instructions, and the instructions, when executed on a computer, causing the computer to perform the steps of the video image segmentation method.

[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0151] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0152] The above-described embodiments are merely used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video image segmentation method, characterized in that: The video image segmentation method comprises: Calculate the optical flow field of the current frame and the next frame in the continuous video frames to generate a motion vector field, calculate the weighted combination of the optical flow divergence and the rotation based on the motion vector field, and generate an entropy generation rate map; Mapping the grayscale information of the current frame into a first virtual morpheme concentration field, mapping the gradient information into a second virtual morpheme concentration field, iteratively evolving the dual morpheme fields, and outputting a texture prediction map for the next frame; Projecting the entropy generation rate map to the next frame coordinate space via optical flow to generate a projected entropy generation rate map, calculating a pixel-level absolute residual map between the texture prediction map and the actual next frame, and fusing the projected entropy generation rate map and the residual map to generate a boundary response map; The segmentation mask of the previous frame is warped to the current frame through optical flow to generate the initial segmentation mask. Adaptive threshold segmentation is performed based on the boundary response map to identify the binary mask of the changed area. Within the range of the change region mask, a region growing algorithm is executed with the boundary response map as a growth constraint, the growing result is fused with the initial segmentation mask, and the final segmentation mask is output through topological skeleton extraction and persistence filtering.

2. The video image segmentation method according to claim 1, characterized in that: include: Perform non-parametric optical flow estimation on the current frame and the next frame in the continuous video frames to obtain the original two-dimensional motion vector field; The full-field motion vector modulus is divided by the maximum motion modulus to generate a normalized optical flow field. The optical flow divergence scalar field is extracted from the normalized optical flow field through the central difference operator, and the optical flow rotation scalar field is extracted from the normalized optical flow field. After taking the absolute values ​​of the optical flow divergence and the rotation value, they are linearly superimposed according to the preset weights to generate the initial entropy generation rate map; The initial entropy generation rate map is cached in the projection queue and output as the entropy generation rate map.

3. The video image segmentation method according to claim 2, characterized in that: include: Map the grayscale value of the current frame to the first preset interval to generate an activated morphogen concentration field, extract the gradient amplitude of the current frame and map it to the second preset interval to generate an inhibited morphogen concentration field, and combine the two to form a dual-channel spatiotemporal evolution field; Configure an isotropic diffusion kernel for the activated morphogen and an inhibition-dominated diffusion kernel for the inhibited morphogen; The local reaction phase and the global diffusion phase are performed alternately on an artificial time scale, and the change rate of the Euclidean distance of the morphometric field between consecutive iterations is calculated. The iteration is terminated when the change rate is lower than the adaptive threshold. The spatial pattern of the steady-state activated morphogen field is extracted and its amplitude distribution is mapped into the texture prediction map of the next frame.

4. The video image segmentation method according to claim 3, characterized in that: Set the Euclidean distance change rate of the morphological field to R k : Where H is the image height, the number of vertical pixels of the video frame; W is the image width, the number of horizontal pixels of the video frame; A k is the current iteration activation concentration value; A k-1 is the activation concentration value of the previous iteration; I k is the current iterative inhibition concentration value; I k-1 is the suppression concentration value of the previous iteration; i and j are pixel coordinates and position index in the image.

5. The video image segmentation method according to claim 3, characterized in that: include: Based on the normalized optical flow field, the pixel values ​​in the entropy generation rate map are forwarded along the direction of the motion vector, and the vorticity conservation is performed to fill the coordinate conflict area caused by occlusion to obtain the projected entropy generation rate map; For the texture prediction image and the actual next frame grayscale image, the brightness-contrast-structure three-channel difference is calculated in the local window, and the three-channel difference is fused to generate an enhanced texture residual image; A dynamic weight mapping function is established between the projected entropy generation rate map and the enhanced texture residual map. The motion weight is increased in the area with high entropy generation rate, and the appearance weight is increased in the area with high texture residual, thus obtaining a pixel-level fusion weight field. The projection entropy generation rate map and the enhanced texture residual map are weightedly superimposed according to the pixel-level fusion weight field, and the superposition result is enhanced by performing double-scale Gaussian difference to obtain the multi-physics field boundary response map; The extreme points whose connected domains are greater than the threshold in the multi-physics field boundary response graph are marked as topological key points, and the boundary response graph is output.

6. The video image segmentation method according to claim 5, characterized in that: include: Based on the normalized optical flow field, the boundary point set of the segmentation mask of the previous frame is transferred through sub-pixel bilinear motion, and vortex kernel preserving interpolation is performed on the hole area caused by motion occlusion to obtain the initial segmentation mask; A dynamic calculation window is defined on the boundary response map, and the double Gaussian kernel difference statistics are performed on the pixel values ​​in each window to automatically solve the optimal segmentation threshold and obtain the threshold mapping map; Perform an XOR logic operation on the initial segmentation mask and the threshold map, extract the area of ​​the connected domain in the operation result that is larger than the motion consistency, and obtain the candidate deformation region mask; Calculate the standard deviation of the optical flow curl within the candidate deformation region, filter the regions where the standard deviation is lower than the vorticity conservation threshold, and obtain the physically verified change region mask; Extract and output the binary mask of the changed area at the mask boundary of the changed area.

7. The video image segmentation method according to claim 6, characterized in that: include: Within the mask range of the changing region, the gradient amplitude of the boundary response map is used as the growth barrier, and geodesic constrained growth is performed from the topological key points to obtain a topologically connected growth mask. The topologically connected growth mask is fused with the initial segmentation mask through morphological closing operation to obtain a candidate segmentation mask. Perform curvature-aware morphological refinement on the candidate segmentation masks to obtain a topological skeleton graph; Calculate the continuous homology barcode of the topological skeleton graph, filter out the homology features whose life cycle is less than the motion consistency threshold, and obtain the skeleton simplified graph; Using the simplified skeleton image as the central axis, the closed boundary is reconstructed to obtain the final segmentation mask.

8. The video image segmentation method according to claim 7, characterized in that: Also includes dynamic computing resource scheduling: Real-time monitoring of the peak density distribution of the entropy generation rate graph and the gradient entropy value of the boundary response graph, dynamically allocating computing resources for each step based on the monitoring data, and obtaining an adaptive computing strategy table; Extract the Betti number sequence of the final segmentation mask of the current frame and the previous three frames, detect the topological mutation event through algebraic topological homology group analysis, and obtain the topological continuity correction signal; Segmentation at three scales is performed in parallel on the same frame, and the topologically consistent regions of the multi-scale masks are fused based on the vorticity conservation criterion to obtain the final segmentation mask that is verified across scales.

Images