A synchronous interaction method and system of a mine-used explosion-proof host and a display terminal

Abstract

The present application relates to the technical field of mine equipment communication and image processing, and particularly relates to a synchronous interaction method and system of a mine explosion-proof host and a display terminal. The method constructs an action response mapping based on historical action states of a controlled mine equipment and corresponding historical video frames, and determines a synchronization reference; generates action response information and a predicted video frame according to a current action state, compares the current video frame with the predicted video frame to obtain a real scene residual; sends the action response information and the real scene residual to the display terminal, and generates a reconstructed video frame by the display terminal in combination with a reference video frame; and when the real scene residual satisfies a reference update condition, updates the synchronization reference. The present application can reduce repeated transmission of predictable picture changes, reduce communication resource occupation, and improve video synchronization accuracy.

Description

Technical Field

[0001] This invention relates to the field of communication and image processing technology for mining equipment, specifically to a method and system for synchronous interaction between a mining explosion-proof host and a display terminal. Background Technology

[0002] With the development of automation and remote control technologies in mining equipment, explosion-proof mining control units typically need to collect the operational status and on-site video of the controlled mining equipment and transmit the relevant data to a display terminal so that operators can simultaneously view the equipment operation process and the on-site environment. Since the underground communication link also needs to carry control commands, equipment status, and alarm information, the continuous transmission of video data can easily consume significant communication resources.

[0003] Existing technologies typically employ complete video stream transmission or encoding based on differences between adjacent video frames. However, the structural position and attitude changes of controlled mining equipment correspond to its operational state. Existing solutions do not fully utilize this correspondence, still requiring repeated transmission of image changes predictable from the equipment's operational state, easily leading to data redundancy. Furthermore, camera vibration, lighting changes, dust obstruction, and on-site target movement can cause deviations between actual video frames and predicted results; if a fixed reference video frame is used for a long period, these deviations may accumulate, affecting the video reconstruction effect of the display terminal.

[0004] Therefore, how to use the operational status of controlled mining equipment to predict structural changes in video frames, distinguish between predictable image changes and actual scene changes, reduce the amount of video transmission data while ensuring the accuracy of video synchronization between the mining explosion-proof host and the display terminal, and update in a timely manner when the synchronization reference fails, has become a technical problem that existing technologies need to solve. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for synchronous interaction between a mine explosion-proof host and a display terminal. By constructing an action response mapping between changes in action state and changes in video frames, predictive video frames are generated based on a synchronization benchmark, and real-scene residuals are extracted. The action response information and real-scene residuals are sent to the display terminal to complete video reconstruction, and the synchronization benchmark is updated according to the residual state. This solves the technical problems of data redundancy, high communication resource consumption, and long-term synchronization error accumulation in mine video transmission.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for synchronous interaction between a mine explosion-proof host and a display terminal, the method comprising:

[0008] Based on the historical action states of the controlled mining equipment and the corresponding historical video frames, an action response mapping between changes in action states and changes in video frames is constructed, and a synchronization benchmark consisting of reference video frames and corresponding reference action states is determined.

[0009] The current action state of the controlled mining equipment is obtained, action response information is generated based on the current action state, the reference action state, and the action response mapping, and a predicted video frame is generated based on the reference video frame and the action response information.

[0010] Obtain the current video frame corresponding to the current action state, and determine the real-scene residual based on the difference between the current video frame and the predicted video frame;

[0011] The motion response information and the real-scene residual are sent to the display terminal, and the display terminal generates a reconstructed video frame based on the reference video frame, the motion response information, and the real-scene residual.

[0012] When the real-scene residual meets the baseline update conditions, the synchronization baseline is updated with the current video frame and the current action state.

[0013] Furthermore, the step of constructing an action response mapping between changes in action state and changes in video frames based on the historical action states of the controlled mining equipment and the corresponding historical video frames includes:

[0014] Each historical action state and its corresponding historical video frame are grouped into a historical sample group according to the time correspondence, and the historical sample group is arranged according to the action execution order.

[0015] Based on the historical sample pairs in the historical sample group, the amount of change in historical action state is determined, and the controlled structural region that changes with the historical action state is determined from the corresponding previous and subsequent historical video frames.

[0016] The image change parameters of the controlled structural region between the preceding and following historical video frames are extracted, and the correspondence between the historical action state change amount and the image change parameters is established to obtain the action response mapping.

[0017] Further, determining the synchronization reference composed of reference video frames and corresponding reference action states includes:

[0018] Based on the changes in the historical action state of each historical sample pair, candidate historical sample groups in which the controlled mining equipment is in a stable action state are determined.

[0019] Reference video frames are determined based on the controlled structural region integrity and image quality of historical video frames in the candidate historical sample group;

[0020] The historical action state corresponding to the reference video frame is determined as the reference action state, and the reference video frame is associated with the reference action state to obtain a synchronization reference.

[0021] Further, generating action response information based on the current action state, the reference action state, and the action response mapping includes:

[0022] Based on the current action state and the reference action state, determine the change in the current action state;

[0023] Determine the first historical action state change and the second historical action state change that are adjacent to the current action state change from the action response mapping, and obtain the first image change parameter and the second image change parameter corresponding to the two respectively.

[0024] Based on the distance relationship between the current action state change and the first historical action state change and the second historical action state change, the parameter interpolation coefficient is determined;

[0025] The first image change parameter and the second image change parameter are interpolated based on the interpolation coefficients to obtain the current image change parameter;

[0026] The current image change parameters and their corresponding controlled structural regions are used as action response information.

[0027] Further, generating a predicted video frame based on the reference video frame and the motion response information includes:

[0028] Based on the controlled structural region in the action response information, the reference video frame is segmented to obtain a structural region image and a background region image.

[0029] The current image change parameters are converted into coordinate transformation relationships for the corresponding controlled structural regions. The pixels in the structural region image are then mapped and resampled according to the coordinate transformation relationships to obtain the transformed structural image.

[0030] The missing areas formed after the displacement of the controlled structure region are filled using neighboring background pixels in the background region image.

[0031] Based on the structural connection relationship between the controlled structural regions, the transformed structural image and the filled background region image are stitched together and fused to obtain the predicted video frame.

[0032] Further, determining the real-scene residual based on the difference between the current video frame and the predicted video frame includes:

[0033] Background feature points are extracted from the uncontrolled structural regions of the current video frame and the predicted video frame, respectively, and local feature descriptors corresponding to each background feature point are generated.

[0034] Calculate the feature distance between the local feature descriptors of each background feature point in the current video frame and the predicted video frame, and form an initial feature point pair with the background feature points whose feature distances satisfy a preset matching condition;

[0035] Geometric consistency screening is performed based on the relative positional relationship between each initial feature point pair, and initial feature point pairs that do not satisfy the same image transformation relationship are eliminated to obtain target feature point pairs;

[0036] The image registration parameters are solved using the target feature point pairs, and the current video frame and the predicted video frame are spatially aligned according to the image registration parameters.

[0037] The image difference between the current video frame and the predicted video frame after spatial alignment is calculated, candidate residual regions are extracted from the image difference, and real-world residuals are generated based on the candidate residual regions.

[0038] Further, extracting candidate residual regions from the image differences includes:

[0039] The motion response mapping is used to predict motion changes in historical video frames. The predicted historical video frames are then compared with the corresponding historical video frames to obtain mapping error samples.

[0040] The mapping error samples are partitioned and statistically analyzed according to the controlled structure region and background region in the video frame to obtain the allowable error range corresponding to each image region.

[0041] The image differences are compared with the allowable error range of the corresponding image region, and the difference pixels that exceed the allowable error range are retained;

[0042] Spatial connectivity aggregation is performed on the retained difference pixels, and discrete difference regions whose region size does not meet the preset size condition are filtered out to obtain candidate residual regions.

[0043] Further, the display terminal generates a reconstructed video frame based on the reference video frame, the motion response information, and the real-scene residual, including:

[0044] The controlled structural region and its corresponding current image change parameters are obtained from the action response information. The controlled structural region in the reference video frame is then mapped and resampled according to the current image change parameters to obtain a transformed structural image.

[0045] Using the image content in the reference video frame located outside the controlled structure region, the empty region formed after the controlled structure region is transformed is filled, and the transformed structure image is fused with the filled reference video frame to obtain the basic reconstructed video frame.

[0046] The residual region position and corresponding pixel difference are obtained from the real-scene residual. The residual region position is mapped to the basic reconstructed video frame. The pixel difference is superimposed on the corresponding pixel in the basic reconstructed video frame to obtain the reconstructed video frame.

[0047] Furthermore, when the real-scene residual meets the baseline update condition, the synchronization baseline is updated with the current video frame and the current action state, including:

[0048] Obtain the real-scene residuals corresponding to multiple consecutive current video frames, and associate the residual regions in adjacent real-scene residuals whose center distance meets the distance condition and whose region overlap ratio meets the overlap condition to obtain the continuous residual region.

[0049] The continuous residual features are obtained by statistically analyzing the number of consecutively occurring frames, the region coverage ratio, and the pixel difference in the continuous residual region.

[0050] When the number of consecutively occurring frames, the area coverage ratio, and the pixel difference satisfy the corresponding baseline update conditions, the corresponding current video frame is determined as a new reference video frame, and the corresponding current action state is determined as a new reference action state.

[0051] The new reference video frame is associated with the new reference motion state, and the synchronization reference is updated.

[0052] A synchronous interaction system for a mining explosion-proof host and a display terminal, used to realize the synchronous interaction method between the mining explosion-proof host and the display terminal, the system comprising:

[0053] The mapping and benchmark determination module constructs an action response mapping based on the historical action states of the controlled mining equipment and the corresponding historical video frames, and determines a synchronization benchmark composed of reference video frames and corresponding reference action states.

[0054] The action response and prediction module is used to generate action response information based on the current action state of the controlled mining equipment, the reference action state, and the action response mapping, and to generate predicted video frames based on the reference video frames and the action response information.

[0055] The real-scene residual determination module is used to determine the real-scene residual based on the difference between the current video frame corresponding to the current action state and the predicted video frame.

[0056] The synchronous interaction module is used to send the motion response information and the real-scene residual to the display terminal, and generate a reconstructed video frame based on the reference video frame, the motion response information and the real-scene residual;

[0057] The benchmark update module is used to update the synchronization benchmark with the current video frame and the current action state when the real-scene residual meets the benchmark update conditions.

[0058] Compared with the prior art, the beneficial effects of the present invention are:

[0059] This invention establishes a motion response mapping between changes in motion state and changes in video frames. It converts predictable changes in the screen caused by equipment actions into motion response information, and only transmits motion response information and real-world residuals. The display terminal then combines these with reference video frames to complete the reconstruction, reducing the repeated transmission of complete video frames, reducing the occupation of underground communication links, and reserving communication resources for control commands, status feedback, and alarm information.

[0060] This invention distinguishes between normal device operation, image offset, and real scene changes by background feature registration, partition error filtering, and residual region aggregation. It updates the synchronization benchmark based on the number of frames, coverage ratio, and pixel difference of continuous residuals, avoiding false updates triggered by instantaneous dust or local reflections, suppressing error accumulation caused by reference image mismatch, and improving the synchronization accuracy and continuity of reconstructed video frames. Attached Figure Description

[0061] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0062] Figure 1 This is a schematic diagram illustrating an application scenario of the synchronous interaction method between a mine explosion-proof host and a display terminal provided in an embodiment of the present invention.

[0063] Figure 2 A flowchart illustrating the synchronous interaction method between a mine explosion-proof host and a display terminal provided in an embodiment of the present invention;

[0064] Figure 3 A schematic diagram illustrating the generation process of motion response prediction video frames provided in an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of the synchronous interaction system between the mine explosion-proof host and the display terminal provided in an embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0067] Example 1

[0068] Please see Figure 1 This invention provides an application scenario for a synchronous interaction method between a mine explosion-proof host and a display terminal, applicable to the remote control and video monitoring of controlled mining equipment underground. This application scenario includes controlled mining equipment, a mine explosion-proof host, a switch, a display terminal, and an operating console.

[0069] The controlled mining equipment is equipped with video acquisition devices and motion status acquisition units. The video acquisition devices capture video frames of the controlled mining equipment itself and its surrounding working environment; the motion status acquisition units acquire the motion status of the robotic arm, telescopic mechanism, rotary mechanism, or other movable structures within the controlled mining equipment. The mining explosion-proof host receives the video frames and motion status data, and establishes communication connections with the display terminal and control console via a switch.

[0070] The mining explosion-proof host is used to generate motion response information and real-scene residuals based on the operating status of the controlled mining equipment and video frames, and then sends the motion response information and real-scene residuals to the display terminal via a switch. Therefore, Figure 1 The lines marked "Action Response Information, Real-Scene Residual" indicate the data transmission direction from the mine explosion-proof host through the switch to the display terminal. The display terminal is used to generate reconstructed video frames based on the received action response information and real-scene residual, combined with the locally stored synchronization reference, and to present the operating screen of the controlled mine equipment and the surrounding working environment.

[0071] The control panel receives operation actions input by the operator via buttons, joysticks, or knobs, and generates control commands based on these actions. These control commands are sent to the mining explosion-proof host via a switch to achieve remote control of the controlled mining equipment. The mining explosion-proof host can also return equipment status, command execution results, or alarm information to the control panel via the switch. The display terminal can also send synchronization requests, benchmark verification results, or interactive confirmation information to the mining explosion-proof host via the switch. The switch is primarily used to forward communication data between the mining explosion-proof host, display terminal, and control panel, and does not participate in video prediction and video reconstruction processing.

[0072] The explosion-proof host for mining can be a mining explosion-proof and intrinsically safe computer with an image Ethernet interface, a data interaction interface, and an equipment communication interface; the switch can be a mining intrinsically safe switch capable of mining network data exchange; the display terminal can be a mining explosion-proof and intrinsically safe monitor with Ethernet communication, video display, recording, and storage functions; and the control panel can be a mining intrinsically safe control panel with a display screen, buttons, joysticks, knobs, and a communication interface.

[0073] In the aforementioned application scenarios, the communication links between the mine explosion-proof host, switch, display terminal, and control console need to simultaneously carry video data, action response information, real-time residuals, control commands, equipment status, and alarm information. Due to mine wiring conditions, communication distance, interface transmission capabilities, and the concurrent transmission of multiple services, the data volume of continuous video frames is typically significantly higher than that of control commands and equipment status data. When continuously transmitting a complete video stream, video data can easily consume a large amount of communication resources, affecting the timely transmission of control commands and equipment feedback information; conversely, reducing the data volume simply by lowering the video frame rate or image quality may result in discontinuous equipment actions or missing details in the displayed image. Therefore, it is necessary to reduce the amount of video data transmitted while ensuring that the video image is synchronized with the actual actions of the controlled mine equipment.

[0074] The technical concept of this invention is based on the correlation between the controlled motion state of mining equipment and changes in video footage. When the video acquisition perspective is relatively stable, changes in the position, posture, and outline of the robotic arm, telescopic mechanism, or rotary mechanism in the video footage are primarily caused by corresponding changes in motion state, thus allowing for prediction based on the motion state. However, changes in the field, such as personnel entry, material movement, dust obstruction, light fluctuations, equipment vibration, or video acquisition equipment offset, cannot be fully represented by the equipment's motion state and will still cause discrepancies between the actual video frames and the motion prediction results.

[0075] Based on the above understanding, this embodiment establishes a synchronization benchmark using a reference video frame and its corresponding reference action state. The changes in the current video frame are categorized into predictable changes caused by the actions of the controlled mining equipment, and real-world changes that cannot be explained by action-response relationships. For predictable changes, the mining explosion-proof host generates action response information based on the current action state; for other differences in the image, the mining explosion-proof host determines the real-world residual based on the difference between the current video frame and the predicted video frame.

[0076] The mine explosion-proof host transmits action response information and real-scene residuals to the display terminal via a switch. The display terminal reconstructs the current video frame using the local synchronization reference, action response information, and real-scene residuals, thereby avoiding continuous transmission of complete video frames and reserving communication resources for control commands, equipment status, and alarm information. When the working environment or video acquisition perspective changes continuously, causing the real-scene residuals to meet the reference update conditions, the mine explosion-proof host updates the synchronization reference with the current video frame and current action status to maintain consistency in subsequent video prediction and video reconstruction.

[0077] It should be noted that the controlled mining equipment described in this embodiment is not limited to a specific type of equipment. As long as it has an action state that can be collected, and this action state can cause changes in the position, posture, or shape of the corresponding structural area in the video image, the synchronous interaction method disclosed in this invention can be applied. This embodiment also does not require a specific video resolution, communication protocol, switch port type, or equipment model as a prerequisite for implementation.

[0078] Please see Figure 2 The present invention provides an embodiment of a method for synchronous interaction between a mine explosion-proof host and a display terminal, the specific steps of which are as follows;

[0079] S1: Based on the historical action states of the controlled mining equipment and the corresponding historical video frames, construct the action response mapping between changes in action state and changes in video frames, and determine the synchronization benchmark composed of reference video frames and corresponding reference action states.

[0080] In this embodiment, the historical action status is output by the controller, encoder, or position detection unit of the controlled mining equipment, and is used to characterize the position, attitude, and action progress of the controlled structure during historical operation. Historical video frames are paired with historical action status according to the acquisition time, so that the action status and video frame in the same historical sample group correspond to the same running segment. For data with inconsistent sampling periods, alignment can be achieved by matching adjacent timestamps or time interpolation. This application does not further limit the specific time synchronization method.

[0081] The mining explosion-proof host compares historical sample groups according to the action execution sequence, identifies controlled structural regions whose position or shape changes with the action state from historical video frames, and extracts displacement, rotation, scale changes, or local deformation parameters of these regions. By associating the changes in historical action states with corresponding image change parameters, an action response map is formed. This map describes the deterministic impact of equipment actions on the video image, eliminating the need to treat all pixel changes of the controlled structure as unknown video content in subsequent interaction processes.

[0082] The synchronization benchmark is determined from historical samples where the equipment is in a stable operating state and the controlled structural area remains intact. Reference video frames provide the image basis for image reconstruction, while reference operating states provide the starting point for calculating changes in operating state; the two maintain a one-to-one correspondence. By unifying the video prediction basis on the mine explosion-proof host side and the video reconstruction basis on the display terminal side using this synchronization benchmark, structural position shifts caused by inconsistencies between the operating state benchmark and the reference image can be avoided.

[0083] S2: Obtain the current action state of the controlled mining equipment, generate action response information based on the current action state, the reference action state, and the action response mapping, and generate a predicted video frame based on the reference video frame and the action response information.

[0084] In this embodiment, the mine explosion-proof host calculates the change in the current action state relative to a reference action state and queries the action response map for the image change parameter corresponding to that change. When the change in the current action state is between two historical action state changes, interpolation can be performed based on the distance between it and the two historical action state changes to obtain the current image change parameter matching the current action state. The action response information consists of the current image change parameter and the controlled structural area it affects. Its data content is mainly used to indicate which structures in the image have changed and the corresponding change methods.

[0085] When generating the predicted video frame, the controlled structural region is extracted from the reference video frame. Coordinate mapping and resampling are performed on the pixels within the region according to the current image change parameters. Background content from the reference video frame is used to fill the gaps created by the structural movement. Finally, the transformed structural region is merged with the background region. The resulting predicted video frame represents the equipment view inferred from the current action state, without considering personnel entry, material movement, dust obstruction, or lighting changes. This process converts the main image changes caused by equipment movement into parameterized information, providing a basis for reducing the repeated transmission of complete video frames.

[0086] S3: Obtain the current video frame corresponding to the current action state, and determine the real-scene residual based on the difference between the current video frame and the predicted video frame.

[0087] In this embodiment, the current video frame and the current action state are correlated according to the acquisition time or control cycle. Considering that there may be installation vibration or slight displacement of the video acquisition equipment during mine operations, the mine explosion-proof host first extracts background feature points from the uncontrolled structural regions of the current video frame and the predicted video frame. Then, it obtains the image registration parameters through feature descriptor matching and geometric consistency screening, allowing the two video frames to be compared under a unified spatial position. This process is used to suppress the interference of overall image displacement on inter-frame differences.

[0088] After spatial alignment, the pixel differences between the current video frame and the predicted video frame are calculated. Combined with the allowable error range formed during historical prediction, fitting errors in the motion response mapping itself, image noise, and discrete difference points are excluded. The retained difference pixels are then connected and aggregated to form a real-scene residual with regional location and pixel difference values. This real-scene residual reflects changes in the scene that cannot be explained by the current action state, rather than repetitive image changes caused by normal device operation, thus concentrating the image data to be transmitted on the newly added or changed parts of the actual scene.

[0089] S4: The motion response information and the real-scene residual are sent to the display terminal, and the display terminal generates a reconstructed video frame based on the reference video frame, the motion response information and the real-scene residual.

[0090] In this embodiment, the mine explosion-proof host encapsulates the action response information and the real-scene residual into interactive data corresponding to the current video frame. The interactive data can be configured with frame number, acquisition time, and synchronization reference identifier, so that the display terminal can identify the correspondence between each data point and the reference video frame. Those skilled in the art will understand that the interactive data can be transmitted using existing Ethernet messages or mine communication messages; this application does not further limit the message format or transmission protocol.

[0091] The display terminal performs coordinate transformation on the controlled structural area in the local reference video frame, consistent with that on the mining explosion-proof host side, based on the action response information, to form a basic reconstructed video frame. Then, according to the area position in the real-scene residual, the corresponding pixel difference is superimposed on the basic reconstructed video frame to obtain the reconstructed video frame. The action response information is responsible for restoring structural changes that can be predicted by the equipment status, while the real-scene residual is responsible for supplementing unpredictable on-site changes, so that the communication link no longer continuously carries a complete video frame, while reserving transmission resources for control commands, status feedback, and alarm information.

[0092] S5: When the real-scene residual meets the baseline update condition, update the synchronization baseline with the current video frame and the current action state.

[0093] In this embodiment, the mine explosion-proof host performs temporal correlation on residual regions in consecutive video frames and counts the number of consecutive frames in which the residual regions appear, the region coverage ratio, and the pixel difference. When the residual only appears in a single frame or does not form a continuous correlation, the original synchronization reference is continued to be used to avoid instantaneous dust, local reflections, or acquisition noise triggering reference switching; when the continuous residual characteristics reach the corresponding update conditions, it indicates that the original reference video frame can no longer accurately represent the current working scene.

[0094] At this point, the current video frame that meets the update conditions is used as the new reference video frame, the corresponding current action state is used as the new reference action state, and the synchronization reference identifier is updated. After acquiring the new reference video frame, the display terminal switches to the corresponding synchronization reference, and subsequent action response information and real-scene residuals are generated relative to the new synchronization reference. Through reference updates, the accumulation of residuals caused by long-term changes in the on-site environment can be eliminated, ensuring that action prediction, residual extraction, and video reconstruction remain continuous and effective.

[0095] Preferably, the step of constructing an action response mapping between changes in action state and changes in video frames based on the historical action states of the controlled mining equipment and the corresponding historical video frames includes:

[0096] S1.1.1: According to the time correspondence, each historical action state and its corresponding historical video frame are combined into a historical sample group, and the historical sample group is arranged according to the action execution order.

[0097] S1.1.2: Based on the historical sample pairs in the historical sample group, determine the amount of change in the historical action state, and determine the controlled structural region that changes with the historical action state from the corresponding previous and subsequent historical video frames.

[0098] S1.1.3: Extract the image change parameters of the controlled structure region between the preceding and following historical video frames, establish the correspondence between the historical action state change amount and the image change parameters, and obtain the action response mapping.

[0099] In this embodiment, the historical motion status refers to the motion feedback data generated by the controlled mining equipment during completed operations, and its data content corresponds to the structural form of the controlled mining equipment. For example, for a mining robotic arm with rotation, pitch, and extension movements, the historical motion status can be characterized by the rotation angle, pitch angle, and extension length, respectively; for mining equipment with lifting or linear pushing mechanisms, the historical motion status can be characterized by the lifting height or linear displacement. The historical motion status is output by the equipment controller, encoder, or position detection unit. The same moving component uses a fixed data unit and sampling reference to ensure that the motion status in different operation processes can be directly compared.

[0100] Historical action states and historical video frames each carry their acquisition time. Using the acquisition time of a historical video frame as the pairing time, the historical action state with the closest time distance is searched in the action state sequence. When the time interval between the two does not exceed a time matching threshold, they are grouped into a historical sample group. The time matching threshold is set based on the video frame acquisition period and the action state acquisition period, and in principle, it should not exceed half of the longer acquisition period. For example, when the video acquisition frame rate is 25 frames per second and the action state sampling frequency is 50 Hz, the time matching threshold can be set to 20 milliseconds. If the pairing time lies between two action state sampling times, time interpolation is performed based on the two sampling times and their action states to obtain a historical action state consistent with the historical video frame acquisition time.

[0101] After time matching is completed, the motion execution process is divided according to the motion command type, motion component, and motion direction, and historical sample groups are arranged according to the acquisition time within each motion execution process. For historical sample groups with opposite rotation directions, extension and retraction, or rise and fall, independent motion sequences are formed to avoid mutual cancellation of image change parameters corresponding to different motion directions. The start and end positions of the motion execution process can be determined by the start and end markers of the control command, or by the moment when the motion state changes from a stable state to a continuously changing state and then back to a stable state.

[0102] Historical sample pairs are formed by selecting two historical sample groups from the same action execution process, one earlier and one later. The interval between historical sample pairs is set according to the action state detection resolution and the video image spatial resolution, ensuring that the change between the preceding and following historical action states exceeds the minimum effective resolution of the corresponding sensor, and that the displacement of the controlled structure in the image reaches at least one pixel. Taking a video of 25 frames per second as an example, historical sample pairs can initially be formed using historical sample groups spaced 1 to 5 video frames apart; when the device's action speed changes, the sample interval is adjusted according to the above effective resolution conditions, without fixing a specific number of frames.

[0103] The change in historical motion state is determined by the historical motion state of the subsequent historical sample group relative to the historical motion state of the previous historical sample group. For a single motion component, the change in historical motion state is expressed as an angle change, displacement change, or length change. For multiple motion components moving together, the state change components of each motion component are calculated separately and arranged in a fixed order to form the change in historical motion state. To avoid the influence of different dimensions on the sample distance calculation, each state change component is scaled uniformly according to the allowable motion range of the corresponding motion component, and the value range after scaling is set to 0 to 1.

[0104] Before determining the controlled structure region, image alignment is performed using fixed background features from preceding and following historical video frames to eliminate overall image shift caused by vibration of the video acquisition equipment. Image alignment can be achieved by feature point matching combined with affine transformation or perspective transformation. Those skilled in the art can choose the appropriate registration method according to the installation method of the video acquisition equipment; this application does not limit the specific feature point operator. After alignment is completed, pixel displacement or feature point displacement in preceding and following historical video frames is calculated, and pixels or feature points whose displacement exceeds the image motion threshold and whose motion direction is consistent with the direction of change of the historical action state are selected as candidate points for controlled structures.

[0105] The image motion threshold is determined based on historical video frames when the equipment is in a stable operating state. Specifically, the pixel displacements of consecutive video frames in a stable operating state are statistically analyzed, and the 95th percentile of the pixel displacement distribution is used as the image motion threshold. This prevents pixel changes caused by camera noise, compression noise, and local illumination fluctuations from being identified as structural motion. Spatial connectivity aggregation is performed on candidate points of the controlled structure, and combined with the equipment outline range of the controlled mining equipment in the video frame, areas containing personnel, materials, and dust located outside the equipment outline are filtered out to obtain the controlled structure region. When a controlled mining equipment has multiple independent moving parts, multiple controlled structure regions are divided according to the movement direction and spatial connectivity of the candidate points, and each controlled structure region is assigned a fixed region identifier.

[0106] Image transformation parameters describe the image transformation relationship of a controlled structural region as it changes position from the previous historical video frame to the next. For controlled structural regions whose contours remain largely unchanged, corresponding feature points are extracted from the preceding and following historical video frames, and translation, rotation, and scale changes are fitted based on the coordinate correspondence of these feature points. For controlled structural regions composed of multiple connected components or exhibiting local pose changes, the region is divided into multiple interconnected sub-regions, and local coordinate transformation parameters are fitted to each sub-region. During the fitting process, mismatched feature points that do not satisfy the same coordinate transformation relationship are removed, and the effectiveness of the image transformation parameters is evaluated by the reprojection deviation of the retained feature points.

[0107] For multiple historical sample pairs corresponding to the same moving parts, movement directions, and similar historical movement state changes, their image change parameters are merged. During merging, samples whose historical movement state changes fall within the same state change interval are grouped together. The median value of each image change parameter in the same group is used as the representative parameter of that state change interval to reduce the impact of single occlusion, illumination fluctuations, and image matching errors. The width of the state change interval is set according to the movement state detection resolution. For example, when the detection resolution for rotation angle is 0.1 degrees, the width of the state change interval can be set to 0.5 degrees; when the detection resolution for extension length is 1 millimeter, the width of the state change interval can be set to 5 millimeters.

[0108] Motion response mapping uses motion components, motion directions, and controlled structural regions as indices to record the correlation between the amount or range of historical motion state changes and the corresponding image change parameters. For continuous motion states, mapping nodes are saved in numerical order of historical motion state changes, enabling parameter interpolation between adjacent mapping nodes. For motion states with discrete levels, corresponding image change parameters are saved according to the change relationship of each level. The effective range of motion response mapping is limited by the minimum and maximum historical motion state changes in the historical samples. Motion state changes exceeding this range are not extrapolated and require the addition of historical samples corresponding to the motion range to expand the mapping range.

[0109] Therefore, the motion response mapping does not represent the statistical changes between ordinary video frames, but rather the correspondence between the changes in the motion state of controlled mining equipment and the changes in the controlled structure in the picture, providing a callable data basis for recovering predictable picture changes based on the current motion state.

[0110] Preferably, determining the synchronization reference composed of reference video frames and corresponding reference action states includes:

[0111] S1.2.1: Based on the changes in the historical action state of each historical sample pair, determine the candidate historical sample group in which the controlled mining equipment is in a stable action state.

[0112] S1.2.2: Determine the reference video frame based on the controlled structural region integrity and image quality of the historical video frames in the candidate historical sample group.

[0113] S1.2.3: Determine the historical action state corresponding to the reference video frame as the reference action state, and associate the reference video frame with the reference action state to obtain a synchronization reference.

[0114] In this embodiment, a stable operating state refers to a situation where the moving parts of the controlled mining equipment do not undergo effective displacement or attitude changes exceeding the state detection error range during a continuous sampling period. A stable operating state does not limit the controlled mining equipment to a stopped state; when the equipment maintains a constant rotation angle, extension length, pitch angle, or other defined attitude, it can also be considered a stable operating state. Establishing a synchronization reference using historical video frames corresponding to the stable operating state ensures a definite correspondence between the controlled structure outline in the reference video frame and the reference operating state, avoiding the use of momentarily blurred images during the operation as reference objects.

[0115] The stabilization conditions are set based on the detection resolution of the motion state acquisition unit and the fluctuation range of the static state. Specifically, from the historical motion states of the controlled mining equipment known to be in a static or attitude-maintaining state, the state difference between adjacent sampling times is statistically analyzed. The 95th percentile of the absolute value of the state difference is compared with the minimum resolution of the motion state acquisition unit, and the larger of the two values ​​is taken as the stabilization threshold for the corresponding motion state component. Taking a rotation angle detection resolution of 0.1 degrees as an example, if the 95th percentile of the difference between adjacent sampling angles in the static state is 0.16 degrees, then the rotation angle stabilization threshold can be set to 0.16 degrees; if the percentile is 0.06 degrees, then the stabilization threshold can be set to 0.1 degrees.

[0116] For controlled mining equipment containing multiple motion state components, each motion state component is assessed to determine whether it meets its corresponding stability threshold. When the changes in each historical motion state in a continuous historical sample pair do not exceed the corresponding stability threshold, the historical sample group covered by the continuous historical sample pair is taken as a candidate historical sample group. The duration of the continuous stable period is set based on the video acquisition frame rate and the equipment's action response speed, and it should cover a sufficient number of historical video frames for image quality comparison. For example, when the video acquisition frame rate is 25 frames per second, a stable state can be required to be maintained for at least 5 consecutive frames, corresponding to a duration of at least 200 milliseconds; for equipment with an action inertia duration exceeding 200 milliseconds, the duration can be set as the time required for the vibration to decay to a stable range after the action stops.

[0117] After the candidate historical sample group is determined, the controlled structure region integrity of the historical video frames is checked first. The controlled structure region integrity characterizes the effective visibility of the controlled structure in the current historical video frame, and its value ranges from 0 to 1. Historical video frames of the same motion component in the same or adjacent motion states can be spatially aligned, and a structural visibility envelope is formed based on the union of the controlled structure regions in each frame. The ratio of the area of ​​the unoccluded controlled structure region, which does not exceed the screen boundary and has valid pixels, to the area of ​​the structural visibility envelope in the current historical video frame is used as the controlled structure region integrity.

[0118] The effective pixels of the controlled structural region can be determined based on contour continuity, local texture consistency, and occlusion detection results. For areas covered by personnel, materials, or dust, local areas with a duration insufficient to the preset occlusion duration and inconsistent with the structural texture can be marked as occluded areas by comparing with other historical video frames under the same stable action state. For structural parts that extend beyond the video frame boundary, they are marked as invisible areas based on the intersection relationship between the contour of the controlled structural region and the frame boundary. For example, the controlled structural region integrity threshold can be set to 0.90, meaning that at least 90% of the visible envelope of the controlled structural region has effective image content. This threshold can be adjusted according to the importance of the controlled structure in video reconstruction; for structural regions whose contour boundaries directly participate in coordinate transformation, it can be set to 0.95.

[0119] Image quality, used to characterize whether historical video frames are suitable as reference images for long-term preservation, can be evaluated by sharpness, exposure effectiveness, and noise level. Sharpness can be determined based on the average response of edge gradients within a controlled structural region; exposure effectiveness can be determined based on the proportion of pixels whose pixel values ​​fall within the effective brightness range, which can be set in conjunction with the quantization range of the video acquisition device. For example, for an 8-bit grayscale or luminance channel, pixels with pixel values ​​between 16 and 239 can be considered as effectively exposed pixels; noise level can be determined based on the difference between adjacent pixels within a stable background region or the pixel fluctuations in consecutive frames.

[0120] Sharpness, exposure effectiveness, and noise level are each converted into evaluation values ​​ranging from 0 to 1, and then weighted and combined to obtain the image quality evaluation value. The weights are set according to the impact of each indicator on the reconstruction accuracy of the reference video frame. For example, the sharpness weight can be set to 0.5, the exposure effectiveness weight to 0.3, and the noise evaluation weight to 0.2. Sharpness directly affects the boundary extraction and pixel resampling of the controlled structural area, so its weight is higher than the other indicators. If there are frequent lighting fluctuations in the video acquisition environment, the exposure effectiveness weight can be increased, but the sum of all weights remains 1.

[0121] The selection of reference video frames involves first performing integrity screening, followed by image quality sorting. Historical video frames whose controlled structural region integrity reaches the integrity threshold are retained as valid candidate frames and arranged in descending order of image quality evaluation value. The valid candidate frame with the highest image quality evaluation value is determined as the reference video frame. When multiple valid candidate frames have the same image quality evaluation value, the historical video frame with the smallest distance between its corresponding historical action state and the median value of the action state within the stable period is selected first to reduce the impact of action state sampling fluctuations on the synchronization benchmark. When no historical video frame meeting the integrity threshold is found in the candidate historical sample group, a new candidate historical sample group is selected from adjacent stable action periods, and the synchronization benchmark is not established based on historical video frames with obvious occlusion or structural defects.

[0122] The reference action state is directly taken from the historical action state corresponding to the reference video frame time. If the acquisition time of the reference video frame is between two action state sampling times, the time interpolation result used when constructing the historical sample group is used, and the action state at other sampling times is not reused. For controlled mining equipment with multiple action components, the reference action state records the state components of each action component in a preset order, so that each state component is consistent with the action state field in the action response mapping.

[0123] When a reference video frame is associated with a reference action state, the reference video frame identifier, acquisition time, reference action state, controlled structure area identifier, and action response mapping version can be recorded together to form synchronization reference data. The mine explosion-proof host and display terminal use the same synchronization reference identifier to call the reference video frame; if the display terminal does not save the corresponding reference video frame, the mine explosion-proof host sends a reference video frame once when the synchronization reference is established. After the synchronization reference is established, action response information and real-world residuals are interpreted based on this synchronization reference until the reference update conditions are met and a new synchronization reference is generated.

[0124] Preferably, action response information is generated based on the current action state, the reference action state, and the action response mapping, including:

[0125] S2.1.1: Determine the change amount of the current action state based on the current action state and the reference action state.

[0126] S2.1.2: Determine the first historical action state change and the second historical action state change that are adjacent to the current action state change from the action response mapping, and obtain the first image change parameter and the second image change parameter corresponding to the two respectively.

[0127] S2.1.3: Determine the parameter interpolation coefficients based on the distance relationship between the current action state change amount and the first historical action state change amount and the second historical action state change amount.

[0128] S2.1.4: Interpolate the first image change parameter and the second image change parameter based on the parameter interpolation coefficient to obtain the current image change parameter.

[0129] S2.1.5: Use the current image change parameters and their corresponding controlled structural regions as action response information.

[0130] In this embodiment, the current action state and the reference action state use action state fields consistent with the action response mapping. Taking a controlled mining equipment with a slewing arm and a telescopic arm as an example, the action state can sequentially record the slewing angle, telescopic length, and pitch angle. The change in the current action state represents the angular or displacement change of each moving component from the reference action state to the current action state, and retains the action direction. For example, if the telescopic length in the reference action state is 120 mm and the telescopic length in the current action state is 440 mm, then the change in the current action state corresponding to the telescopic length is a change of 320 mm along the extension direction; if the current telescopic length is 80 mm, then it corresponds to a change of 40 mm along the retraction direction.

[0131] For periodic angular states, the angle change is determined according to the actual direction of movement. With a reference rotation angle of 355 degrees, a current rotation angle of 5 degrees, and motion feedback indicating that the slewing arm crosses the zero-degree position in the positive direction, the rotation angle change is 10 degrees in the positive direction. This avoids misinterpreting continuous movements crossing the zero angle point as a reverse 350-degree movement. When multiple moving parts move simultaneously, the change components of each motion state are arranged according to a predetermined field order to form the current motion state change, and this field order is consistent with the field order of historical motion state changes.

[0132] The motion response mapping records the historical motion state changes and their corresponding image change parameters according to the motion component, motion direction, and controlled structure region. The first and second historical motion state changes are not arbitrary historical values, nor are they named according to the order of acquisition time. Instead, they refer to two historical mapping nodes adjacent to the current motion state change within the mapping branch corresponding to the same motion component, the same motion direction, and the same controlled structure region.

[0133] For an action process represented by a single action variable, the mapping nodes can be arranged according to the numerical order of the changes in historical action states. The historical action state change that is no more than the current action state change and is closest to it is designated as the first historical action state change, and the historical action state change that is no less than the current action state change and is closest to it is designated as the second historical action state change. For example, if the current change in extension length is 320 mm, and the historical changes adjacent to it in the action response mapping are 300 mm and 350 mm, then 300 mm is the first historical action state change, and 350 mm is the second historical action state change.

[0134] For an action process characterized by multiple action variables, the state change components are first scaled uniformly according to the allowable action range of each action variable. Then, the state distance between the current action state change and each historical action state change is calculated. Two historical mapping nodes with the smallest state distance and which can form an interpolation interval near the current action state change are selected as the first and second historical action state changes, respectively. If the action response mapping is established separately for each action variable, adjacent historical mapping nodes can be selected independently for each action variable, and then the image change results can be merged according to the corresponding controlled structural region.

[0135] The first image transformation parameter is the image transformation result of the controlled structure region corresponding to the change in the first historical action state, and the second image transformation parameter is the image transformation result of the controlled structure region corresponding to the change in the second historical action state. The image transformation parameters may include the horizontal displacement, vertical displacement, rotation angle, and scale change ratio of the controlled structure region in the video frame coordinate system; when the controlled structure undergoes local deformation, it may also include local displacement or local coordinate transformation parameters of the structural sub-region. The first and second image transformation parameters use the same field structure and measurement method, enabling corresponding interpolation between identical fields.

[0136] For example, when the telescopic arm extends 300 mm, its controlled structural area moves horizontally by 42 pixels, vertically by -6 pixels, and rotates clockwise by 1.8 degrees relative to the reference video frame, with a scale change ratio of 1.04. These values ​​constitute the first image change parameters. When the telescopic arm extends 350 mm, the controlled structural area moves horizontally by 49 pixels, vertically by -7 pixels, and rotates clockwise by 2.1 degrees, with a scale change ratio of 1.05. These values ​​constitute the second image change parameters. The -6 pixels and -7 pixels represent the movement of the controlled structural area along the decreasing direction of the video frame's vertical coordinate.

[0137] The interpolation coefficients represent the relative position of the current action state change between the first and second historical action state changes, and their values ​​range from 0 to 1. The closer the current action state change is to the first historical action state change, the larger the proportion of the first image change parameter in the interpolation result; the closer the current action state change is to the second historical action state change, the larger the proportion of the second image change parameter in the interpolation result.

[0138] Taking the aforementioned stretching length as an example, the current change of 320 mm falls between 300 mm and 350 mm, and the change range of 300 mm to 350 mm accounts for 40%. Therefore, the interpolation coefficient for the second image change parameter is 0.4, and the interpolation coefficient for the first image change parameter is 0.6. After interpolating the horizontal displacement, vertical displacement, rotation angle, and scale change ratio, we obtain a horizontal displacement of 44.8 pixels, a vertical displacement of -6.4 pixels, a rotation angle of 1.92 degrees, and a scale change ratio of 1.044. When there are non-integer results in the pixel displacement, pixel resampling is used when generating the predicted video frame, and integer rounding is not performed during the motion response information generation stage.

[0139] When the change in the current motion state matches a historical change in the motion state, the image change parameters corresponding to that historical change are directly used without interpolation. When the change in the current motion state exceeds the effective range of the motion response mapping, boundless extrapolation is not performed using the two outermost historical mapping nodes; instead, the corresponding controlled structure region is marked as an area outside the mapping range. The actual image change in this region can be propagated by the real-scene residual, or the motion response mapping can be supplemented after obtaining historical samples covering this motion range to prevent the extrapolation result from deviating from the actual image position of the controlled structure.

[0140] The current image change parameters are organized according to the controlled structural regions. When a motion component corresponds to a controlled structural region, the region identifier is associated with the horizontal displacement, vertical displacement, rotation angle, and scale change ratio obtained by interpolation. When a motion component corresponds to multiple interconnected structural sub-regions, the identifiers of each structural sub-region and their local image change parameters are recorded respectively, and the connection relationship between each structural sub-region is preserved.

[0141] The motion response information consists of a controlled structure region identifier, current image change parameters, and a synchronization reference identifier. The controlled structure region identifier indicates the specific image region that the display terminal should process from the reference video frame. The current image change parameters describe the image transformation required when this region changes from the reference motion state to the current motion state. The synchronization reference identifier ensures that the motion response information is consistent with the corresponding reference video frame and reference motion state. Therefore, the first historical motion state change, the second historical motion state change, and the corresponding two sets of image change parameters are all implemented as specific historical sample nodes in the motion response mapping, rather than being used merely as abstract interpolation objects.

[0142] For preferred options, please refer to [link / reference]. Figure 3 This invention provides a schematic diagram of the motion response prediction video frame generation process. Generating a prediction video frame based on the reference video frame and the motion response information includes:

[0143] S2.2.1: According to the controlled structural region in the action response information, the reference video frame is segmented to obtain a structural region image and a background region image.

[0144] S2.2.2: Convert the current image change parameters into coordinate transformation relationships for the corresponding controlled structural region, and perform coordinate mapping and resampling on the pixels in the structural region image according to the coordinate transformation relationships to obtain a transformed structural image.

[0145] S2.2.3: Use neighboring background pixels in the background region image to fill the empty area formed after the controlled structure region is displaced.

[0146] S2.2.4: According to the structural connection relationship between the controlled structural regions, the transformed structural image and the filled background region image are spliced ​​and fused to obtain the predicted video frame.

[0147] In this embodiment, the action response information carries a controlled structural region identifier and corresponding current image change parameters. The controlled structural region identifier is consistent with the region range determined when constructing the action response mapping, and can be represented by region boundary coordinates, region pixel set, or region mask. The region mask has the same pixel size as the reference video frame. Pixels within the controlled structural region are assigned valid identifiers, while pixels outside the region are assigned invalid identifiers, thereby extracting the structural region image from the reference video frame; the portion of the reference video frame not covered by the region mask forms the background region image.

[0148] When a controlled mining equipment comprises multiple movable parts, a controlled structural region is configured for each movable part. For robotic arm-type equipment, the regions can be divided according to the structural hierarchy of the fixed base, rotary arm, telescopic arm, and end effector. For lifting mechanisms, the lifting platform and its follower connectors can be included in the corresponding controlled structural region. The boundaries of adjacent regions can be set along the actual connection positions of the equipment components, and a preset width of overlapping pixel bands is reserved on both sides of the region boundaries. The overlapping pixel bands are used to handle the boundary connection after region transformation, and their width can be set according to the video resolution and structural contour extraction error, for example, 3 to 5 pixels in a 1920×1080 pixel video frame.

[0149] The horizontal displacement, vertical displacement, rotation angle, and scale change ratio in the current image change parameters are converted into coordinate transformation relationships in the reference video frame coordinate system. For controlled structural regions whose contours remain unchanged, two-dimensional rigid transformation or affine transformation can be used; for regions where local morphological changes occur due to stretching, joint rotation, etc., the controlled structural region can be divided into several structural sub-regions, and local coordinate transformation relationships can be configured for each. Those skilled in the art can select existing coordinate transformation methods according to the motion form of the controlled structure; this application does not limit the specific transformation matrix type.

[0150] Coordinate mapping takes the pixel coordinates of the structural region in the reference video frame as input and calculates its target coordinates in the prediction video frame. Target coordinates typically contain non-integer values, therefore pixel resampling is required at the target location. For example, bilinear interpolation can be used, assigning pixel weights based on the distances to four integer pixel locations surrounding the target coordinates to obtain the pixel value at the target location. For structural contour edges, resampling can be performed synchronously in conjunction with region masks to ensure that the transformed structural contour matches the transformed structural image, preventing misalignment between the structural image and region boundaries.

[0151] After the controlled structure region moves from the reference position to the target position, a gap is formed at the original position. The extent of the gap is determined by the difference between the region mask before and after the transformation. During filling, neighboring background pixels are extracted from the background region image outside the boundary of the gap, and filling weights are assigned according to the spatial distance between the neighboring background pixels and the pixels to be filled. For tunnel walls, ground surfaces, or fixed equipment shells with continuous textures, neighborhood interpolation can be used to complete the filling; for backgrounds with obvious edge directions, background pixels can be selected along the edge extension direction for filling to reduce edge interruptions.

[0152] The selection range of neighboring background pixels is set according to the width of the missing area. For example, a background band with a width of 5 to 15 pixels can be selected outside the boundary of the missing area, and background pixels located in the same texture direction as the pixel to be filled are preferred. When the selection range contains people, materials or other unstable targets, the corresponding pixels can be excluded based on the controlled structure area and fixed background area division results in the reference video frame, and only the effective pixels in the background area image are retained for filling.

[0153] When combining multiple transformed structural images, the processing order is determined according to the structural connection relationships between the controlled structural regions. Taking a robotic arm as an example, processing can proceed sequentially from the fixed base to the end effector, ensuring that the connection ends of later structural regions correspond to the connection ends of earlier structural regions. If different transformed structural images overlap at the target location, the coverage order is determined based on the front-to-back occlusion relationship of the device structure; structural regions closer to the video acquisition viewpoint cover structural regions located behind them.

[0154] At the junction of the transformed structural image and the filled background area image, an overlapping pixel band is set to cover both the inner and outer sides of the structural area boundary. The width of the overlapping pixel band is determined based on the boundary positioning error of the controlled structural area and the pixel resampling range. Specifically, while the device maintains a stable operating state, the contour positions of the same controlled structural area in consecutive video frames are statistically analyzed, and the 95th percentile value of the corresponding contour point position deviation is taken as the boundary positioning error; the width of the overlapping pixel band is the sum of twice the boundary positioning error and the resampling neighborhood radius, rounded up to an integer pixel. When bilinear resampling is used and the boundary positioning error is 1.4 pixels, the resampling neighborhood radius is 1 pixel, and the width of the overlapping pixel band can be set to 4 pixels.

[0155] For any target pixel within the overlapping pixel band, calculate the shortest pixel distance from the target pixel to both the inner and outer boundaries of the overlapping pixel band. The inner boundary is the boundary where the pixel values ​​of the transformed structure image are fully utilized, and the outer boundary is the boundary where the pixel values ​​of the filled background region image are fully utilized. The ratio of the target pixel's distance to the outer boundary to the sum of its distances to the inner and outer boundaries is used as the structure pixel weight, and the ratio of the target pixel's distance to the inner boundary to the sum of these distances is used as the background pixel weight. Both weights range from 0 to 1, and their sum is 1. The closer the target pixel is to the inner boundary, the closer its structure pixel weight is to 1; the closer it is to the outer boundary, the closer its background pixel weight is to 1.

[0156] For example, if the overlap pixel band width is 4 pixels, the shortest distance from a target pixel to its inner boundary is 1 pixel, and the shortest distance to its outer boundary is 3 pixels, then the structure pixel weight is 0.75, and the background pixel weight is 0.25. The pixel values ​​of each color channel at the target location in the transformed structure image and the filled background region image are multiplied by their respective weights and then summed to obtain the fused pixel value. For 8-bit images, if the fusion result exceeds the range of 0 to 255, it is limited to that range and rounded to an integer pixel value.

[0157] Pixels located inside the overlapping pixel band directly use the pixel values ​​of the transformed structure image, while pixels located outside the overlapping pixel band directly use the pixel values ​​of the filled background region image. When multiple controlled structure regions overlap at the target location, the transformed structure images are first covered according to the structural connection relationship and a predetermined occlusion order. Then, the boundary fusion described above is performed on the outermost structure contour and the background region. After completing the coordinate transformation of the controlled structure regions, filling of missing regions, and splicing boundary fusion, the predicted video frame is obtained.

[0158] Preferably, determining the real-scene residual based on the difference between the current video frame and the predicted video frame includes:

[0159] S3.1: Extract background feature points from the uncontrolled structural regions of the current video frame and the predicted video frame respectively, and generate local feature descriptors corresponding to each background feature point.

[0160] S3.2: Calculate the feature distance between the local feature descriptors of each background feature point in the current video frame and the predicted video frame, and form an initial feature point pair by matching the background feature points whose feature distances meet the preset matching conditions.

[0161] S3.3: Based on the relative positional relationship between each initial feature point pair, perform geometric consistency screening, remove initial feature point pairs that do not satisfy the same image transformation relationship, and obtain the target feature point pairs.

[0162] S3.4: Solve the image registration parameters using the target feature point pairs, and spatially align the current video frame and the predicted video frame according to the image registration parameters.

[0163] S3.5: Calculate the image difference between the current video frame and the predicted video frame after spatial alignment, extract candidate residual regions from the image difference, and generate real-world residuals based on the candidate residual regions.

[0164] In this embodiment, the uncontrolled structure region refers to the image area in the video frame that is not covered by the controlled structure region mask. This mainly corresponds to the tunnel walls, ground, fixed supports, and the surfaces of fixed equipment that do not change position with equipment movement. The controlled structure region does not participate in image registration to prevent normal movements of robotic arms, telescopic mechanisms, or rotating components from being mistakenly interpreted as overall displacement of the video acquisition equipment. For non-fixed targets such as personnel, transported materials, and dust clumps, they can be temporarily excluded from the uncontrolled structure region based on motion detection results in consecutive video frames.

[0165] Background feature points are extracted from pixel locations with abrupt grayscale changes or texture variations. This embodiment employs the ORB feature extraction method to detect corner points in the grayscale images of the current and predicted video frames, and generates binary local feature descriptors based on the grayscale comparison results within the corner point's neighborhood. The number of feature points can be set according to the video resolution. For example, for a 1920×1080 pixel video frame, the maximum number of feature points is set to 1000, and uncontrolled structural regions are divided into regular grids. Each grid retains the feature points ranked first by response value to prevent feature points from concentrating in a single texture region. When using other existing feature extraction methods that can output feature locations and local feature descriptors, the feature extraction range is also limited by the uncontrolled structural region mask.

[0166] For any background feature point in the current video frame, calculate the feature distance between its local feature descriptor and each background feature point in the predicted video frame. When using binary descriptors, the feature distance is the number of different binary bits in the two descriptors. The nearest and second nearest matching feature points are determined according to the feature distance from smallest to largest. If the ratio of the nearest to the second nearest feature distance does not exceed 0.75, the matching relationship is retained. The ratio threshold is set based on the registration samples collected under stable device conditions and can be selected between 0.70 and 0.80; this embodiment uses 0.75 to exclude feature points that do not have a unique correspondence in repetitive texture regions.

[0167] To prevent multiple current background feature points from pointing to the same predicted background feature point, a bidirectional consistency check is performed. A feature point in the current video frame is matched with a feature point in the predicted video frame, and this predicted feature point is also matched with the current feature point during reverse retrieval; only then do these two form an initial feature point pair. The initial feature point pair obtained after this process may still contain incorrect matches caused by dust occlusion, localized reflections, and repetitive textures; therefore, further filtering is performed based on the spatial relationships between feature points.

[0168] Geometric consistency screening employs a random sampling consistency method. Each time, feature point pairs that meet the quantity requirements for solving image transformation parameters are extracted from the initial feature point pairs. The candidate transformation relationship between the current video frame and the predicted video frame is solved, and then the reprojection positions of the remaining initial feature points after the candidate transformation are calculated. If the distance between the reprojection position and the corresponding feature point position does not exceed the reprojection error threshold, the initial feature point pair is recorded as a consistent feature point pair. The reprojection error threshold is set based on the video resolution and the background feature point positioning error; for a 1920×1080 pixel video frame, it can be set to 3 pixels.

[0169] The random sampling number can be set to 500, and the candidate transformation relation with the largest number of consistent feature point pairs is selected as the target transformation relation. When the number of consistent feature point pairs is the same, the candidate transformation relation with the smallest average reprojection error of consistent feature point pairs is selected. The target feature points correspond to no fewer than 12 pairs, and the distribution range covers no less than 30% of the width and height of the video frame to avoid determining the image transformation based solely on feature points in a local area. If the above conditions are not met, the current video frame does not undergo registration transformation, and the valid image registration parameters corresponding to the previous current video frame are used.

[0170] When the video acquisition equipment is fixedly installed and mainly affected by translation, slight rotation, and imaging scale fluctuations, the image registration parameters adopt two-dimensional affine transformation parameters, including horizontal translation, vertical translation, planar rotation, scaling, and shearing components. Based on the target feature point pairs, the image registration parameters are re-solved using the minimum reprojection error criterion, and these parameters are used to transform the current video frame to the coordinate space of the predicted video frame. When the spatial transformation generates non-integer pixel coordinates, bilinear interpolation is used to determine the pixel values; positions outside the original video frame range after transformation are marked as invalid pixels and do not participate in image difference calculations.

[0171] After spatial alignment, the two video frames are converted to the same color space, and the brightness and color differences are calculated pixel by pixel. To avoid a single color channel change directly forming a residual, the brightness difference is used as the primary difference component, and the differences between the two color components are used as supplementary components, which are then converted into image differences with the same size as the video frames. Invalid pixels, preset fusion bands at controlled structural boundaries, and registration areas with insufficient feature points are not directly used as candidate residual regions.

[0172] Image differences are filtered according to the allowable error range corresponding to the controlled structural region and the background region. Pixels with differences exceeding the allowable error range are retained, and adjacent pixels with differences are spatially connected and aggregated. Hole filling and boundary smoothing are performed on the connected regions to obtain candidate residual regions. The real-scene residual records the position of each candidate residual region in the coordinate space of the predicted video frame, the region mask, and the pixel difference between the current video frame and the predicted video frame. Thus, the display terminal can superimpose the corresponding pixel difference onto the corresponding position of the basic reconstructed video frame according to the position of the residual region.

[0173] Preferably, extracting candidate residual regions from the image differences includes:

[0174] S3.5.1: Use the motion response mapping to predict motion changes in historical video frames, compare the predicted historical video frames with the corresponding historical video frames, and obtain mapping error samples.

[0175] S3.5.2: Perform partitioned statistics on the mapping error samples according to the controlled structure region and background region in the video frame to obtain the allowable error range corresponding to each image region.

[0176] S3.5.3: Compare the image differences with the allowable error range of the corresponding image region, and retain the difference pixels that exceed the allowable error range.

[0177] S3.5.4: Spatial connectivity aggregation is performed on the retained difference pixels, and discrete difference regions whose region size does not meet the preset size condition are filtered out to obtain candidate residual regions.

[0178] In this embodiment, mapping verification samples are divided from historical samples collected during the construction of the action response mapping. These mapping verification samples do not participate in the determination of corresponding image change parameters and are collected under conditions where the video acquisition device is fixed in position and there is no personnel entering, material movement, or continuous dust obstruction around the controlled mining equipment. For any pair of mapping verification samples, the earlier historical video frame is used as the historical reference video frame. The action response mapping is queried based on the change in historical action states between the preceding and following historical action states to obtain the image change parameters of the corresponding controlled structural region. Then, the historical reference video frame is subjected to region transformation, gap filling, and boundary fusion according to the generation method of the predicted video frame to obtain the historical predicted video frame.

[0179] The predicted historical video frames are spatially aligned with the later actual historical video frames, and compared pixel-by-pixel using the same image difference calculation method as the current and predicted video frames. The resulting pixel differences do not reflect changes in the actual work scene and primarily originate from parameter fitting deviations in the motion response mapping, boundary positioning deviations of controlled structure regions, pixel resampling errors, and video acquisition noise. The image difference between each predicted historical video frame and its corresponding actual historical video frame is treated as a mapping error sample.

[0180] When calculating the mapping error samples, the video frame is divided into controlled structural regions and a background region according to the controlled structural region identifiers stored in the motion response mapping. Multiple controlled structural regions are counted separately, without merging error data corresponding to different motion components. For example, the rotary arm region, the telescopic arm region, and the end effector region each form an independent error sample set; valid pixels not covered by any controlled structural region form the background region error sample set. Overlapping pixel bands located at region stitching boundaries are included in the statistics of the corresponding controlled structural region to ensure that normal pixel deviations caused by boundary fusion are included within the allowable error range.

[0181] The permissible error range for each image region is determined based on the pixel difference distribution among all mapping error samples in that region. When the image difference is non-negative, the lower limit of the permissible error range is set to zero, and the 99th percentile of the pixel difference distribution is used as the upper limit of the permissible error. Using the 99th percentile preserves the main fluctuations in historical normal prediction errors while avoiding a small amount of abnormal noise directly raising the upper limit of the permissible error. The number of mapping verification samples is no less than 100 sets, and the number of effective pixels participating in the statistics for each controlled structural region is no less than 10,000; if the above number is not reached, mapping verification samples continue to be collected.

[0182] For example, if the 99th percentile of the mapping error pixel values ​​in the telescopic arm region is 18 after sorting, then the allowable error range for this region is set to 0 to 18; if the 99th percentile of the background region is 7, then the allowable error range for the background region is set to 0 to 7. For image differences represented using 8-bit brightness differences, the upper limit of the allowable error ranges from 0 to 255. Different controlled structure regions can have different upper limits of allowable errors, without using a uniform threshold, to accommodate differences in motion amplitude, contour complexity, and resampling error among various moving parts.

[0183] When performing pixel-by-pixel judgment on the image differences between the current video frame and the predicted video frame, the image region to which the pixel belongs is first determined based on its coordinates, and then the allowable error range corresponding to that image region is called. When the pixel difference does not exceed the upper limit of the allowable error, it is regarded as a normal deviation allowed by motion prediction and image processing; when the pixel difference exceeds the upper limit of the allowable error, the pixel and its actual pixel difference are retained. Positions located in areas with invalid spatial alignment, video frame boundary extrapolation areas, or lacking effective reference pixels are not included in the judgment to avoid identifying invalid pixels as changes in the real scene.

[0184] The retained difference pixels are spatially aggregated according to the eight-neighbor connectivity rule. Any two difference pixels that are adjacent in the horizontal, vertical, or diagonal direction are included in the same connected region. For single-pixel gaps formed by texture discontinuities within the same real-world object, a closing operation can be performed first. The closing operation uses a 3×3 pixel structuring element to connect adjacent difference pixels with a spacing of no more than 1 pixel and fill single-pixel holes within the connected region. The closing operation is not iterated repeatedly to prevent independent real-world change regions from being merged.

[0185] The preset size condition is limited by the pixel area of ​​connected regions, and its setting is based on the area distribution of residual discrete difference regions under conditions of no real-world changes. For difference pixels in the mapping verification sample that exceed the allowable error range, the same connected component aggregation is performed, the pixel area of ​​each discrete difference region is calculated, and the 99th percentile of the area distribution plus 1 pixel is taken as the minimum region area. For example, if the 99th percentile of the discrete difference region area in the mapping verification sample is 14 pixels, then the minimum region area is set to 15 pixels. For 1920×1080 pixel video frames, this threshold can also be set to 16 pixels based on the actual verification results.

[0186] Connected regions whose pixel area does not reach the minimum region area are filtered out; connected regions whose pixel area reaches the minimum region area are used as candidate residual regions, and their region boundaries, region masks, pixel coordinates, and corresponding pixel differences are retained. Thus, the candidate residual regions represent continuous image changes that exceed the inherent prediction error of motion response mapping and the normal fluctuations of image processing, and can be used to generate real-world residuals.

[0187] Preferably, the display terminal generates a reconstructed video frame based on the reference video frame, the motion response information, and the real-scene residual, including:

[0188] S4.1: Obtain the controlled structure region and the corresponding current image change parameters from the action response information, and perform coordinate mapping and pixel resampling on the controlled structure region in the reference video frame according to the current image change parameters to obtain the transformed structure image.

[0189] S4.2: Using the image content in the reference video frame located outside the controlled structure region, fill the empty region formed after the controlled structure region is transformed, and fuse the transformed structure image with the filled reference video frame to obtain the basic reconstructed video frame.

[0190] S4.3: Obtain the location of the residual region and the corresponding pixel difference from the real-scene residual, map the location of the residual region to the basic reconstructed video frame, and superimpose the pixel difference onto the corresponding pixel in the basic reconstructed video frame to obtain the reconstructed video frame.

[0191] In this embodiment, the display terminal pre-stores the reference video frame corresponding to the current synchronization reference identifier, the controlled structural region mask, and the region connection relationship. Upon receiving the action response information, it first verifies whether the synchronization reference identifier carried therein is consistent with the local reference video frame. If they are consistent, video frame reconstruction is performed; if they are inconsistent, the use of the action response information is suspended, and the mine explosion-proof host is requested to send the corresponding reference video frame to avoid mismatch of structural regions and image change parameters under different synchronization references.

[0192] Each controlled structural region in the action response information is configured with a region identifier and current image change parameters. The display terminal retrieves the corresponding region mask from the reference video frame based on the region identifier and extracts the pixels covered by the region mask. The horizontal displacement, vertical displacement, rotation angle, and scale change ratio recorded in the current image change parameters are expressed using the reference video frame coordinate system, thus eliminating the need for coordinate conversion between the mine explosion-proof host and the display terminal. For a controlled structural region containing multiple structural sub-regions, local image change parameters are read separately according to the structural sub-region identifiers.

[0193] Coordinate mapping and pixel resampling employ the same processing rules as the predicted video frame generation process. Structural pixels in the reference video frame are transformed to the target position according to the current image change parameters; when the target position is a non-integer coordinate, bilinear interpolation is used to calculate the target pixel value. The region mask undergoes coordinate transformation and resampling simultaneously to form a transformed region mask. The transformed structural image retains only the pixels covered by the transformed region mask; the remaining positions are set to transparent or invalid states for combination with the background image.

[0194] The vacant area formed after the displacement of the controlled structure region is determined by the difference between the region mask before and after transformation. The display terminal selects neighboring background pixels from the background region image of the reference video frame for filling. The selection range and filling method of the neighboring background pixels are consistent with the parameters used by the mine explosion-proof host when generating the prediction video frame. For multiple controlled structure regions, they are processed sequentially according to the structural connection relationship and occlusion order saved in the reference video frame, so that the structure region closer to the video acquisition viewpoint covers the structure region following it.

[0195] The transformed structural image and the filled reference video frame are fused at the stitching boundary using the overlapping pixel band width and boundary fusion rules stored in the synchronization reference. The display terminal does not recalculate the fusion parameters but directly uses the boundary width and weight determination method used by the mine explosion-proof host to generate the predicted video frame. This ensures that the basic reconstructed video frame and the predicted video frame on the mine explosion-proof host side have consistent pixel results under the same input conditions. The basic reconstructed video frame is formed after the transformation, filling, and fusion of all controlled structural regions are completed.

[0196] The real-scene residual is transmitted separately according to residual regions. Each residual region records at least the coordinates of its bounding box, the region mask, and the corresponding pixel difference. The location of the residual region is represented using the coordinate space of the predicted video frame, which is consistent with the coordinate space of the base reconstructed video frame. For example, if the top-left corner coordinates of the residual region's bounding box are 620 pixels horizontally and 315 pixels vertically, with a width of 48 pixels and a height of 36 pixels, then the display terminal reads the region mask within the corresponding rectangular area of ​​the base reconstructed video frame and performs interpolation only on the pixels within the mask.

[0197] Pixel difference represents the signed difference between the spatially aligned pixel value of the current video frame and the predicted pixel value of the video frame. For 8-bit color video frames, the differences between the luminance and color channels can be recorded separately, with each channel difference ranging from -255 to +255. When using the RGB color space, the differences between the red, green, and blue color channels can also be recorded separately. The mining explosion-proof host and display terminal use the same color space and channel arrangement order to avoid channel misalignment when differences are superimposed.

[0198] After superimposing the pixel differences onto the corresponding pixels of the base reconstructed video frame, the pixel values ​​of each channel are limited within a certain range. Values ​​less than 0 are rounded to 0, values ​​greater than 255 are rounded to 255, and values ​​between 0 and 255 are rounded to integer pixel values. Superposition is performed separately for residual regions where there is no overlap; if there is overlap, superposition is performed once based on the region merging result generated by the mining explosion-proof host for the real-world residuals, without repeatedly adding differences to the same pixel.

[0199] Motion response information and real-scene residuals can also carry the same current video frame number. The display terminal only combines data with the same frame number to generate reconstructed video frames, avoiding the mixing of motion response information and real-scene residuals from different acquisition times. When a current video frame only receives motion response information but not real-scene residuals, the basic reconstructed video frame can be used as the display result for that frame; when the corresponding real-scene residual arrives, pixel correction is performed according to the same frame number. Thus, the reconstructed video frame simultaneously contains the controlled structural changes recovered from the current motion state and the on-site changes supplemented by the real-scene residuals.

[0200] Preferably, when the real-scene residual meets the baseline update condition, the synchronization baseline is updated with the current video frame and the current action state, including:

[0201] S5.1: Obtain the real-scene residuals corresponding to multiple consecutive current video frames, and associate the residual regions in adjacent real-scene residuals where the distance between the center of the region meets the distance condition and the region overlap ratio meets the overlap condition to obtain the continuous residual region.

[0202] S5.2: Calculate the number of consecutive frames, the coverage ratio of the region, and the pixel difference of the continuous residual region to obtain the continuous residual feature.

[0203] S5.3: When the number of consecutively occurring frames, the area coverage ratio, and the pixel difference satisfy the corresponding baseline update conditions, the corresponding current video frame is determined as the new reference video frame, and the corresponding current action state is determined as the new reference action state.

[0204] S5.4: Associate the new reference video frame with the new reference action state, and update the synchronization reference.

[0205] In this embodiment, each real-scene residual records the region mask, region bounding box, region center coordinates, and pixel difference of the residual region. The region center coordinates can be determined by averaging the coordinates of all pixels within the region mask. The real-scene residuals are arranged according to the frame number of the current video frame, and only two real-scene residuals with adjacent frame numbers are compared to determine whether the same scene change persists over time.

[0206] The region center distance is the pixel distance between the center coordinates of two residual regions in two adjacent frames. The distance condition is set based on the image registration error and the allowable positional change of the continuous target within a video acquisition cycle. Specifically, in a verification video where the synchronization reference is valid and the working scene remains stable, adjacent residual regions that can be confirmed to belong to the same object are associated, their region center distances are calculated, and the 95th percentile value of the region center distance is added to the image registration reprojection error threshold to obtain the center distance threshold. For example, when the 95th percentile value is 5 pixels and the reprojection error threshold is 3 pixels, the center distance threshold is set to 8 pixels.

[0207] The region overlap ratio is represented by the proportion of pixels in the intersection of two adjacent residual region masks to the total number of pixels in their union, ranging from 0 to 1. A region overlap ratio closer to 1 indicates a more consistent spatial location and shape between the two residual regions. The overlap ratio threshold is set based on the distribution of region overlap ratios for the same persistent target in the verification video, and can be taken as the 5th percentile of that distribution. For example, if 95% of the samples for the same persistent target have a region overlap ratio of at least 0.48, then the overlap ratio threshold is set to 0.48.

[0208] Adjacent residual regions are considered to be in the same continuous change only when the center distance between regions does not exceed the center distance threshold and the region overlap ratio is not lower than the overlap ratio threshold. When a current residual region simultaneously satisfies multiple associated objects, the object with the largest region overlap ratio is associated first; when the region overlap ratios are the same, the object with the smallest center distance is associated. Residual regions sequentially associated in multiple consecutive video frames constitute a continuous residual region, and the frame number corresponding to each residual region is retained. When no residual region satisfying the association conditions is detected in any video frame, the continuous statistics of the corresponding continuous residual region are terminated.

[0209] The consecutive occurrence frame count refers to the number of consecutively associated frames from the first appearance of the persistent residual region to the current video frame. The region coverage ratio characterizes the spatial occupancy of the persistent residual relative to the effective video frame. Specifically, it is calculated by dividing the pixel area of ​​the residual region in each associated frame by the effective pixel area of ​​the corresponding video frame, and then averaging the resulting ratios. The effective pixel area does not include invalid boundary regions generated by spatial alignment. The pixel difference is the average of the absolute pixel differences of each effective pixel within the persistent residual region, and this average is calculated again for consecutive associated frames. This value characterizes the degree of deviation of the persistent change from the average image of the predicted video frames.

[0210] The consecutive occurrence frame count threshold is set based on the duration of instantaneous dust, localized reflections, and short-term occlusions in the mine scene. The longest consecutive occurrence frame count of the non-permanent residual region can be counted in the verification video where the synchronization benchmark remains valid, and this frame count is incremented by 1 to serve as the consecutive occurrence frame count threshold. For example, if the video capture frame rate is 25 frames per second, and the longest duration of the non-permanent residual region in the verification video is 1.2 seconds, corresponding to 30 frames, then the consecutive occurrence frame count threshold is set to 31 frames.

[0211] The region coverage ratio threshold and pixel difference threshold are determined based on the mapping verification samples and the statistical results of the real-scene residuals during the effective period of the synchronization benchmark. The region coverage ratio threshold can be taken as the 99th percentile of the region coverage ratio of the non-permanent residuals. For example, if this percentile is 0.018, the region coverage ratio threshold is set to 0.018. The pixel difference threshold can be taken as the 99th percentile of the average absolute pixel difference corresponding to normal mapping errors and short-term scene disturbances. For example, for an 8-bit image, if this percentile is 21, the pixel difference threshold is set to 21 pixels.

[0212] When the number of frames, the area coverage ratio, and the pixel difference all consecutively reach the corresponding thresholds, it is determined that the original synchronization benchmark can no longer adequately represent the current video frame. All three conditions must be met simultaneously to exclude fixed noise that meets the duration requirement but only covers a small number of pixels, and situations where the coverage area meets the requirement but the pixel deviation does not exceed normal fluctuations.

[0213] To ensure that the new reference video frame has a stable structural position and usable image quality, from the consecutive current video frames that meet the baseline update conditions, the current video frame with the highest image quality evaluation value, whose current action state meets the aforementioned stability conditions, whose controlled structural region integrity reaches the integrity threshold, is selected as the new reference video frame, and its current action state at the acquisition time is used as the new reference action state. If no current video frame that meets the above conditions has yet appeared, the original synchronization baseline is retained until a current video frame that meets the requirements is obtained.

[0214] The new reference video frame, new reference action state, controlled structure area identifier, and action response mapping version are associated, and a new synchronization reference identifier is generated. The mine explosion-proof host sends the new reference video frame and synchronization reference identifier to the display terminal. After the display terminal saves the data and returns confirmation information, both parties use the new synchronization reference to process the action response information and real-world residuals. Before confirmation, the original synchronization reference is still used to avoid reference video frame mismatch during synchronization reference switching.

[0215] Example 2

[0216] Please see Figure 4The present invention provides an embodiment of a synchronous interaction system between a mine explosion-proof host and a display terminal, the system comprising:

[0217] The mapping and benchmark determination module constructs an action response mapping based on the historical action states of the controlled mining equipment and the corresponding historical video frames, and determines a synchronization benchmark composed of reference video frames and corresponding reference action states.

[0218] The action response and prediction module is used to generate action response information based on the current action state of the controlled mining equipment, the reference action state, and the action response mapping, and to generate predicted video frames based on the reference video frames and the action response information.

[0219] The real-scene residual determination module is used to determine the real-scene residual based on the difference between the current video frame corresponding to the current action state and the predicted video frame.

[0220] The synchronous interaction module is used to send the motion response information and the real-scene residual to the display terminal, and generate a reconstructed video frame based on the reference video frame, the motion response information and the real-scene residual;

[0221] The benchmark update module is used to update the synchronization benchmark with the current video frame and the current action state when the real-scene residual meets the benchmark update conditions.

[0222] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for synchronous interaction between a mine explosion-proof host and a display terminal, characterized in that, The method includes: Based on the historical action states of the controlled mining equipment and the corresponding historical video frames, an action response mapping between changes in action states and changes in video frames is constructed, and a synchronization benchmark consisting of reference video frames and corresponding reference action states is determined. The current action state of the controlled mining equipment is obtained, action response information is generated based on the current action state, the reference action state, and the action response mapping, and a predicted video frame is generated based on the reference video frame and the action response information. Obtain the current video frame corresponding to the current action state, and determine the real-scene residual based on the difference between the current video frame and the predicted video frame; The motion response information and the real-scene residual are sent to the display terminal, and the display terminal generates a reconstructed video frame based on the reference video frame, the motion response information, and the real-scene residual. When the real-scene residual meets the baseline update conditions, the synchronization baseline is updated with the current video frame and the current action state.

2. The synchronous interaction method between the mine explosion-proof host and the display terminal according to claim 1, characterized in that, The process of constructing an action response mapping between changes in action state and changes in video frames based on the historical action states of the controlled mining equipment and the corresponding historical video frames includes: Each historical action state and its corresponding historical video frame are grouped into a historical sample group according to the time correspondence, and the historical sample group is arranged according to the action execution order. Based on the historical sample pairs in the historical sample group, the amount of change in historical action state is determined, and the controlled structural region that changes with the historical action state is determined from the corresponding previous and subsequent historical video frames. The image change parameters of the controlled structural region between the preceding and following historical video frames are extracted, and the correspondence between the historical action state change amount and the image change parameters is established to obtain the action response mapping.

3. The synchronous interaction method between the mine explosion-proof host and the display terminal according to claim 2, characterized in that, The determination of the synchronization reference, which consists of reference video frames and corresponding reference action states, includes: Based on the changes in the historical action state of each historical sample pair, candidate historical sample groups in which the controlled mining equipment is in a stable action state are determined. Reference video frames are determined based on the controlled structural region integrity and image quality of historical video frames in the candidate historical sample group; The historical action state corresponding to the reference video frame is determined as the reference action state, and the reference video frame is associated with the reference action state to obtain a synchronization reference.

4. The synchronous interaction method between the mine explosion-proof host and the display terminal according to claim 1, characterized in that, Action response information is generated based on the current action state, the reference action state, and the action response mapping, including: Based on the current action state and the reference action state, determine the change in the current action state; Determine the first historical action state change and the second historical action state change that are adjacent to the current action state change from the action response mapping, and obtain the first image change parameter and the second image change parameter corresponding to the two respectively. Based on the distance relationship between the current action state change and the first historical action state change and the second historical action state change, the parameter interpolation coefficient is determined; The first image change parameter and the second image change parameter are interpolated based on the interpolation coefficients to obtain the current image change parameter; The current image change parameters and their corresponding controlled structural regions are used as action response information.

5. The synchronous interaction method between the mine explosion-proof host and the display terminal according to claim 4, characterized in that, Generate a predicted video frame based on the reference video frame and the motion response information, including: Based on the controlled structural region in the action response information, the reference video frame is segmented to obtain a structural region image and a background region image. The current image change parameters are converted into coordinate transformation relationships for the corresponding controlled structural regions. The pixels in the structural region image are then mapped and resampled according to the coordinate transformation relationships to obtain the transformed structural image. The missing areas formed after the displacement of the controlled structure region are filled using neighboring background pixels in the background region image. Based on the structural connection relationship between the controlled structural regions, the transformed structural image and the filled background region image are stitched together and fused to obtain the predicted video frame.

6. The synchronous interaction method between the mine explosion-proof host and the display terminal according to claim 1, characterized in that, Determining the real-scene residual based on the difference between the current video frame and the predicted video frame includes: Background feature points are extracted from the uncontrolled structural regions of the current video frame and the predicted video frame, respectively, and local feature descriptors corresponding to each background feature point are generated. Calculate the feature distance between the local feature descriptors of each background feature point in the current video frame and the predicted video frame, and form an initial feature point pair with the background feature points whose feature distances satisfy a preset matching condition; Geometric consistency screening is performed based on the relative positional relationship between each initial feature point pair, and initial feature point pairs that do not satisfy the same image transformation relationship are eliminated to obtain target feature point pairs; The image registration parameters are solved using the target feature point pairs, and the current video frame and the predicted video frame are spatially aligned according to the image registration parameters. The image difference between the current video frame and the predicted video frame after spatial alignment is calculated, candidate residual regions are extracted from the image difference, and real-world residuals are generated based on the candidate residual regions.

7. The synchronous interaction method between the mine explosion-proof host and the display terminal according to claim 6, characterized in that, Extracting candidate residual regions from the image differences includes: The motion response mapping is used to predict motion changes in historical video frames. The predicted historical video frames are then compared with the corresponding historical video frames to obtain mapping error samples. The mapping error samples are partitioned and statistically analyzed according to the controlled structure region and background region in the video frame to obtain the allowable error range corresponding to each image region. The image differences are compared with the allowable error range of the corresponding image region, and the difference pixels that exceed the allowable error range are retained; Spatial connectivity aggregation is performed on the retained difference pixels, and discrete difference regions whose region size does not meet the preset size condition are filtered out to obtain candidate residual regions.

8. The synchronous interaction method between the mine explosion-proof host and the display terminal according to claim 1, characterized in that, The display terminal generates a reconstructed video frame based on the reference video frame, the motion response information, and the real-scene residual, including: The controlled structural region and its corresponding current image change parameters are obtained from the action response information. The controlled structural region in the reference video frame is then mapped and resampled according to the current image change parameters to obtain a transformed structural image. Using the image content in the reference video frame located outside the controlled structure region, the empty region formed after the controlled structure region is transformed is filled, and the transformed structure image is fused with the filled reference video frame to obtain the basic reconstructed video frame. The residual region position and corresponding pixel difference are obtained from the real-scene residual. The residual region position is mapped to the basic reconstructed video frame. The pixel difference is superimposed on the corresponding pixel in the basic reconstructed video frame to obtain the reconstructed video frame.

9. The synchronous interaction method between the mine explosion-proof host and the display terminal according to claim 1, characterized in that, When the real-scene residual meets the baseline update condition, the synchronization baseline is updated with the current video frame and the current action state, including: Obtain the real-scene residuals corresponding to multiple consecutive current video frames, and associate the residual regions in adjacent real-scene residuals whose center distance meets the distance condition and whose region overlap ratio meets the overlap condition to obtain the continuous residual region. The continuous residual features are obtained by statistically analyzing the number of consecutively occurring frames, the region coverage ratio, and the pixel difference in the continuous residual region. When the number of consecutively occurring frames, the area coverage ratio, and the pixel difference satisfy the corresponding baseline update conditions, the corresponding current video frame is determined as a new reference video frame, and the corresponding current action state is determined as a new reference action state. The new reference video frame is associated with the new reference motion state, and the synchronization reference is updated.

10. A synchronous interaction system for a mining explosion-proof host and a display terminal, used to implement the synchronous interaction method for a mining explosion-proof host and a display terminal as described in any one of claims 1 to 9, characterized in that, The system includes: The mapping and benchmark determination module constructs an action response mapping based on the historical action states of the controlled mining equipment and the corresponding historical video frames, and determines a synchronization benchmark composed of reference video frames and corresponding reference action states. The action response and prediction module is used to generate action response information based on the current action state of the controlled mining equipment, the reference action state, and the action response mapping, and to generate predicted video frames based on the reference video frames and the action response information. The real-scene residual determination module is used to determine the real-scene residual based on the difference between the current video frame corresponding to the current action state and the predicted video frame. The synchronous interaction module is used to send the motion response information and the real-scene residual to the display terminal, and generate a reconstructed video frame based on the reference video frame, the motion response information and the real-scene residual; The benchmark update module is used to update the synchronization benchmark with the current video frame and the current action state when the real-scene residual meets the benchmark update conditions.

Images