Method and system for remote coordination of binocular vision police
By constructing heterogeneous cross-baseline virtual stereo pairs and reversing multi-view geometric inconsistencies, the problems of depth completion and accurate command feedback in obscured areas of the police remote collaborative command system were solved. This enabled reliable spatial analysis and accurate command overlay in obscured areas, improving the accuracy and traceability of police remote collaborative command.
Patent Information
- Application Number
- CN202611055903.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-16
- Publication Date
- 2026-11-17
AI Technical Summary
Existing remote collaborative command systems for police affairs cannot obtain reliable depth assessment and accurate command reversal in obscured areas under heterogeneous spatiotemporal references and obstruction conditions, resulting in the command end being unable to accurately determine the spatial relationship between the target object and the front and back in the obscured area.
By constructing a heterogeneous cross-baseline virtual stereo pair, and utilizing the complementary observations of a first-view acquisition device and a fixed-point binocular acquisition device, the geometric inconsistency of multiple views is reversed into an occlusion-depth judgment signal. The command annotations are then back-projected from the binocular depth coordinate system to the first-view coordinate system, enabling reliable spatial judgment of occluded areas and accurate overlay of commands.
In the obscured area, depth completion and reliable judgment of spatial relationship were achieved, ensuring that the command was accurately projected back to the field of vision. This improved the accuracy of spatial judgment and command transmission in remote collaborative command, forming a complete closed loop of on-site perception, remote judgment, command superposition and manual confirmation.
Smart Images

Figure REF-OBJ-1784169860729-000119 
Figure REF-OBJ-1784169860729-000120 
Figure REF-OBJ-1784169860729-000121
Abstract
Description
Technical Field
[0001] This invention relates to the fields of intelligent policing equipment and computer vision processing technology, and in particular to a method and system for remote collaborative command of MR and binocular vision policing. Background Technology
[0002] With the increasing demands for national security and social governance, public security work places higher requirements on on-site perception, remote command, and collaborative response capabilities. In scenarios such as security for large-scale events, three-dimensional patrols of key areas, and emergency response to sudden incidents, the command center needs to grasp the on-site situation in real time and accurately, especially accurately judging the spatial relationship between the target and surrounding obstacles. For example, whether the target is in front of or behind cover, the depth structure of areas where crowds gather, and whether passageways are blocked. This spatial relationship is a key basis for command and decision-making, and this is precisely the weakest link in existing police video systems.
[0003] Chinese patent application CN107040761A discloses a command center control system for public security bureaus. This system uses a splicing display and controller to display fixed urban security surveillance video in real-time on a large command screen, and enables police force dispatch through the control center, main venue, and several branch venues. However, this solution relies on a fixed-view, two-dimensional surveillance image, which does not carry depth information. When a target object is obscured by an obstacle, the command center cannot determine from the two-dimensional image whether the target object is in front of or behind the obstruction. The obstructed area remains an unanalyzable blind spot in this solution, and the command center loses its perception capability precisely in the area where spatial analysis is most needed.
[0004] Chinese patent application CN103245322A discloses a ranging method and system based on binocular stereo vision. This scheme obtains the positional relationship between two fixed points and constructs a ranging model with a single fixed point as the origin, and combines heterogeneous sensor fusion to achieve target ranging in complex environments. However, in this scheme, the binocular cameras are fixedly arranged and their baselines and intrinsic parameters are consistent. The stereo matching process treats the multi-view geometric inconsistencies caused by occlusion as noise that reduces the accuracy of depth estimation and suppresses or discards them. Therefore, the occluded area is directly judged as an invalid depth area, and no judgment output can be provided on the spatial relationship between front and back within the occluded area.
[0005] Chinese patent application CN114647315A discloses a human-computer interaction method based on AR glasses for museum tours. This method uses AR glasses comprising a video acquisition unit, a visual tracking unit, a gesture recognition unit, a voice recognition unit, and a positioning unit. It achieves local real-time interaction between the wearer and the AR glasses through three control modes: eye tracking, gesture, and voice. However, this solution is a local human-computer interaction method for the wearer. The first-person view captured by this method neither establishes a stereoscopic observation relationship with a remote fixed-point binocular camera, nor does it invert and project the annotations generated by the remote command terminal in the depth coordinate system back to the wearer's field of vision. Therefore, it cannot solve the problems of remotely judging the spatial relationship of obstructed areas and accurately projecting commands back to the wearer.
[0006] Therefore, existing technologies either employ fixed two-dimensional monitoring without depth, or use single-source binocular ranging which invalidates obstructions, or use local AR interaction without remote stereoscopic analysis. Their common fundamental bottleneck lies in the fact that, in police work conditions where the spatiotemporal references of the first-person view acquisition device and the fixed-point binocular acquisition device are heterogeneous, and there is significant obstruction on-site, the command center cannot obtain reliable depth analysis capabilities for obstructed areas, nor can it accurately project commands back to the field of vision. Therefore, there is an urgent need for a remote collaborative command solution for police work that enables the command center to obtain reliable depth analysis of obstructed areas and accurately project commands back to the field of vision under conditions of heterogeneous spatiotemporal references and significant obstruction. Summary of the Invention Addressing the core technical bottlenecks in existing technologies where remote collaborative command in police operations suffers from heterogeneous spatiotemporal references between first-view acquisition devices and fixed-point binocular acquisition devices, and where significant obstructions exist on-site, the command center cannot obtain reliable depth assessment of obstructed areas and accurately project commands back to the on-site motion field of view. This invention provides a method and system for remote collaborative command in police operations using MR and binocular vision. By constructing the first-view acquisition device and the fixed-point binocular acquisition device as a heterogeneous cross-baseline virtual stereo pair, and inverting multi-view geometric inconsistencies from noise to be suppressed into obstruction-depth assessment signals, and then inverting command annotations from the binocular depth coordinate system back to the first-view coordinate system, without changing the existing hardware form of the first-view acquisition device and the fixed-point binocular acquisition device, this invention achieves reliable spatial assessment of obstructed areas and accurate on-site overlay of commands by the command center from the principle level of obstruction geometric inversion and cross-coordinate system depth back projection.
[0007] The technical solution of this invention is as follows: The method for remote collaborative command in MR and binocular vision policing includes the following steps: Step 1: Collect first-view motion video and first-view pose signal of the duty personnel through first-view acquisition equipment, collect binocular video of the target area through fixed-point binocular acquisition equipment, and generate a depth map of the target area based on binocular parallax. Step 2: Perform time synchronization, spatial association, and target object encoding on the first-view motion video, the first-view pose signal, and the target area depth map to establish a mapping relationship between the first-view coordinate system and the binocular depth coordinate system; Step 3: Based on the mapping relationship, construct the first view acquisition device and the fixed-point binocular acquisition device into a heterogeneous cross-baseline virtual stereo pair, and perform complementary depth enhancement on the same target object to obtain an enhanced depth map; Step 4: Determine the reprojection residual region generated by the heterogeneous cross-baseline virtual stereo pair observing the same target object as the occlusion candidate region, and reverse the occlusion candidate region from the noise to be suppressed in binocular stereo matching to the occlusion-depth judgment signal; Step 5: Generate comprehensive situational information based on the enhanced depth map and the occlusion-depth analysis signal, and generate command annotations in the binocular depth coordinate system based on the comprehensive situational information; Step 6: Based on the first viewpoint pose signal, the command annotation is inverted from the binocular depth coordinate system and projected back to the first viewpoint coordinate system to obtain the back-transmitted annotation, which is then superimposed and displayed on the first viewpoint acquisition device.
[0008] Furthermore, when the occlusion candidate region is inverted into an occlusion-depth judgment signal, the depth gradient along the depth direction of the reprojection residual region is extracted within the occlusion candidate region. The direction of the depth gradient is determined as the spatial relationship between the same target object and the occlusion object. Based on the spatial relationship, the occlusion-depth judgment signal including the occlusion region position, the order of precedence, and the occlusion boundary is generated. When the first view acquisition device moves relative to the fixed-point binocular acquisition device and the acquisition frame rate is inconsistent, the reprojection residual region is decomposed into an occlusion component and a spatiotemporal registration error component. With the depth temporal continuity of the same target object on the depth map of the target region as a constraint, the spatiotemporal registration error component is separated and eliminated by performing change point detection and positioning of the registration jump time on the depth temporal sequence and assigning attenuation weight to its reprojection residual, while retaining the occlusion component.
[0009] Furthermore, for pixels in the occlusion candidate area that lack direct depth due to parallax loss, depth extrapolation is performed along the epipolar direction based on the spatial relationship between the front and back and the boundary of the same target object to obtain the completed depth, which is then incorporated into the enhanced depth map. When the pose jitter of the first view acquisition device causes the return annotation landing point to change, a joint confidence of the landing point is constructed based on the completed depth and the first view pose signal, and spatiotemporal filtering is performed on the return annotation landing point to make the return annotation landing points of multiple first view acquisition devices converge to the same target object.
[0010] This invention also provides a remote collaborative command system for MR and binocular vision policing, comprising: The MR terminal is used to collect first-person perspective motion video and first-person perspective pose signal of the duty personnel, and to overlay the returned annotations on the MR terminal. A fixed-point binocular device is used to acquire binocular video of a target area and generate a depth map of the target area based on binocular parallax. The command platform is used to synchronize the motion video from the first viewpoint, the pose signal from the first viewpoint, and the depth map of the target area in time, spatially associate them, and encode the target object to establish a mapping relationship. It constructs the MR terminal and the fixed-point binocular device into a heterogeneous cross-baseline virtual stereo pair and performs complementary depth enhancement to obtain an enhanced depth map. It determines the reprojection residual region as an occlusion candidate region and inverts it into an occlusion-depth judgment signal, thereby generating comprehensive situational information and generating command annotations in the binocular depth coordinate system. The annotation feedback module is used to invert and project the command annotation back to the first viewpoint coordinate system based on the first viewpoint pose signal to obtain the feedback annotation; The manual confirmation module is used to receive manual confirmation signals from on-duty personnel regarding the returned annotations; The log auditing module is used to keep logs of the entire process.
[0011] The beneficial effects of this invention are as follows: First, by constructing a heterogeneous cross-baseline virtual stereo pair between a first-person perspective acquisition device and a fixed-point binocular acquisition device, and performing complementary depth enhancement on the same target object, depth completion of the blind spots in single-source binocular obstruction is achieved. The mechanism is as follows: the fixed-point binocular acquisition device has a fixed baseline; when the target object is obstructed, neither the left nor right eyes can observe the target object's surface, resulting in a lack of depth in the obstructed area. Meanwhile, the first-person perspective acquisition device worn by the personnel is positioned in a different spatial location and observation direction, creating a third observation constraint on the same target object—a different baseline and line of sight from the fixed-point binocular acquisition device. The heterogeneous cross-baseline virtual stereo pair formed by these two devices can utilize the lateral observation from the first-person perspective to complete the surface depth of the obstructed area that the fixed-point binocular device cannot observe. Compared to a fixed-baseline, consistent binocular ranging scheme that directly outputs invalid depth in the obstructed area, this scheme can obtain effective depth coverage in the obstructed area, fundamentally expanding the assessable spatial range.
[0012] Second, by identifying the reprojection residual region generated by heterogeneous cross-baseline virtual stereo pairs observing the same target object as an occlusion candidate region and inverting it from the noise to be suppressed in binocular stereo matching into an occlusion-depth judgment signal, a reliable judgment of the spatial relationship between the front and back of the occlusion region is achieved. The mechanism is as follows: In the conventional processing paradigm of binocular stereo matching, geometric inconsistencies caused by occlusion between multiple views contaminate the cost volume and reduce the accuracy of depth estimation, and are therefore suppressed as noise. However, this geometric inconsistency itself is the geometric projection encoding of the occlusion relationship in space; the position of the residual region corresponds to the position where occlusion occurs, and the gradient of the residual along the depth direction points to the front-back order of the target object and the occluder. This scheme no longer suppresses this inconsistency, but instead inversely derives the position, front-back order, and occlusion boundary of the occlusion region, transforming the originally discarded blind spot into an information-rich area. Compared to two-dimensional monitoring screens that cannot determine the front-back relationship of the target object in the occlusion region, this scheme enables the command end to obtain a clear spatial judgment conclusion in the occlusion region.
[0013] Third, by inverting command annotations from the binocular depth coordinate system back to the first-view coordinate system based on the first-view pose signal, precise overlay of remote commands onto the on-site moving field of vision is achieved. The mechanism is as follows: the command annotations are generated in the binocular depth coordinate system carrying depth information and inherently possess clear 3D spatial anchor points; through the coordinate system mapping relationship established by the first-view pose signal, these 3D anchor points can be inverted and projected onto the correct pixel position in the on-duty personnel's current moving field of vision, ensuring the annotations are firmly attached to the 3D target object on-site rather than simply pasted onto a 2D image. Even if the target object is in an occluded area, it can still fall into the correct spatial position based on the completed depth. Compared to human-computer interaction only within the local field of vision and lacking cross-coordinate system depth back-projection capabilities, this solution enables remotely generated commands to accurately fall onto the real target object in the on-duty personnel's field of vision.
[0014] Fourth, the above three technical features generate a synergistic effect that exceeds the sum of their individual effects. Specifically: the heterogeneous cross-baseline virtual stereo pair provides the depth of completion for the occluded area; the occlusion-depth judgment signal resolves the geometric inconsistencies of the completed depth area into spatial relationships; and the depth inversion and reprojection of command annotations relies on the aforementioned completed depth as a three-dimensional anchor point to obtain a stable landing point in the occluded area. The three constitute a closed loop in which heterogeneous stereo enhancement provides depth-inconsistency inversion to generate judgment-depth reprojection with accurate landing points. Any single link cannot achieve reliable judgment and accurate reprojection in the occluded area. Only the synergy of the three can transform the occluded area from a blind spot into an information area that can be judged, annotated, and handled. This significantly improves the spatial judgment accuracy and command transmission precision of remote collaborative command in police work with a large amount of occlusion, realizing a complete closed loop of on-site perception, remote judgment, command overlay, manual confirmation, and log recording.
[0015] Fifth, by organizing various objects and their spatial relationships at the police scene into a police spatial situation map during the generation of comprehensive situational information, and integrating it to obtain a comprehensive risk level, the command end not only obtains the credible depth of the obstructed area, but also can conduct risk assessment and generate structured command instructions based on a unified node and edge structure. Combined with review and security auditing according to a unified timeline, a complete closed loop of on-site perception, remote assessment, instruction overlay, manual confirmation, log recording, and post-event review is formed, further improving the traceability and standardization of remote collaborative command in police affairs. Attached Figure Description
[0016] Figure 1 This is a flowchart of the remote collaborative command method for MR and binocular vision police services of the present invention; Figure 2 This is an architecture diagram of the MR and binocular vision remote collaborative command system for police affairs according to the present invention.
[0017] Figure 3 This is a schematic diagram illustrating the construction of a police spatial situation map according to the present invention; Figure 4 This is a schematic diagram of the MR situation labeling and execution feedback closed loop of the present invention; Figure 5 This is a schematic diagram of the overall deployment architecture of the system of this invention. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0019] The overall processing flow of the method of the present invention is as follows: Figure 1 As shown, the modular composition of the system of the present invention is as follows: Figure 2 As shown.
[0020] Example 1: Implementation of methods in the security scenario of large-scale events.
[0021] This embodiment uses the security of a large-scale sporting event as an application scenario to describe in detail the complete implementation process of the MR and binocular vision-based remote collaborative command method for policing of the present invention. During a large-scale event held at a stadium, the security command center deploys several personnel wearing first-person perspective acquisition devices at key locations such as the stadium entrance, stands, VIP area, and surrounding security checkpoints. Several fixed-point binocular acquisition devices are also deployed at the stadium's outer entrance passages and surrounding key areas. Both are connected to the command platform via an encrypted transmission channel, forming the remote collaborative command link for policing according to the present invention.
[0022] Step S1: Acquisition of first-view motion video, first-view pose signal and target area depth map.
[0023] The first-person perspective (FPPS) acquisition device worn by the duty personnel captures high-definition audio and video in real time from the front of their field of vision, resulting in a first-person perspective motion video. In this embodiment, the FPS motion video is captured at a resolution of 1920×1080 pixels and a frame rate of 30fps. Simultaneously, the built-in inertial measurement unit and positioning unit of the FPS acquisition device acquire the spatial position and attitude of the device in real time, outputting a first-person perspective pose signal. This first-person perspective pose signal includes three position components (longitude, latitude, and altitude) and three attitude components (pitch angle, yaw angle, and roll angle), forming a six-degree-of-freedom pose description. The unit for the attitude components is deg, and the unit for the position components is meters (m).
[0024] Simultaneously, the left and right eyes of the fixed-point binocular acquisition device synchronously acquire binocular video of the target area using a known baseline distance and a known intrinsic parameter matrix. The baseline distance of the fixed-point binocular acquisition device is set to 0.12m, the focal length of the left and right eyes is 1200 pixels, and the acquisition frame rate is 25fps. The fixed-point binocular acquisition device calculates the depth of the target area pixel-by-pixel based on the binocular disparity between the left and right eye images, generating a depth map of the target area. The conversion from binocular disparity to depth uses the well-known triangulation relationship, i.e., depth is inversely proportional to disparity and directly proportional to the product of the baseline and the focal length. In this embodiment, the depth accuracy of the target area depth map at a 10m observation distance is better than 0.05m. It should be noted that due to occlusion, the target area depth map exhibits disparity loss in areas blocked by occluded objects because neither the left nor right eyes can observe the surface of the target object. The depth value in these areas is invalid, and this loss is precisely what needs to be addressed in subsequent steps.
[0025] Step S2: Time synchronization, spatial association, and target object encoding. Since the first-person view acquisition device and the fixed-point binocular acquisition device are different physical devices, use different acquisition frame rates (30fps for the former and 25fps for the latter), and the first-person view acquisition device is in motion, the data acquired by the two are inconsistent in terms of time and spatial reference. Therefore, a unified spatiotemporal reference must be established first.
[0026] In terms of time synchronization, the command platform assigns a timestamp from a unified clock source to each frame of the first-person motion video and each frame of the target area depth map, and aligns them according to the nearest timestamp, so that the first-person frame and the target area depth frame participating in subsequent fusion correspond to the same physical time. In this embodiment, the time alignment residual is controlled within 20ms.
[0027] In terms of spatial correlation, the command platform establishes a mapping relationship between the first-view coordinate system and the binocular depth coordinate system. The first-view coordinate system is a moving coordinate system with the optical center of the first-view acquisition device as its origin, which changes in real time with the first-view pose signal; the binocular depth coordinate system is a fixed coordinate system with the optical center of the left eye of the fixed-point binocular acquisition device as its origin. The command platform uses the position and attitude of the first-view acquisition device in the geographic coordinate system given by the first-view pose signal, as well as the calibration extrinsic parameters of the fixed-point binocular acquisition device, to solve the mapping relationship between the two coordinate systems described by the rotation matrix and the translation vector.
[0028] Regarding target object coding, the command platform assigns a unique target object code to each target object detected within the field of view of the fixed-point binocular acquisition equipment, and maintains consistency of the code for the same target object in the first-view motion video and the depth map of the target area, providing an object index that runs through the entire process for subsequent heterogeneous stereo observation, occlusion analysis and annotation re-projection.
[0029] Step S3: Construction of Heterogeneous Cross-Baseline Virtual Stereo Pair and Complementary Depth Enhancement. This step is one of the core innovations of this invention. Conventional binocular stereo vision systems consist of two cameras with fixed baselines, consistent intrinsic parameters, and rigid connections; however, this invention treats a moving first-view acquisition device and a fixed-position binocular acquisition device as a pair of cameras, constructing a heterogeneous cross-baseline virtual stereo pair. It is called heterogeneous cross-baseline because the baselines of this pair of cameras are not fixed and change in real time with the movement of the first-view acquisition device, and their intrinsic parameters and frame rates are inconsistent. It is this heterogeneity that allows the first-view acquisition device to observe the same target object from different spatial positions and line-of-sight directions, providing complementary observations of areas missing by the fixed-position binocular acquisition device due to occlusion.
[0030] Based on the mapping relationship established in step two, the command platform projects pixels of the same target object in the first-view motion video onto the binocular depth coordinate system, registering them pixel-by-pixel with the corresponding region in the target area depth map. For regions with valid depth, a weighted fusion is performed, primarily using the depth from the fixed-point binocular acquisition device and secondarily using the first-view observation; for regions lacking depth due to occlusion, depth estimation is supplemented using lateral observations from the first-view acquisition device. This complementary depth enhancement employs the following confidence-weighted fusion: ,in: To enhance depth maps at the pixel level The enhanced depth value at the location is a scalar with a value range of [0.3, 100] and a unit of m. It is calculated by this formula and represents the final depth of the pixel after fusing fixed-point binocular observation and first-view observation. is the pixel coordinate in the image, a two-dimensional vector with dimension 2, whose value range is determined by the image resolution, dimensionless, and represents the pixel position currently being processed; For the depth map of the target region in pixels The binocular depth value given by the fixed-point binocular acquisition device is a scalar with a value range of [0.3, 100] and a unit of m. It is obtained by calculating the binocular disparity in step one. The value of 0 is taken at the occlusion and missing parts to indicate invalidity, which represents the fixed-point binocular observation component. The first-person view acquisition device projects the data onto pixels using a mapping relationship. The estimated first-view depth value is a scalar with a value range of [0.3, 100] and a unit of m. It is estimated by combining the first-view motion video with the mapping relationship and represents the first-view observation component. For fixed-point binocular observation at pixels The confidence weight at a given point is a scalar with a value range of [0, 1]. It is dimensionless, takes a value close to 1 when the binocular depth is valid, and takes a value of 0 when there is occlusion. It is determined by the matching cost of binocular disparity and represents the confidence level of fixed-point binocular observation. For first-person perspective observation in pixels The confidence weight at a given point is a scalar, ranging from [0, 1], dimensionless, and determined by the angle between the observation line of sight from the first-view acquisition device to that pixel and the normal to the surface of the obstruction. The closer the angle is to orthogonal, the more reliable the observation, representing the reliability of the first-view lateral observation. Through the above complementary depth enhancement, in the region where the depth of the fixed-point binocular acquisition device is effective... Approaching 1 The primary focus is on achieving accurate fusion results; in occluded or missing areas... Invalid and Setting it to 0 means the fusion result is entirely based on first-person perspective observations. This provides the depth information, thus filling in the missing occluded areas and obtaining an enhanced depth map with a wider coverage.
[0031] Step S4: The reprojected residual region is identified as an occlusion candidate region and inverted into an occlusion-depth assessment signal. This step is the core and most counterintuitive innovation of this invention. In the conventional processing paradigm of binocular stereo matching, when the position of the same spatial point is inconsistent between two observations of a heterogeneous cross-baseline virtual stereo pair, a mismatch will be formed in the cost volume. This multi-view geometric inconsistency caused by occlusion is usually regarded as noise contaminating depth estimation and is either weighted and suppressed or directly masked out. This invention takes the opposite approach: instead of suppressing this geometric inconsistency, it utilizes it as a geometric encoding of the occlusion relationship.
[0032] The command platform projects the pixels of the same target object in the first-view motion video onto the binocular depth coordinate system through a mapping relationship, and compares them with the corresponding positions in the target area depth map to calculate the reprojection residual. The connected regions with larger reprojection residuals are the reprojection residual regions generated when heterogeneous cross-baseline virtual stereo pairs observe the same target object, and the command platform identifies these reprojection residual regions as occlusion candidate areas.
[0033] After identifying candidate occlusion areas, the command platform extracts the depth gradient along the depth direction of the reprojection residual region within these areas. The direction of the depth gradient is then used to determine the spatial relationship between the target object and the occluding object: if the depth gradient crosses the target object boundary from near to far, it indicates that the occluding object is in front of the target object; conversely, if the depth gradient crosses the target object boundary from near to far, it indicates that the target object is in front of the occluding object. Based on this spatial relationship, the command platform generates an occlusion-depth analysis signal, which includes the location of the occlusion area, the order of the occluding object and the same target object, and the occlusion boundary. In this way, geometric inconsistencies that would normally be discarded in conventional processing are transformed into useful signals for the command center to analyze occlusion relationships, and the occlusion area is transformed from a blind spot into an information-rich area.
[0034] Refinement of Step S4: Separation of Spatiotemporal Registration Error Components and Occlusion Components. In actual police work, the first-person view acquisition device is worn by the officer on duty and moves with him, while the relatively stationary binocular acquisition device is in motion, and the two acquisition frame rates are inconsistent (30fps and 25fps). This heterogeneity brings about a subtle problem: the reprojection residual region calculated in Step 4 contains both occlusion components truly caused by occlusion and spatiotemporal registration error components caused by the movement of the first-person view acquisition device and the inconsistent frame rate, resulting in the two data streams not strictly corresponding to the same physical moment. If the entire reprojection residual region is indiscriminately treated as an occlusion candidate region, the spatiotemporal registration error component will be misjudged as occlusion, leading to false judgments.
[0035] To address this, before extracting the depth gradient, the command platform first decomposes the reprojection residual region into occlusion components and spatiotemporal registration error components. Then, it separates and removes the spatiotemporal registration error component based on the temporal continuity of the depth sequence of the same target object on the target region's depth map. This is because true occlusion is determined by scene geometry and manifests as a continuous deviation in the depth time sequence; while spatiotemporal registration error is caused by the motion of the first-view acquisition device and manifests as discontinuous jumps in the depth time sequence accompanying sudden changes in motion.
[0036] Specifically, the command platform performs change point detection on the depth time series of the same target object to locate the registration jump moments caused by the movement of the first-view acquisition equipment. The change point detection uses the following sliding double-window mean difference statistic: ,in: For deep time series at time The change point detection statistic is a scalar with a value range of [0, 100] and a unit of m. It is calculated by this formula and represents the time interval. The larger the abrupt change in the mean depth between two consecutive time windows, the more likely that moment is a registration jump moment. is the frame time sequence number of the depth time series, a scalar positive integer with a value range of [L+1, N-L+1], dimensionless, representing the current detection time; The length of the sliding window is a scalar positive integer, set to 5 in this embodiment, with a range of [3, 10]. It is dimensionless and determined by the acquisition frame rate and the typical movement speed of the personnel on duty. If the value is too small, the statistical value is easily affected by single-frame noise. If it is too large, it will smooth out the real jumps, representing the number of frames participating in the mean statistics; Let be the summation frame number within the window, a scalar positive integer, dimensionless, and iterate through the time points in the two summation formulas respectively. Previous Frame and self-time Rising frame; For the same target object in the frame The representative depth value at a given location is a scalar, ranging from [0.3, 100], with units of meters. It is obtained from the median depth of the region corresponding to the encoded target object in the target region depth map, representing the depth of the target object within the frame. Distance; Summation symbol Indicates to from to Accumulate frame by frame. Indicates to from to Accumulate frame by frame; This indicates the absolute value operation. When... When the preset transition threshold is exceeded, the command platform determines the timing. For the registration jump moment, the reprojection residual corresponding to that moment is identified as a spatiotemporal registration error component. The command platform assigns a decay weight to the reprojection residual corresponding to the registration jump moment and then discards it. ,in: The occlusion component retained after removing the spatiotemporal registration error component is at time... The residual value, a scalar, ranges from [0, 50] and is in pixels. It is calculated using this formula and represents the time interval. The reprojection residuals are truly attributable to occlusion; For a moment The original reprojection residual value is a scalar with a value range of [0, 50] and a unit of pixel. It is calculated from the reprojection residual in step four and represents the original residual that does not distinguish between occlusion and registration error. For a moment The attenuation weight is a scalar with a value range of (0, 1). It is dimensionless and is calculated by this formula. It takes a value close to 0 at the registration jump moment to eliminate registration error and a value close to 1 at the non-jump moment to retain the occlusion component, which represents the degree of preservation of the original residual. For the natural constant An exponential function with base 0; Let be the attenuation steepness coefficient, a scalar value, which is 0.8 in this embodiment, with a value range of [0.3, 2.0], and the unit is . The discrimination of the jump and non-jump statistics is determined. If the value is too large, the attenuation will be too steep, causing the occlusion components near the critical point to be mistakenly deleted. If the value is too small, the registration error will not be completely eliminated, which indicates the steepness of the attenuation curve; The definition is the same as the aforementioned change point detection formula; The threshold for determining the transition is a scalar value, set to 0.3 in this embodiment, with a range of [0.1, 1.0] and a unit of meters (m). This threshold is determined by the non-transition time period. The statistical upper bound is determined, representing the critical statistic between quasi-jumps and normal fluctuations in the distribution region. After the above separation and elimination, the remaining occlusion component... Only the corresponding regions are used as final occlusion candidate areas for depth gradient extraction and spatial relationship determination, thus avoiding false occlusion assessments caused by the movement of the first-person view acquisition equipment. The separated spatiotemporal registration error components reflect the relative pose deviation of the heterogeneous cross-baseline virtual stereo pair at that moment, which can be further used for online calibration and correction of heterogeneous equipment, ensuring that the mapping relationship remains accurate during the continuous movement of on-duty personnel.
[0037] Step S5: Depth extrapolation and completion of occluded areas, generation of comprehensive situational information and command annotations. After steps three and four, the enhanced depth map has obtained the first-view completion depth in most occluded areas. However, in depth occluded areas that are also unobservable by the first-view acquisition equipment, there are still pixels without direct depth due to parallax loss. For these pixels, the command platform performs depth extrapolation and completion based on the spatial relationship between the occlusion and depth analysis signal and the boundary of the same target object along the epipolar direction. The epipolar geometry constrains the correspondence of the same spatial point in the two observations, ensuring that the completed depth is consistent with the known occlusion relationship. The depth extrapolation and completion adopts the following weighted extrapolation based on the epipolar direction and the occlusion relationship: ,in: For pixels with missing parallax in the occluded area The completion depth obtained by extrapolation is a scalar with a value range of [0.3, 100] and a unit of m. It is calculated by this formula and represents the depth of the missing pixel after completion. The coordinates of the missing disparity pixels to be filled in the occluded area are given. They are two-dimensional vectors with a dimension of 2 and no dimension, representing the current position of the pixel to be filled in. For along the pixel The polar direction is selected as the first The coordinates of a reference pixel with known depth are two-dimensional vectors with a dimension of 2 and no dimension, obtained by sampling along the epipolar direction. Reference pixel The enhancement depth value at that location is defined in the same way as the aforementioned complementary depth enhancement formula, and the unit is m; The total number of reference pixels selected along the epipolar direction is a scalar positive integer, which is 8 in this embodiment. The value range is [4, 16], which is dimensionless and is determined by the typical width of the occlusion area. It represents the number of reference pixels participating in the extrapolation. , All are summation indices of reference pixels, which are scalar positive integers with a value range of [1, M] and are dimensionless; Reference pixel The extrapolated weight is a scalar with a value range of [0,1] and is dimensionless. It is calculated by this formula and represents the contribution of the reference pixel to the completion depth. Here, is the distance attenuation coefficient, a scalar value, taken as 0.5 in this embodiment, with a range of [0.2, 1.5], and the unit is . It is determined by the sharpness of the depth change at the occlusion boundary. If the value is too large, only the nearest neighbor reference pixel will be effective, leading to overfitting during extrapolation. If the value is too small, the distant reference pixels will introduce an incorrect depth across the occlusion boundary, which indicates the locality of the extrapolation; Reference pixel To the pixel to be completed The pixel distance along the epipolar line is a scalar with a value range of [1, 64] and a unit of pixel. It is calculated from the pixel coordinate difference and represents the distance between the reference pixel and the pixel to be completed. The meaning is the same as ; For the natural constant An exponential function with base 0; This is a function indicating the consistency of the relationship before and after occlusion. It is a scalar, taking the value 0 or 1, and is dimensionless. When the reference pixel... With the pixels to be completed A value of 1 is assigned when the reference pixel is on the same side of the occlusion boundary (i.e., in the same order), and 0 otherwise. The order is determined by the occlusion-depth analysis signal, characterizing whether the reference pixel can be used for extrapolation across the occlusion boundary. (Summarization symbol) Indicates to From 1 to Accumulate them one by one. The command platform will complete the depth. The enhanced depth map is incorporated to obtain an enhanced depth map that also has complete depth in occluded areas. The command platform generates comprehensive situational information based on this enhanced depth map and the occlusion-depth analysis signal. This comprehensive situational information includes the on-site viewpoint, the 3D position of the target object in the binocular depth coordinate system, the spatial relationship between the target object and the occluder, and risk warnings derived from the spatial relationship. Command personnel generate command annotations in the binocular depth coordinate system based on the comprehensive situational information. These command annotations include an annotation number, target object code, annotation type, 3D coordinates in the binocular depth coordinate system, display style, and effective time. Annotation types include text prompts, target boxes, directional arrows, and path guide lines.
[0038] Step S6: Inversion and Retrospective Projection of Command Annotations and Stable Display of Retrospective Annotations. After the command annotations are generated in the binocular depth coordinate system, they need to be retrospectively projected onto the first-person perspective motion view of the personnel on duty. Based on the first-person perspective pose signal, the command platform inverts and projects the 3D coordinates of the command annotations from the binocular depth coordinate system to the first-person perspective coordinate system through a mapping relationship, and then projects them onto the imaging plane of the first-person perspective acquisition device to obtain the pixel landing points of the retrospective annotations in the first-person perspective motion video. The retrospective annotations are then superimposed and displayed on the first-person perspective acquisition device. Because the command annotations carry 3D anchor points provided by the enhanced depth map, the retrospective annotations can be firmly attached to the 3D target objects on site, and even if the target objects are in occluded areas, they can still fall into the correct spatial position based on the completed depth.
[0039] However, the first-person view acquisition equipment is worn by personnel on duty, and its pose is subject to jitter. Direct reverse projection will cause the returned annotation landing point to jump between adjacent frames, affecting the personnel's viewing experience. Therefore, when the pose jitter of the first-person view acquisition equipment causes the returned annotation landing point to jump, the command platform constructs a joint confidence score for the landing point based on the completion depth and the first-person view pose signal, and performs spatiotemporal filtering on the returned annotation landing point based on the joint confidence score. The joint confidence score and spatiotemporal filtering are calculated as follows: ,in: To return the annotation landing point at the 1st The joint confidence score of the frame's landing point is a scalar with a value range of (0, 1] and is dimensionless. It is calculated by this formula. The higher the confidence score, the more reliable the landing point of the current frame is and the more it should be adopted. It represents the reliability of the current frame's reversal result. The frame number of the first-person perspective motion video is a scalar positive integer, dimensionless, representing the current frame being processed. For the first The local standard deviation of the completion depth of the target object to which the frame return annotation is attached is a scalar, ranging from [0, 5], with units of meters. It is derived from the completion depth. The statistics obtained within the labeled neighborhood represent the stability of the completion depth; the more unstable the depth, the lower the confidence level. For the first Frame relative to the first The magnitude of the first-view pose change of a frame is a scalar with a value range of [0, 2]. It is dimensionless (the position component is normalized and then weighted and synthesized with the attitude component). It is calculated from the difference between adjacent frames of the first-view pose signal and represents the pose jitter amplitude. The greater the jitter, the lower the confidence level. The L2 norm of a vector; Here, is the deep instability penalty coefficient, a scalar, which is set to 0.4 in this embodiment, with a value range of [0.1, 1.0], and the unit is . The strength of the penalty for depth instability is determined by the typical noise level of the completed depth. The pose jitter penalty coefficient is a scalar, which is 0.6 in this embodiment and ranges from [0.1, 1.5]. It is dimensionless and is determined by the typical jitter amplitude of the first-view acquisition device, representing the penalty intensity for pose jitter. For the first The coordinates of the returned annotation landing point output after the frame is spatiotemporally filtered are two-dimensional vectors with a dimension of 2 and a unit of pixels. They are calculated by this formula and represent the final superimposed annotation landing point. For the first The original landing point coordinates obtained by direct frame re-projection are two-dimensional vectors with a dimension of 2 and a unit of pixels. They are obtained from the re-projection in step six and represent the unfiltered original landing point. For the first The landing point coordinates output after the frame is spatiotemporally filtered are two-dimensional vectors with a dimension of 2 and a unit of pixels. They are the filtered output of the previous frame and represent the historical landing points. For the natural constant An exponential function with base 0.
[0040] Through the above spatiotemporal filtering, the joint confidence of the landing point is obtained when the pose jitter is large or the completion depth is unstable. As the position decreases, the output landing point retains more of the historical landing point from the previous frame to suppress jumps; when the pose is stable and the depth is reliable... The output landing point closely follows the current back-projection result to ensure consistency. Furthermore, when multiple first-person view acquisition devices generate back-transmission annotations for the same target object, since the back-transmission annotations of each device are all anchored to the same three-dimensional position corresponding to the encoding of the same target object in the binocular depth coordinate system, after their respective spatiotemporal filtering, the landing points of multiple back-transmission annotations converge to the projection of the same three-dimensional position in their respective fields of view. This achieves consistent convergence of the back-transmission annotation landing points of multiple first-person view acquisition devices for the same target object, avoiding contradictions between the annotations of the same target object seen by different duty personnel.
[0041] Step S7: Manual Confirmation and Handling Feedback. After reviewing the transmitted annotations using the first-person view acquisition device, officers manually confirm the information based on the actual situation on site. The command platform receives the manual confirmation signal from the officers regarding the transmitted annotations. This signal can be one of the following: confirmation, rejection, modification, supplementary explanation, request for support, or handling completed. The command platform transmits the manual confirmation signal back to the command terminal that generated the command annotation and updates the comprehensive situation information based on the signal, ensuring that command personnel and all on-site terminals are simultaneously aware of the latest handling progress. The system output of this invention is only for auxiliary prompts; the final handling decision is made independently by on-site officers in accordance with the law. This invention does not involve automated law enforcement, automated punishment, or automated arrest functions.
[0042] Step S8: Encrypted Transmission and Log Recording. The data transmission relied upon in the above steps is executed via a heterogeneous dual-channel encrypted channel. In this embodiment, the heterogeneous dual channel is composed of a 5G network slicing channel and a public security dedicated network channel, which serve as backups for each other and are scheduled according to data priority. When the heterogeneous dual-channel encrypted channel is interrupted, the first-view motion video and target area depth map are locally cached by the first-view acquisition device and the fixed-point binocular acquisition device. After recovery, the local cache is synchronized to the command terminal to ensure data integrity. The command platform records the entire process of first-view motion video and target area depth map acquisition, transmission, command annotation generation, return annotation resubmission, manual confirmation, and handling feedback. Each log entry includes a log number, timestamp, task number, device number, target object code, operation type, and operation content, forming a traceable collaborative command closed loop.
[0043] Example 2: Implementation of the method in the scenario of remote three-dimensional inspection of key areas.
[0044] This embodiment uses remote three-dimensional border patrol as an application scenario. Its characteristics lie in the significant distance between the personnel on duty and the key observation area, and the dense obstructions such as vegetation and embankments within the observation area, making the analysis of the spatial relationship between the obstructed areas particularly crucial. Several fixed-point binocular acquisition devices are deployed along the border to achieve comprehensive coverage without blind spots. Personnel on duty wear first-person view acquisition devices while patrolling the patrol routes.
[0045] During the patrol, the fixed-point binocular acquisition equipment generated a depth map of the target area and found a suspicious target moving about 80m from the border, but it was sometimes obscured by earthen embankments. The command platform, following the steps described in Example 1, constructs a heterogeneous cross-baseline virtual stereo pair between the first-view acquisition device worn by the patrol personnel and the fixed-point binocular acquisition device. It uses the patrol personnel's lateral first-view observation to complete part of the depth of the embankment-occluded area, obtaining an enhanced depth map. The reprojection residual region of the target object's edge is identified as an occlusion candidate area, and depth gradients are extracted to determine that the suspicious target object is located behind the embankment. An occlusion-depth analysis signal is generated, including the occlusion area location, front-back order, and occlusion boundary. Simultaneously, change point detection is performed on the depth time series to eliminate spatiotemporal registration error components caused by the patrol personnel's movement. For disparity-deficient pixels still unobservable deep within the embankment, depth extrapolation is performed along the epipolar direction to complete the complete spatial position of the suspicious target object, generating corresponding comprehensive situational information and command annotations. The command annotations are inverted and projected back to the patrol personnel's first-view motion field of view. The back-transmitted annotations are stably attached to the spatial position of the suspicious target object behind the embankment. When another reinforcement patrol personnel approaches from a different direction, their back-transmitted annotations converge to the same three-dimensional position as the original patrol personnel's back-transmitted annotations. Patrol personnel conduct manual confirmation based on on-site observations and select the nearest dispatch point. The system updates the overall situational information accordingly, and the entire process is logged. This embodiment overcomes the bottleneck of traditional two-dimensional monitoring and single-source binocular cameras being unable to determine the spatial relationship between front and back in occluded areas by coordinating heterogeneous cross-baseline virtual stereo pairs with occlusion-depth analysis signals.
[0046] Example 3: Implementation of methods in emergency response scenarios.
[0047] This embodiment uses emergency response to a sudden public safety incident in an urban commercial district as its application scenario. Its characteristics include a complex on-site structure, frequent obstructions, and dense crowds, placing the highest demands on depth assessment of obstructed areas and consistency of multi-terminal annotation. Multiple on-site personnel wearing first-person view (FPV) acquisition devices initiate FPV live streaming, while nearby fixed-point binocular acquisition devices automatically switch to emergency mode to acquire binocular video at a higher frame rate and generate a depth map of the target area.
[0048] The command platform processes data according to steps one through eight of Example 1: First-person perspective acquisition devices of multiple on-site personnel are used to construct heterogeneous cross-baseline virtual stereo pairs with nearby fixed-point binocular acquisition devices to complement and complete the depth maps obscured by building components and crowds; geometric inconsistencies in obscured areas are reversed into obscuration-depth analysis signals to determine whether evacuation routes are obstructed by obstacles and what kind of obstruction structure the trapped personnel are behind; spatiotemporal registration error components are eliminated through change point detection when multiple on-site personnel move rapidly; epipolar depth extrapolation is performed to complete the depth obscured areas, thereby accurately delineating a 50m radius three-dimensional warning zone and planning evacuation routes along unobstructed safety passages; command annotations are inverted and projected back to the first-person perspective acquisition devices of on-site personnel and reinforcement forces, and spatiotemporal filtering of landing points is performed on on-site personnel with severe pose jitter to ensure that the landing points of the returned annotations from multiple on-site personnel for the same trapped person and the same warning boundary converge and become consistent. After reviewing the transmitted annotations, on-site personnel discovered an obstacle in the western passageway. They manually confirmed the obstacle by selecting the correct type and adding information suggesting a detour via the eastern passageway. The command platform then updated the overall situational awareness and regenerated evacuation route annotations, which were then transmitted back to all terminals. The entire process was logged. This example fully demonstrates the value of the synergy between heterogeneous cross-baseline virtual stereo enhancement, occlusion-depth analysis signal inversion, and command annotation depth retransmission in emergency response to complex occlusion scenarios.
[0049] Example 4: MR and binocular vision remote collaborative command system for policing.
[0050] This embodiment provides a detailed description of the MR and binocular vision-based remote collaborative command system for policing that implements the above-described method. The processing logic executed by each module in this embodiment is consistent with the corresponding steps in Embodiments 1 to 3; the similarities will not be repeated. The focus is on describing the composition, connection relationships, and data flow of each module. This system includes an MR terminal, a fixed-point binocular device, a command platform, a labeling and feedback module, a manual confirmation module, and a log auditing module. These modules are connected via a heterogeneous dual-channel encrypted channel, forming a collaborative command closed loop encompassing on-site perception, remote analysis, command overlay, manual confirmation, and log recording.
[0051] The MR terminal, worn by personnel on duty, is used to collect first-person perspective motion video and pose signals, and then overlays the returned annotations onto the MR terminal. The MR terminal includes a first-person perspective acquisition unit, a pose acquisition unit, a field interaction unit, and a display unit. The first-person perspective acquisition unit collects high-definition audio and video from the personnel's field of vision, outputting first-person perspective motion video at a resolution of 1920×1080 pixels and a frame rate of 30fps. The pose acquisition unit collects the spatial position and attitude of the MR terminal, outputting first-person perspective pose signals, including three position components (longitude, latitude, and altitude) and three attitude components (pitch, yaw, and roll), forming a six-degree-of-freedom pose description. The field interaction unit collects voice commands, gestures, and physical button inputs from personnel on duty, used to trigger manual confirmation. The display unit uses an optical perspective scheme to overlay the returned annotations onto the personnel's actual field of vision, with a display resolution of no less than 1920×1080 pixels. As the first-view acquisition device of this system, the MR terminal acquires first-view motion video and first-view pose signal, which are then uploaded to the command platform via a heterogeneous dual-channel encrypted channel to participate in the construction of heterogeneous cross-baseline virtual stereo pairs. The first-view pose signal is also used by the annotation and feedback module for command annotation inversion and reprojection.
[0052] A fixed-point binocular device is deployed in key areas requiring spatial awareness to acquire binocular video of the target area and generate a depth map of the target area based on binocular parallax. The fixed-point binocular device includes a left-eye camera unit, a right-eye camera unit, and a depth calculation unit. The left and right-eye camera units synchronously acquire binocular video of the target area using a known baseline distance and a known intrinsic parameter matrix. In this embodiment, the baseline distance is 0.12m, the focal length of the left and right eyes is 1200 pixels, and the acquisition frame rate is 25fps. The depth calculation unit calculates the depth pixel-by-pixel based on binocular parallax using a stereo matching method known in the art to generate a depth map of the target area, achieving a depth accuracy better than 0.05m at an observation distance of 10m. As the fixed-point binocular acquisition device of this system, the target area depth map output by the fixed-point binocular device is uploaded to the command platform via a heterogeneous dual-channel encrypted channel. In areas with effective depth, it serves as the primary depth source for complementary depth enhancement; in occluded or missing areas, it is supplemented by first-person perspective observations from the MR terminal.
[0053] The command platform is the core processing and decision-making module of this system. It is used to perform time synchronization, spatial correlation, and target object encoding on first-view motion video, first-view pose signal, and target area depth map to establish the mapping relationship between the first-view coordinate system and the binocular depth coordinate system. It constructs a heterogeneous cross-baseline virtual stereo pair between the MR terminal and the fixed-point binocular device and performs complementary depth enhancement on the same target object to obtain an enhanced depth map. It determines the reprojection residual area generated by the heterogeneous cross-baseline virtual stereo pair observing the same target object as the occlusion candidate area and reverses the occlusion candidate area from the noise to be suppressed in the binocular stereo matching to the occlusion-depth judgment signal. Based on the enhanced depth map and the occlusion-depth judgment signal, it generates comprehensive situational information and generates command annotations in the binocular depth coordinate system.
[0054] The command platform includes a time synchronization unit, a spatial association unit, a target object encoding unit, a stereo enhancement unit, an occlusion analysis unit, a situation fusion unit, and a command generation unit. The time synchronization unit, spatial association unit, and target object encoding unit respectively perform the timestamp alignment, mapping relationship solving, and target object encoding described in step two of Embodiment 1. The stereo enhancement unit performs the complementary depth enhancement described in step three of Embodiment 1, constructing a heterogeneous cross-baseline virtual stereo pair between the MR terminal and the fixed-point binocular device. In the effective depth area of the fixed-point binocular device, the binocular depth is the primary method, while in the occlusion-deficient area, first-view observation is used to complete the image, resulting in an enhanced depth map. The occlusion assessment unit executes the processing described in step four and its refinement in embodiment one, as well as step five. First, constrained by the temporal continuity of the same target object, it eliminates spatiotemporal registration error components caused by MR terminal motion through change point detection, retaining the occlusion component. Then, it extracts the spatial relationship before and after the depth gradient determination within the occlusion candidate area, generating an occlusion-depth assessment signal including the occlusion area position, before and after sequence, and occlusion boundary. Finally, it performs depth extrapolation and completion along the epipolar direction for disparity-deficient pixels within the occlusion area to obtain the completed depth, which is then incorporated into the enhanced depth map. The situation fusion unit generates comprehensive situation information based on the enhanced depth map and the occlusion-depth assessment signal, including the on-site viewpoint, the 3D position of the target object, the before and after spatial relationship, and risk warnings, and displays it on the command center's large screen. The command generation unit generates command annotations in the binocular depth coordinate system according to the commands of the command personnel. The command annotations include annotation number, target object code, annotation type, 3D coordinates, display style, and effective time.
[0055] The annotation return module is used to invert and project command annotations from the binocular depth coordinate system back to the first-view coordinate system based on the first-view pose signal, thus obtaining the returned annotations. The annotation return module includes a coordinate inversion unit, a display adaptation unit, and a landing point stabilization unit, which respectively perform the coordinate inversion projection, display style adaptation, and landing point spatiotemporal filtering described in step six of Embodiment 1. Because the command annotations carry 3D anchor points containing the depth of the occluded area, the returned annotations can be attached to the correct spatial position of the 3D target object on site, and the landing points of returned annotations from multiple MR terminals for the same target object converge consistently. The returned annotations output by the annotation return module are sent to the display unit of the MR terminal for superimposed display via a heterogeneous dual-channel encrypted channel.
[0056] The manual confirmation module receives manual confirmation signals from on-duty personnel in response to the returned annotations. The module includes a confirmation input unit and a feedback transmission unit. The confirmation input unit receives the manual confirmation signal input by on-duty personnel via the MR terminal's on-site interaction unit. The manual confirmation signal can be one of the following: confirmation, rejection, modification, supplementary explanation, request for support, or action already taken. The feedback transmission unit transmits the manual confirmation signal back to the command platform's instruction generation unit, which updates the comprehensive situational information based on the signal. The system's output is only for auxiliary prompts; the final decision rests with on-site personnel who independently judge and confirm the action in accordance with the law. This system does not involve automated law enforcement, automated punishment, or automated arrest functions.
[0057] The log auditing module is used to record the entire process of acquiring, transmitting, generating command annotations, retransmitting returned annotations, manually confirming and responding to feedback from first-person perspective motion videos and target area depth maps. The module records data categorized by acquisition, transmission, command, confirmation, feedback, and anomalies. Each log entry includes a log number, timestamp, task number, device number, target object code, operation type, and operation content, forming a traceable collaborative command closed loop.
[0058] Data transmission between the above modules is performed through a heterogeneous dual-channel encrypted channel. The heterogeneous dual-channel encrypted channel consists of two physical channels that serve as backups for each other and are scheduled according to data priority. In the event of an interruption, the MR terminal and the fixed-point binocular device locally cache the first-view motion video and the target area depth map, and synchronize them to the command platform after recovery to ensure data integrity.
[0059] In summary, this system acquires first-view motion video and pose signals from an MR terminal, generates a depth map of the target area using a fixed-point binocular device, constructs a heterogeneous cross-baseline virtual stereo pair on the command platform and reverses the geometric inconsistencies of occlusion areas into occlusion-depth analysis signals, an annotation feedback module reverts the command annotation depth back to the system, a manual confirmation module completes on-site confirmation, and a log auditing module records the entire process. These modules work together to achieve reliable depth analysis of occluded areas and accurate on-site overlay of commands by the command end under heterogeneous spatiotemporal references and significant occlusion conditions. The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A remote collaborative command method for MR and binocular vision policing, characterized in that, include: The system acquires first-person perspective motion video and first-person perspective pose signal of the personnel on duty through first-person perspective acquisition equipment, acquires binocular video of the target area through fixed-point binocular acquisition equipment, and generates a depth map of the target area based on binocular parallax. The motion video from the first viewpoint, the pose signal from the first viewpoint, and the depth map of the target area are synchronized in time, spatially correlated, and the target object is encoded to establish a mapping relationship between the first viewpoint coordinate system and the binocular depth coordinate system. Based on the mapping relationship, the first view acquisition device and the fixed-point binocular acquisition device are constructed as a heterogeneous cross-baseline virtual stereo pair, and complementary depth enhancement is performed on the same target object to obtain an enhanced depth map; The reprojection residual region generated by the heterogeneous cross-baseline virtual stereo pair observing the same target object is determined as the occlusion candidate region, and the occlusion candidate region is reversed from the noise to be suppressed by binocular stereo matching into the occlusion-depth judgment signal. Based on the enhanced depth map and the occlusion-depth analysis signal, comprehensive situational information is generated, and based on the comprehensive situational information, command annotations are generated in the binocular depth coordinate system; Based on the first viewpoint pose signal, the command annotation is inverted from the binocular depth coordinate system and projected back to the first viewpoint coordinate system to obtain the back-transmitted annotation, which is then superimposed and displayed on the first viewpoint acquisition device.
2. The MR and binocular vision-based remote collaborative command method for policing according to claim 1, characterized in that, The step of inverting the occlusion candidate region from the noise to be suppressed in binocular stereo matching to the occlusion-depth judgment signal includes: Within the occlusion candidate region, the depth gradient along the depth direction of the reprojection residual region is extracted, and the direction of the depth gradient is determined as the spatial relationship between the same target object and the occluder. The occlusion-depth judgment signal is generated based on the spatial relationship between the front and back. The occlusion-depth judgment signal includes the location of the occlusion area, the front and back order of the occluding object and the same target object, and the occlusion boundary.
3. The MR and binocular vision-based remote collaborative command method for policing according to claim 2, characterized in that, Before extracting the depth gradient along the depth direction of the reprojection residual region, the method further includes: When the first view acquisition device moves relative to the fixed-point binocular acquisition device and the acquisition frame rate is inconsistent, the reprojection residual region is decomposed into occlusion component and spatiotemporal registration error component. Using the temporal continuity of the depth of the same target object on the depth map of the target region as a constraint, the spatiotemporal registration error component is separated and eliminated, while the occlusion component is retained as the occlusion candidate region. The step of separating and removing the spatiotemporal registration error component includes: performing change point detection on the depth time series of the same target object, locating the registration jump moment caused by the movement of the first view acquisition device in the depth time series, and assigning a decay weight to the reprojection residual corresponding to the registration jump moment to remove it, thereby obtaining the occlusion component after removing the spatiotemporal registration error component.
4. The MR and binocular vision-based remote collaborative command method for policing according to claim 3, characterized in that, Before generating comprehensive situational information based on the enhanced depth map and the occlusion-depth analysis signal, the method further includes: For pixels in the occlusion candidate region that have no direct depth due to parallax loss, depth extrapolation and completion are performed along the epipolar direction based on the spatial relationship between the front and back of the occlusion-depth judgment signal and the boundary of the same target object to obtain the completion depth of the occlusion candidate region. The completion depth is incorporated into the enhanced depth map; The step of obtaining and overlaying the returned annotations onto the first viewpoint acquisition device includes: when the pose jitter of the first viewpoint acquisition device causes the landing point of the returned annotations to change, constructing a joint confidence score of the landing point based on the completion depth and the first viewpoint pose signal, and performing spatiotemporal filtering on the landing point of the returned annotations based on the joint confidence score of the landing point, so that the landing points of the returned annotations generated by multiple first viewpoint acquisition devices for the same target object converge to be consistent; and calculating the depth confidence score of the depth map of the target area, displaying a depth uncertainty mark for the corresponding area when the depth confidence score is lower than a preset threshold, and correcting the depth of the corresponding area based on at least one of auxiliary sensor data, historical frame data and remote manual confirmation results before incorporating it into the enhanced depth map.
5. The MR and binocular vision-based remote collaborative command method for policing according to claim 1, characterized in that, The process of generating comprehensive situational information based on the enhanced depth map and the occlusion-depth analysis signal includes: abstracting the same target object and at least one of the on-site police officers, parties involved, vehicles, shelters, passages, hazard sources, and evidence into nodes in the police spatial situational map; abstracting the distance relationships, occlusion relationships, line-of-sight relationships, passage relationships, hazard sector relationships, control relationships, and command relationships between the nodes into edges in the police spatial situational map; and using the police spatial situational map as the comprehensive situational information. Each node includes object type, three-dimensional spatial coordinates, movement status, risk label, and confidence level; and each edge includes distance value, occlusion status, passage status, hazard level, and timestamp. Furthermore, the process involves calculating a comprehensive risk level based on the nodes and edges. The comprehensive risk level is obtained by fusing at least two of the following: distance risk, action risk, crowd risk, occlusion risk, passage risk, and procedural risk, and then writing it into the comprehensive situational information.
6. The MR and binocular vision-based remote collaborative command method for police affairs according to claim 1, characterized in that, The step of generating command annotations in the binocular depth coordinate system based on the comprehensive situational information includes generating structured command instructions based on the comprehensive situational information and mapping the structured command instructions to the command annotations. The structured command instructions include the command object, command action, target coordinates or target area, display level, urgency level, effective time and feedback requirements. When the feedback annotations are superimposed on the first view acquisition device, the feedback annotations include at least one of the following display objects: target highlight box, risk heat box, ground safety position circle, path arrow, no-entry sector, bunker marker, distance scale, program prompt bar and legal provision prompt card. Each of the display objects is anchored to the three-dimensional spatial position corresponding to the same target object in the field of view of the first view acquisition device according to the enhanced depth map.
7. The MR and binocular vision-based remote collaborative command method for policing according to claim 1, characterized in that, Also includes: Receive manual confirmation signals from on-duty personnel regarding the returned annotations, wherein the manual confirmation signal is one of the following: confirmation, rejection, modification, supplementary explanation, request for support, or action already taken; The manual confirmation signal is transmitted back to the command terminal that generated the command annotation, and the comprehensive situation information is updated based on the manual confirmation signal.
8. The MR and binocular vision-based remote collaborative command method for police affairs according to claim 1, characterized in that, The transmission upon which the time synchronization, spatial association, and target object encoding depend is performed via a heterogeneous dual-channel encrypted channel. When the heterogeneous dual-channel encrypted channel is interrupted, the first-view motion video and the target area depth map are locally cached. After the heterogeneous dual-channel encrypted channel is restored, the locally cached first-view motion video and the target area depth map are synchronized to the command end. The method further includes: logging the entire process of acquiring and transmitting the first-view motion video and the depth map of the target area, generating the command annotations, retransmitting the returned annotations, and manually confirming and handling feedback. Each log entry includes a log number, timestamp, task number, device number, target object code, operation type, and operation content.
9. A remote collaborative command system for MR and binocular vision policing, used to implement the remote collaborative command method for MR and binocular vision policing as described in any one of claims 1-8, characterized in that, include: The MR terminal is used to collect first-person perspective motion video and first-person perspective pose signal of the duty personnel, and to overlay the returned annotations on the MR terminal. A fixed-point binocular device is used to acquire binocular video of a target area and generate a depth map of the target area based on binocular parallax. The command platform is used to perform time synchronization, spatial correlation, and target object encoding on the first-view motion video, the first-view pose signal, and the target area depth map to establish a mapping relationship between the first-view coordinate system and the binocular depth coordinate system. Based on the mapping relationship, the MR terminal and the fixed-point binocular device are constructed as a heterogeneous cross-baseline virtual stereo pair, and complementary depth enhancement is performed on the same target object to obtain an enhanced depth map. The reprojection residual region generated by the heterogeneous cross-baseline virtual stereo pair observing the same target object is determined as an occlusion candidate region, and the occlusion candidate region is inverted from the noise to be suppressed in the binocular stereo matching to an occlusion-depth judgment signal. Based on the enhanced depth map and the occlusion-depth judgment signal, comprehensive situational information is generated, and command annotations are generated in the binocular depth coordinate system. The annotation feedback module is used to invert and project the command annotation from the binocular depth coordinate system back to the first viewpoint coordinate system based on the first viewpoint pose signal to obtain the feedback annotation; The manual confirmation module is used to receive manual confirmation signals from on-duty personnel regarding the returned annotations; The log auditing module is used to record logs of the entire process of acquiring and transmitting the first-view motion video and the depth map of the target area, generating the command annotations, retransmitting the returned annotations, and manually confirming and handling feedback. The debriefing management module is used to store the first-view motion video, the target area depth map, the enhanced depth map, comprehensive situation information, command annotations, feedback annotations and manual confirmation signals according to a unified timeline, and generate an interactive debriefing interface; The security audit module is used to manage identity authentication, access control, data anonymization, transmission encryption, instruction logging, and log auditing.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the MR and binocular vision police remote collaborative command method as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Distance measurement method and system based on binocular stereo vision
CN103245322A
Command center control system applied to public security bureau
CN107040761A
Man-machine interaction method based on museum guide AR glasses
CN114647315A
