Security monitoring system of duty center based on the integration of virtual reality and augmented reality

Abstract

The present invention belongs to the field of security monitoring technology, and specifically relates to a duty center security monitoring system based on the fusion of virtual reality and augmented reality, including: a data acquisition module, a preprocessing module, an image processing module, a VR projection module and an AR module; the image processing module is used to splice multiple images into a seamless panoramic image through a homography matrix estimation algorithm; the VR projection module is used to establish a spatial projection of a two-dimensional image to a three-dimensional scene; the AR module is used to generate corresponding virtual objects according to business needs, and superimpose the virtual objects to the corresponding world coordinate system position in the three-dimensional scene to form a virtual-real fusion AR video. The preprocessing module is used to perform noise reduction and optimization of video acquisition, and the seamless splicing technology of the image processing module and the VR projection module is combined to provide a high-definition and realistic virtual reality scene, providing a panoramic and immersive monitoring experience for the duty personnel.

Description

Technical Field

[0001] The present invention belongs to the technical field of security monitoring, and in particular, relates to a security monitoring system for a duty center based on the fusion of virtual reality and augmented reality. Background Art

[0002] With the advancement of science and technology and the development of virtual reality (VR) technology, the security field has gradually shifted from traditional passive monitoring to active defense and intelligent management; and is widely used in warehouse fire protection, industrial park security, public place security and other scenarios. However, the following shortcomings still exist in complex scenarios:

[0003] 1. The images provided by traditional surveillance cameras are usually fixed perspectives or multi-angle switching within a limited range, which makes it difficult to display a panoramic view and monitor the entire area.

[0004] Second, the monitoring screen and related data (such as temperature, humidity, smoke alarm, etc.) are usually displayed separately, requiring the on-duty personnel to switch views or refer to multiple information sources at the same time, resulting in low information integration efficiency. Summary of the invention

[0005] In order to solve the technical problems that traditional surveillance cameras are difficult to monitor the entire area in a panoramic manner; and the monitoring screen and related data are displayed separately, resulting in low information integration efficiency, the present invention provides a duty center security monitoring system based on the fusion of virtual reality and augmented reality. Through the AR module, key information (such as smoke concentration, temperature and humidity data, fire equipment status, etc.) is superimposed on the three-dimensional scene in the form of virtual objects, eliminating the need for frequent interface switching operations, and improving the convenience and efficiency of information acquisition.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] The duty center security monitoring system based on the integration of virtual reality and augmented reality includes data acquisition module, preprocessing module, image processing module, VR projection module and AR module;

[0008] The data acquisition module includes a camera and a sensor. The camera is used to collect video data; the sensor is used to detect environmental data;

[0009] The preprocessing module is used to preprocess the collected video data, including video format unification and noise removal;

[0010] The image processing module is used to stitch multiple images into a seamless panoramic image through a homography matrix estimation algorithm;

[0011] Preferably, the implementation process of the image processing module includes:

[0012] Use appropriate feature detectors to identify features in each image and generate feature descriptors for similarity matching of subsequent images;

[0013] In adjacent images, similar feature point pairs are found based on feature descriptors;

[0014] The distance metric method is used to determine the similarity between feature point pairs;

[0015] Based on the feature point pairs, the homography matrix between each pair of images is calculated using the least squares method or RANSAC;

[0016] Align all images to a common reference plane using the computed homography matrix;

[0017] Image blending technology is used to smooth the boundaries between aligned images and eliminate stitching artifacts to create a seamless panoramic image.

[0018] Preferably, the feature detector is a SIFT or ORB detection algorithm.

[0019] Preferably, the distance measurement method is the Euclidean distance method, and the value of the similarity is 0.7-0.8.

[0020] Preferably, the homography matrix is ​​used to describe the planar mapping relationship between two adjacent images; the mapping relationship expression of the homography matrix is:

[0021]

[0022] Where H is the homography matrix, a 3×3 matrix that describes the planar projection transformation relationship between two images, and is used to map a point [x1, y1, 1] in one image to a corresponding point [x2, y2, 1] in another image.

[0023] Preferably, the specific process of aligning all images to a common reference plane using the calculated homography matrix includes:

[0024] Select the image in the middle as the reference plane to avoid excessive deformation;

[0025] transforming image coordinates to a common reference plane;

[0026] Align the mapped pixels through pixel interpolation (bilinear or bicubic interpolation);

[0027] Starting from the reference image, the other images are aligned sequentially through homography transformation.

[0028] Preferably, the image blending technology is gradient domain fusion or multi-band blending technology.

[0029] The VR projection module is used to establish the spatial projection of two-dimensional images to three-dimensional scenes;

[0030] Preferably, the specific implementation process of the VR projection module includes:

[0031] S1) defining a camera coordinate system through the external parameter information of the camera, wherein the origin is located at the optical center of the camera, and the x, y and z axes of the camera are aligned with its image plane and the optical axis;

[0032] S2) defining a three-dimensional universal world coordinate system, which is used as a reference system for the three-dimensional scene;

[0033] S3) Calculate the transformation matrix between the world coordinate system and the camera coordinate system; the expression is:

[0034]

[0035] In the formula, [X c , Y c , Z c ] represents the camera coordinate system; [X w , Y w , Z w ] represents the world coordinate system; R is a 3×3 rotation matrix, which represents the rotation of the camera coordinate system relative to the world coordinate system; T is a 3×1 translation vector, which represents the displacement of the origin of the camera coordinate system relative to the world coordinate system; 0 and 1 are used to represent the expansion of homogeneous coordinates;

[0036] S4) extract key features (points, lines, planes) from 2D video frames and generate 3D point clouds from 2D video frames using structure from motion (SfM) or multi-view stereo (MVS) techniques;

[0037] S5) The internal parameter matrix of the camera is established through the internal parameters of the camera; its expression is:

[0038]

[0039] Where: f x and f y is the effective focal length in the horizontal and vertical directions (usually determined by the actual focal length f and the pixel scale factor); c x and c y is the coordinate of the principal point of the image (the pixel position of the center point of the image); 0 and 1 represent the expansion of homogeneous coordinates;

[0040] S6) Project the 3D point cloud in the camera coordinate system onto the 2D image plane to obtain its pixel coordinates; the calculation formula is:

[0041]

[0042] Where: [Xp , Y p ] is the normalized coordinate of the camera's image plane; [u, v] is the pixel coordinate; K is the camera's intrinsic parameter matrix;

[0043] S7) back-projecting the pixel coordinates of the 3D point cloud into the camera coordinate system, and then transforming them into the world coordinate system.

[0044] Preferably, the external parameters of the camera include position, attitude angle and PTZ, which are used to define the attitude of the camera in the 3D world coordinate system;

[0045] The internal parameters of the camera include focal length, image center point and pixel scaling ratio, which are used to define the geometric characteristics of the 2D image captured by the camera.

[0046] The AR module is used to generate corresponding virtual objects according to business needs, and superimpose the virtual objects to the corresponding world coordinate system positions in the three-dimensional scene to form an AR video that integrates virtual and real.

[0047] Preferably, the specific implementation process of the AR module includes:

[0048] Generate corresponding virtual objects according to the needs of warehouse fire protection business, including warehouse partition labeling, storage item information labeling, sensor data labeling and fire protection equipment status labeling;

[0049] Through the image overlay algorithm, the virtual object is superimposed on the corresponding world coordinate system position in the three-dimensional scene with appropriate transparency or occlusion relationship, forming an AR video that integrates virtuality and reality.

[0050] Preferably, the sensor data annotation includes real-time data display of smoke sensors and temperature and humidity sensors.

[0051] Beneficial effects of the present invention:

[0052] 1. The data acquisition and preprocessing modules perform video acquisition and noise reduction to improve data quality and processing efficiency; the image processing module and VR projection module provide high-definition and realistic virtual reality scenes through seamless splicing and spatial projection optimization, providing on-duty personnel with a panoramic and immersive monitoring experience.

[0053] 2. The AR module directly superimposes key information in the form of virtual objects in the three-dimensional scene, eliminating the need to switch between different interfaces, making it convenient for on-duty personnel to quickly obtain the required information.

[0054] 3. Improve the convenience of security management through intelligent scene and data monitoring and detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0056] Figure 1 It is a schematic diagram of the framework structure of the duty center security monitoring system based on the fusion of virtual reality and augmented reality of the present invention.

[0057] Figure 2 This is a flow chart of the VR projection module in the duty center security monitoring system based on the fusion of virtual reality and augmented reality in the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figure 1-Figure 2 As shown, the duty center security monitoring system based on the fusion of virtual reality and augmented reality includes a data acquisition module, a preprocessing module, an image processing module, a VR projection module and an AR module;

[0060] The data acquisition module includes a camera and a sensor. The camera is used to collect video data; the sensor is used to detect environmental data;

[0061] Specifically, the data acquisition module is mainly composed of two types of equipment: cameras and sensors, which are used to collect video data and environmental data respectively. They are used to support the image basis of VR space projection and the virtual objects of AR augmented reality respectively.

[0062] Taking the security monitoring needs of warehouse fire protection as an example, the present invention arranges high-definition network cameras, panoramic cameras or PTZ cameras at various key locations in the warehouse, including entrances and exits, main passages, around fire protection equipment and high-risk areas.

[0063] In addition, the internal parameters of the camera (focal length, image principal point position, etc.) are fixed after configuration and used to calculate the projection internal parameter matrix. The external parameters of the camera (position, attitude angle, pan / tilt angle) are calculated according to the specific image and used for subsequent space conversion and augmented reality mapping calculations.

[0064] In the specific implementation process, the data acquisition module also has a network transmission function, which transmits the video data to the preprocessing module through the RTSP or ONVIF protocol. The transmitted video data is marked with a timestamp for the subsequent spatial mapping and synchronization of virtual object superposition. Specifically, the time synchronization protocol (such as NTP) is used to calibrate the data acquisition time of the camera and sensor. This ensures that the multimodal data collected by different devices are aligned on the time axis.

[0065] The preprocessing module is used to preprocess the collected video data, including video format unification and noise removal;

[0066] Specifically, the specific process of video format unification is: standardizing the video resolution to a target resolution defined by the system (such as 1920x1080).

[0067] The video frame rate is unified (such as 30FPS) to meet the needs of real-time processing.

[0068] Synchronize video streams to avoid frame loss or inconsistent frame rates across multiple cameras.

[0069] Noise removal mainly uses inter-frame consistency filtering algorithms (such as optical flow method) to smooth time series noise; improve the signal-to-noise ratio of video images and remove light interference.

[0070] As an optional implementation, the preprocessing step also includes key frame extraction, using a motion detection algorithm (such as optical flow method or frame difference method) to select frames with significant motion areas, and then extracting key frames with more effective information to improve the processing efficiency of subsequent modules.

[0071] The image processing module is used to stitch multiple images into a seamless panoramic image through a homography matrix estimation algorithm;

[0072] Preferably, the implementation process of the image processing module includes:

[0073] Use appropriate feature detectors to identify features in each image and generate feature descriptors for similarity matching of subsequent images;

[0074] In adjacent images, similar feature point pairs are found based on feature descriptors;

[0075] The distance metric method is used to determine the similarity between feature point pairs;

[0076] Based on the feature point pairs, the homography matrix between each pair of images is calculated using the least squares method or RANSAC;

[0077] Align all images to a common reference plane using the computed homography matrix;

[0078] Furthermore, the image in the middle position is selected as the reference plane to avoid excessive deformation;

[0079] transforming image coordinates to a common reference plane;

[0080] Align the mapped pixels through pixel interpolation (bilinear or bicubic interpolation);

[0081] Starting from the reference image, the other images are aligned sequentially through homography transformation.

[0082] In addition, the alignment results of all images can be optimized to reduce the cumulative error.

[0083] Image blending technology is used to smooth the boundaries between aligned images and eliminate stitching artifacts to create a seamless panoramic image.

[0084] Specifically, the data acquisition module uses multiple cameras to collect images covering the panoramic scene, ensuring that there is a certain overlap area between the images to facilitate feature matching.

[0085] The feature detector is used to identify the significant features (such as corners and edges) in each image, and a descriptor (such as a 128-dimensional SIFT vector) is generated to facilitate subsequent similarity matching. The Euclidean distance metric is used to calculate similarity. Then the optimal matching point corresponding to each feature point in the other image is found. Among them, the matching point pairs are screened, and the threshold is set to 0.7-0.8 (based on the feature point distance ratio). Using the coordinates of the feature point pairs, the least squares method is used to fit the initial homography matrix. That is, randomly sampled point pairs and fit the homography matrix. Then the number of inliers in each group of points is calculated, and the model with the most inliers is selected as the final result.

[0086] Next, the image alignment is to use the homography matrix to align the mapped pixels through pixel interpolation (bilinear or bicubic interpolation). Finally, the image blending technology is used to eliminate the stitching traces and generate a seamless panoramic image.

[0087] Furthermore, the feature detector is a SIFT (Scale Invariant Feature Transform) or ORB (Oriented Fast Binary Feature) detection algorithm. Among them, SIFT is suitable for processing complex scene images and has strong scale and rotation invariance. ORB has high computational efficiency and is suitable for real-time processing scenes.

[0088] Furthermore, the distance measurement method is the Euclidean distance method, and the similarity value is 0.7-0.8.

[0089] Furthermore, the homography matrix is ​​used to describe the planar mapping relationship between two adjacent images; the mapping relationship expression of the homography matrix is:

[0090]

[0091] Where H is the homography matrix, a 3×3 matrix that describes the planar projection transformation relationship between two images, and is used to map a point [x1, y1, 1] in one image to a corresponding point [x2, y2, 1] in another image.

[0092] Furthermore, the image blending technology is a gradient domain fusion or a multi-band blending technology.

[0093] Specifically, the image processing module realizes efficient and accurate panoramic image stitching function through feature detection, feature matching, homography matrix calculation, image alignment and hybrid processing, providing high-quality visual data support for the subsequent VR and AR modules of the system.

[0094] The VR projection module is used to establish the spatial projection of two-dimensional images to three-dimensional scenes;

[0095] Furthermore, the specific implementation process of the VR projection module includes:

[0096] S1) defining a camera coordinate system through the external parameter information of the camera, wherein the origin is located at the optical center of the camera, and the x, y and z axes of the camera are aligned with its image plane and the optical axis;

[0097] S2) defining a three-dimensional universal world coordinate system, which is used as a reference system for the three-dimensional scene;

[0098] S3) Calculate the transformation matrix between the world coordinate system and the camera coordinate system; the expression is:

[0099]

[0100] In the formula, [X c , Y c , Z c ] represents the camera coordinate system; [X w , Y w , Z w ] represents the world coordinate system; R is a 3×3 rotation matrix, which represents the rotation of the camera coordinate system relative to the world coordinate system; T is a 3×1 translation vector, which represents the displacement of the origin of the camera coordinate system relative to the world coordinate system; 0 and 1 are used to represent the expansion of homogeneous coordinates;

[0101] Specifically, when calculating the transformation matrix between the world coordinate system and the camera coordinate system, a coordinate transformation process is usually involved.

[0102] The rotation matrix R is calculated from the camera's attitude angles (yaw, pitch, roll) and is defined using the Euler formula.

[0103] Translation Vector Determined by the position of the camera. At the same time, the PTZ parameters (the rotation angle of the pan / tilt) can further adjust the rotation matrix R.

[0104] Through the transformation matrix, any camera coordinate point [X c , Y c , Z c ] is converted to the world coordinate system point [X w , Y w , z w ], making it possible to relate the real world and the camera perspective.

[0105] S4) extract key features (points, lines, planes) from 2D video frames and generate 3D point clouds from 2D video frames using structure from motion (SfM) or multi-view stereo (MVS) techniques;

[0106] S5) The internal parameter matrix of the camera is established through the internal parameters of the camera; its expression is:

[0107]

[0108] Where: f x and f y is the effective focal length in the horizontal and vertical directions (usually determined by the actual focal length f and the pixel scale factor); c x and c y is the coordinate of the principal point of the image (the pixel position of the center point of the image); 0 and 1 represent the expansion of homogeneous coordinates;

[0109] S6) Project the 3D point cloud in the camera coordinate system onto the 2D image plane to obtain its pixel coordinates; the calculation formula is:

[0110]

[0111] Where: [X p , Y p ] is the normalized coordinate of the camera's image plane; [u, v] is the pixel coordinate; K is the camera's intrinsic parameter matrix;

[0112] Specifically, the camera's intrinsic parameter matrix describes the camera's internal geometric characteristics, including focal length, image center point, and pixel scaling. c , Y c , Z c ], and the pixel coordinates [u, v] are obtained through steps S5-S6); wherein [X p , Y p ] is based on the pinhole camera model, which projects the three-dimensional point onto the imaging plane of the camera and normalizes the coordinates. The normalized coordinates are then converted to pixel coordinates [u, v].

[0113] Finally, we need to infer the world coordinate point [X] in three-dimensional space from the two-dimensional pixel coordinate [u, v]. w , Y w , Z w ], which is calculated by combining the depth value Zc and the external parameter matrix (rotation matrix R and translation vector T).

[0114] S7) back-projecting the pixel coordinates of the 3D point cloud into the camera coordinate system, and then transforming them into the world coordinate system.

[0115] Therefore, the VR projection module can realize projection from three-dimensional space to two-dimensional image, and reverse projection from two-dimensional image to three-dimensional space through the combination of camera intrinsic matrix and external matrix. In this way, the VR projection module can realize the transformation from two-dimensional image to three-dimensional space through reverse projection and space conversion matrix according to the collected image.

[0116] Furthermore, the external parameters of the camera include position, attitude angle and PTZ, which are used to define the attitude of the camera in the 3D world coordinate system;

[0117] The internal parameters of the camera include focal length, image center point and pixel scaling ratio, which are used to define the geometric characteristics of the 2D image captured by the camera.

[0118] The AR module is used to generate corresponding virtual objects according to business needs, and superimpose the virtual objects to the corresponding world coordinate system positions in the three-dimensional scene to form an AR video that integrates virtual and real.

[0119] Furthermore, the specific implementation process of the AR module includes:

[0120] Generate corresponding virtual objects according to the needs of warehouse fire protection business, including warehouse partition labeling, storage item information labeling, sensor data labeling and fire protection equipment status labeling;

[0121] Specifically, in the fire safety scenario of a large warehouse, combined with specific business needs, the generated virtual objects should be intuitive and functional, and be able to assist security personnel in quickly understanding the scenario, locating problems, and making decisions. The following is a classification and expansion analysis of virtual objects:

[0122] By superimposing dynamic borders, the warehouse is divided into several functional areas and each area is distinguished by color. Specific storage areas are marked in the panoramic image to display the type of items in the area, flammability level, and storage specifications for chemicals.

[0123] Combined with the smoke sensors, temperature and humidity sensors deployed in the warehouse, relevant values ​​are superimposed in real time.

[0124] Mark the specific location of firefighting equipment (such as fire extinguishers and sprinkler heads) and update their status in real time.

[0125] The above objects are used for interactive operations and can also be set to allow users to click a certain position in the panoramic video to trigger the virtual model to expand the details.

[0126] Through the image overlay algorithm, the virtual object is superimposed on the corresponding world coordinate system position in the three-dimensional scene with appropriate transparency or occlusion relationship, forming an AR video that integrates virtuality and reality.

[0127] In the specific implementation, image overlay algorithms such as alpha blending or light mapping;

[0128] Transparency processing adjusts the lighting and shadow effects of virtual objects to make them consistent with the real scene through lighting calculations (such as using an ambient lighting model or physically based rendering (PBR)).

[0129] Occlusion relationship processing uses depth buffer or occlusion map to handle the front-to-back relationship between virtual objects and real scenes to achieve a more realistic overlay effect.

[0130] Finally, the virtual-reality fusion video is output to a display device (such as a monitoring screen or AR device).

[0131] Furthermore, the sensor data annotation includes real-time data display of smoke sensors and temperature and humidity sensors. Sensors are mainly used to collect real-time data related to the warehouse environment, including temperature, humidity, smoke, and other possible environmental changes.

[0132] In summary, the data acquisition and preprocessing module of the present invention performs video acquisition and noise reduction, etc., to improve data quality and processing efficiency; the image processing module generates high-precision panoramic images through technologies such as feature extraction, matching, homography matrix calculation and image stitching. The VR projection module combines the calibration of internal and external parameters of the camera, three-dimensional point cloud projection and back-projection calculation to complete the geometric mapping of two-dimensional images and three-dimensional scenes, and finally presents the results in a virtual reality environment. In addition, virtual objects that meet business needs are superimposed to form an AR video that integrates virtual and real.

[0133] The present invention gives full play to the advantages of computer vision, virtual reality and augmented reality technology, and provides an efficient and practical solution for intelligent monitoring scenarios of video security.

[0134] In the specific embodiments provided in the present application, it should be understood that the disclosed systems, modules and methods can be implemented in other ways. For example, the module embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0136] If the process or calculation steps are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

[0137] In the description of the specification, the description with reference to the terms "in a specific implementation process", "example", "specifically", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0138] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. The duty center security monitoring system based on the integration of virtual reality and augmented reality is characterized by: Including data acquisition module, preprocessing module, image processing module, VR projection module and AR module; The data acquisition module includes a camera and a sensor. The camera is used to collect video data; the sensor is used to detect environmental data; The preprocessing module is used to preprocess the collected video data, including video format unification and noise removal; The image processing module is used to stitch multiple images into a seamless panoramic image through a homography matrix estimation algorithm; The VR projection module is used to establish the spatial projection of two-dimensional images to three-dimensional scenes; The AR module is used to generate corresponding virtual objects according to business needs, and superimpose the virtual objects to the corresponding world coordinate system positions in the three-dimensional scene to form an AR video that integrates virtual and real.

2. The duty center security monitoring system based on the fusion of virtual reality and augmented reality according to claim 1 is characterized by: The implementation process of the image processing module includes: Use appropriate feature detectors to identify features in each image and generate feature descriptors for similarity matching of subsequent images; In adjacent images, similar feature point pairs are found based on feature descriptors; The distance metric method is used to determine the similarity between feature point pairs; Based on the feature point pairs, the homography matrix between each pair of images is calculated using the least squares method or RANSAC; Align all images to a common reference plane using the computed homography matrix; Image blending technology is used to smooth the boundaries between aligned images and eliminate stitching artifacts to create a seamless panoramic image.

3. The duty center security monitoring system based on the fusion of virtual reality and augmented reality according to claim 2 is characterized by: The feature detector is a SIFT or ORB detection algorithm.

4. The duty center security monitoring system based on the fusion of virtual reality and augmented reality according to claim 2 is characterized by: The distance measurement method is the Euclidean distance method, and the similarity value is 0.7-0.

8.

5. The duty center security monitoring system based on the fusion of virtual reality and augmented reality according to claim 2 is characterized by: The mapping relationship expression of the homography matrix is: Where H is the homography matrix, a 3×3 matrix that describes the planar projection transformation relationship between two images, and is used to map a point [x1, y1, 1] in one image to a corresponding point [x2, y2, 1] in another image.

6. The duty center security monitoring system based on the fusion of virtual reality and augmented reality according to claim 2 is characterized by: The image blending technology is a gradient domain fusion or a multi-band blending technology.

7. The duty center security monitoring system based on the fusion of virtual reality and augmented reality according to claim 1 is characterized by: The specific implementation process of the VR projection module includes: S1) defining a camera coordinate system through the external parameter information of the camera, wherein the origin is located at the optical center of the camera, and the x, y and z axes of the camera are aligned with its image plane and the optical axis; S2) defining a three-dimensional universal world coordinate system, which is used as a reference system for the three-dimensional scene; S3) Calculate the transformation matrix between the world coordinate system and the camera coordinate system; the expression is: In the formula, [X c , Y c , Z c ] represents the camera coordinate system; [X w , Y w , Z w ] represents the world coordinate system; R is a 3×3 rotation matrix, which represents the rotation of the camera coordinate system relative to the world coordinate system; T is a 3×1 translation vector, which represents the displacement of the origin of the camera coordinate system relative to the world coordinate system; 0 and 1 are used to represent the expansion of homogeneous coordinates; S4) extracting key features from the two-dimensional video frames and generating 3D point clouds from the two-dimensional video frames using motion structure or multi-view stereo technology; S5) The internal parameter matrix of the camera is established through the internal parameters of the camera; its expression is: Where: f x and S y is the effective focal length in the horizontal and vertical directions; c x and c y is the coordinate of the principal point of the image; 0 and 1 represent the expansion of homogeneous coordinates; S6) Project the 3D point cloud in the camera coordinate system onto the 2D image plane to obtain its pixel coordinates; the calculation formula is: Where: [X p , Y p ] is the normalized coordinate of the camera's image plane; [u, v] is the pixel coordinate; K is the camera's intrinsic parameter matrix; S7) back-projecting the pixel coordinates of the 3D point cloud into the camera coordinate system, and then transforming them into the world coordinate system.

8. The duty center security monitoring system based on the fusion of virtual reality and augmented reality according to claim 7 is characterized by: The external parameters of the camera include position, attitude angle and PTZ; the internal parameters of the camera include focal length, image center point and pixel scaling ratio.

9. The duty center security monitoring system based on the fusion of virtual reality and augmented reality according to claim 1 is characterized by: The specific implementation process of the AR module includes: Generate corresponding virtual objects according to the needs of warehouse fire protection business, including warehouse partition labeling, storage item information labeling, sensor data labeling and fire protection equipment status labeling; Through the image overlay algorithm, the virtual object is superimposed on the corresponding world coordinate system position in the three-dimensional scene with appropriate transparency or occlusion relationship, forming an AR video that integrates virtual and real.

10. The duty center security monitoring system based on the fusion of virtual reality and augmented reality according to claim 9 is characterized in that: The sensor data annotation includes real-time data display of smoke sensors and temperature and humidity sensors.

Images