Mine monitoring video playing method and device and electronic equipment

Abstract

The invention discloses a mine monitoring video playing method and device and electronic equipment, and relates to the technical field of digital twinning and the technical field of intelligent mines, and the method comprises the steps: obtaining the pixel value of a pixel point of a to-be-played video frame of a monitoring video, the monitoring video being obtained by shooting a designated region of a mine; determining each pixel point of which the pixel value is changed compared with that of the previous video frame in the video frame to be played; the pixel value of the same pixel point in a point cloud model loaded in the three-dimensional scene model of the mine is updated by using the pixel value of each pixel point of which the pixel value changes, an updated three-dimensional scene model is obtained, the point cloud model is generated based on a video frame of a monitoring video, and the updated three-dimensional scene model is obtained. The point cloud model is loaded at a position corresponding to a specified area in the three-dimensional scene model; and rendering the updated three-dimensional scene model in real time. By adopting the scheme, the effect of playing the mine monitoring video is improved.

Description

Technical Field

[0001] The present application relates to the fields of digital twin technology and intelligent mine technology, and in particular to a mine monitoring video playback method, device and electronic equipment. Background Art

[0002] With the continuous advancement of smart mine construction, the presentation of mine information is gradually changing from the traditional two-dimensional plane to three-dimensional space. Usually, a three-dimensional mine model is constructed based on the mine design CAD drawings, videos and pictures of the mine site and other relevant materials to build a three-dimensional mine scene to show the overall picture of the mine and achieve mine transparency.

[0003] With the development of network technology, mining areas have also built video surveillance systems with wider coverage to provide real-time monitoring images and assist mine personnel in responding quickly in emergency situations.

[0004] Video surveillance is one of the important means of mine production supervision, and its role cannot be ignored. However, traditional mine video surveillance technology often uses a flat list visualization method, which lacks the three-dimensional reality of the mine and cannot quickly locate the monitored area. In addition, although the three-dimensional scene of the mine can be displayed by placing three-dimensional models, it cannot display the mining and operation conditions of the mine in real time. Summary of the invention

[0005] The embodiments of the present application provide a mine monitoring video playback method, device and electronic device to solve the problem of poor mine monitoring video playback effect in the prior art.

[0006] The present application provides a method for playing a mine monitoring video, including:

[0007] Obtaining pixel values ​​of pixel points of a video frame to be played of a surveillance video, wherein the surveillance video is obtained by shooting a designated area of ​​a mine;

[0008] Determine each pixel point in the to-be-played video frame whose pixel value has changed compared with a previous video frame;

[0009] Using the pixel value of each pixel point whose pixel value has changed, updating the pixel value of the same pixel point in the point cloud model loaded in the three-dimensional scene model of the mine, to obtain an updated three-dimensional scene model, wherein the point cloud model is generated based on the video frame of the monitoring video, and the point cloud model is loaded at a position corresponding to the designated area in the three-dimensional scene model;

[0010] The updated three-dimensional scene model is rendered in real time.

[0011] Furthermore, the point cloud model is generated by the following steps:

[0012] Obtain video key frames of the monitoring video collected from a specified area of a mine, where the video key frames are video frames in a continuous sequence of video frames that can better represent the scene of the specified area compared to other video frames;

[0013] Identify the image feature points of the three-dimensional scene map of the mine and the video key frames;

[0014] Perform feature point matching according to the attribute values of the image feature points of the three-dimensional scene map and the video key frames;

[0015] Assign the three-dimensional coordinates of the image feature points in the three-dimensional scene map to the matching image feature points in the video key frames;

[0016] Generate the three-dimensional coordinates of other pixel points of the video key frames based on the three-dimensional coordinates of the image feature points of the video key frames assigned with three-dimensional coordinates;

[0017] Generate a point cloud model according to the three-dimensional coordinates and pixel values of the pixel points of the video key frames.

[0018] Further, after generating the point cloud model according to the three-dimensional coordinates and pixel values of the pixel points of the video key frames, it further includes:

[0019] Load the point cloud model to the corresponding position of the three-dimensional scene model according to the three-dimensional coordinates of the pixel points of the point cloud model to obtain a three-dimensional scene model loaded with the point cloud model.

[0020] Further, the identifying the image feature points of the three-dimensional scene map of the mine and the video key frames includes:

[0021] Use the SIFT algorithm to identify the image feature points of the three-dimensional scene map of the mine and the video key frames, and obtain a feature point list of the three-dimensional scene map and a feature point list of the video key frames;

[0022] The feature point list contains the index and descriptor vector of the image feature point.

[0023] Further, performing feature point matching according to the attribute values of the image feature points of the three-dimensional scene map and the video key frames to obtain the feature point matching relationship between the three-dimensional scene map and the video key frames includes:

[0024] Based on the index and descriptor vector of the image feature points included in the feature point list of the three-dimensional scene map and the feature point list of the video key frames, calculate the distance between the descriptor vectors of each image feature point of the three-dimensional scene map and each image feature point of the video key frames as the descriptor distance;

[0025] According to the matching strategy that the smaller the descriptor distance is, the greater the matching degree of two image feature points is, determine the image feature points that match between the three-dimensional scene graph and the video key frame.

[0026] Further, generating the three-dimensional coordinates of other pixels of the video key frame based on the three-dimensional coordinates of the image feature points with three-dimensional coordinates assigned to the video key frame includes:

[0027] Based on the three-dimensional coordinates of the image feature points with three-dimensional coordinates assigned to the video key frame, use the coordinate interpolation algorithm to generate the three-dimensional coordinates of other pixels of the video key frame.

[0028] Further, obtaining the pixel values of the pixels of the to-be-played video frame of the surveillance video includes:

[0029] Obtain the pixel values of the pixels of the to-be-played video frames at the same moment of multiple surveillance videos, where the multiple surveillance videos are obtained by respectively shooting multiple specified areas of the mine;

[0030] Updating the pixel values of the same pixels in the point cloud model loaded in the three-dimensional scene model of the mine using the pixel values of the pixels whose pixel values have changed includes:

[0031] For each surveillance video, use the pixel values of the pixels whose pixel values have changed in the to-be-played video frame of this surveillance video to update the pixel values of the same pixels in the point cloud model corresponding to this surveillance video loaded in the three-dimensional scene model of the mine, and the point cloud model corresponding to this surveillance video is generated based on the video frames of this surveillance video.

[0032] An embodiment of the present application further provides a mine surveillance video playback device, including:

[0033] A video frame acquisition module, configured to obtain the pixel values of the pixels of the to-be-played video frame of the surveillance video, where the surveillance video is obtained by shooting a specified area of the mine;

[0034] A pixel value comparison module, configured to determine the pixels whose pixel values have changed compared to the previous video frame in the to-be-played video frame;

[0035] A point cloud model update module, configured to use the pixel values of the pixels whose pixel values have changed to update the pixel values of the same pixels in the point cloud model loaded in the three-dimensional scene model of the mine to obtain an updated three-dimensional scene model, where the point cloud model is generated based on the video frames of the surveillance video, and the point cloud model is loaded at the position corresponding to the specified area in the three-dimensional scene model;

[0036] A model rendering module, configured to render the updated three-dimensional scene model in real time.

[0037] Further, it further includes:

[0038] A key frame acquisition module, configured to acquire video key frames of a surveillance video collected from a specified area of a mine, where the video key frames are video frames in a sequence of consecutive video frames that can better represent the scene of the specified area compared to other video frames;

[0039] A feature point recognition module, configured to recognize image feature points of the three-dimensional scene map of the mine and the video key frames;

[0040] A feature point matching module, configured to perform feature point matching according to the attribute values of the image feature points of the three-dimensional scene map and the video key frames;

[0041] A coordinate assignment module, configured to assign the three-dimensional coordinates of the image feature points in the three-dimensional scene map to the matching image feature points in the video key frames;

[0042] A coordinate generation module, configured to generate the three-dimensional coordinates of other pixel points of the video key frames based on the three-dimensional coordinates of the image feature points of the video key frames assigned with three-dimensional coordinates;

[0043] A point cloud model generation module, configured to generate a point cloud model according to the three-dimensional coordinates and pixel values of each pixel point of the video key frames.

[0044] Further, it further includes:

[0045] A model loading module, configured to load the point cloud model to the corresponding position of the three-dimensional scene model according to the three-dimensional coordinates of each pixel point of the point cloud model, to obtain a three-dimensional scene model loaded with the point cloud model.

[0046] Further, the feature point recognition module is specifically configured to use the SIFT algorithm to recognize the image feature points of the three-dimensional scene map of the mine and the video key frames, to obtain a feature point list of the three-dimensional scene map and a feature point list of the video key frames;

[0047] The feature point list contains the index and descriptor vector of the image feature point.

[0048] Further, the feature point matching module is specifically configured to calculate the distance of the descriptor vectors between each image feature point of the three-dimensional scene map and each image feature point of the video key frames based on the index and descriptor vector of the image feature points included in the feature point list of the three-dimensional scene map and the feature point list of the video key frames, as the descriptor distance;

[0049] According to the matching strategy that the smaller the descriptor distance is, the greater the matching degree of two image feature points is, determine the image feature points that match between the three-dimensional scene graph and the video key frame.

[0050] Further, the coordinate generation module is specifically configured to generate the three-dimensional coordinates of other pixels of the video key frame by using a coordinate interpolation algorithm based on the three-dimensional coordinates of the image feature points assigned with three-dimensional coordinates of the video key frame.

[0051] Further, the video frame acquisition module is specifically configured to acquire the pixel values of the pixels of the video frames to be played at the same moment of multiple monitoring videos, and the multiple monitoring videos are obtained by respectively shooting multiple specified areas of the mine;

[0052] Further, the point cloud model update module is specifically configured to, for each monitoring video, use the pixel values of the pixels whose pixel values change in the video frame to be played of the monitoring video to update the pixel values of the same pixel in the point cloud model corresponding to the monitoring video loaded in the three-dimensional scene model of the mine, and the point cloud model corresponding to the monitoring video is generated based on the video frames of the monitoring video.

[0053] An embodiment of the present application further provides an electronic device, including a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is prompted by the machine-executable instructions to implement the mine monitoring video playing method described in any one of the above.

[0054] An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the mine monitoring video playing method described in any one of the above is implemented.

[0055] An embodiment of the present application further provides a computer program product including instructions, and when it runs on a computer, it causes the computer to execute the mine monitoring video playing method described in any one of the above.

[0056] The beneficial effects of the present application include:

[0057] In the method provided by the embodiments of the present application, for the video frames to be played in the monitored video to be displayed, based on the pixel values of the pixel points therein, the pixel points in the video frames to be played whose pixel values have changed compared to the previous video frame are determined, and the pixel values of the pixel points whose pixel values have changed are used to update the pixel values of the same pixel points in the point cloud model loaded in the three-dimensional scene model of the mine, so as to obtain an updated three-dimensional scene model, and the updated three-dimensional scene model is rendered in real time. In this method, the point cloud model is generated based on the video frames of the monitored video, and the point cloud model can represent three-dimensional effects. Therefore, by loading the point cloud model in the three-dimensional scene model and rendering the three-dimensional scene model in real time, the display of the three-dimensional effects of the video frames to be played can be realized. Moreover, since the point cloud model is loaded at the position corresponding to the specified area in the three-dimensional scene model, combining the scenes around the point cloud model in the three-dimensional scene model of the mine can enable the viewer to more accurately know the position of the scene captured by the monitored video, that is, the effect of playing the mine monitored video is improved.

[0058] Other features and advantages of the present application will be described in the following specification, and some of them will become obvious from the specification or be understood by implementing the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures specifically pointed out in the written specification, claims, and drawings. Description of the Drawings

[0059] The drawings are used to provide a further understanding of the present application, and constitute a part of the specification. They are used together with the embodiments of the present application to explain the present application, but do not constitute a limitation to the present application. In the drawings:

[0060] Figure 1 It is a flowchart of the method for playing a mine monitored video provided by the embodiments of the present application;

[0061] Figure 2 It is a schematic diagram of the three-dimensional scene model loaded with a point cloud model rendered in the embodiments of the present application;

[0062] Figure 3 It is a flowchart of generating a point cloud model in the embodiments of the present application;

[0063] Figure 4 It is a flowchart of generating a three-dimensional scene map of the mine and calculating the three-dimensional coordinates of the pixel points in the three-dimensional scene map in the embodiments of the present application;

[0064] Figure 5 It is a schematic structural diagram of the device for playing a mine monitored video provided by the embodiments of the present application;

[0065] Figure 6 It is a schematic structural diagram of the device for playing a mine monitored video provided by another embodiment of the present application;

[0066] Figure 7 A structural schematic diagram of the electronic device provided by the embodiment of the present application. Specific embodiments

[0067] In order to provide an implementation solution for improving the effect of playing mine monitoring videos, the embodiments of the present application provide a method, a device and an electronic device for playing mine monitoring videos. The preferred embodiments of the present application are described below with reference to the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. And without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0068] The embodiments of the present application provide a method for playing mine monitoring videos, as Figure 1 shown, including:

[0069] Step 11: Obtain the pixel values of the pixels of the video frame to be played in the monitoring video, where the monitoring video is obtained by shooting a specified area of the mine;

[0070] Step 12: Determine the pixels whose pixel values have changed in the video frame to be played compared to the previous video frame;

[0071] Step 13: Use the pixel values of the pixels whose pixel values have changed to update the pixel values of the same pixels in the point cloud model loaded in the three-dimensional scene model of the mine, and obtain an updated three-dimensional scene model. The point cloud model is generated based on the video frames of the monitoring video, and the point cloud model is loaded at the position corresponding to the specified area in the three-dimensional scene model;

[0072] Step 14: Render the updated three-dimensional scene model in real time.

[0073] By adopting the above method for playing mine monitoring videos provided by the embodiments of the present application, since the point cloud model is generated based on the video frames of the monitoring video, and the point cloud model can represent three-dimensional effects, therefore, by loading the point cloud model in the three-dimensional scene model and rendering the three-dimensional scene model in real time, the display of the three-dimensional effects of the content of the video frame to be played can be realized. And because the point cloud model is loaded at the position corresponding to the specified area in the three-dimensional scene model, therefore, by combining the scenes around the point cloud model in the three-dimensional scene model of the mine, the position of the scene captured by the monitoring video can be known more accurately, that is, the effect of playing the mine monitoring video is improved.

[0074] In the embodiments of the present application, a three-dimensional scene model can be created in advance for the actual scene of the mine, and the three-dimensional scene model can realize the display of the three-dimensional effects of the mine scene.

[0075] Cameras can be arranged at multiple positions in the actual scenario of a mine to respectively capture surveillance videos. Each camera corresponds to a specified area being captured, that is, the captured surveillance videos correspond one-to-one with the specified areas.

[0076] For the captured surveillance videos, based on the video frames of the surveillance videos, a point cloud model representing the scene of the specified area corresponding to the surveillance video can also be generated, and the point cloud model can be loaded into the position corresponding to the specified area in the three-dimensional scene model.

[0077] When there are multiple surveillance videos and correspondingly multiple point cloud models are generated, the multiple point cloud models can all be loaded into the corresponding positions in the three-dimensional scene model.

[0078] In an embodiment of the present application, for step 11 above, the pixel values of the pixel points of the to-be-played video frames at the same moment of multiple surveillance videos can be obtained. The multiple surveillance videos are respectively captured for multiple specified areas of the mine, and the multiple to-be-played video frames obtained from the multiple surveillance videos are all captured at the same moment.

[0079] Correspondingly, for step 13 above, for each surveillance video, the pixel values of the pixel points whose pixel values change in the to-be-played video frame of the surveillance video can be used to update the pixel values of the same pixel points in the point cloud model corresponding to the surveillance video loaded in the three-dimensional scene model of the mine. The point cloud model corresponding to the surveillance video is generated based on the video frames of the surveillance video.

[0080] After updating the point cloud model, multiple latest point cloud models are loaded in the obtained three-dimensional scene model, and the multiple latest point cloud models can represent the content of the multiple to-be-played video frames. Thus, by real-time rendering the updated three-dimensional scene model, synchronous playback of the multiple to-be-played video frames is achieved.

[0081] And in the three-dimensional scene model rendered in real time, the video frames of multiple surveillance videos are all located in their respective corresponding specified areas. Combining with the scenes around the point cloud models in the three-dimensional scene model of the mine, viewers can more accurately know the positions of the scenes captured by each surveillance video.

[0082] Figure 2 The schematic diagram of the three-dimensional scene model loaded with the point cloud model rendered by adopting the above mine surveillance video playback method provided by the embodiment of the present application, where an example including one surveillance video is used.

[0083] In an embodiment of the present application, the following method flow is proposed for generating the point cloud model, as Figure 3 shown, including the following steps:

[0084] Step 31: Obtain the video key frames of the surveillance video captured from a specified area of the mine. The video key frames are the video frames in a continuous sequence of video frames that can better represent the scene of the specified area compared to other video frames.

[0085] Video is a continuous sequence of images, usually played at a certain frame rate to display a dynamic visual effect. The standard frame rate is 30 FPS (frames per second), that is, one second contains 30 images, and this frame rate is sufficient to capture smooth videos. Video key frames are representative frames in the video sequence that can help users quickly locate key scenes in the video. Usually, frames with relatively stable motion or little scene change and containing the main objects and activities in the scene are selected for subsequent processing.

[0086] To improve the efficiency of subsequent feature point matching, in this step, video frames that are as consistent as possible with the scene objects included in the specified area of the 3D scene map of the mine can be selected as the video key frames.

[0087] In the embodiments of this application, the video key frames can be selected manually. In this step, the video key frames manually selected are directly obtained and subsequent operations are performed.

[0088] Step 32: Identify the image feature points of the 3D scene map of the mine and the video key frames.

[0089] In this step, specifically, the SIFT (Scale-Invariant Feature Transform) algorithm can be used to identify the image feature points of the 3D scene map of the mine and the video key frames, obtaining the feature point list of the 3D scene map and the feature point list of the video key frames. And the feature point list contains the index and descriptor vector of the image feature points.

[0090] Furthermore, the following attribute information of the image feature points can also be included in the feature point list:

[0091] Feature point coordinates pt: in the form of a tuple (x, y), where x is the abscissa and y is the ordinate, with the upper left corner of the image as the origin, the right direction as the x-axis, and the downward direction as the y-axis;

[0092] Feature point response intensity response: represents the degree to which the point is a feature point, quantifying the significance or prominence of the feature point relative to its neighborhood;

[0093] Feature point neighborhood diameter size: within the neighborhood with the size of the feature point as the diameter, the response intensity of the feature point is the strongest.

[0094] Feature point descriptor des: That is, the descriptor vector. In the SIFT algorithm, each feature point has a 128-dimensional descriptor vector. With the feature point as the center, the feature point neighborhood is divided into a 16*16 pixel area, and then divided into 4*4 sub-areas without overlap. For each sub-area, a gradient histogram in 8 directions is calculated (each direction is at an interval of 45 degrees, that is, an 8-dimensional feature vector). Finally, all 8-dimensional feature vectors are concatenated to obtain a 128-dimensional descriptor vector for subsequent feature point matching using the Flann (Fast Library for Approximate Nearest Neighbors) algorithm.

[0095] Step 33: Perform feature point matching according to the attribute values of the image feature points of the three-dimensional scene graph and the video key frame.

[0096] In this step, based on the indexes and descriptor vectors of the image feature points included in the feature point list of the three-dimensional scene graph and the feature point list of the video key frame, the distance between the descriptor vectors of each image feature point of the three-dimensional scene graph and each image feature point of the video key frame can be calculated as the descriptor distance.

[0097] According to the matching strategy that the smaller the descriptor distance, the greater the matching degree of the two image feature points, determine the matching image feature points between the three-dimensional scene graph and the video key frame.

[0098] Specifically, the Flann algorithm can be used to read the three-dimensional scene graph, the feature point list of the three-dimensional scene graph, the video key frame, and the feature point list of the video key frame respectively for feature point matching:

[0099] Traverse the feature point lists of the three-dimensional scene graph and the video key frame. According to the above feature point descriptor des, calculate the distance between the descriptor vectors of each feature point as the descriptor distance. If the ratio of the closest distance to the second closest distance is less than a threshold (usually 0.7 to 0.8), it can be determined that this is a better match. Finally, obtain a list of matching relationships of the feature points in the three-dimensional scene graph and the video key frame. The matching relationship object mainly includes 3 attributes: queryIdx, trainIdx, and distance:

[0100] queryIdx: The index of the feature point of the three-dimensional scene graph in the feature point list of the three-dimensional scene graph;

[0101] trainIdx: The index of the feature point matched in the video key frame in the feature point list of the video key frame;

[0102] distance: The distance between the feature points in the 3D scene graph and the feature points in the video key frame. The smaller the value, the greater the matching degree of the two feature points.

[0103] Furthermore, these attributes can also be used to draw the feature points and the feature point matching situation in the 3D scene graph and the video key frame, obtaining a matching image for subsequent manual inspection of whether the feature point matching is correctly used.

[0104] During the process of manual inspection of feature point matching, the feature point matching situation can be viewed according to the above matching image. When there is an incorrect match, the corresponding feature points in the feature point list are deleted according to the index in the matching relationship, and the above matching process is performed again, thereby ensuring the accuracy of feature point matching.

[0105] Step 34: Assign the 3D coordinates of the image feature points in the 3D scene graph to the matching image feature points in the video key frame.

[0106] In this step, according to the obtained feature point matching relationship, the 3D coordinates of the pixel points in the 3D scene graph can be assigned to the matching pixel points in the video key frame.

[0107] Step 35: Generate the 3D coordinates of the other pixel points in the video key frame based on the 3D coordinates of the image feature points in the video key frame assigned with 3D coordinates.

[0108] In this step, specifically, based on the 3D coordinates of the image feature points in the video key frame assigned with 3D coordinates, a coordinate interpolation algorithm can be used to generate the 3D coordinates of the other pixel points in the video key frame.

[0109] Through this step, the coordinate interpolation algorithm is used to fill in the missing data points, thereby smoothing the data and improving the data resolution to ensure that all pixel points in the video key frame have 3D coordinates, thus improving the display effect of the point cloud model loaded in the subsequent mine 3D scene model.

[0110] Step 36: Generate a point cloud model according to the 3D coordinates and pixel values of each pixel point in the video key frame.

[0111] A point cloud model is a collection composed of a large number of 3D coordinate points (X, Y, Z) and is used to represent the shape and structure of an object. The point cloud model can also contain auxiliary information such as color, intensity, and normal.

[0112] In this step, based on the PCL library (PCL is an open-source library for processing 3D point cloud data), the video key frame with 3D coordinates can be read to generate a point cloud model. Each point in the point cloud model records the 3D coordinates (X, Y, Z), pixel value, and unique identifier (which can be used to update the pixel value of the pixel points in the point cloud model subsequently).

[0113] Step 37: Load the point cloud model to the corresponding position in the three-dimensional scene model according to the three-dimensional coordinates of each pixel point of the point cloud model, and obtain a three-dimensional scene model with the point cloud model loaded.

[0114] Since the video data of the surveillance video usually contains a large amount of redundant information, such as the similarity between consecutive frames, in the embodiments of the present application, by extracting key video frames, the accuracy of feature point matching is improved, so that the generated point cloud model can better display the video content.

[0115] In the method for playing a mine surveillance video provided by the embodiments of the present application, in a 3D rendering engine, according to the change of the pixel value of the pixel point in the surveillance video frame, correspondingly, modify the pixel value of the corresponding pixel point in the point cloud model. Specifically, it can be as follows:

[0116] In the 3D rendering engine, when the scene needs to view the surveillance video, traverse the video frames. The 3D rendering engine obtains each video frame, traverses the pixel points of the video frame, and determines whether the pixel value of the previous frame pixel point changes compared with the pixel value of the current frame pixel point. After determining which pixel points in the current frame have changed pixel values, according to the unique identifier of the pixel point, update the pixel value of the corresponding pixel point in the corresponding point cloud model (this method can improve the calculation efficiency and the rendering efficiency of the 3D rendering engine). Finally, in the three-dimensional scene model of the mine, by loading the point cloud model and modifying the pixel value of the pixel point in the point cloud model, replace the traditional flat list-style method of loading the surveillance video to view the real-time surveillance video in the three-dimensional scene of the mine, that is, realize the real-time fusion of the surveillance video and the three-dimensional scene model, thereby improving the playing effect of the mine surveillance video.

[0117] In the embodiments of the present application, the following process can be adopted Figure 4 as shown to generate a three-dimensional scene map of the mine and calculate the three-dimensional coordinates of the pixel points in the three-dimensional scene map, including the following steps:

[0118] Step 41: Produce a three-dimensional scene model of the mine.

[0119] In this step, Blender software can be used to determine information such as the layout, size, structure, color, material, and texture of the model according to the collected CAD plane drawings of the mine, on-site photos or videos of the mine, etc., and construct a three-dimensional scene model of the mine (such as three-dimensional models of ore unloading stations on the mine surface, repair shops inside roadways, etc.).

[0120] Step 42: Set up a three-dimensional scene and render a three-dimensional scene map and a depth map.

[0121] The Blender software can be used to adjust the placement position and orientation of the 3D model according to the collected relevant materials such as mine site videos and photos, and in combination with the actual positions of mine monitoring cameras, so as to build a real mine 3D scene.

[0122] By configuring the lighting of the scene to obtain the best lighting and perspective, at the same time, the Blender software provides a variety of rendering engines, such as Eevee (real-time rendering) and Cycles (physically based rendering). Select a suitable rendering engine and set rendering parameters such as the resolution, number of samples, and selection of the "depth" channel of the image to render the 3D scene map and depth map.

[0123] The depth map is a special image that contains the distance information from each pixel point in the scene to the camera and can be used for the calculation of the 3D coordinates of the pixel points in the subsequent 3D scene map.

[0124] Step 43: Calculate the 3D coordinates of the pixel points in the 3D scene map.

[0125] Considering that the 3D scene map and the depth map are different rendered images of the same 3D scene, the pixel points in the two images correspond one by one, and the pixel points of the depth map record the distance to the camera. Therefore, the 3D coordinates of each pixel point in the depth map can be calculated, and then the 3D coordinates of the pixel points in the 3D scene map can be obtained. Use the following steps in combination with the relevant camera parameters to obtain the 3D coordinates of the depth image pixel points:

[0126] First, calculate the coordinates of the pixel point in the normalized device (NDC, Normalized Device Coordinates) coordinate system. The NDC coordinate range is from -1 to 1, that is, convert the pixel coordinates of the pixel points in the depth map into normalized coordinates (NDC coordinates).

[0127] In this step, the following formula can be used to calculate the normalized coordinates:

[0128]

[0129] where (u, v) are the pixel coordinates, starting from the upper left corner, u is horizontal to the right, and v is vertical downward, (sx, sy) are the width and height of the sensor size, and (x n , y n ) are the normalized coordinates.

[0130] Then, convert the normalized coordinates into camera coordinates in the camera coordinate system.

[0131] In this step, the following formula can be used to convert the normalized coordinates into camera coordinates:

[0132] X c = d(u, v)·x n ;

[0133] Y c = d(u, v)·y n ;

[0134] Z c = d(u, v);

[0135] where d(u, v) is the depth value of the pixel point (u, v), that is, the distance from the camera to the observed point, and (X c , Y c , Z c ) are the camera coordinates.

[0136] In this step, it is assumed that the origin of the camera coordinate system is the camera position, and the Z-axis points in the front direction of the camera.

[0137] Then, the camera coordinates are converted into world coordinates in the world coordinate system, and the world coordinates are used as three-dimensional coordinates.

[0138] In this step, the following formula can be used to convert the camera coordinates into world coordinates:

[0139]

[0140] where R is the rotation matrix in the camera parameters, T is the translation vector in the camera parameters, and (X w , Y w , Z w ) are the world coordinates.

[0141] By using the above-mentioned mine monitoring video playing method provided by the embodiments of the present application, real-time fusion of the monitoring video and the three-dimensional scene of the mine can be achieved, so as to more vividly and intuitively display the mine mining and operation conditions, help the monitoring personnel to timely discover potential safety hazards, help the management personnel to better understand the production process, timely adjust the production strategy, and improve the production efficiency.

[0142] Based on the same inventive concept, according to the mine monitoring video playing method provided by the above embodiments of the present application, correspondingly, another embodiment of the present application further provides a mine monitoring video playing device, and its structural schematic diagram is as Figure 5 shown, and specifically includes:

[0143] A video frame acquisition module 501, configured to acquire the pixel values of the pixel points of the video frame to be played in the monitoring video, where the monitoring video is obtained by shooting a specified area of the mine;

[0144] A pixel value comparison module 502, configured to determine each pixel point in the video frame to be played whose pixel value has changed compared with the previous video frame;

[0145] The point cloud model update module 503 is used to update the pixel values of the same pixel points in the point cloud model loaded in the three-dimensional scene model of the mine by using the pixel values of the pixel points whose pixel values have changed, so as to obtain an updated three-dimensional scene model. The point cloud model is generated based on the video frames of the monitoring video, and the point cloud model is loaded at the position corresponding to the specified area in the three-dimensional scene model;

[0146] The model rendering module 504 is used to render the updated three-dimensional scene model in real time.

[0147] Furthermore, as Figure 6 shown, it further includes:

[0148] The key frame acquisition module 505 is used to acquire the video key frames of the monitoring video collected for the specified area of the mine. The video key frames are the video frames that can better represent the scene of the specified area among the continuous video frame sequences;

[0149] The feature point recognition module 506 is used to identify the image feature points of the three-dimensional scene map of the mine and the video key frames;

[0150] The feature point matching module 507 is used to perform feature point matching according to the attribute values of the image feature points of the three-dimensional scene map and the video key frames;

[0151] The coordinate assignment module 508 is used to assign the three-dimensional coordinates of the image feature points in the three-dimensional scene map to the matching image feature points in the video key frames;

[0152] The coordinate generation module 509 is used to generate the three-dimensional coordinates of other pixel points of the video key frames based on the three-dimensional coordinates of the image feature points of the video key frames assigned with three-dimensional coordinates;

[0153] The point cloud model generation module 510 is used to generate a point cloud model according to the three-dimensional coordinates and pixel values of the pixel points of the video key frames.

[0154] Furthermore, as Figure 6 shown, it further includes:

[0155] The model loading module 511 is used to load the point cloud model to the corresponding position of the three-dimensional scene model according to the three-dimensional coordinates of the pixel points of the point cloud model, so as to obtain a three-dimensional scene model loaded with the point cloud model.

[0156] Furthermore, the feature point recognition module 506 is specifically used to use the SIFT algorithm to identify the image feature points of the three-dimensional scene map of the mine and the video key frames, so as to obtain the feature point list of the three-dimensional scene map and the feature point list of the video key frames;

[0157] The feature point list contains the indices and descriptor vectors of the image feature points.

[0158] Further, the feature point matching module 507 is specifically configured to calculate the distances of the descriptor vectors between each image feature point of the 3D scene graph and each image feature point of the video key frame based on the indices and descriptor vectors of the image feature points included in the feature point list of the 3D scene graph and the feature point list of the video key frame, as the descriptor distances;

[0159] According to the matching strategy that the smaller the descriptor distance indicates the greater the matching degree of the two image feature points, determine the image feature points that match between the 3D scene graph and the video key frame.

[0160] Further, the coordinate generation module 509 is specifically configured to generate the 3D coordinates of each other pixel point of the video key frame by using a coordinate interpolation algorithm based on the 3D coordinates of the image feature points assigned with 3D coordinates of the video key frame.

[0161] Further, the video frame acquisition module 501 is specifically configured to acquire the pixel values of the pixel points of the to-be-played video frames at the same moment of multiple monitoring videos, and the multiple monitoring videos are obtained by respectively shooting multiple specified areas of the mine;

[0162] Further, the point cloud model update module 503 is specifically configured to, for each monitoring video, use the pixel values of the pixel points whose pixel values change in the to-be-played video frame of the monitoring video to update the pixel values of the same pixel points in the point cloud model corresponding to the monitoring video loaded in the 3D scene model of the mine, and the point cloud model corresponding to the monitoring video is generated based on the video frames of the monitoring video.

[0163] The functions of the above modules can correspond to Figures 1 to 4 the corresponding processing steps in the shown process, which will not be elaborated here.

[0164] The mine monitoring video playback device provided by the embodiments of the present application can be implemented by a computer program. Those skilled in the art should be able to understand that the above module division method is only one of many module division methods. If divided into other modules or not divided into modules, as long as the mine monitoring video playback device has the above functions, it should be within the protection scope of the present application.

[0165] The embodiments of the present application also provide an electronic device, such as Figure 7As shown, it includes a processor 71 and a machine-readable storage medium 72. The machine-readable storage medium 72 stores machine-executable instructions that can be executed by the processor 71. The processor 71 is prompted by the machine-executable instructions to implement the mine monitoring video playback method described in any one of the above.

[0166] An embodiment of this application also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the mine monitoring video playback method described in any one of the above is implemented.

[0167] An embodiment of this application also provides a computer program product containing instructions. When it runs on a computer, it causes the computer to execute the mine monitoring video playback method described in any one of the above.

[0168] The machine-readable storage medium in the above electronic device may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0169] The above processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0170] Each embodiment in this specification is described in a related manner. For the same and similar parts between each embodiment, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the partial description of the method embodiments for the relevant parts.

[0171] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0172] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general purpose computers, special purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0173] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are performed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or combinations of blocks.

[0175] Obviously, those skilled in the art can make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A mine monitoring video playback method, characterized in that: include: Obtaining pixel values ​​of pixel points of a video frame to be played of a surveillance video, wherein the surveillance video is obtained by shooting a designated area of ​​a mine; Determine each pixel point in the to-be-played video frame whose pixel value has changed compared with a previous video frame; Using the pixel value of each pixel point whose pixel value has changed, updating the pixel value of the same pixel point in the point cloud model loaded in the three-dimensional scene model of the mine, to obtain an updated three-dimensional scene model, wherein the point cloud model is generated based on the video frame of the monitoring video, and the point cloud model is loaded at a position corresponding to the designated area in the three-dimensional scene model; The updated three-dimensional scene model is rendered in real time.

2. The method according to claim 1, characterized in that The point cloud model is generated by the following steps: Acquire a video key frame of a surveillance video collected from a designated area of ​​a mine, wherein the video key frame is a video frame in a continuous video frame sequence that is more capable of representing a scene of the designated area than other video frames; Identifying the three-dimensional scene graph of the mine and the image feature points of the video key frame; Matching feature points according to the attribute values ​​of the image feature points of the three-dimensional scene graph and the video key frame; Assigning the three-dimensional coordinates of the image feature points in the three-dimensional scene graph to the matching image feature points in the video key frame; Based on the three-dimensional coordinates of the image feature points of the video key frame assigned with the three-dimensional coordinates, generating the three-dimensional coordinates of other pixel points of the video key frame; A point cloud model is generated according to the three-dimensional coordinates and pixel values ​​of each pixel point of the video key frame.

3. The method according to claim 2, characterized in that After generating the point cloud model according to the three-dimensional coordinates and pixel values ​​of each pixel point of the video key frame, the method further includes: According to the three-dimensional coordinates of each pixel point of the point cloud model, the point cloud model is loaded into the corresponding position of the three-dimensional scene model to obtain the three-dimensional scene model loaded with the point cloud model.

4. The method according to claim 2, characterized in that The identifying of the three-dimensional scene graph of the mine and the image feature points of the video key frame includes: Using the SIFT algorithm, the three-dimensional scene graph of the mine and the image feature points of the video key frame are identified to obtain a feature point list of the three-dimensional scene graph and a feature point list of the video key frame; The feature point list contains the indices and descriptor vectors of the image feature points.

5. The method according to claim 4, characterized in that According to the attribute values ​​of the image feature points of the three-dimensional scene graph and the video key frame, feature point matching is performed to obtain a matching relationship between the feature points of the three-dimensional scene graph and the video key frame, including: Based on the indexes and descriptor vectors of the image feature points included in the feature point list of the three-dimensional scene graph and the feature point list of the video key frame, calculate the distance of the descriptor vector between each image feature point of the three-dimensional scene graph and each image feature point of the video key frame as the descriptor distance; According to a matching strategy in which a smaller descriptor distance indicates a greater matching degree between two image feature points, image feature points matching between the three-dimensional scene graph and the video key frame are determined.

6. The method according to claim 2, characterized in that The generating of the three-dimensional coordinates of other pixel points of the video key frame based on the three-dimensional coordinates of the image feature points assigned with the three-dimensional coordinates of the video key frame comprises: Based on the three-dimensional coordinates of the image feature points of the video key frame assigned with three-dimensional coordinates, a coordinate interpolation algorithm is used to generate the three-dimensional coordinates of other pixel points of the video key frame.

7. The method according to claim 1, characterized in that The step of obtaining pixel values ​​of pixel points of a video frame to be played of a surveillance video includes: Obtaining pixel values ​​of pixel points of a video frame to be played at the same time of multiple surveillance videos, wherein the multiple surveillance videos are obtained by shooting multiple designated areas of the mine respectively; The method of using the pixel value of each pixel point whose pixel value has changed to update the pixel value of the same pixel point in the point cloud model loaded in the three-dimensional scene model of the mine comprises: For each surveillance video, the pixel value of each pixel point where the pixel value of the video frame to be played of the surveillance video has changed is used to update the pixel value of the same pixel point in the point cloud model corresponding to the surveillance video loaded in the three-dimensional scene model of the mine. The point cloud model corresponding to the surveillance video is generated based on the video frame of the surveillance video.

8. A mine monitoring video playback device, characterized in that: include: A video frame acquisition module, used to acquire pixel values ​​of pixel points of a video frame to be played of a monitoring video, wherein the monitoring video is obtained by shooting a designated area of ​​a mine; A pixel value comparison module, used to determine each pixel point in the to-be-played video frame whose pixel value has changed compared with the previous video frame; A point cloud model updating module, used to update the pixel value of the same pixel point in the point cloud model loaded in the three-dimensional scene model of the mine by using the pixel value of each pixel point whose pixel value has changed, so as to obtain an updated three-dimensional scene model, wherein the point cloud model is generated based on the video frame of the monitoring video, and the point cloud model is loaded at a position corresponding to the designated area in the three-dimensional scene model; The model rendering module is used to render the updated three-dimensional scene model in real time.

9. An electronic device, characterized in that: The method comprises a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is prompted by the machine-executable instructions to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images