Mine panoramic monitoring system based on array type hundred million-level pixel image intelligent perception

Abstract

The invention discloses a mine panoramic monitoring system based on array type hundred-million-level pixel image intelligent perception. The mine panoramic monitoring system comprises an array image acquisition module which acquires multiple paths of original video signals of a mine scene through a plurality of hundred-million-level pixel cameras which are spatially arranged; the edge preprocessing module receives multiple paths of original video signals and generates preprocessed video signals and preliminary target signals; the panoramic splicing module receives the preprocessed video signals and fuses multiple paths of video streams through a real-time image splicing algorithm to generate unified panoramic video signals; the intelligent sensing module receives the panoramic video signal and the preliminary target signal and generates an intelligent analysis result signal; and the early warning and interaction module receives an intelligent analysis result signal so as to realize alarm output and user operation feedback. According to the mine panoramic monitoring system based on array type hundred million-level pixel image intelligent perception, the problem that view, precision and real-time intelligent analysis cannot be considered in mine panoramic monitoring can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and intelligent monitoring, and particularly relates to a mine panoramic monitoring system based on arrayed hundred-million-pixel image intelligent perception. BACKGROUND

[0002] The mine production environment is complex, the region is vast, and there are a large number of dynamic changes of dangerous sources, which puts forward very high requirements for safety production monitoring. The traditional mine monitoring system mainly relies on deploying multiple ordinary or high-definition cameras to realize the monitoring of local areas by independently deploying and controlling key points. This mode has many inherent defects: first, limited by the limited field of view of a single camera, if you want to realize the coverage of the entire mine operation surface, you need to install a large number of cameras, which not only leads to high system construction cost and complex wiring, but also cannot form a unified and coherent panoramic situation awareness due to the mutual fragmentation of the pictures, and the monitoring personnel need to frequently switch attention between different screens, which is easy to cause visual fatigue and miss important information. Secondly, the traditional camera has a bottleneck in resolution. When shooting a large range of scenes, although it can "see" that there are targets in the distance, it cannot "see clearly" the details, for example, it cannot accurately identify the specific behavior of the personnel in the distance, whether they wear safety equipment, or it cannot distinguish the subtle abnormal state of the equipment. This contradiction of "seeing but not seeing clearly" greatly reduces the value of the monitoring system in early warning. Moreover, the existing system mostly stays at the level of passive video recording and manual monitoring, and is extremely dependent on the monitoring personnel to keep focused and make correct judgments at all times. In the face of a large amount of and boring video pictures, manual monitoring is inefficient and slow to react, and it is difficult to realize the immediate discovery and response to safety hazards. Although some systems try to introduce simple moving detection alarms, they are easy to produce a large number of false alarms due to light changes, animal intrusion and other disturbances, and are not practical. In addition, if you try to directly use a single hundred-million-pixel camera to improve the resolution, you will face the problem of narrow field of view and inability to consider the panorama, and the huge amount of data generated will put a huge pressure on network transmission and background processing systems, resulting in high system delay and inability to meet the urgent needs of real-time early warning. Therefore, the field of mines urgently needs an innovative monitoring solution that can integrate ultra-wide-angle seamless coverage, ultra-high-definition detail capture, and real-time intelligent analysis and early warning capabilities. SUMMARY

[0003] In view of the above prior art defects, the purpose of the present application is to provide a mine panoramic monitoring system based on arrayed hundred million pixel image intelligent perception, which is used to solve the problem that the field of view, precision and real-time intelligent analysis cannot be considered in mine panoramic monitoring. By deploying an arrayed hundred million pixel camera group, high-definition video streams covering the entire mine are collected; the edge computing node is used to pre-process and preliminarily screen the video streams, thereby reducing the central load; the central server is used to splice multiple video streams in real time to form a unified panoramic image; and a deep learning model optimized for mine scenes is used to automatically identify and analyze the equipment, personnel and abnormal behaviors in the panoramic video, and finally the digital twin interface is used to realize linkage warning and interaction, thereby realizing the leap from "seeing" to "seeing clearly and understanding".

[0004] The present application provides a mine panoramic monitoring system based on arrayed hundred million pixel image intelligent perception, comprising: An array image acquisition module acquires multiple original video signals of the mine scene through multiple hundred million pixel cameras arranged in space; An edge pre-processing module receives the multiple original video signals and performs image pre-processing and preliminary target detection on the signals to generate pre-processed video signals and preliminary target signals; A panoramic splicing module receives the pre-processed video signals and fuses multiple video streams through a real-time image splicing algorithm to generate a unified panoramic video signal; An intelligent perception module receives the panoramic video signal and the preliminary target signal, and uses a deep learning model to identify and analyze the targets and abnormal behaviors in the mine scene to generate an intelligent analysis result signal; A warning and interaction module receives the intelligent analysis result signal and generates a warning control signal and a visual interaction signal according to the analysis result to realize alarm output and user operation feedback.

[0005] In an embodiment of the present application, the multiple hundred million pixel cameras in the array image acquisition module are arranged in a ring-shaped space arrangement in the key area of the mine, ensuring that there is an overlapping area between the fields of view of adjacent cameras to cover the panorama, and the size of the overlapping area is optimized to adapt to the undulating terrain and obstacle shielding of the mine, thereby ensuring that the multiple original video signals can seamlessly capture the entire working surface. The camera installation position takes into account the different heights and angles of the mining surface, transportation channel and dump, and is stably fixed through a fixed support to prevent image shaking due to vibration or environmental influence. The array image acquisition module is also equipped with an environment adaptation component that can automatically adjust exposure and focal length to cope with changes in mine day and night lighting and dust interference, ensuring that the collected video signals remain high definition and consistent under different working conditions, providing a reliable data foundation for subsequent processing.

[0006] In an embodiment of the present application, the image preprocessing performed by the edge preprocessing module includes distortion correction on the multi-channel raw video signal to eliminate lens distortion and color balance processing to unify the tone and brightness differences of different cameras, the preliminary target detection uses a lightweight neural network model to identify moving targets and significant abnormalities in the video stream in real time, the lightweight neural network model is trained on mine scene data and can quickly filter out potential risk objects such as personnel or vehicles, and the detection results are transmitted synchronously with the preprocessed video signal, the edge preprocessing module also integrates a caching mechanism to temporarily store processing data to cope with network fluctuations, ensuring the continuity and stability of the preprocessed video signal and the preliminary target signal, and reducing data transmission delay.

[0007] In an embodiment of the present application, the real-time image stitching algorithm used by the panoramic stitching module is based on feature point extraction and matching technology, first extracts multi-scale image features from the preprocessed video signal, then calculates the transformation matrix between images to achieve accurate alignment through a feature point matching algorithm, and uses a multi-band fusion method to eliminate stitching seams and lighting differences to generate smooth panoramic video signals, the panoramic stitching module also integrates a parallel processing architecture that uses multi-threading technology to process multiple video streams simultaneously, ensuring that the stitching process still maintains real-time performance under high load, and the panoramic video signal output is in a scalable format that allows users to dynamically adjust the field of view in the panoramic view.

[0008] In an embodiment of the present application, the deep learning model used by the intelligent perception module is a convolutional neural network structure optimized specifically for mine environments, capable of identifying multiple target types including mining equipment, transport vehicles, staff and safety equipment, while analyzing abnormal behaviors such as equipment failure, personnel entering dangerous areas, or signs of slope sliding, the deep learning model performs multi-modal analysis by fusing panoramic video signals and preliminary target signals to improve recognition accuracy and robustness, the intelligent perception module also includes a behavior prediction sub-module that infers the future motion trajectory of the target based on historical data and learning models, thereby generating early warning prompts, and the intelligent analysis result signal includes target position, category and risk level information.

[0009] In an embodiment of the present application, the warning and interaction module generates multi-level warning control signals based on the intelligent analysis result signal, including low-level reminders, medium-level warnings and high-level alarms, corresponding to different risk levels and implemented through sound and light devices, platform notifications or mobile terminal push, the visual interaction signal provides a graphical user interface that displays panoramic video and intelligent analysis overlay information, allowing users to define areas of interest and view details through touch or mouse operations, the warning and interaction module also integrates a log recording function to store all warning events and user operation records for subsequent audit and analysis, and the module supports remote access, allowing administrators to monitor and interact in real time on different terminals.

[0010] In one embodiment of the invention, the system further comprises a digital map integration module that receives the panoramic video signal and the intelligent analysis result signal and performs coordinate alignment and overlay with the mine's two-dimensional map or three-dimensional model to generate a dynamic digital twin, the digital map integration module allows users to set virtual electronic fences on the digital twin to define safety zones and danger boundaries, automatically triggers early warning control signals when the intelligent perception module detects targets entering or leaving these zones, the digital twin supports time backtracking function to reproduce historical scenes for accident investigation and process optimization.

[0011] In one embodiment of the invention, the system supports dynamic enhancement of interest areas, when users select specific areas in the panoramic video through the early warning and interaction module, the system automatically calls high-resolution raw video signals of corresponding cameras in the array image acquisition module, performs digital zoom processing and enlarges display while maintaining image detail clarity, the dynamic enhancement function allows simultaneous processing of multiple interest areas and independent analysis and early warning for each area, the processing is performed in real time without interrupting the panoramic video stream, ensuring the continuity and comprehensiveness of monitoring.

[0012] In one embodiment of the invention, the system further comprises a data storage and audit module that receives and stores dynamic context signals including multi-channel raw video signals, preprocessed video signals, preliminary target signals, panoramic video signals, intelligent analysis result signals, and early warning control signals to form a complete data chain, the data storage and audit module indexes and compresses stored data to support fast retrieval and playback, while generating audit logs recording all processing steps and decision basis for compliance checks and performance evaluation, the module also provides data export interfaces for external systems to access and analyze.

[0013] In one embodiment of the invention, the abnormal behavior analysis of the intelligent perception module includes monitoring of mine slope stability, by continuously analyzing surface texture changes and displacement signs in the panoramic video signal to identify potential landslide or collapse risks, the analysis combines time series data models to compare image differences at different time points to detect minor changes, the intelligent perception module also fuses with external sensor data such as vibration or inclination sensors to improve the reliability of anomaly detection, the results are used to generate early warning signals to guide on-site personnel to take preventive measures. The application provides a mine panoramic monitoring system based on array type hundred million pixel image intelligent perception. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0015] Figure 1 FIG. 1 is a system architecture diagram of the mine panoramic monitoring system based on array type hundred million pixel image intelligent perception. DETAILED DESCRIPTION

[0016] The following will illustrate the embodiments of the present application by specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure of the present specification. The present application can also be implemented or applied by different specific embodiments, and each detail in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0017] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components in actual implementation. The type, number and proportion of each component in actual implementation can be arbitrarily changed, and the layout type of the components can also be more complex.

[0018] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application, however, it is obvious for those skilled in the art that the embodiments of the present application can be implemented without these specific details, and in other embodiments, the known structures and devices are shown in the form of block diagrams instead of details to avoid making the embodiments of the present application difficult to understand.

[0019] Please refer to Figure 1The application discloses a mine panoramic monitoring system based on array-level billion-pixel image intelligent perception, as shown in the drawing, comprising an array image acquisition module, a plurality of billion-pixel cameras arranged in space are used to collect multi-channel original video signals of a mine scene by the array image acquisition module; an edge preprocessing module, the edge preprocessing module receives the multi-channel original video signals, and performs image preprocessing and preliminary target detection on the signals to generate preprocessed video signals and preliminary target signals; a panoramic splicing module, the panoramic splicing module receives the preprocessed video signals, and fuses the multi-channel video streams by a real-time image splicing algorithm to generate a unified panoramic video signal; an intelligent perception module, the intelligent perception module receives the panoramic video signal and the preliminary target signal, and uses a deep learning model to identify and analyze targets and abnormal behaviors in the mine scene to generate an intelligent analysis result signal; an early warning and interaction module, the early warning and interaction module receives the intelligent analysis result signal, and generates an early warning control signal and a visual interaction signal according to the analysis result to realize alarm output and user operation feedback.

[0020] Figure 1As shown, the system starts with the array image acquisition module. This module is not simply a stack of multiple cameras, but a carefully designed and systematic one. Its core lies in the combination of the two key features: "array" and "billion-pixel". "Array" means that these cameras are arranged in a certain geometric pattern in space, such as the common ring array, fan array or multi-layer stereo array, the fundamental purpose of which is to ensure that there is enough overlap between the field of view of each camera and that of the adjacent camera. This overlap is a prerequisite for seamless splicing later. "Billion-pixel" gives each observation point unprecedented detail capture capability, allowing the capture of small targets in the distance, such as the operation of personnel or the reading of instruments on equipment, while still capturing the vast scene. The "multi-channel raw video signal" output by this module is the source and data basis for all intelligent processing of the system, and its high quality and high coverage directly determine the upper limit of the performance of the final system. Next, the edge preprocessing module undertakes the crucial tasks of "refining" and "reducing" data. It is deployed in a location physically close to the array image acquisition module and can be understood as the "local brain" of each camera or each group of cameras. The amount of multi-channel raw video signal data received by this module is extremely large, and if it is directly transmitted to the central server, it will put a huge pressure on the enterprise network bandwidth and cause significant delay. Therefore, the edge preprocessing module performs two core tasks: image preprocessing and preliminary target detection. Image preprocessing includes lens distortion correction to eliminate the image distortion caused by wide-angle lenses and ensure the geometric accuracy of subsequent splicing; it also includes color balance processing to compensate for inconsistencies in picture color and brightness caused by individual differences between different cameras and different light angles, laying the foundation for generating visually consistent panoramic images. Preliminary target detection usually runs an optimized lightweight neural network model that can quickly scan video frames, identify moving objects or significant abnormal areas (such as sudden smoke, moving vehicles or personnel), and generate "preliminary target signals" containing target location and rough category. This step is equivalent to a quick screening in the vast amount of data, marking out "areas that may be worth attention". Finally, the module outputs the optimized "preprocessed video signal" and the "preliminary target signal" as a prompt signal.

[0021] In particular, the panorama stitching module is the key to the transition from "multi-view" to "unified panorama". It receives pre-processed video signals from each edge preprocessing module, which have been corrected and equalized. The module runs complex real-time image stitching algorithms, the core steps of which include feature point detection (finding unique corner points, edges, etc. in each frame of image), feature point matching (finding corresponding feature points in the overlapping area of different images), image transformation matrix calculation (calculating how to project one image into the coordinate system of another image according to matching points), and finally multi-band image fusion. The fusion technology aims to eliminate the stitching seam and the exposure difference between different images, generating a single panoramic picture that is visually continuous, smooth and seamless. The output "panoramic video signal" is a dynamic, unified visual context covering the entire monitoring area, providing global situational awareness for the monitoring personnel. Further, the intelligent perception module is the "wisdom center" of the entire system. It does not work in isolation, but efficiently integrates two types of input signals: one is the panoramic video signal from the panorama stitching module, which provides the global context; the other is the preliminary target signal from the edge preprocessing module, which provides the local focus. This dual-input design greatly improves the efficiency and accuracy of analysis. The module has built-in deep learning models, such as complex models based on convolutional neural networks or visual transformers, which have been extensively trained and optimized for the complex environment of a mine, including mine cards, excavators, conveyors, miners, rocks, dust, and other unique elements. Using the panoramic video signal, it can accurately identify, classify, and continuously track each target in the picture; at the same time, combined with the guidance of the preliminary target signal, it can conduct more in-depth analysis of the marked area. In addition, the core function of this module is "abnormal behavior analysis", which goes beyond simple target recognition and can understand the behavior patterns of targets in the time and space dimensions, such as determining whether personnel have entered a dangerous area that is prohibited, whether equipment has been abnormally started or stopped at non-scheduled times, whether vehicles have deviated from the planned route, etc. All these analysis results are aggregated and structured into "intelligent analysis result signals". Finally, the warning and interaction module is the bridge between the system and the user, responsible for converting analysis results into actionable insights. It receives intelligent analysis result signals and generates corresponding "warning control signals" according to the pre-set rule base. This warning mechanism is usually multi-level, which may include popping up warning boxes on the monitoring screen, triggering on-site sound and light alarms, sending push notifications to the manager's mobile phone, etc., thereby achieving differentiated responses to different risk levels. At the same time, the "visualization interaction signal" it generates drives the graphical user interface, presenting the panoramic video, the target recognition box superimposed on it, the behavior trajectory, the risk area, and the real-time warning information to the user. The interface usually supports rich interactive operations, such as clicking on a target to view details, drawing an electronic fence, replaying historical events, etc., giving users powerful monitoring management and investigation capabilities.

[0022] Further, the further limitation and concretization of the array image acquisition module, the protection focus of which is the spatial configuration, deployment strategy and environmental adaptability design of the module, these details are important guarantees to ensure that the system can run stably and reliably in the real and complex mine environment. Multiple billion-pixel cameras adopt "ring space arrangement mode". This is a high-efficiency layout strategy verified by practice. The cameras are arranged around the key areas of the mine (such as open-pit mining, main transportation platform or dump), which can capture the images of the surrounding environment from 360 degrees. Compared with simple parallel or opposite arrangement, the core technical advantage of ring arrangement is that it systematically ensures that there is an overlapping area between the fields of view of adjacent cameras. This overlapping area is not dispensable, but the mathematical basis for high-precision panoramic stitching. The stitching algorithm needs to use the common features in the overlapping area to calculate the relative position and orientation relationship between the cameras (i.e. external parameter calibration), and accurately "stitch" these images with different angles together. The "size of the overlapping area is optimized" means that before the system is deployed, the optimal installation position and orientation angle of each camera need to be calculated and determined through software simulation or on-site measurement according to the specific mine terrain (such as undulating high slopes, deep pit bottoms), existing permanent obstacles (such as buildings, large equipment) and areas that need to be monitored, to ensure that the overlapping area can meet the needs of the stitching algorithm, and will not cause resource waste and insufficient field of view due to excessive overlapping. The deployment strategy for the unique terrain of the mine is further elaborated. "The camera installation position takes into account the different heights and angles of the mining face, transportation channel and dump", which reveals that the deployment process is not a simple horizontal ring. For example, in order to effectively monitor the high and steep mining slope, cameras may need to be deployed at high and low positions to form a three-dimensional monitoring network; for a long transportation channel, cameras need to be arranged along the line to ensure uninterrupted tracking of moving vehicles. This multi-level and multi-angle deployment scheme is to overcome the problem of line-of-sight obstruction caused by the large undulation of the mine terrain. In addition, "stabilized fixation through fixed supports" is a key implementation detail. The mine environment is full of vibrations caused by the operation of heavy equipment and blasting operations, which can cause image blurring or shaking, seriously affecting the subsequent stitching accuracy and intelligent recognition accuracy. Therefore, a dedicated anti-vibration support (which may use shock-absorbing materials or mechanical damping structures) is necessary hardware support to ensure the stability of image quality. The "environmental adaptation components" of the module are emphasized. The lighting conditions of the mine environment are extremely variable, from the strong direct sunlight at noon to complete darkness at night, and are also disturbed by dust and water vapor all year round. Ordinary automatic exposure algorithms are prone to failure in such large light ratio and turbid medium environments. The environmental adaptation components here mean that the cameras are integrated with or externally connected to more advanced sensors and algorithms, such as those with wide dynamic range to simultaneously capture details in shadow and highlight areas, and automatic white balance and image dehazing algorithms optimized for mine dust environments.Meanwhile, the selection of auto-focus or fixed-focus lens also needs to ensure that the image is always clear at the typical monitoring distance. These measures work together to ensure that the video signal output "remains clear and consistent under different working conditions", providing a reliable and high-quality data source for the downstream processing module, which is a prerequisite for the entire system to function effectively in practical applications.

[0023] In an embodiment of the present application, the "image preprocessing" includes two specific and key steps: "distortion correction" and "color balance". The lens distortion, especially the barrel or pincushion distortion inherent in wide-angle lenses, will make the straight lines in the image curved at the edges, which will have a disastrous impact on the subsequent image stitching, resulting in incorrect matching of feature points and misalignment of stitching seams. Therefore, distortion correction is to restore the true geometric shape of the scene by using a mathematical model to perform geometric transformation on the image with the pre-calculated camera intrinsic parameters (focal length, principal point, distortion coefficient), which straightens the curved lines and is the absolute prerequisite for achieving high-precision seamless stitching. "Color balance" aims to solve the problem of visual consistency. Due to the hardware differences of different cameras, the differences in light angles caused by different installation orientations, and the slight differences in white balance settings, the directly acquired multi-channel raw video signals will have visible differences in color, brightness and contrast. If such pictures are directly stitched, even if the geometric positions are perfectly aligned, they will look very awkward due to the obvious color blocks and brightness jumps, affecting the visual effect and subsequent analysis. The color balance algorithm dynamically adjusts the color mapping curve of each signal by analyzing the overlapping area between images or global statistical information, making it consistent in hue, brightness and saturation, and laying the foundation for generating a visually harmonious panorama. The technical path adopted by "preliminary target detection" is a "lightweight neural network model". This is the best balance point between limited edge computing resources and real-time requirements. Unlike large and complex deep learning models deployed on central servers, lightweight models greatly reduce computational complexity and memory usage while maintaining comparable detection accuracy by reducing the number of network layers, channels, or using efficient network structures (such as MobileNet, SqueezeNet, etc.). This allows it to run smoothly on edge computing devices with limited power consumption and computing power, enabling real-time analysis of video streams. What is particularly important is that the model is "trained on mine scene data", which means that its recognition ability is specific to mine targets, such as its ability to more accurately distinguish between mine trucks and ordinary engineering vehicles, identify workers wearing safety helmets, or ignore common interference such as flying birds and cloud shadows. The output "preliminary target signal" can be a data structure containing target bounding box coordinates, class confidence and timestamp. Although this signal is rough, it provides valuable attention guidance for the intelligent perception module in the later stage, telling the central server "where there may be targets worth further analysis", thereby avoiding the central server's blind and high-computational-cost search of the entire image and optimizing the allocation of system computing resources.

[0024] As Figure 1Further limitations of the core algorithm of the panoramic stitching module are illustrated, and the key technologies and processing architecture that enable real-time, seamless, and high-quality panoramic video generation are elaborated. The technical implementation of this module is the core converter that transforms multiple independent video streams into a unified situational awareness for the entire system, and its performance directly determines the accuracy, smoothness, and visual quality of the final panoramic image. First, it is clear that the real-time image stitching algorithm used by this module is based on "feature point extraction and matching technology". This is a complex and precise multi-step calculation process. Feature point extraction is the first step, and the algorithm (such as improved SIFT, SURF, or ORB, etc.) will find unique image structures that remain stable under different viewing angles and lighting conditions, such as corner points, edge intersection points, or specific texture patterns, from each pre-processed video signal frame. In the challenging environment of a mine, which may have large areas of monotonous sky, rock, and other low-texture regions, the algorithm needs to be optimized to improve the number and uniformity of feature points. Then comes the feature point matching, which finds corresponding partners for the extracted feature points in the overlapping area of adjacent camera video frames. This is a large-scale data association problem, usually solved by calculating the similarity of feature descriptors. Based on these successfully matched feature point pairs, the algorithm can accurately calculate the transformation matrix (usually a homography matrix) that describes the geometric relationship between images, which defines how to project a pixel point in one image into the coordinate system of another image, thus achieving precise alignment of multiple images. However, alignment alone is not enough, as lens vignetting, exposure differences, and other factors may cause noticeable seams and brightness / color jumps between directly stitched images. Therefore, the algorithm further adopts a "multi-band fusion method". This advanced fusion technology does not simply perform linear blending at the seam, but rather decomposes the image into different frequency bands (e.g., high-frequency details and low-frequency color information) and performs fusion operations independently on each band. The smooth fusion of low-frequency information eliminates lighting differences, while the clever handling of high-frequency information preserves image clarity and details, resulting in a visually smooth, continuous, and seamless panoramic video signal. In addition, to address the enormous computational pressure brought by multi-channel billion-pixel video streams, claim 4 emphasizes the integration of a parallel processing architecture that utilizes multi-threading technology to process multiple video streams simultaneously. This means that the stitching task is divided into multiple sub-tasks (such as feature extraction for multiple video streams, multiple matching calculations, and processing of multiple fusion regions), and executed simultaneously by multiple cores of the processor, greatly improving computational efficiency. This parallelization design is crucial to ensuring that the stitching process can keep up with high-frame-rate video streams and meet real-time requirements, allowing the system to continuously output smooth panoramic images even under peak loads with extremely high data throughput. Finally, it is mentioned that the panoramic video signal output is in a scalable format, which usually means that techniques commonly used in map services such as pyramid tiles are used to store and transmit the generated panoramic image in multiple resolution levels.This enables the user to quickly load the data at the resolution required for the current view when interacting, thus enabling smooth zooming and panning operations, allowing the user to quickly focus on the local details of interest from a macroscopic overview, greatly improving the user experience and monitoring efficiency.

[0025] As Figure 1As shown, the internal structure and in-depth development of the intelligent perception module define the core intelligent engine that enables the system to move from "perception" to "cognition". Instead of simply detecting targets, the module aims to achieve deep and contextualized scene understanding. First, it is clear that the deep learning model used is a "convolutional neural network structure", which is a widely validated and highly effective architecture in the field of image recognition. However, more importantly, the model is "optimized specifically for mine environments". This means that during the model training phase, a large amount of meticulously annotated mine scene image dataset is used, which contains various types of mining equipment (such as electric shovels, hydraulic excavators, mine dump trucks), transport vehicles, workers, and their safety equipment (such as safety helmets, reflective clothing), and other unique targets. Through this targeted training, the model learns to ignore common disturbances in the mine environment (such as ore piles, dust), and can accurately distinguish between different devices that look similar, with recognition accuracy and robustness far superior to general target detection models. The core functions of this module include the identification of "multiple target types" and the analysis of "abnormal behavior". Target identification is the foundation, as it provides the system with a list and location of all key elements in the scene. Abnormal behavior analysis represents a higher level of intelligence, as it requires the model to not only recognize static targets, but also understand the dynamic patterns of targets in the time and space dimensions. For example, analyzing whether a device is in an abnormal state of movement or long-term stationary (which may indicate a malfunction); monitoring whether a worker has entered a blast warning zone, a device blind area, or other clearly prohibited dangerous areas; or even identifying small, gradually expanding cracks or signs of surface soil sliding through continuous observation of slope images, all fall within the scope of abnormal behavior analysis. It is also revealed that the module has a key working mode: "multi-modal analysis by fusing panoramic video signals and preliminary target signals". This is an efficient and intelligent data processing strategy. The preliminary target signal comes from the edge, which acts as a fast scout and marks areas "that may have something". After receiving this signal, the intelligent perception module does not blindly rescan the entire panoramic image, but can prioritize and focus on using its more powerful model resources to conduct in-depth analysis and verification of these marked areas. This collaborative mechanism ensures the depth of analysis and optimizes the allocation of computing resources, achieving a balance between accuracy and efficiency. In addition, the "behavior prediction sub-module" contained within the module is described, which further enhances the system's ability from "post-event alert" to "pre-event warning". Based on the current motion state (such as position, speed, direction) and historical trajectory data of the target, the sub-module uses built-in learning models (such as Kalman filter, recurrent neural network, or long short-term memory network) to infer the possible motion path of the target in the near future. For example, predicting whether a moving mine truck will cross paths with a moving worker, or whether a person located on the slope has a tendency to slip and fall.With this predictive ability, the system can "thus generate early warning prompts" to give valuable reaction time for on-site personnel to take risk avoidance measures. Ultimately, the "intelligent analysis result signal" output by this module is a structured, information-rich data package that not only includes what the target is and where it is, but also includes its identity, real-time state, behavior label, and estimated risk level, providing accurate decision-making basis for subsequent warning and interaction modules.

[0026] Further, the function refinement and expansion of the early warning and interaction module define how the system converts intelligent analysis results into effective action instructions and intuitive user experience, which is the last and crucial link to realize human-machine collaborative decision-making. The core of this module lies in the multi-level response mechanism and the richness of the interaction interface. First, it is specified that the "multi-level early warning control signal is generated according to the intelligent analysis result signal". This multi-level early warning mechanism (such as low-level reminder, medium-level warning and high-level alarm) is the key to realize precise management, avoiding the "wolf came" effect, and ensuring that events of different severity can be responded accordingly. For example, a person wandering in a normal area may trigger a low-level reminder, only with icon flashing in the software interface; while the same person entering the blasting danger zone will immediately trigger a high-level alarm, linking the on-site high-pitched alarm and the responsible person's mobile phone message. The output channels of early warning are diversified, including "sound and light equipment" (such as rotating warning light, buzzer) on site, "platform notification" (such as pop-up window, voice broadcast) in monitoring center and "mobile terminal push" (such as sending to the management personnel's mobile phone through a special APP or SMS), which ensures that critical information can be received in time and reliably. Secondly, the "visual interactive signal" driven graphical user interface is described in depth. This interface is the main window of the system and user interaction, which is not simply displaying the original video, but deeply superimposing and fusing panoramic video and intelligent analysis results. Users can directly see the various targets (personnel, vehicles, etc.), their unique identity, real-time motion trajectory, and the system's determination of risk area contour on the panoramic screen. This augmented reality display greatly reduces the cognitive load of the monitoring personnel, making them have a clear overall situation. The interface's interaction ability is a manifestation of its powerful function, which "supports users to define the area of interest and view details through touch or mouse operation", which gives users the ability to actively explore and focus. In addition, the "log recording function" of the module is emphasized, which is an important feature related to the system's traceability and continuous improvement. All triggered early warning events, system analysis results, and user's key operations (such as confirming the alarm, manually marking the area) will be automatically and completely recorded and stored, forming an audit clue with a time stamp. These log data have immeasurable value for post-accident investigation, responsibility definition, operation process optimization, and iterative optimization of the system's own algorithms. Finally, the "remote access" support capability of the module breaks the geographical constraints of monitoring work, allowing security officers or management to master the on-site situation and perform necessary interactive operations in real time through computers, tablets and other terminals anywhere with network, greatly improving the emergency response speed and management flexibility.

[0027] As Figure 1As shown, this is a major enhancement and sublimation of the system function, which upgrades the traditional video monitoring to the level of "digital twin", realizing the deep integration and bidirectional mapping of the physical world and the information world. The core task of this module is to associate abstract video information with concrete geographic spatial information. It receives two key inputs: one is the "panoramic video signal" that provides visual images of the real world, and the other is the "intelligent analysis result signal" that provides semantic scene understanding (i.e. the location, category, ID of the target, etc.). The module accurately "pastes" these video data and analysis results onto the two-dimensional map or three-dimensional model of the mine through pre-calibrated coordinate conversion parameters. This process generates a "dynamic digital twin" that is updated synchronously with the physical mine and dynamically changes. In this digital twin, users no longer see only video images, but a virtual mine that contains all real-time dynamic information. For example, you can accurately see on the three-dimensional model which slope each mine truck is driving on and how fast it is going, and where each worker is active. One of the core applications is to allow users to set "virtual electronic fences" on this digital twin. This is fundamentally different from drawing lines on pure video images. Electronic fences are safety rules based on real geographic coordinates and have clear spatial boundaries. Users can easily define blast areas, equipment maintenance areas, high-risk slope areas, etc. on the digital map and set corresponding trigger rules (such as any personnel entering, specific equipment leaving, etc.). When the location information of the target identified by the intelligent perception module violates the pre-set rules of the spatial relationship of these electronic fences, the system will "automatically trigger an early warning control signal". This alarm mechanism based on geographic information is more direct and reliable than simply analyzing behavior in video images. More importantly, it reveals that this digital twin has a "time backtracking function". This can be understood as that the system has recorded a complete and interactive "4D movie" (three-dimensional space plus time dimension) for the operation of the entire mine. Users can choose any time point in the past, and the system will reproduce the entire digital twin scene state at that moment, including the location of all targets, movement trajectories, and the panoramic video snapshot at that time. This function has revolutionary significance for "accident investigation and process optimization". Investigators can trace back the entire process of the accident like using a time machine, accurately analyze the event chain; managers can also review the production process, analyze vehicle scheduling efficiency, equipment utilization, etc. to find bottlenecks and optimize production organization. In summary, the digital map integration module protected by claim 7 greatly enhances the system's spatial management capability, rule definition capability, and historical tracking capability by building a mine digital twin, upgrading the panoramic monitoring system from a safety tool to a comprehensive intelligent management and control platform that integrates safety, production, and management.

[0028] The mine panoramic monitoring system based on array type hundred million pixel image intelligent perception of the application, through the data aggregation module, the original office behavior signal after desensitization is collected from multiple office software interfaces; the efficiency modeling module processes these signals by using a pre-trained personal efficiency model to generate efficiency portrait signals reflecting the real-time concentration and work load of employees; the intelligent scheduling module generates personalized workflow recommendation signals based on the portrait and the rules of the central strategy library, and intelligently arranges high-cognitive-demand tasks in the high-efficiency period of employees; finally, the optimization suggestions and efficiency insights are fed back to the user in a visual form through the interactive presentation module, forming a closed-loop optimization system from data collection, intelligent analysis, decision recommendation to interactive feedback, thereby realizing dynamic personalized adaptation of work arrangement.

[0029] Therefore, through the mine panoramic monitoring system based on array type hundred million pixel image intelligent perception of the application, the existing office management system can be solved. The problem of rigid static, unable to perceive the individual work state and cognitive law of employees, leading to unreasonable task arrangement, low cooperation efficiency and frequent interruption of individual deep work.

[0030] The above embodiments only exemplarily illustrate the principles and effects of the application, and are not used to limit the application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought disclosed by the application should be covered by the claims of the application.

Claims

1. A panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception, characterized in that, include: An array image acquisition module, which acquires multiple raw video signals of a mining scene through multiple spatially arranged cameras with hundreds of millions of pixels; An edge preprocessing module receives the multiple raw video signals and performs image preprocessing and preliminary target detection on the signals to generate preprocessed video signals and preliminary target signals. A panoramic stitching module receives the preprocessed video signal and fuses multiple video streams using a real-time image stitching algorithm to generate a unified panoramic video signal. The intelligent sensing module receives the panoramic video signal and the preliminary target signal, uses a deep learning model to identify and analyze targets and abnormal behaviors in the mining scene, and generates intelligent analysis result signals. The early warning and interaction module receives the intelligent analysis result signal and generates early warning control signals and visual interaction signals based on the analysis results to realize alarm output and user operation feedback.

2. The panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception as described in claim 1, characterized in that, The array image acquisition module comprises multiple cameras with hundreds of millions of pixels arranged in a circular spatial pattern in key areas of the mine. This ensures that there is an overlapping area between the fields of view of adjacent cameras to cover the entire panorama. The size of the overlapping area is optimized to adapt to the undulating terrain and obstructions in the mine, thereby ensuring that multiple raw video signals can seamlessly capture the entire working face. The camera installation positions take into account the different heights and angles of the mining face, transportation channels, and spoil heaps, and are stably fixed with fixed brackets to prevent image jitter caused by vibration or environmental influences. The array image acquisition module is also equipped with an environmental adaptation component that can automatically adjust exposure and focus to cope with changes in day and night lighting and dust interference in the mine, ensuring that the acquired video signals maintain high definition and consistency under different working conditions, providing a reliable data foundation for subsequent processing.

3. The panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception as described in claim 1, characterized in that, The image preprocessing performed by the edge preprocessing module includes distortion correction of multiple raw video signals to eliminate lens distortion, and color equalization to unify the hue and brightness differences between different cameras. The preliminary target detection uses a lightweight neural network model to identify moving targets and significant anomalies in the video stream in real time. The lightweight neural network model is trained with mining scene data and can quickly screen out potential risk objects such as people or vehicles. The detection results are transmitted synchronously with the preprocessed video signal. The edge preprocessing module also integrates a caching mechanism to temporarily store processed data to cope with network fluctuations, ensuring the continuity and stability of the preprocessed video signal and the preliminary target signal, and reducing data transmission latency.

4. The panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception as described in claim 1, characterized in that, The panoramic stitching module uses a real-time image stitching algorithm based on feature point extraction and matching technology. First, it extracts multi-scale image features from the pre-processed video signal, and then calculates the transformation matrix between images through a feature point matching algorithm to achieve precise alignment. The stitching algorithm uses a multi-band fusion method to eliminate stitching seams and lighting differences, generating a smooth panoramic video signal. The panoramic stitching module also integrates a parallel processing architecture, using multi-threading technology to process multiple video streams simultaneously, ensuring that the stitching process maintains real-time performance under high load. The panoramic video signal output is in a scalable format, allowing users to dynamically adjust the field of view in the panoramic view.

5. The panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception as described in claim 1, characterized in that, The intelligent sensing module utilizes a convolutional neural network deep learning model specifically optimized for mining environments. It can identify various target types, including mining equipment, transport vehicles, workers, and safety equipment. Simultaneously, it analyzes abnormal behaviors such as equipment malfunctions, personnel entering dangerous areas, or signs of slope slippage. The deep learning model performs multimodal analysis by fusing panoramic video signals and preliminary target signals, improving recognition accuracy and robustness. The intelligent sensing module also includes a behavior prediction submodule, which infers the future trajectory of targets based on historical data and the learning model, thereby generating early warning prompts. The intelligent analysis result signal includes target location, category, and risk level information.

6. The panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception as described in claim 1, characterized in that, The early warning and interaction module generates multi-level early warning control signals based on intelligent analysis results, including low-level alerts, medium-level warnings, and high-level alarms, each corresponding to different risk levels. Alarm output is achieved through audio-visual devices, platform notifications, or mobile terminal push notifications. The visual interactive signal provides a graphical user interface, displaying panoramic video and intelligent analysis overlay information. Users can select areas of interest and view details via touch or mouse operation. The early warning and interaction module also integrates a log recording function, storing all early warning events and user operation records for subsequent auditing and analysis. The module supports remote access, allowing administrators to monitor and interact in real time on different terminals.

7. The panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception as described in claim 1, characterized in that, The system also includes a digital map integration module, which receives panoramic video signals and intelligent analysis results signals, and aligns and overlays them with the coordinates of a two-dimensional map or three-dimensional model of the mine to generate a dynamic digital twin. The digital map integration module allows users to set virtual electronic fences on the digital twin, defining safe areas and dangerous boundaries. When the intelligent sensing module detects a target entering or leaving these areas, it automatically triggers an early warning control signal. The digital twin supports time rewind functionality, which can reproduce historical scenes for accident investigation and process optimization.

8. The panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception as described in claim 1, characterized in that, The system supports dynamic enhancement of regions of interest. When a user selects a specific region in the panoramic video through the warning and interaction module, the system automatically calls the high-resolution raw video signal from the corresponding camera in the array image acquisition module, performs digital zoom processing, and magnifies the display while maintaining the clarity of image details. The dynamic enhancement function allows multiple regions of interest to be processed simultaneously, and each region is analyzed and warned independently. The processing is performed in real time without interrupting the panoramic video stream, ensuring the continuity and comprehensiveness of monitoring.

9. The panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception according to claim 1, characterized in that, The system also includes a data storage and auditing module, which receives and stores dynamic context signals, including multiple raw video signals, pre-processed video signals, preliminary target signals, panoramic video signals, intelligent analysis result signals, and early warning control signals, forming a complete data chain. The data storage and auditing module indexes and compresses the stored data, supports fast retrieval and playback, and generates audit logs to record all processing steps and decision-making basis for compliance checks and performance evaluations. The module also provides a data export interface for easy access and analysis by external systems.

10. The panoramic monitoring system for mines based on array-type billion-pixel image intelligent perception according to claim 1, characterized in that, The abnormal behavior analysis of the intelligent sensing module includes monitoring the stability of mine slopes. By continuously analyzing changes in surface texture and displacement signs in panoramic video signals, it identifies potential landslide or collapse risks. The analysis combines a time-series data model to compare image differences at different time points to detect minute changes. The intelligent sensing module also fuses data with external sensor data, such as vibration or tilt sensors, to improve the reliability of anomaly detection. The results are used to generate early warning signals to guide on-site personnel in taking preventive measures.

Images