A dynamic target tracking method and system for vehicle-mounted intrinsically safe video monitoring for mines

Abstract

The present invention provides a dynamic target tracking method and system for vehicle-mounted intrinsically safe video monitoring for mines, which relates to the field of video monitoring target tracking technology, including real-time video streams obtained through vehicle-mounted cameras; preprocessing the real-time video streams; fine-grained tracking feature extraction of detection target results; sorting captured events according to timestamps to obtain the real-time position and motion trend of mobile targets; using the Kalman filter in the TLD target movement algorithm for short-term prediction to obtain an updated target model; using a motion detection algorithm to process to obtain the motion characteristics of the mobile target; applying a video tracking algorithm based on background segmentation to process to obtain specific event prediction results, combining the motion characteristics of the mobile target and the specific event prediction results, and finally obtaining dynamic target tracking results. The present invention improves the performance of the vehicle-mounted intrinsically safe video monitoring system for mines in dynamic target tracking, and realizes accurate tracking of dynamic targets in mines.

Description

Technical Field

[0001] The present invention relates to the technical field of video monitoring target tracking, and in particular to a dynamic target tracking method and system for vehicle-mounted mining intrinsically safe video monitoring. Background Art

[0002] With the continuous deepening of industrialization and automation, the safety and efficiency of mine operations have gradually received attention. In the mine monitoring system, dynamic target tracking technology is one of the key technologies to ensure the safety of miners and optimize the operation process. In recent years, with the development of computer vision and pattern recognition technology, the application of dynamic target tracking technology in the field of mine monitoring has been significantly improved.

[0003] Traditional monitoring systems usually use wired connections, which can easily cause line damage in complex environments such as mines. The internal environment of mines is complex, with dim light, high dust, and large temperature changes. Traditional video monitoring technology is difficult to work stably in such an environment, and sometimes affects the monitoring quality. For example, the environment in mines is harsh, such as high dust and strong vibration, which requires extremely high stability and reliability of equipment, and many existing equipment cannot adapt. However, existing video monitoring systems often focus on video recording rather than real-time analysis, lack the ability to respond quickly to emergencies, and often need to upload video data to the central server for analysis when processing video data, resulting in slow processing speed and unable to meet the needs of real-time monitoring. In addition, the target tracking algorithm is easy to lose the target in complex backgrounds, such as rocks and vehicles of similar colors in mines, affecting the monitoring effect. In addition, existing systems often lack effective early warning mechanisms and cannot make accurate predictions and timely responses before danger occurs. Therefore, existing technologies face many challenges in special environments such as mines, and these factors seriously affect the stability and reliability of monitoring systems. Summary of the invention

[0004] The purpose of the present invention is to provide a dynamic target tracking method and system for vehicle-mounted intrinsically safe video monitoring for mining, so as to improve the above problems. In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present application provides a dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring, comprising:

[0006] A real-time video stream obtained by a vehicle-mounted camera, wherein the real-time video stream includes a visible light video and an event stream captured by an event camera, wherein the visible light video is a visible light activity scene in the mine, including miner activities, vehicle operation, and the status of other visible objects, and the event stream captured by the event camera includes the location and time information of brightness changes recorded when pixel brightness changes;

[0007] Based on the pixel brightness changes in the event stream captured by the event camera, an event stream containing a timestamp is generated, and the captured events are sorted according to the timestamps to generate an event stream with high temporal resolution; the real-time video stream is preprocessed, wherein the preprocessing includes noise reduction and illumination enhancement, to obtain an optimized video frame, and the optimized video frame is analyzed using a lightweight deep learning model to obtain a detection target result, and the detection target result is subjected to fine-grained tracking feature extraction based on the high temporal resolution event stream and visible light video to obtain the first feature of the surface interest point of the target result, wherein the first feature includes local texture, shape information and motion mode, and the detection target result includes preliminary positioning results of miners, vehicle targets and other visible objects;

[0008] According to the first feature of the surface interest point, the moving target in the event stream is processed by the fast tracking algorithm to obtain the real-time position and movement trend of the moving target, and the Kalman filter in the TLD target movement algorithm is used for short-term prediction to obtain the short-term movement trajectory of the moving target. After the online learning mechanism, the updated target model is obtained;

[0009] Based on the updated target model and real-time video frames, the motion detection algorithm is used to obtain the motion state of the moving target, wherein the processing process includes analyzing the changes in the pixels in the image sequence to identify the moving target and extract its features; according to the motion state and the updated target model, the motion features of the moving target are obtained through the adaptive tracking algorithm;

[0010] According to the color distribution characteristics of specific events, a video tracking algorithm based on background segmentation is applied to obtain the prediction results of specific events. The processing process includes calculating the color histogram of the dynamic target, searching for the area that best matches this histogram in the next frame, and dynamically adjusting the size and shape of the target box; combining the motion characteristics of the moving target and the prediction results of specific events, and finally obtaining the dynamic target tracking results through multi-scale fusion processing and update correction.

[0011] Preferably, the optimized video frame is analyzed using a lightweight deep learning model to obtain a detection target result; fine-grained tracking feature extraction is performed on the detection target result to obtain a first feature of a surface interest point of the target result, which includes:

[0012] The optimized video frames are processed by the YOLOv5-Lite model, which obtains preliminary positioning results of miners, vehicle targets, and other visible objects by multiplying the input frame with the pre-trained weights and applying an activation function;

[0013] Based on the preliminary positioning results, the optimized metric of the target candidate area is obtained through distance intersection and union calculation, and the bounding box of the target candidate area is obtained through non-maximum suppression processing, where the processing process includes calculating the intersection and union ratio of the predicted box and removing the overlapping boxes based on the preset threshold;

[0014] By using the scale-invariant feature transformation algorithm and calculating the gradient direction histogram of the local image in the bounding box, the features that can still be recognized and matched under the conditions of illumination change and occlusion rotation are extracted, and the features are recorded as the first features of the interest points on the target surface.

[0015] Preferably, the scale-invariant feature transformation algorithm is used to calculate the gradient direction histogram of the local image in the bounding box, extract features that can still be recognized and matched under the conditions of illumination change and occlusion rotation, and record the features as the first features of the interest points on the target surface, including:

[0016] Based on the bounding box, a differential Gaussian pyramid is constructed using the Gaussian pyramid technology to obtain the first key point position, wherein the first key point position includes a point representing a corner point, an edge or other significant feature of the target in the mine in the scale space, wherein the differential Gaussian pyramid is obtained by calculating the difference between two adjacent layers of the Gaussian pyramid;

[0017] According to the differential Gaussian pyramid, the scale space of each pixel at the first key point position is detected to obtain the second key point position, wherein each pixel includes an extreme point compared with 26 adjacent pixels in the surrounding adjacent scale layers;

[0018] Based on the gradient direction around the second key point position, a unique direction of each second key point position is obtained by calculating the gradient direction histogram, wherein a 16x16 pixel neighborhood is taken around the second key point position, and the gradient direction and magnitude of each pixel are calculated, and an 8-direction histogram is generated in each 4x4 sub-region to assign a main direction to the second key point position;

[0019] According to the main direction of the second key point position, a feature descriptor is generated in the 16x16 pixel neighborhood around it, so as to obtain features that can still be recognized and matched under conditions of illumination changes and occlusion rotation, and recorded as the first feature of the interest point on the target surface. The descriptor is constructed by calculating the histogram of the gradient direction in each 4x4 sub-region, and finally forming a 128-dimensional feature vector.

[0020] Preferably, the method of processing the moving target in the event stream by a fast tracking algorithm based on the first feature of the surface interest point to obtain the real-time position and movement trend of the moving target includes:

[0021] Calculating a color histogram of the moving target area based on the first feature of the surface interest point;

[0022] Calculate the area similar to the color histogram of the moving target in the search window to determine the current position of the moving target, obtain the matching result, and update the position of the moving target according to the matching result;

[0023] Based on the updated position of the moving target, the movement trend of the moving target is analyzed, wherein the movement trend includes movement direction and speed.

[0024] Preferably, the Kalman filter in the TLD target movement algorithm is used for short-term prediction to obtain the short-term motion trajectory of the moving target, and an updated target model is obtained through an online learning mechanism, which includes:

[0025] Based on the real-time position and motion trend of the moving target, the target state is optimally estimated and calculated after being processed by the Kalman filter to obtain the short-term motion trajectory of the moving target. The calculation process includes a prediction step and an update step. The prediction step is to estimate the position and speed of the moving target in the next frame, and the update step is to adjust the prediction according to the new observation data.

[0026] According to the short-term motion trajectory of the moving target, positive and negative samples are collected to form a training data set for model updating, where positive samples come from the detection results of the tracked target, and negative samples come from the area identified as non-target;

[0027] The online stochastic gradient descent algorithm is applied to process the training data set, and the model parameters are updated online to adapt to the changes in the target appearance, thereby obtaining an updated target model that reflects the current state of the moving target.

[0028] Preferably, the color distribution characteristics of the specific event are processed by a video tracking algorithm based on background segmentation to obtain a specific event prediction result, wherein the processing process includes calculating the color histogram of the dynamic target and searching for the area that best matches the histogram in the next frame to dynamically adjust the size and shape of the target frame, including:

[0029] Calculate the color histogram of specific events in the mine to obtain the color feature description of the target, where the color distribution characteristics include color concentration, distribution range and color uniformity;

[0030] Based on the color feature description of the target, search for the area in the next frame that best matches the calculated color histogram to obtain the potential position of the target. The histogram matching process includes comparing the color histogram of each pixel with the target histogram to find the best matching area.

[0031] According to the potential position of the target, the CamShift algorithm is used to dynamically adjust the target box according to the color distribution of the target, which includes initializing the target box. In subsequent frames, the color histogram is used to track the target. The position and size of the target box are iteratively updated to match the color distribution of the target. The target box position is iteratively updated until the pixel color angle that best matches the target color histogram is obtained, thereby obtaining a specific event prediction result.

[0032] In the second aspect, the present application also provides a dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring, including:

[0033] Acquisition module: used to acquire real-time video streams through vehicle-mounted cameras, where the real-time video streams include visible light video and event streams captured by event cameras, where the visible light video is a visible light activity scene in the mine, including miner activities, vehicle operation, and the status of other visible objects, and the event stream captured by the event camera includes the location and time information of brightness changes recorded when pixel brightness changes;

[0034] Detection and extraction module: used to generate an event stream containing timestamps based on the pixel brightness changes in the event stream captured by the event camera, and sort the captured events according to the timestamps to generate an event stream with high temporal resolution; preprocess the real-time video stream, where the preprocessing includes noise reduction and illumination enhancement, to obtain optimized video frames, use a lightweight deep learning model to analyze the optimized video frames to obtain detection target results, and perform fine-grained tracking feature extraction on the detection target results based on the high temporal resolution event stream and visible light video to obtain the first feature of the surface interest point of the target result, where the first feature includes local texture, shape information and motion mode, and the detection target result includes preliminary positioning results of miners, vehicle targets and other visible objects;

[0035] Prediction module: It is used to process the moving targets in the event stream according to the first feature of the surface interest point through the fast tracking algorithm to obtain the real-time position and movement trend of the moving target, and use the Kalman filter in the TLD target movement algorithm to make short-term predictions to obtain the short-term movement trajectory of the moving target. After the online learning mechanism, the updated target model is obtained;

[0036] Identification module: used to obtain the motion state of the moving target by using the motion detection algorithm based on the updated target model and real-time video frames. The motion detection algorithm processing process includes analyzing the changes in pixels in the image sequence to identify the moving target and extract its features; according to the motion state and the updated target model, the motion features of the moving target are obtained by processing with the adaptive tracking algorithm;

[0037] Tracking module: It is used to target the color distribution characteristics of specific events and apply a background segmentation-based video tracking algorithm to obtain specific event prediction results. The processing includes calculating the color histogram of the dynamic target and searching for the area that best matches this histogram in the next frame, dynamically adjusting the size and shape of the target box; combining the motion characteristics of the moving target and the specific event prediction results, and finally obtaining the dynamic target tracking results through multi-scale fusion processing and update correction.

[0038] In a third aspect, the present application also provides a dynamic target tracking device for vehicle-mounted mining intrinsically safe video monitoring, including:

[0039] Memory for storing computer programs;

[0040] A processor is used to implement the steps of the dynamic target tracking method of the vehicle-mounted mining intrinsically safe video monitoring when executing the computer program.

[0041] In a fourth aspect, the present application further provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned dynamic target tracking method based on vehicle-mounted mining intrinsically safe video monitoring are implemented.

[0042] The beneficial effects of the present invention are:

[0043] The present invention utilizes a vehicle-mounted computing platform to perform front-end processing of the video, reduces data transmission delays, and improves the real-time performance of the system. It focuses on points of interest on the target surface through fine-grained tracking technology to improve the accuracy of target detection. It utilizes an event camera to capture dynamic changes in the scene and generates event streams with high temporal resolution to achieve efficient tracking of fast-moving targets. It uses a target detection and tracking algorithm based on deep learning to improve the accuracy of target detection and tracking. It also applies a video tracking algorithm based on background segmentation to dynamically adjust the size and shape of the target frame according to the color distribution of the target, thereby improving the tracking capability of events such as ore slides and gas leaks, achieving dynamic monitoring and intelligent analysis of coal mine disaster events, and improving the accuracy and response speed of early warnings.

[0044] The present invention uses a lightweight deep learning model to analyze the optimized video frames. The present invention can accurately detect targets such as miners and vehicles, and can maintain high accuracy even in a complex environment such as a mine. By performing fine-grained tracking feature extraction on the detected target, the first feature including local texture, shape information and motion pattern is obtained. These features are highly robust to lighting changes and occlusions, and help to maintain a stable tracking effect in harsh environments. By utilizing the high-temporal resolution event stream captured by the event camera and combining it with a fast tracking algorithm, the present invention can track fast-moving targets in real time, and is particularly suitable for dynamic monitoring of equipment or personnel in mines. By performing short-term prediction through the Kalman filter in the TLD algorithm and updating the target model in combination with an online learning mechanism, the present invention can adapt to the dynamic changes of the target motion and improve the accuracy and adaptability of tracking.

[0045] The present invention applies the CamShift algorithm to dynamically adjust the size and shape of the target frame according to the color distribution characteristics of specific events such as ore slides and gas leaks, so as to accurately track these events, issue early warnings in time, and improve the safety of mines; by performing front-end processing and feature extraction on the on-board computing platform, the dependence on the central server is reduced, the data transmission delay is reduced, the real-time performance and reliability of the system are improved, and the use of computing resources is also optimized; through multi-scale fusion processing and update correction, the motion characteristics of dynamic targets and the prediction results of specific events are comprehensively utilized to improve the intelligent decision-making level of the system and achieve more accurate target tracking.

[0046] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or be understood by implementing the embodiments of the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 It is a schematic flow chart of a dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring described in an embodiment of the present invention;

[0049] Figure 2 It is a structural schematic diagram of a dynamic target tracking system for vehicle-mounted intrinsically safe video monitoring for mining according to an embodiment of the present invention;

[0050] Figure 3 It is a schematic structural diagram of a dynamic target tracking device for vehicle-mounted intrinsically safe video monitoring for mining as described in an embodiment of the present invention.

[0051] In the figure: 701, acquisition module; 702, detection and extraction module; 703, prediction module; 704, recognition module; 705, tracking module; 800, dynamic target tracking equipment for vehicle-mounted intrinsically safe video monitoring for mining; 801, processor; 802, memory; 803, multimedia component; 804, I / O interface; 805, communication component. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0054] Embodiment 1:

[0055] This embodiment provides a dynamic target tracking method for vehicle-mounted intrinsically safe video monitoring for mining.

[0056] See also Figure 1 , the figure shows that the method includes step S100, step S200, step S300, step S400 and step S500.

[0057] S100, a real-time video stream obtained by a vehicle-mounted camera, wherein the real-time video stream includes a visible light video and an event stream captured by an event camera, wherein the visible light video is a visible light activity scene in a mine, including miner activities, vehicle operation, and the status of other visible objects, and the event stream captured by the event camera includes location and time information of brightness changes recorded when pixel brightness changes;

[0058] It can be understood that in this step, the event stream captured by the event camera: captured by the event camera, which is different from the traditional camera. The event camera only records events when the pixel brightness changes, and generates a timestamp-based event stream. These event streams contain the location and time information of the brightness change. The data difference between the two is: visible light video provides continuous video frames, which is suitable for capturing details and color information in the scene, but the data volume is large and the processing requirements are high; the event stream provides asynchronous, event-driven data, which is suitable for capturing rapid changes and has low requirements for computing resources, but lacks the detailed information of traditional videos.

[0059] S200. Based on the pixel brightness changes in the event stream captured by the event camera, an event stream containing a timestamp is generated, and the captured events are sorted according to the timestamps to generate an event stream with high temporal resolution; the real-time video stream is preprocessed, wherein the preprocessing includes noise reduction and illumination enhancement, to obtain optimized video frames, and the optimized video frames are analyzed using a lightweight deep learning model to obtain detection target results, and the detection target results are subjected to fine-grained tracking feature extraction based on the high temporal resolution event stream and visible light video to obtain a first feature of a surface interest point of the target result, wherein the first feature includes local texture, shape information and motion pattern, and the detection target result includes preliminary positioning results of miners, vehicle targets and other visible objects.

[0060] Among them, the target surface interest points refer to those points on the surface of the target object with unique and significant features in image processing and computer vision. These points are usually easy to identify and track in the image because they have obvious texture changes or color changes in the surrounding area, which makes them stand out from other image areas. In target tracking and recognition, interest points are key features because they help: feature matching: matching the same object between different images, even if the object is rotated, scaled or the lighting changes; target positioning: accurately locating and tracking the target object in the image; motion estimation: analyzing the movement pattern and direction of the target object. Common interest points include corner points, edge points or other significant texture change points. In practical applications, various algorithms can be used to detect these points, such as Harris corner detector, FAST or SIFT.

[0061] In this step, based on the pixel brightness changes in the event stream captured by the event camera, an event stream containing timestamps is generated, and the captured events are sorted according to timestamps to generate an event stream with high temporal resolution, which includes:

[0062] By utilizing the characteristics of the event camera, the position and time information of the brightness change when the pixel brightness changes are recorded to form an event stream with high time resolution. The calculation formula of the sorting process is as follows:

[0063]

[0064] In the formula, Sorted represents a sorted event stream, Captured represents the raw event stream captured from the event camera, sort is a sorting function, l For each event's timestamp, x Represents a single event in an event stream, x . timestamp Indicates an event x The timestamp of the function. The key is a parameter used to specify a function.

[0065] It can be understood that step S200 includes S201, S202 and S203, wherein:

[0066] S201. The optimized video frame is processed by the YOLOv5-Lite model. The model obtains the preliminary positioning results of miners, vehicle targets and other visible objects by multiplying the input frame with the pre-trained weight and applying the activation function. The calculation formula for target detection by the YOLOv5-Lite model is as follows:

[0067]

[0068] In the formula, bi represents the predicted value of the bounding box, Wi and bi ′ is the weight and bias parameters of the model, s is the sigmoid activation function, x Represents the input feature map;

[0069] S202, based on the preliminary positioning result, the optimized metric of the target candidate area is obtained by distance intersection and union calculation, and the bounding box of the target candidate area is obtained by non-maximum suppression processing, wherein the processing process includes calculating the intersection and union of the predicted box and removing the overlapping box based on a preset threshold, wherein the calculation formula of the non-maximum suppression processing is as follows:

[0070]

[0071] In the formula, B 1 and B 2 represents two bounding boxes, IoU is the intersection over union ratio, and area is the area;

[0072] S203, using a scale-invariant feature transformation algorithm, by calculating the gradient direction histogram of the local image in the bounding box, extracting features that can still be recognized and matched under conditions of illumination changes and occlusion rotation, and recording the features as the first features of the interest points on the target surface.

[0073] It should be noted that features with strong robustness refer to features that can still be accurately identified and matched under various environmental changes, such as changes in lighting conditions, image blur, occlusion, rotation, scale changes, etc. These features are very important for target recognition and tracking because they provide a capability to work stably under different conditions.

[0074] In this embodiment, step S203 also includes S2031, S2032, S2033 and S2034, wherein:

[0075] S2031. Based on the bounding box, a differential Gaussian pyramid is constructed using the Gaussian pyramid technology to obtain a first key point position, wherein the first key point position includes a point representing a corner point, an edge or other significant feature of a target in the mine in the scale space, wherein the differential Gaussian pyramid is obtained by calculating the difference between two adjacent layers of the Gaussian pyramid, and the calculation formula is as follows:

[0076] DoG( x , y , s )= G ( x , y , ks )− G ( x , y , s )

[0077] Where DoG is the Gaussian difference function, G ( x , y , s ) is a Gaussian function, s is the standard deviation of the Gaussian function, k is a constant that controls the scale change, x and y is the coordinate of the pixel in the image, ks Compared to the current scale s Large scales are used to construct blurred images of adjacent scales in the Gaussian pyramid;

[0078] Therefore, the Gaussian difference function actually calculates the difference between two Gaussian blurred images of different scales. This difference image can highlight the local features in the image, such as edges and corners, which are invariant to scale transformation. In the SIFT algorithm, the Gaussian difference function is used to identify potential key point locations.

[0079] S2032, performing extreme value detection on the scale space of each pixel point at the first key point position according to the differential Gaussian pyramid to obtain the second key point position, wherein each pixel point includes an extreme value point compared with 26 adjacent pixels at surrounding adjacent scale layers;

[0080] It should be noted that the differential Gaussian pyramid is a pyramid consisting of a series of images obtained by Gaussian filtering and scaling the original image. Each layer of images represents a different scale (blur degree), and each layer of images is more blurred than the next layer of images. The first key point position is In the differential Gaussian pyramid, each pixel is compared with its surrounding 8 pixels in the same scale layer and 18 pixels in the upper and lower adjacent scale layers (a total of 26 pixels) to determine whether the pixel is a local extreme point (i.e., the local maximum or minimum value of brightness). The purpose of extreme value detection is to find points that are significantly prominent in brightness, which may be potential key points. If a pixel is higher or lower in brightness than the surrounding 26 pixels in the current scale layer of the differential Gaussian pyramid, then it is an extreme point. The second key point position is the pixel position identified as an extreme point in the first round of detection, which becomes the final key point position after further screening and precise positioning. These positions represent points with significant features in the image, such as corners, edges, etc. Scale space refers to the representation of an image at different scales. In the SIFT algorithm, the scale space is implemented through a Gaussian pyramid, where each scale layer represents a blurred version of the image at that scale.

[0081] In summary, this sentence describes how the SIFT algorithm identifies potential key point locations by comparing the brightness values ​​of each pixel in the DoG pyramid with its surrounding neighboring pixels, which will be further processed in subsequent steps to determine the final key points.

[0082] S2033, based on the gradient direction around the second key point position, by calculating the gradient direction histogram, a unique direction of each second key point position is obtained, wherein a neighborhood of 16x16 pixels is taken around the second key point position, and the gradient direction and magnitude of each pixel are calculated, and an 8-direction histogram is generated in each 4x4 sub-region, and a main direction is assigned to the second key point position;

[0083] It should be noted that in the SIFT (Scale Invariant Feature Transform) algorithm, the 8-direction histogram refers to the gradient direction histogram calculated for each sub-region in the 4x4 pixel neighborhood around the key point. This histogram is used to describe the local features of the image. Among them, the meaning of the 8-direction histogram is: Direction assignment: In image processing, in order to make the feature descriptor invariant to rotation, it is necessary to assign a main direction to each key point; Gradient calculation: For each pixel around the key point, calculate its gradient vector (including the magnitude and direction of the gradient), and the gradient direction is the direction pointing to the fastest change in image brightness; Histogram construction: Divide the 4x4 neighborhood into 8 quadrants, each quadrant covers an angle range of 45 degrees, and construct a histogram for each quadrant to count the number of gradient directions falling in the quadrant; 8 directions: These 8 directions are 0°, 45°, 90°, 135°, 180°, 225°, 270° and 315°, where each quadrant corresponds to a direction interval, which is used to count the gradient information in that direction.

[0084] It can be understood that during the calculation process, for each pixel around the key point, its gradient direction is calculated, and then according to the gradient direction, it is determined which of the 8 quadrants the pixel belongs to, and finally the count is increased in the histogram of the corresponding quadrant (direction). In this way, even if the image is rotated, the histograms of the 8 directions can still describe the same features because they are based on the local gradient direction, not the absolute coordinate direction. The generated 8-direction histogram is used to construct a feature descriptor for the key point, which can uniquely identify the key point in the image and remain stable when the image is scaled and rotated.

[0085] S2034. Based on the main direction of the second key point position, a feature descriptor is generated in a 16x16 pixel neighborhood around it, so as to obtain a feature that can still be recognized and matched under conditions of illumination changes and occlusion rotation, and is recorded as the first feature of the interest point on the target surface, wherein the descriptor is constructed by calculating the histogram of the gradient direction in each 4x4 sub-region, and finally forming a 128-dimensional feature vector.

[0086] Among them, the "128-dimensional feature vector" is a set of numerical values ​​used in the SIFT algorithm to describe the features of key points in an image. It contains the image information around the point and is used in the subsequent image matching and recognition process. These 128 dimensions describe the gradient direction histogram information around the key point, and each dimension corresponds to the gradient statistics of a direction in a 4x4 sub-region in the 16x16 pixel area around the key point. Among them, the connection with "and recording the feature as the first feature of the target surface interest point" is that this 128-dimensional feature vector is a detailed description of the key point in the image (i.e., the target surface interest point). It captures the local image features around the key point, including information such as gradient direction and size, and these feature descriptors enable the algorithm to identify and match the same target surface interest point under different scales, directions and lighting conditions. In the SIFT algorithm, each key point is assigned a main direction, and the 16x16 pixel neighborhood around it is divided into 16 sub-regions. Each sub-region calculates an 8-directional gradient histogram, and these histograms are combined to form a 128-dimensional feature vector. It can be understood that this vector provides a unique "fingerprint" for each key point in the image, so that it can be accurately identified and matched during the image matching process. The feature vectors included in the "128-dimensional feature vector" have good adaptability to changes in dust, water vapor and dim light conditions commonly found in mines. These 128-dimensional feature vectors will be normalized to reduce the impact of lighting changes and improve the stability and accuracy of feature matching, which also helps to improve the reliability of mine monitoring systems.

[0087] S300, according to the first feature of the surface interest point, the moving target in the event stream is processed by a fast tracking algorithm to obtain the real-time position and movement trend of the moving target, and the Kalman filter in the TLD target movement algorithm is used for short-term prediction to obtain the short-term movement trajectory of the moving target, and an updated target model is obtained through an online learning mechanism;

[0088] Use event-stream-based tracking algorithms, such as optical flow or Kalman filter, to track fast-moving targets. This method can quickly respond to the target's motion changes, achieve real-time tracking, and obtain the real-time position and motion trend of fast-moving targets through algorithm processing. This information is crucial for predicting potential safety accidents.

[0089] It can be understood that step S300 includes S301, S302 and S303, wherein:

[0090] S301, based on the first feature of the surface interest point, calculate the color histogram of the moving target area, wherein the calculation formula is as follows:

[0091]

[0092] In the formula, H is the color histogram of the target area, N is the total number of pixels in the area, i is the color angle, d is the Dirac delta function used to construct the color histogram, i is the current number of items in the summation process, I will For the i The color angle of each pixel;

[0093] S302, calculating the area similar to the color histogram of the moving target in the search window to determine the current position of the moving target, obtain a matching result, and update the position of the moving target according to the matching result. The calculation formula is as follows:

[0094]

[0095] In the formula, S is the similarity within the search window, Hw is the color histogram within the search window, Hello is the color histogram of the target model, x , y are pixel coordinates within the image or search window, where x and y Traverse each pixel in the search window;

[0096] In this step, it can be understood that the real-time position of the target is determined by finding the window position with the greatest similarity.

[0097] S303: Analyze the movement trend of the moving target based on the updated position of the moving target, where the movement trend includes movement direction and speed.

[0098] In this step, the movement trend includes the movement direction and speed, wherein the speed calculation formula is as follows:

[0099]

[0100] in, v is the target speed, d For in time t The distance the target moves.

[0101] In mine monitoring, speed calculation can help understand how fast or slow a target is moving.

[0102] The direction calculation formula is as follows:

[0103]

[0104] Among them, θ represents the direction of movement of the target, Δy and Δx are the displacement of the target on the y-axis and x-axis respectively. The calculation of direction is very important for tracking the moving trajectory of the target and predicting its possible route. Through these two steps, the mine monitoring system can track the position of targets such as miners and vehicles in real time and analyze their movement trends, thereby improving the safety and efficiency of mine operations.

[0105] It can be understood that step S300 also includes S304, S305 and S306, wherein:

[0106] S304, based on the real-time position and motion trend of the moving target, after processing by the Kalman filter, the target state is optimally estimated and calculated to obtain the short-term motion trajectory of the moving target, wherein the calculation process includes a prediction step and an update step, the prediction step is to estimate the position and speed of the moving target in the next frame, and the update step is to adjust the prediction according to the new observation data;

[0107] It should be noted that the prediction step: estimating the position and speed of the target in the next frame uses the following formula:

[0108]

[0109] In the formula, x ^ k ∣ k −1 is the time step k The prior state estimate of is the state transfer matrix, For the time step k −1 posterior state estimate, is the control matrix, For control input.

[0110] Update step: adjust the predictions based on the new observations using the following formula:

[0111]

[0112] In the formula, For the time step k The posterior state estimate of is the Kalman gain, is the observed value, x ^ k ∣ k −1 is the time step k The prior state estimate of is the observation matrix.

[0113] S305, collecting positive and negative samples according to the short-term motion trajectory of the moving target to form a training data set for model updating, wherein the positive samples come from the detection results of the tracked target, and the negative samples come from the area identified as a non-target;

[0114] S306, applying an online stochastic gradient descent algorithm to process the training data set, updating the model parameters online to adapt to changes in the target appearance, thereby obtaining an updated target model that reflects the current state of the moving target.

[0115] It can be understood that in the mine monitoring scenario of this step, this means that whenever a new video frame or sensor data arrives, the model can be updated immediately to adapt to the changes in the dynamic targets in the mine. In specific applications, the SGD update process can be described as the following steps:

[0116] First, according to the state of the previous model, the model parameters θ are initialized. After the online learning mechanism, for each newly arrived sample ( xi , yes ), calculate the loss function L ( i , xi , yes ) About parameters i The gradient of ∇ θL ( i , xi , yes ), and then use the stochastic gradient descent update rule to update the model parameters:

[0117]

[0118] In the formula, or is the learning rate, a hyperparameter that controls the step size of each update, i new represents the updated model parameters, i old represents the model parameters before updating, is the loss function L In the current parameters i old, for the sample ( xi , yes ) gradient; then after online learning, it may be necessary to dynamically adjust the learning rate or , to adapt to changes in the target. This step can be achieved through a learning rate decay strategy or an adaptive learning rate algorithm. Finally, after the model is updated, the performance of the model on the new data is evaluated. If the model performs poorly, the learning rate or model structure may need to be further adjusted.

[0119] In summary, in mine monitoring scenarios, SGD can be used to update target detection and tracking models in real time to adapt to the dynamic changes of targets such as miners and vehicles. For example, if the model performs poorly on new samples, SGD can be used to quickly adjust the model parameters to improve the accuracy and robustness of tracking. This method is particularly suitable for environments such as mines where the data stream is constantly changing because it can quickly respond to changes in the appearance of the target, such as changes caused by dust, lighting changes, or target occlusion.

[0120] S400, based on the updated target model and the real-time video frame, the motion detection algorithm is used to obtain the motion state of the moving target, wherein the motion detection algorithm processing process includes analyzing the changes in the pixels in the image sequence to identify the moving target and extract its features; according to the motion state and the updated target model, the motion features of the moving target are obtained through the adaptive tracking algorithm processing.

[0121] It can be understood that the step S400 includes S401, S402, S403 and S404, wherein:

[0122] S401. According to the updated target model and the real-time video frame, preliminary motion information of the moving target is obtained through processing by the inter-frame difference algorithm. The inter-frame difference algorithm detects the moving target by calculating the pixel-level difference between consecutive video frames:

[0123]

[0124] In the formula, It is the current frame, is the previous frame, and Motion is the motion information. This step uses the pixel changes caused by the target motion in the mine environment to provide a basis for subsequent feature extraction;

[0125] S402: According to the motion information obtained in step S401, the motion vector and speed of the moving target are obtained through processing based on the optical flow algorithm. The optical flow algorithm estimates the motion characteristics of the target by analyzing its displacement in consecutive frames:

[0126]

[0127] In the formula, is the position of the target in the current frame, is the position of the previous frame, Δ t is the time interval, Flow represents the motion vector of the target. The motion vector extracted in this step reflects the motion trend of the target in the mine environment.

[0128] S403, according to the motion vector extracted in step S402, the updated target model is obtained by processing through a Kalman filter, wherein the Kalman filter is a recursive algorithm, which estimates the target state through prediction and update steps;

[0129] S404: According to the target model updated in step S403, the accurate motion characteristics of the moving target are obtained through processing by the adaptive tracking algorithm, wherein the adaptive tracking algorithm can dynamically adjust the tracking parameters according to the motion characteristics of the target to adapt to the complex and changeable environment in the mine. The calculation formula is as follows:

[0130]

[0131] In the formula, Track represents the tracking result, f is the adaptive tracking function, Updated Model is the updated target model, and Motion Features is the motion features. This step ensures stable and accurate tracking of moving targets in a mine environment.

[0132] S500. Based on the color distribution characteristics of a specific event, a video tracking algorithm based on background segmentation is applied to obtain a specific event prediction result, wherein the processing process includes calculating the color histogram of the dynamic target, searching for the area that best matches the histogram in the next frame, and dynamically adjusting the size and shape of the target frame; combining the motion characteristics of the moving target and the specific event prediction result, and finally obtaining the dynamic target tracking result through multi-scale fusion processing and update correction.

[0133] It is understandable that in this step, the color distribution characteristics of a specific event usually refer to the unique color change patterns that occur when a specific event occurs, such as a rock fall or a gas leak. For example, a rock fall may generate a large amount of dust, which may appear as a sudden change in color or a significant increase in grayscale value in a specific area in the video; a gas leak may appear as abnormal light or color changes in the video. These changes in color and brightness can serve as important clues for event detection.

[0134] When dealing with such specific events, these color distribution characteristics are usually analyzed and used to identify and track events. This includes: color histogram analysis: by calculating the number of pixels of each color in the image, a color histogram is generated to understand the overall color distribution of the image; color moment calculation: color moment is a statistic that describes the color distribution of an image, including the mean, variance, and skewness of the color; color space conversion: converting the image from RGB color space to HSV or other color space to better highlight specific color features; color clustering: using algorithms such as K-Means to quantify image colors and cluster the colors in the image into several main color categories; color correlation graph: analyzing the correlation between pixel colors and describing the spatial distribution and relationship of colors in the image.

[0135] Through the above methods, the characteristics of specific events can be captured and described from the perspective of color, thereby achieving the identification and tracking of these events.

[0136] It can be understood that step S500 includes S501, S502 and S503, which include:

[0137] S501, calculating a color histogram of a specific event in a mine, and obtaining a color feature description of the target, wherein the color distribution characteristics include color concentration, distribution range, and color uniformity;

[0138] It should be noted that "specific events" refer to various abnormal situations that may occur in the mine, such as ore slides, gas leaks, etc. These events are usually accompanied by color changes, such as: Ore slides: A large amount of coal dust may be generated, which may appear in a specific color, such as black or gray. Gas leaks: Gas leaks may be accompanied by special color changes, such as yellow or orange tones, because gas leaks may react with other substances in the air to produce color changes.

[0139] Among them, color distribution characteristics refer to the distribution pattern of colors in these specific events, such as color concentration, distribution range, color uniformity, etc. The color feature description of the target is the quantification of these color distribution characteristics, which includes: color frequency distribution: that is, the frequency of occurrence of different colors in the image; color spatial distribution: the distribution of colors in the image space, such as whether they are concentrated or dispersed; color brightness and saturation: the brightness and vividness of the color.

[0140] S502, based on the color feature description of the target, searching for the area in the next frame that best matches the calculated color histogram to obtain the potential position of the target, wherein the histogram matching process includes comparing the color histogram of each pixel with the target histogram to find the best matching area;

[0141] It should be noted that the matching process of the best matching area usually involves the following steps: Calculate the target histogram: First calculate the color histogram of the target area (such as the gas leakage area); Search the next frame: Search for the area that best matches the target histogram in the next frame of the video; Compare histograms: Compare the color histogram of each candidate area with the target histogram, usually using a histogram comparison algorithm such as correlation matching, chi-square test, Bayesian classification, etc.; Determine the best matching area: Select the area with the highest matching degree with the target histogram as the best matching area. Among them, the best matching area refers to the area whose color histogram is most similar to the target histogram in the current frame, and this area is most likely to be the location where the target event occurred.

[0142] S503, according to the potential position of the target, the target frame is dynamically adjusted according to the color distribution of the target using the CamShift algorithm, which includes initializing the target frame, using the color histogram to track the target in subsequent frames, iteratively updating the position and size of the target frame to match the color distribution of the target, iteratively updating the new target frame position until the pixel color angle that best matches the target color histogram is obtained, and obtaining a specific event prediction result, which is calculated by the following formula:

[0143]

[0144] In the formula, is the updated target position, For all possible ( x , y ) coordinates to find the coordinate value that maximizes the following expression, I will is the pixel color angle in the current frame that matches the target color histogram, i is a specific color angle in the target color histogram, N is the total number of pixels in the search window, For i From 1 to N All the values ​​of are accumulated, d is the Dirac function, which is used to count the number of pixels at each color angle.

[0145] In this step, the process of dynamically adjusting the target frame according to the color distribution of the target generally involves the following steps: Initialize the target frame: determine the approximate position and size of the target in the initial frame; Color histogram tracking: in subsequent frames, use the color histogram to track the target; CamShift algorithm: use the CamShift algorithm to adjust the target frame. The CamShift algorithm iteratively updates the position and size of the target frame to match the color distribution of the target; Update the target frame: in each iteration, the algorithm calculates a new target frame position based on the pixel color angle that best matches the target color histogram in the current frame; Adapt to target changes: As the target moves and changes, the target frame will dynamically adjust to keep tracking the target. In summary, in this way, the algorithm can continue to track the target and maintain the accuracy of the target frame even if the target moves in the video or its color distribution changes.

[0146] In this step, in combination with specific events in the mine (such as ore slides, gas leaks, etc.), a prediction model (possibly based on machine learning or statistical analysis) is used to predict the occurrence of these events and analyze their impact on the target motion. Then, through the target tracking algorithm, multiple features are usually fused to improve the accuracy and robustness of tracking. For example, color features, fast directional gradient histograms, and local binary pattern features can be combined, and these features can be integrated through linear weighting or other fusion strategies. In terms of scale fusion, since the target may change in scale (i.e., size) in the video, it is necessary to track the target at different scales, which is achieved by constructing a scale filter that can respond to the characteristics of the target at different scales and fuse these responses to estimate the scale and position of the target.

[0147] It should be noted that during the target tracking process, tracking drift may occur due to occlusion, illumination changes or other interference factors. To deal with this situation, a self-correction mechanism can be introduced, such as peak-to-sidelobe ratio check. When an abnormal tracking output is detected, the self-correction mechanism is activated to correct the tracking output, so as to accurately re-track the target. Then, as the target features change, the tracking model also needs to be updated accordingly, which is achieved through online learning or incremental learning, that is, updating the model parameters in each frame or every few frames to adapt to the dynamic changes of the target.

[0148] It can be understood that through the above steps, the final dynamic target tracking result is a continuous time series of the position, scale and motion state of the target in the video sequence, which can be used for real-time monitoring, early warning system or further analysis and processing. In the actual mining vehicle video monitoring system, these tracking results can be used to improve the safety of mine operations, such as preventing accidents such as collisions or landslides by monitoring the motion state of the mine car, or taking timely measures by detecting early signs of specific events.

[0149] Embodiment 2:

[0150] like Figure 2 As shown, this embodiment provides a dynamic target tracking system for vehicle-mounted mining intrinsically safe video monitoring, see Figure 2 The system comprises:

[0151] Acquisition module 701: used to acquire a real-time video stream through a vehicle-mounted camera, wherein the real-time video stream includes a visible light video and an event stream captured by an event camera, wherein the visible light video is a visible light activity scene in a mine, including miner activities, vehicle operation, and the status of other visible objects, and the event stream captured by the event camera includes the location and time information of brightness changes recorded when pixel brightness changes;

[0152] Detection and extraction module 702: used to generate an event stream containing timestamps based on pixel brightness changes in the event stream captured by the event camera, and sort the captured events according to the timestamps to generate an event stream with high temporal resolution; preprocess the real-time video stream, wherein the preprocessing includes noise reduction and illumination enhancement, obtain optimized video frames, use a lightweight deep learning model to analyze the optimized video frames, obtain detection target results, perform fine-grained tracking feature extraction on the detection target results based on the high temporal resolution event stream and visible light video, and obtain the first feature of the surface interest point of the target result, wherein the first feature includes local texture, shape information and motion mode, wherein the detection target result includes preliminary positioning results of miners, vehicle targets and other visible objects;

[0153] Prediction module 703: used to process the moving target in the event stream according to the first feature of the surface interest point through a fast tracking algorithm to obtain the real-time position and movement trend of the moving target, and use the Kalman filter in the TLD target movement algorithm to perform short-term prediction to obtain the short-term movement trajectory of the moving target, and obtain an updated target model through an online learning mechanism;

[0154] Identification module 704: used to obtain the motion state of the moving target by using the motion detection algorithm based on the updated target model and the real-time video frame, wherein the motion detection algorithm processing process includes analyzing the changes of pixels in the image sequence to identify the moving target and extract its features; according to the motion state and the updated target model, the motion features of the moving target are obtained by using the adaptive tracking algorithm;

[0155] Tracking module 705: used to apply the background segmentation-based video tracking algorithm to the color distribution characteristics of a specific event to obtain a specific event prediction result, wherein the processing process includes calculating the color histogram of the dynamic target, searching for the area that best matches the histogram in the next frame, and dynamically adjusting the size and shape of the target frame; combining the motion characteristics of the moving target and the specific event prediction result, and finally obtaining the dynamic target tracking result through multi-scale fusion processing and update correction.

[0156] Specifically, the detection and extraction module 702 includes:

[0157] The first processing unit is used to process the optimized video frames through the YOLOv5-Lite model. The model obtains the preliminary positioning results of miners, vehicle targets and other visible objects by multiplying the input frame with the pre-trained weights and applying the activation function. The calculation formula for target detection by the YOLOv5-Lite model is as follows:

[0158]

[0159] In the formula, bi represents the predicted value of the bounding box, Wi and bi ′ is the weight and bias parameters of the model, s is the sigmoid activation function, x Represents the input feature map;

[0160] The second processing unit is used to obtain the optimization metric of the target candidate area based on the preliminary positioning result through distance intersection and union calculation, and obtain the bounding box of the target candidate area through non-maximum suppression processing, wherein the processing process includes calculating the intersection and union ratio of the predicted box and removing the overlapping box based on a preset threshold, wherein the calculation formula of the non-maximum suppression processing is as follows:

[0161]

[0162] In the formula, B 1 and B 2 represents two bounding boxes, IoU is the intersection over union ratio, and area is the area;

[0163] Extraction unit: It is used to use the scale-invariant feature transformation algorithm to calculate the gradient direction histogram of the local image in the bounding box, extract the features that can still be recognized and matched under the conditions of illumination change and occlusion rotation, and record the features as the first features of the interest points on the target surface.

[0164] Specifically, the extraction unit includes:

[0165] Construction unit: used to construct a differential Gaussian pyramid based on the bounding box using the Gaussian pyramid technology to obtain the first key point position, wherein the first key point position includes the corner points, edges or other significant features of the target in the mine in the scale space, wherein the differential Gaussian pyramid is obtained by calculating the difference between two adjacent layers of the Gaussian pyramid, and the calculation formula is as follows:

[0166] DoG( x , y , s )= G ( x , y , ks )− G ( x , y , s )

[0167] Where DoG is the Gaussian difference function, G ( x , y , s ) is a Gaussian function, s is the standard deviation of the Gaussian function, k is a constant that controls the scale change, x and y is the coordinate of the pixel in the image, ks Compared to the current scale s Large scales are used to construct blurred images of adjacent scales in the Gaussian pyramid;

[0168] Detection unit: used to perform extreme value detection on the scale space of each pixel point at the first key point position according to the differential Gaussian pyramid to obtain the second key point position, wherein each pixel point includes an extreme value point compared with 26 adjacent pixels of the surrounding adjacent scale layers;

[0169] A first calculation unit is used to obtain a unique direction of each second key point position based on the gradient direction around the second key point position by calculating the gradient direction histogram, wherein a 16x16 pixel neighborhood is taken around the second key point position, and the gradient direction and magnitude of each pixel are calculated, and an 8-direction histogram is generated in each 4x4 sub-region to assign a main direction to the second key point position;

[0170] Generation unit: used to generate feature descriptors in a 16x16 pixel neighborhood around the second key point according to the main direction of the position, so as to obtain features that can still be recognized and matched under conditions of illumination changes and occlusion rotation, and recorded as the first feature of the interest point on the target surface. The descriptor is constructed by calculating the histogram of the gradient direction in each 4x4 sub-region, and finally forming a 128-dimensional feature vector.

[0171] Specifically, the detection and extraction module 702 includes:

[0172] Recording unit: It is used to utilize the characteristics of the event camera to record the position and time information of the brightness change when the pixel brightness changes, forming an event stream with high time resolution. The calculation formula of the sorting process is as follows:

[0173]

[0174] In the formula, Sorted represents a sorted event stream, Captured represents the raw event stream captured from the event camera, sort is a sorting function, l For each event's timestamp, x Represents a single event in an event stream, x . timestamp Indicates an event x The timestamp of the function. The key is a parameter used to specify a function.

[0175] Specifically, the prediction module 703 includes:

[0176] The second calculation unit is used to calculate the color histogram of the moving target area based on the first feature of the surface interest point, wherein the calculation formula is as follows:

[0177]

[0178] In the formula, H is the color histogram of the target area, N is the total number of pixels in the area, i is the color angle, d is the Dirac delta function used to construct the color histogram, i is the current number of items in the summation process, I will For the i The color angle of each pixel;

[0179] Matching update unit: used to calculate the area similar to the color histogram of the moving target in the search window to determine the current position of the moving target, obtain the matching result, and update the position of the moving target according to the matching result. The calculation formula is as follows:

[0180]

[0181] In the formula, S is the similarity within the search window, Hw is the color histogram within the search window, Hello is the color histogram of the target model, x , yare pixel coordinates within the image or search window, where x and y Traverse each pixel in the search window;

[0182] Analysis unit: used to analyze the movement trend of the moving target based on the updated position of the moving target, wherein the movement trend includes the movement direction and speed.

[0183] Specifically, the prediction module 703 includes:

[0184] Estimation unit: used to perform optimal estimation calculation of the target state based on the real-time position and motion trend of the moving target after processing by the Kalman filter, and obtain the short-term motion trajectory of the moving target. The calculation process includes a prediction step and an update step. The prediction step is to estimate the position and speed of the moving target in the next frame, and the update step is to adjust the prediction according to the new observation data;

[0185] Training unit: used to collect positive and negative samples according to the short-term motion trajectory of the moving target to form a training data set for model updating, where the positive samples come from the detection results of the tracked target, and the negative samples come from the area identified as non-target;

[0186] Update adaptation unit: used to apply the online stochastic gradient descent algorithm to process the training data set, update the model parameters online, adapt to the changes in the target appearance, and thus obtain an updated target model that reflects the current state of the moving target.

[0187] Specifically, the tracking module 705 includes:

[0188] Description unit: used to calculate the color histogram of a specific event in the mine and obtain the color feature description of the target, where the color distribution characteristics include color concentration, distribution range and color uniformity;

[0189] Search and match unit: used to search for the area in the next frame that best matches the calculated color histogram based on the color feature description of the target, and obtain the potential position of the target. The histogram matching process includes comparing the color histogram of each pixel with the target histogram to find the best matching area;

[0190] Iteration unit: It is used to dynamically adjust the target frame according to the target's potential position and the color distribution of the target using the CamShift algorithm, which includes initializing the target frame. In subsequent frames, the color histogram is used to track the target. The position and size of the target frame are iteratively updated to match the color distribution of the target. The target frame position is iteratively updated until the pixel color angle that best matches the target color histogram is obtained. The calculation formula is as follows:

[0191]

[0192] In the formula, is the updated target position, For all possible ( x , y ) coordinates to find the coordinate value that maximizes the following expression, I will is the pixel color angle in the current frame that matches the target color histogram, i is a specific color angle in the target color histogram, N is the total number of pixels in the search window, For i From 1 to N All the values ​​of are accumulated, d is the Dirac function, which is used to count the number of pixels at each color angle.

[0193] It should be noted that, regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0194] Embodiment 3:

[0195] Corresponding to the above method embodiments, this embodiment also provides a dynamic target tracking device for vehicle-mounted intrinsically safe video monitoring for mines. The dynamic target tracking device for vehicle-mounted intrinsically safe video monitoring for mines described below and the dynamic target tracking method for vehicle-mounted intrinsically safe video monitoring for mines described above can be referenced to each other.

[0196] Figure 3 FIG. 8 is a block diagram of a dynamic target tracking device 800 for vehicle-mounted intrinsically safe video monitoring for mining according to an exemplary embodiment. Figure 3 As shown, the vehicle-mounted intrinsically safe video monitoring dynamic target tracking device 800 for mining includes: a processor 801 and a memory 802. The vehicle-mounted intrinsically safe video monitoring dynamic target tracking device 800 for mining also includes one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0197] The processor 801 is used to control the overall operation of the vehicle-mounted intrinsically safe video monitoring dynamic target tracking device 800 to complete all or part of the steps in the above-mentioned vehicle-mounted intrinsically safe video monitoring dynamic target tracking method. The memory 802 is used to store various types of data to support the operation of the vehicle-mounted intrinsically safe video monitoring dynamic target tracking device 800, which may include, for example, instructions for any application or method operating on the vehicle-mounted intrinsically safe video monitoring dynamic target tracking device 800, as well as application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 802 or sent via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 804 provides an interface between the processor 801 and other interface modules, and the above-mentioned other interface modules can be a keyboard, a mouse or buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 805 is used for wired or wireless communication between the dynamic target tracking device 800 of the vehicle-mounted mining intrinsically safe video monitoring and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 805 can include: a Wi-Fi module, a Bluetooth module or an NFC module.

[0198] In an exemplary embodiment, the dynamic target tracking device 800 for vehicle-mounted intrinsically safe video monitoring for mining can be implemented by one or more application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned dynamic target tracking method for vehicle-mounted intrinsically safe video monitoring for mining.

[0199] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned method for dynamic target tracking of vehicle-mounted intrinsically safe video monitoring for mining are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 802 including program instructions, and the above-mentioned program instructions may be executed by the processor 801 of the dynamic target tracking device 800 for vehicle-mounted intrinsically safe video monitoring for mining to complete the above-mentioned method for dynamic target tracking of vehicle-mounted intrinsically safe video monitoring for mining.

[0200] Embodiment 4:

[0201] Corresponding to the above method embodiment, a readable storage medium is also provided in this embodiment. The readable storage medium described below and the dynamic target tracking method for vehicle-mounted intrinsically safe video monitoring for mining described above can refer to each other.

[0202] A computer program is stored on the readable storage medium, and when the computer program is executed by the processor, the steps of the dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring in the above method embodiment are implemented.

[0203] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0204] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0205] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A dynamic target tracking method for vehicle-mounted intrinsically safe video monitoring for mining, characterized in that: include: S100, a real-time video stream obtained by a vehicle-mounted camera, wherein the real-time video stream includes a visible light video and an event stream captured by an event camera, wherein the visible light video is a visible light activity scene in a mine, including miner activities, vehicle operation, and the status of other visible objects, and the event stream captured by the event camera includes location and time information of brightness changes recorded when pixel brightness changes; S200, based on the pixel brightness changes in the event stream captured by the event camera, generate an event stream containing a timestamp, and sort the captured events according to the timestamps to generate an event stream with high temporal resolution; preprocess the real-time video stream, wherein the preprocessing includes noise reduction and illumination enhancement, obtain optimized video frames, use a lightweight deep learning model to analyze the optimized video frames, obtain detection target results, perform fine-grained tracking feature extraction on the detection target results according to the high temporal resolution event stream and visible light video, and obtain the first feature of the surface interest point of the target result, wherein the first feature includes local texture, shape information and motion mode, wherein the detection target result includes preliminary positioning results of miners, vehicle targets and other visible objects; S300, according to the first feature of the surface interest point, the moving target in the event stream is processed by a fast tracking algorithm to obtain the real-time position and movement trend of the moving target, and the Kalman filter in the TLD target movement algorithm is used for short-term prediction to obtain the short-term movement trajectory of the moving target, and an updated target model is obtained through an online learning mechanism; S400, based on the updated target model and the real-time video frame, a motion detection algorithm is used to obtain the motion state of the moving target, wherein the motion detection algorithm processing process includes analyzing the changes of pixels in the image sequence to identify the moving target and extract its features; according to the motion state and the updated target model, the motion features of the moving target are obtained through the adaptive tracking algorithm processing; Step S400 further includes the following steps S401 to S404: S401. According to the updated target model and the real-time video frame, preliminary motion information of the moving target is obtained through processing by the inter-frame difference algorithm. The inter-frame difference algorithm detects the moving target by calculating the pixel-level difference between consecutive video frames: In the formula, It is the current frame, is the previous frame, and Motion is the motion information. This step uses the pixel changes caused by the target motion in the mine environment to provide a basis for subsequent feature extraction; S402: According to the motion information obtained in step S401, the motion vector and speed of the moving target are obtained through processing based on the optical flow algorithm. The optical flow algorithm estimates the motion characteristics of the target by analyzing its displacement in consecutive frames: In the formula, is the position of the target in the current frame, is the position of the previous frame, Δ t is the time interval, Flow represents the motion vector of the target. The motion vector extracted in this step reflects the motion trend of the target in the mine environment; S403, according to the motion vector extracted in step S402, the updated target model is obtained by processing through a Kalman filter, wherein the Kalman filter is a recursive algorithm, which estimates the target state through prediction and update steps; S404: According to the target model updated in step S403, the accurate motion characteristics of the moving target are obtained through processing by the adaptive tracking algorithm, wherein the adaptive tracking algorithm can dynamically adjust the tracking parameters according to the motion characteristics of the target to adapt to the complex and changeable environment in the mine. The calculation formula is as follows: In the formula, Track represents the tracking result, f is the adaptive tracking function, Updated Model is the updated target model, and Motion Features is the motion features. This step ensures stable and accurate tracking of moving targets in a mine environment; S500. Based on the color distribution characteristics of the set event, a video tracking algorithm based on background segmentation is applied to obtain the set event prediction result, wherein the processing process includes calculating the color histogram of the dynamic target, searching for the area that best matches the histogram in the next frame, and dynamically adjusting the size and shape of the target frame; combining the motion characteristics of the moving target and the set event prediction result, and finally obtaining the dynamic target tracking result through multi-scale fusion processing and update correction.

2. The dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring according to claim 1 is characterized in that: The pixel brightness changes in the event stream captured by the event camera are used to generate an event stream containing a timestamp, and the captured events are sorted according to the timestamps to generate an event stream with a high time resolution, including: By utilizing the characteristics of the event camera, the position and time information of the brightness change when the pixel brightness changes are recorded to form an event stream with high time resolution. The calculation formula of the sorting process is as follows: In the formula, Esorted represents a sorted event stream, Ecaptured represents the raw event stream captured from the event camera, sort is a sorting function, λ For each event's timestamp, x Represents a single event in an event stream, x . timestamp Indicates an event x The timestamp of the function. The key is a parameter used to specify a function.

3. The dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring according to claim 1 is characterized in that: The optimized video frames are analyzed using a lightweight deep learning model to obtain the detection target result, and the detection target result is subjected to fine-grained tracking feature extraction based on the high-temporal-resolution event stream and visible light video to obtain the first feature of the surface interest point of the target result, which includes: The optimized video frames are processed by the YOLOv5-Lite model, which obtains preliminary positioning results of miners, vehicle targets, and other visible objects by multiplying the input frame with the pre-trained weights and applying an activation function. The calculation formula for target detection by the YOLOv5-Lite model is as follows: In the formula, bi represents the predicted value of the bounding box, Wi and bi ′ is the weight and bias parameters of the model, σ is the sigmoid activation function, x Represents the input feature map; Based on the preliminary positioning results, the optimized metric of the target candidate area is obtained through distance intersection and union calculation, and the bounding box of the target candidate area is obtained through non-maximum suppression processing. The processing process includes calculating the intersection and union of the predicted box and removing the overlapping boxes based on the preset threshold. The calculation formula of non-maximum suppression processing is as follows: In the formula, B 1 and B 2 represents two bounding boxes, IoU is the intersection over union ratio, and area is the area; By using the scale-invariant feature transformation algorithm and calculating the gradient direction histogram of the local image in the bounding box, the features that can still be recognized and matched under the conditions of illumination change and occlusion rotation are extracted, and the features are recorded as the first features of the interest points on the target surface.

4. The dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring according to claim 3 is characterized in that: The scale-invariant feature transformation algorithm is used to calculate the gradient direction histogram of the local image in the bounding box, extract features that can still be recognized and matched under the conditions of illumination change and occlusion rotation, and record the features as the first features of the interest points on the target surface, including: Based on the bounding box, a differential Gaussian pyramid is constructed using the Gaussian pyramid technology to obtain the first key point position, where the first key point position includes the corner points, edges or other significant features of the target in the mine in the scale space. The differential Gaussian pyramid is obtained by calculating the difference between two adjacent layers of the Gaussian pyramid. The calculation formula is as follows: Where DoG is the Gaussian difference function, G ( x , y , σ ) is a Gaussian function, σ is the standard deviation of the Gaussian function, k is a constant that controls the scale change, x and y is the coordinate of the pixel in the image, kσ Compared to the current scale σ Large scales are used to construct blurred images of adjacent scales in the Gaussian pyramid; According to the differential Gaussian pyramid, the scale space of each pixel at the first key point position is detected to obtain the second key point position, wherein each pixel includes an extreme point compared with 26 adjacent pixels in the surrounding adjacent scale layers; Based on the gradient direction around the second key point position, a unique direction of each second key point position is obtained by calculating the gradient direction histogram, wherein a 16x16 pixel neighborhood is taken around the second key point position, and the gradient direction and magnitude of each pixel are calculated, and an 8-direction histogram is generated in each 4x4 sub-region to assign a main direction to the second key point position; According to the main direction of the second key point position, a feature descriptor is generated in the 16x16 pixel neighborhood around it, so as to obtain features that can still be recognized and matched under conditions of illumination changes and occlusion rotation, and recorded as the first feature of the interest point on the target surface. The descriptor is constructed by calculating the histogram of the gradient direction in each 4x4 sub-region, and finally forming a 128-dimensional feature vector.

5. The dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring according to claim 1 is characterized in that: According to the first feature of the surface interest point, the moving target in the event stream is processed by a fast tracking algorithm to obtain the real-time position and movement trend of the moving target, including: Based on the first feature of the surface interest point, the color histogram of the moving target area is calculated, where the calculation formula is as follows: In the formula, H is the color histogram of the target area, N is the total number of pixels in the area, θ is the color angle, δ is the Dirac delta function used to construct the color histogram, i is the current number of items in the summation process, θi For the i The color angle of each pixel; Calculate the area similar to the color histogram of the moving target in the search window to determine the current position of the moving target, obtain the matching result, and update the position of the moving target based on the matching result. The calculation formula is as follows: In the formula, S is the similarity within the search window, H is the color histogram within the search window, Hi is the color histogram of the target model, x , y are pixel coordinates within the image or search window, where x and y Traverse each pixel in the search window; Based on the updated position of the moving target, the movement trend of the moving target is analyzed, wherein the movement trend includes movement direction and speed.

6. The dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring according to claim 1 is characterized in that: The Kalman filter in the TLD target movement algorithm is used for short-term prediction to obtain the short-term motion trajectory of the moving target, and the updated target model is obtained through the online learning mechanism, which includes: Based on the real-time position and motion trend of the moving target, the target state is optimally estimated and calculated after being processed by the Kalman filter to obtain the short-term motion trajectory of the moving target. The calculation process includes a prediction step and an update step. The prediction step is to estimate the position and speed of the moving target in the next frame, and the update step is to adjust the prediction according to the new observation data. According to the short-term motion trajectory of the moving target, positive and negative samples are collected to form a training data set for model updating, where positive samples come from the detection results of the tracked target, and negative samples come from the area identified as non-target; The online stochastic gradient descent algorithm is applied to process the training data set, and the model parameters are updated online to adapt to the changes in the target appearance, thereby obtaining an updated target model that reflects the current state of the moving target.

7. The dynamic target tracking method for vehicle-mounted mining intrinsically safe video monitoring according to claim 1 is characterized in that: The color distribution characteristics of the set event are processed by applying a video tracking algorithm based on background segmentation to obtain a set event prediction result, wherein the processing process includes calculating the color histogram of the dynamic target, searching for the area that best matches the histogram in the next frame, and dynamically adjusting the size and shape of the target frame, including: Calculate the color histogram of the set event in the mine to obtain the color feature description of the target, where the color distribution characteristics include color concentration, distribution range and color uniformity; Based on the color feature description of the target, search for the area in the next frame that best matches the calculated color histogram to obtain the potential position of the target. The histogram matching process includes comparing the color histogram of each pixel with the target histogram to find the best matching area. According to the potential position of the target, the CamShift algorithm is used to dynamically adjust the target frame according to the color distribution of the target, which includes initializing the target frame. In subsequent frames, the color histogram is used to track the target. The position and size of the target frame are iteratively updated to match the color distribution of the target. The target frame position is iteratively updated until the pixel color angle that best matches the target color histogram is obtained. The set event prediction result is obtained. The calculation formula is as follows: In the formula, is the updated target position, For all possible ( x , y ) coordinates to find the coordinate value that maximizes the following expression, θi is the pixel color angle in the current frame that matches the target color histogram, θ is a set color angle in the target color histogram, N is the total number of pixels in the search window, For i From 1 to N All the values ​​of are accumulated, δ is the Dirac function, which is used to count the number of pixels at each color angle.

8. A dynamic target tracking system for vehicle-mounted intrinsically safe video monitoring for mining, based on the dynamic target tracking method for vehicle-mounted intrinsically safe video monitoring for mining according to claim 1, characterized in that: include: Acquisition module: used to acquire real-time video streams through vehicle-mounted cameras, where the real-time video streams include visible light video and event streams captured by event cameras, where the visible light video is a visible light activity scene in the mine, including miner activities, vehicle operation, and the status of other visible objects, and the event stream captured by the event camera includes the location and time information of brightness changes recorded when pixel brightness changes; Detection and extraction module: used to generate an event stream containing timestamps based on the pixel brightness changes in the event stream captured by the event camera, and sort the captured events according to timestamps to generate an event stream with high time resolution; Preprocess the real-time video stream, including noise reduction and illumination enhancement, to obtain optimized video frames, analyze the optimized video frames using a lightweight deep learning model, and obtain detection target results. Perform fine-grained tracking feature extraction on the detection target results based on high-temporal-resolution event streams and visible light videos to obtain the first features of the surface interest points of the target results, wherein the first features include local texture, shape information, and motion patterns. The detection target results include preliminary positioning results of miners, vehicle targets, and other visible objects; Prediction module: It is used to process the moving targets in the event stream according to the first feature of the surface interest point through the fast tracking algorithm to obtain the real-time position and movement trend of the moving target, and use the Kalman filter in the TLD target movement algorithm to make short-term predictions to obtain the short-term movement trajectory of the moving target. After the online learning mechanism, the updated target model is obtained; Identification module: used to obtain the motion state of the moving target by using the motion detection algorithm based on the updated target model and real-time video frames. The motion detection algorithm processing process includes analyzing the changes in pixels in the image sequence to identify the moving target and extract its features; according to the motion state and the updated target model, the motion features of the moving target are obtained by processing with the adaptive tracking algorithm; Tracking module: It is used to apply the video tracking algorithm based on background segmentation to the color distribution characteristics of the set event to obtain the set event prediction result. The processing process includes calculating the color histogram of the dynamic target, searching for the area that best matches this histogram in the next frame, and dynamically adjusting the size and shape of the target box; combining the motion characteristics of the moving target and the set event prediction result, through multi-scale fusion processing and update correction, finally obtaining the dynamic target tracking result.

9. The dynamic target tracking system for vehicle-mounted mining intrinsically safe video monitoring according to claim 8 is characterized in that: The detection and extraction module includes: The first processing unit is used to process the optimized video frames through the YOLOv5-Lite model. The model obtains the preliminary positioning results of miners, vehicle targets and other visible objects by multiplying the input frame with the pre-trained weights and applying the activation function. The calculation formula for target detection by the YOLOv5-Lite model is as follows: In the formula, bi represents the predicted value of the bounding box, Wi and bi ′ is the weight and bias parameters of the model, σ is the sigmoid activation function, x Represents the input feature map; The second processing unit is used to obtain the optimization metric of the target candidate area based on the preliminary positioning result through distance intersection and union calculation, and obtain the bounding box of the target candidate area through non-maximum suppression processing, wherein the processing process includes calculating the intersection and union ratio of the predicted box and removing the overlapping box based on a preset threshold, wherein the calculation formula of the non-maximum suppression processing is as follows: In the formula, B 1 and B 2 represents two bounding boxes, IoU is the intersection over union ratio, and area is the area; Extraction unit: It is used to use the scale-invariant feature transformation algorithm to calculate the gradient direction histogram of the local image in the bounding box, extract the features that can still be recognized and matched under the conditions of illumination change and occlusion rotation, and record the features as the first features of the interest points on the target surface.

10. The dynamic target tracking system for vehicle-mounted mining intrinsically safe video monitoring according to claim 9 is characterized in that: The extraction unit comprises: Construction unit: used to construct a differential Gaussian pyramid based on the bounding box using the Gaussian pyramid technology to obtain the first key point position, wherein the first key point position includes the corner points, edges or other significant features of the target in the mine in the scale space, wherein the differential Gaussian pyramid is obtained by calculating the difference between two adjacent layers of the Gaussian pyramid, and the calculation formula is as follows: Where DoG is the Gaussian difference function, G ( x , y , σ ) is a Gaussian function, σ is the standard deviation of the Gaussian function, k is a constant that controls the scale change, x and y is the coordinate of the pixel in the image, kσ Compared to the current scale σ Large scales are used to construct blurred images of adjacent scales in the Gaussian pyramid; Detection unit: used to perform extreme value detection on the scale space of each pixel point at the first key point position according to the differential Gaussian pyramid to obtain the second key point position, wherein each pixel point includes an extreme value point compared with 26 adjacent pixels of the surrounding adjacent scale layers; A first calculation unit is used to obtain a unique direction of each second key point position based on the gradient direction around the second key point position by calculating the gradient direction histogram, wherein a 16x16 pixel neighborhood is taken around the second key point position, and the gradient direction and magnitude of each pixel are calculated, and an 8-direction histogram is generated in each 4x4 sub-region to assign a main direction to the second key point position; Generation unit: used to generate feature descriptors in a 16x16 pixel neighborhood around the second key point according to the main direction of the position, so as to obtain features that can still be recognized and matched under conditions of illumination changes and occlusion rotation, and recorded as the first feature of the interest point on the target surface. The descriptor is constructed by calculating the histogram of the gradient direction in each 4x4 sub-region, and finally forming a 128-dimensional feature vector.

Images