Omnibearing safe driving early warning system for mining rubber-tyred vehicle

Abstract

The invention relates to the technical field of driving early warning, in particular to a mining rubber-tyred vehicle omnibearing safe driving early warning system which comprises an image reflection monitoring module, a brightness gradient recognition module, a path deviation extraction module, a control trend judgment module and a risk linkage early warning module. According to the method, reflective characteristics and edge contour information are comprehensively analyzed on the basis of image data, brightness gradient distribution evolution trend identification is introduced to distinguish static and moving light source interference signals, and continuous tracking of a pixel displacement direction and a lane center deviation trend is combined to determine a lane center deviation trend. The behavior consistency judgment is carried out with the assistance of a reverse verification mode of a steering angular velocity change trend and a path offset relationship, so that dynamic coupling identification among obstacle stability, an offset trend and a driving control behavior can be realized, and the early perception capability of potential risks in a complex working condition is effectively improved; and the joint early warning judgment of the actual driving deviation and the obstacle risk in the running environment of the mining rubber-tyred vehicle is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of driving warning, in particular to a mine rubber-tyred vehicle all-around safe driving warning system. BACKGROUND

[0002] The technical field of driving warning relates to the related technology of protecting driving safety by monitoring and prompting potential risk factors in real time during vehicle operation, including driving state perception, environment information acquisition, abnormal state judgment and driver behavior warning, which is widely applied to the safe operation scenes of different types of vehicles such as urban buses, engineering machinery, dangerous goods transport vehicles and mine vehicles. Among them, the mine rubber-tyred vehicle all-around safe driving warning system refers to the warning system built on mine transport vehicles to deal with driving safety problems in harsh environments, which usually detects obstacles in close range by setting ultrasonic probes, judges target objects by using infrared sensors, assists drivers in observing non-direct field of view areas by using video acquisition heads, and realizes preliminary processing and prompting of the above problems by combining driver fatigue determination standards based on fixed logic judgment.

[0003] In the existing driving process of rubber-tyred vehicles, obstacles are primarily identified by relying on single sensing methods such as ultrasonic, infrared and video images, and fatigue is determined based on fixed logic thresholds, which cannot dynamically analyze the interaction logic between environmental reflection characteristics and vehicle deviation trends. When there are high-reflective objects or non-fixed light source disturbances, false triggering problems are prone to occur. Moreover, in low-illumination complex conditions such as underground tunnels, the observation accuracy of video acquisition in non-direct field of view coverage areas is limited, causing obstacle identification lag or deviation, making it difficult to timely perceive the linkage risks between path deviation behavior and abnormal behavior, and affecting the comprehensive judgment of potential high-risk states. SUMMARY

[0004] The purpose of the present application is to solve the problems existing in the prior art and to provide a mine rubber-tyred vehicle all-around safe driving warning system.

[0005] In order to achieve the above purpose, the present application adopts the following technical scheme: a mine rubber-tyred vehicle all-around safe driving warning system comprises:

[0006] The image reflection monitoring module obtains the image frame data of the front camera of the mine rubber-tyred vehicle, extracts the average reflection intensity of the center pixels and the average reflection intensity of the edge pixels in the detection area, compares whether the difference value amplitude of the two matches the retroreflective coefficient, confirms the reflective obstacle outline boundary, and generates a static obstacle boundary identification result;

[0007] The brightness gradient identification module extracts the brightness level distribution values of the corresponding areas in the continuous three frames based on the static obstacle boundary identification result, detects and excludes the moving light source interference signal through gradient change, and generates a fixed obstacle stability analysis result.

[0008] The path offset extraction module obtains the pixel coordinates of the left and right boundary lines in the image frame corresponding to the time label of the fixed obstacle stability analysis result, calculates the displacement direction of the horizontal center point of the image relative to the center position of the lane, extracts the offset direction marker and determines whether continuous offset has occurred, and generates a path center line offset trend record.

[0009] The control trend judgment module obtains the steering wheel angular velocity threshold data for the corresponding time period based on the path centerline offset trend record, determines whether the steering wheel return trend has an opposite correction relationship with the path offset direction, and generates a control correction consistency recognition result.

[0010] As a further aspect of the present invention, the static obstacle boundary recognition result includes the location of reflective obstacle boundaries, distribution data of abnormal reflective intensity areas, and edge contour feature curvature; the fixed obstacle stability analysis result includes records of brightness direction gradient change trends, gradient change continuity indicators, and light source interference elimination information; the path centerline offset trend record includes the displacement direction of the image center point, records of path centerline change trends, and offset duration; and the control correction consistency recognition result includes the relationship between the steering wheel return direction and the offset direction, the return action delay status, and the degree to which the control behavior deviates from expectations.

[0011] As a further aspect of the present invention, the retroreflection coefficient is specifically the minimum brightness requirement for reflective markings, with a mining standard of ≥300 mcd·lx. -1 ·m -2 The reflective barrier contour boundary is determined using Canny edge detection; the gradient change detection is performed using directional gradient histogram feature matching.

[0012] As a further aspect of the present invention, the image reflection monitoring module includes:

[0013] The image frame acquisition submodule acquires image frame data from the front-facing camera of the mining rubber-tired vehicle in the underground roadway environment, extracts the center pixels and edge pixels within the detection area, and obtains the pixel set of the image detection area;

[0014] The reflection intensity calculation submodule calculates the average reflection intensity of the pixels in the central region and the average reflection intensity of the pixels in the edge region based on the pixel set of the image detection area, determines whether the difference between the two meets the minimum brightness requirement threshold of the reflective mark, calculates the composite intensity value of the brightness difference, filters out the image area features that meet the reflective intensity benchmark conditions, and generates the regional reflection difference measurement result.

[0015] Based on the regional reflection difference measurement results, the boundary recognition submodule performs Gaussian filtering on the image frame, extracts the edge pixel gradient values, constructs an edge intensity map, determines whether the closed strong edge meets the lane departure warning standard conditions, obtains the edge morphology of the continuous region, and generates static obstacle boundary recognition results.

[0016] As a further aspect of the present invention, the brightness gradient recognition module includes:

[0017] The brightness distribution extraction submodule, based on the static obstacle boundary recognition results, locates the set of pixel coordinates of the same region in three consecutive frames of images, collects the set of pixel reflection intensity values ​​in the region in each frame, calculates the average reflection intensity value of the corresponding region in each frame, establishes the inter-frame brightness sequence distribution of the region, and generates the inter-frame brightness sequence of the region.

[0018] The gradient direction detection submodule, based on the inter-frame brightness sequence of the region, obtains the change direction between the first and second frames and the change direction between the second and third frames according to the average reflection intensity value of adjacent frames in the three frames. It judges the two change directions. If the two directions are consistent, it is marked as a unidirectional change sequence; otherwise, it is marked as a fluctuating sequence, and the brightness change direction consistency marking result is obtained.

[0019] The light source interference elimination submodule obtains all regions marked as fluctuation sequences based on the brightness change direction consistency marking results, calculates the ratio of the brightness change amplitude between frames, eliminates regions with a change ratio greater than the set moving light source disturbance threshold, retains unidirectional slowly changing regions, and generates fixed obstacle stability analysis results.

[0020] As a further aspect of the present invention, the path offset extraction module includes:

[0021] The boundary coordinate extraction submodule acquires image frames with time labels consistent with the fixed obstacle stability analysis results, extracts the abscissa and ordinate sequences of the left and right boundary line pixels from the start point to the end point, calculates the abscissa values ​​of the intersection points of the left and right boundary lines with the bottom row of the image on the bottom cross section of the image, uses the abscissa value of the center pixel of the corresponding row image width as the reference center point, calculates the distance difference between the center point and the intersection point of the two boundary lines, uses the sign of the difference as the displacement direction mark, and generates a record of the relative displacement direction of the lane.

[0022] The centerline offset determination submodule obtains the relative displacement direction of the lane center at consecutive image positions based on the lane relative displacement direction record, calculates the difference sequence between the relative displacement directions of each frame in turn, and performs a continuous direction determination based on whether the absolute value of the direction change is continuously non-zero and has the same sign, marks the continuous offset state, and obtains the path continuous offset determination result.

[0023] The path trend recording submodule extracts the sequence of horizontal coordinate displacement values ​​of the lane center point in each consecutive frame based on the path continuous offset determination result, records the positions of the maximum and minimum values ​​in the sequence, calculates the slope of the trend line of the sequence, and if the slope is greater than a set threshold, it is determined to be a state of significant change in path trend, and generates a path center line offset trend record.

[0024] As a further aspect of the present invention, the manipulation trend judgment module includes:

[0025] The angular velocity threshold extraction submodule records the corresponding time period according to the offset trend of the path centerline, obtains the original angular velocity sampling value sequence of the vehicle steering wheel in the corresponding time interval, filters all effective angular velocity data points whose absolute value is not less than the set threshold, calculates the average value and records the maximum value, and generates effective angular velocity amplitude interval data.

[0026] The steering wheel angular velocity trend judgment submodule, based on the effective angular velocity amplitude range data, compares the offset direction information recorded in the path centerline offset trend record to determine whether the direction of the time point corresponding to the maximum value in the steering wheel angular velocity sequence is opposite to the offset direction, and determines it as a delayed correction or no correction action, and obtains the steering wheel angular velocity trend opposite relationship judgment result.

[0027] The control consistency identification submodule determines the result based on the opposite direction return relationship, marks the time period without correction or with delayed correction as the deviation control behavior period, extracts the start frame and end frame number in each deviation control behavior period, establishes the correction delay amount in the segment as the frame difference between the offset trend occurrence point and the angular velocity peak point, calculates the delay duration and combines it with the maximum angular velocity amplitude to generate the control correction consistency identification result.

[0028] As a further aspect of the present invention, the system further includes:

[0029] The risk linkage early warning module determines whether the current obstacle area is accompanied by uncorrected path behavior based on the control correction consistency recognition result and the fixed obstacle stability analysis result. If both conditions are met, it is marked as a high-risk state, and outputs the sound and light module trigger command and the current image frame early warning label data to generate all-round safe driving early warning information for mining rubber-tired vehicles.

[0030] As a further aspect of the present invention, the all-round safe driving warning information for the mining rubber-tired vehicle includes high-risk status marking records, sound and light module trigger commands, and image frame warning annotation data.

[0031] As a further aspect of the present invention, the risk linkage early warning module includes:

[0032] The joint state determination submodule matches the frame number based on the time tag in the operation correction consistency identification result and the fixed obstacle stability analysis result, extracts whether the delay correction or no correction state and the obstacle area stability boundary identification result exist simultaneously in the same image frame number, marks it as a high-risk frame, and generates a high-risk synchronization state record.

[0033] The warning instruction output submodule, based on the high-risk synchronization status record, triggers the sound and light output control module according to the time point corresponding to each frame number, writes the signal trigger instruction and the instruction type, constructs a frame number-signal lookup dictionary, and obtains the sound and light signal trigger instruction data.

[0034] The image annotation generation submodule reads the original pixel matrix of the image frame according to the image frame number marked in the audio-visual signal trigger command data, overlays the image annotation information, and generates image frame warning annotation data by marking the area as the outer bounding box of the obstacle boundary.

[0035] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0036] This invention comprehensively analyzes reflective features and edge contour information based on image data, introduces brightness gradient distribution evolution trend recognition to distinguish interference signals from stationary and moving light sources, combines continuous tracking of pixel displacement direction and lane center offset trend, and supplements it with a reverse verification method of steering angular velocity change trend and path offset relationship to determine behavior consistency. This enables dynamic coupling recognition of obstacle stability, offset trend and driving control behavior, effectively improves the early perception capability of potential risks in complex working conditions, enhances the discrimination accuracy under non-ideal field of vision and dynamic lighting conditions, avoids false alarms and missed alarms caused by visual interference, and ensures joint early warning judgment of actual driving deviation and obstacle risk in the operating environment of mining rubber-tired vehicles. Attached Figure Description

[0037] Figure 1 This is a system flowchart of the present invention;

[0038] Figure 2 This is a system module diagram of the present invention;

[0039] Figure 3 This is a flowchart of the image reflection monitoring module of the present invention;

[0040] Figure 4 This is a flowchart of the brightness gradient recognition module of the present invention;

[0041] Figure 5 This is a flowchart of the path offset extraction module of the present invention;

[0042] Figure 6 This is a flowchart of the trend judgment module of the present invention;

[0043] Figure 7 This is a flowchart of the risk linkage early warning module of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0045] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0046] Please see Figure 1 and Figure 2 A comprehensive safety driving warning system for mining rubber-tired vehicles includes:

[0047] The image reflection monitoring module acquires image frame data from the front-facing camera of the mining rubber-tired vehicle in the underground roadway environment, extracts the average reflection intensity of the center pixel and the average reflection intensity of the edge pixel within the detection area, and compares whether the difference between the two matches the retroreflection coefficient (minimum brightness requirement for reflective markings, mining standard ≥300mcd·lx). -1 ·m -2 The system confirms whether there are reflective obstacle outlines within the area (using the Canny edge detection algorithm, Gaussian filtering σ=1.5, high-low threshold ratio 1:3, and executing lane departure warning standards), and generates static obstacle boundary identification results.

[0048] The brightness gradient recognition module extracts the brightness level distribution values ​​of the corresponding region in three consecutive frames based on the static obstacle boundary recognition results. It compares whether the brightness gradient direction change between frames remains unidirectional. It determines whether the gradient change is continuously increasing or decreasing by gradient change detection (using directional gradient histogram feature matching). It eliminates interference signals from moving light sources and generates the stability analysis results of fixed obstacles.

[0049] The path offset extraction module obtains the pixel coordinates of the left and right boundary lines in the image frame corresponding to the time label of the fixed obstacle stability analysis result, calculates the displacement direction of the horizontal center point of the image relative to the center position of the lane, extracts the current offset direction marker, tracks the change direction of the path center line at a fixed frame rate, determines whether continuous offset has occurred, and generates a path center line offset trend record.

[0050] The control trend judgment module obtains the steering angular velocity threshold (compliant with ISO19365:2016 setting ≥50° / s) data recorded by the steering wheel angle sensor for the corresponding time period based on the path centerline offset trend record. It then determines whether the steering wheel return trend has an opposite correction relationship with the path offset direction. If the offset trend does not reverse or there is a delayed correction, it is determined that the current control behavior has a deviation from the expected problem, and a control correction consistency recognition result is generated.

[0051] The risk linkage early warning module determines whether the current obstacle area is accompanied by uncorrected path behavior based on the control correction consistency identification result and the fixed obstacle stability analysis result. If both conditions are met, it is marked as a high-risk state, and outputs the sound and light module trigger command and the current image frame early warning label data to generate all-round safe driving early warning information for mining rubber-tired vehicles.

[0052] The static obstacle boundary recognition results include the location of reflective obstacle boundaries, distribution data of abnormal reflective intensity areas, and edge contour feature curvature. The fixed obstacle stability analysis results include records of brightness direction gradient change trends, gradient change continuity indicators, and light source interference elimination information. The path centerline offset trend records include the displacement direction of the image center point, records of path centerline change trends, and offset duration. The control correction consistency recognition results include the relationship between the steering wheel return direction and the offset direction, the return action delay status, and the degree of deviation of control behavior from expectations. The all-round safe driving warning information for mining rubber-tired vehicles includes high-risk status marker records, audio-visual module trigger commands, and image frame warning annotation data.

[0053] Please see Figure 2 and Figure 3 The image reflectance monitoring module includes:

[0054] The image frame acquisition submodule acquires image frame data from the front-facing camera of the mining rubber-tired vehicle in the underground roadway environment, extracts the center pixels and edge pixels within the detection area, and obtains the pixel set of the image detection area;

[0055] To acquire image frame data from the front-facing camera of a mining rubber-tired vehicle in an underground roadway environment, a CCD industrial-grade camera mounted above the front of the vehicle needs to continuously capture frames of the area in front, with a frame rate set to 30fps and a resolution of 1920×1080 per frame. The program defines the detection area of ​​each image frame as a 400×400 pixel region extending from the center of the image, further dividing it into a central region and an edge region. The central region is a 200×200 pixel area surrounding the center point, and the edge regions are four-sided rectangles with an outer band width of 100 pixels each. Image coordinates are used to locate all pixels in the central and edge regions, and their grayscale values ​​are extracted. The grayscale value range is 0–1023, which is converted to reflectance intensity values ​​(unit: mcd·lx) after system calibration. -1 ·m -2 The conversion factor is set to 1 grayscale unit → 1 mcd·lx according to the camera's factory calibration table. -1 ·m -2 Therefore, the grayscale value is directly regarded as the reflection intensity. For example, the pixel value captured in the central region of image frame 1 is R. c ={320, 310, 300}, with R pixels collected in the edge region. e ={290, 275, 265}, and the continuous pixel samples within the region are R ={310, 300, 305, 290, 295, 280, 285, 295, 300}, thereby generating the pixel set of the image detection region.

[0056] Table 1. Sample Table of Image Frame Brightness Features

[0057]

[0058] Table 1 lists the sample values ​​of pixel reflectance intensity in representative areas of the three frames of images and the sampling set used for standard deviation calculation, providing a data basis for subsequent analysis of average reflectance intensity difference and regional deviation.

[0059] The reflection intensity calculation submodule, based on the pixel set of the image detection region, calculates the average reflection intensity of pixels in the central region and the average reflection intensity of pixels in the edge region, and determines whether the difference between the two meets the minimum brightness requirement threshold for reflective markings, using the following formula:

[0060]

[0061] The composite intensity value of brightness difference is obtained through calculation. Image region features that meet the reflectance intensity benchmark conditions are selected, and regional reflectance difference measurement results are generated, where ΔR represents the composite intensity value of brightness difference, and R... ci Represents the reflection intensity of the i-th center pixel. n represents the reflection intensity of the j-th edge pixel. c With n eR represents the number of pixels at the center and edge, respectively. k Let k be the reflection intensity of the k-th pixel within the region. R represents the average reflection intensity of the region's pixels. th Minimum brightness threshold for reflective signs (300 mcd·lx) -1 ·m -2 ), where n is the total number of pixels in the region;

[0062] Based on the pixel set of the image detection region, the mean value of the pixel reflection intensity sets in the center and edge regions is calculated separately. The average reflection intensity of the center region is:

[0063]

[0064] The average reflection intensity in the edge region is:

[0065]

[0066] The difference for the first term is:

[0067]

[0068] The average regional reflection intensity is:

[0069]

[0070] The second item is the standard deviation of brightness:

[0071]

[0072] The third item is the difference from the threshold:

[0073]

[0074] The final degree of difference in synthesis is:

[0075] ΔR=33.33+4.08+5=42.41mcd·lx -1 ·m -2 ;

[0076] It can provide a unified dimensional quantification standard for boundary determination and ultimately generate regional reflectance difference measurement results.

[0077] The composite intensity of brightness difference is a single numerical index obtained by comprehensively evaluating the reflectivity of the image detection area across multiple dimensions. Its core significance lies in uniformly measuring the average brightness difference at different spatial locations (center and edge) within the area, the fluctuation of the overall brightness distribution within the area (i.e., brightness consistency), and the deviation of the overall brightness level of the area from a preset standard reflectivity threshold. This reflects whether there are obvious brightness anomalies or structural reflectivity features in the detection area of ​​the current image, thus serving as a preliminary criterion for determining the existence of reflective contour boundaries. A larger value indicates a more significant brightness difference in the area, more prominent geometric features of the reflective structure, and a greater likelihood of corresponding to a static target or obstacle with boundary characteristics. Therefore, the "composite intensity of brightness difference" plays a crucial role in bridging optical feature extraction and obstacle boundary identification in the entire image reflectivity monitoring process, taking into account brightness contrast, consistency, and standard compliance, and possessing clear physical basis and engineering significance.

[0078] The overall operational logic of the formula is based on the quantitative fusion of multi-dimensional brightness difference features. The first term... This represents the difference in average reflectance intensity between the central and edge regions of an image. The absolute value operation is used to eliminate the directionality of the difference, ensuring the index only reflects the magnitude of the numerical difference. (The second term...) The standard deviation of the reflection intensity of all pixels within the region is used to measure the variability of pixel brightness in spatial distribution. It is calculated by summing the squared deviations and then taking the square root. This method aims to highlight the drastic changes in brightness while maintaining consistent units. (The third term...) This is used to evaluate the overall brightness level of the area compared to the minimum brightness threshold R of the reflective signage. th The degree of deviation is determined by using absolute value calculations to ensure the symmetry of the deviation. The three sub-items correspond to the spatial gradient difference of brightness distribution, local consistency fluctuation, and standard deviation, respectively. Finally, the three items are integrated by addition to form a unified brightness difference measurement index ΔR, which enables a comprehensive judgment on the saliency characteristics of the reflective area.

[0079] The boundary recognition submodule extracts the edge pixel gradient values ​​after performing Gaussian filtering on the image frame based on the regional reflectance difference measurement results, constructs an edge intensity map, determines whether the closed strong edge meets the lane departure warning standard conditions, obtains the edge morphology of continuous regions, and generates static obstacle boundary recognition results.

[0080] Calling the above ΔR = 42.41mcd·lx -1 ·m -2Image filtering and edge detection are performed. Gaussian filtering with σ = 1.5 and a 5×5 convolution window are used. For any pixel P(x,y) in the image frame, its surrounding pixel values ​​are selected and weighted convolved with the Gaussian kernel G(i,j). The filtered pixel values ​​are:

[0081]

[0082] If the center pixel value is 310, and the adjacent pixels have values ​​of 305, 308, 309, etc., and the weighted sum after Gaussian kernel operation is 15375, with a normalization factor of 50, then:

[0083]

[0084] The edge gradient is calculated in the horizontal direction using the central difference method.

[0085] G x =P(x+1,y)-P(x-1,y);

[0086] The vertical direction is:

[0087] G y =P(x,y+1)-P(x,y-1);

[0088] For example, G x =6, G y =4, then the edge strength is:

[0089]

[0090] Set the low and high thresholds T with a threshold ratio of 1:3. L =8,T H =24. If there are pixels that satisfy G>24, they are marked as strong edge points. Connect all strong edge points to form a boundary chain. Extract the contour area and aspect ratio of the closed curve contour. Assume that the closed boundary contains 34 consecutive pixels and forms a 3:1 ratio rectangular border. Record the coordinate set, the average brightness of the contour pixels, the area ratio and other geometric indicators of the boundary box. Finally, output the static obstacle boundary recognition result.

[0091] Please see Figure 2 and Figure 4 The brightness gradient recognition module includes:

[0092] The brightness distribution extraction submodule, based on the static obstacle boundary recognition results, locates the set of pixel coordinates of the same region in three consecutive frames of images, collects the set of pixel reflection intensity values ​​in the region in each frame, calculates the average reflection intensity value of the corresponding region in each frame, establishes the inter-frame brightness sequence distribution of the region, and generates the inter-frame brightness sequence of the region.

[0093] Based on the static obstacle boundary recognition results, the pixel coordinate set of the static boundary region in three frames of images is called to obtain the image data of frames 8, 9, and 10. By matching the coordinates of the upper left and lower right corners of the boundary rectangle in the recognition results, the corresponding region in each frame is determined. The set of reflection intensity values ​​of all pixels in this region in each frame is extracted, and the reflection intensity values ​​are expressed in mcd·lx. -1 ·m -2 The average reflection intensity of the region in each of the three frames is calculated by taking the grayscale values ​​from the original image and converting them using the camera response curve. Assuming an image resolution of 1920×1080 pixels, a boundary region width of 80 pixels, and a height of 40 pixels, containing a total of 3200 pixels, the reflection intensity of each of the 3200 pixels in each frame is summed and divided by the number of pixels to obtain the average reflection intensity value of that region in the three frames. Let the sets of pixel grayscale values ​​extracted from the three frames be: Frame 8: {300, 302, 301...}, average 301.5; Frame 9: {308, 306, 310...}, average 308.2; Frame 10: {316, 317, 315...}, average 316.0. The calculation process for the average value of each frame uses... in R represents the average reflection intensity of frame f. k Let be the reflection intensity value of the k-th pixel in the region, and n be the total number of pixels in the region (3200). After calculation, the average reflection intensity values ​​of the three frames are combined into a sequence [301.5, 308.2, 316.0] to establish the brightness change time series of the corresponding region in the three frames of images, thus obtaining the inter-frame brightness sequence of the region.

[0094] The gradient direction detection submodule is based on the regional inter-frame brightness sequence. According to the average reflection intensity value of adjacent frames in three frames, it obtains the change direction between the first frame and the second frame, and the change direction between the second frame and the third frame. It judges the two change directions. If the two directions are consistent, it is marked as a unidirectional change sequence; otherwise, it is marked as a fluctuating sequence, and the brightness change direction consistency marking result is obtained.

[0095] Based on the regional inter-frame brightness sequence, the brightness gradient change trend is determined according to the direction of the difference in average reflection intensity between adjacent frames in the frame sequence. First, the difference between frames 8 and 9 is calculated as Δ1 = 308.2 - 301.5 = 6.7. Then, the difference between frames 9 and 10 is calculated as Δ2 = 316.0 - 308.2 = 7.8. Since Δ1 > 0 and Δ2 > 0, the change direction between the two frames is consistent, and is recorded as a positive change. Further, it is determined whether the two change directions are consistent, i.e., whether the signs are the same. If they are consistent, it is defined as a unidirectional change sequence; if they are inconsistent, it is defined as a fluctuating sequence. In this example, Δ1 and Δ2 have the same sign. The meaning is a unidirectional increase. If a trend of difference less than 0 appears in subsequent frames, such as Δ3 = 310.0 - 316.0 = -6.0, then the direction is reversed and recorded as a fluctuation sequence. A consistency judgment logic for the change direction is set for this judgment process. If the product of the two differences Δ1·Δ2 > 0, then they are consistent; otherwise, they are inconsistent. This rule is applicable to the trend judgment of any three consecutive frames. By judging the consistency of the direction of two consecutive sets of differences, the directional stability index of the region brightness in the time dimension is obtained. This index is used as the basis for judging whether to continue to retain the region as a static region. Finally, the consistency mark result of the brightness change direction is obtained.

[0096] The light source interference elimination submodule obtains all regions marked as fluctuation sequences based on the consistency marking results of brightness change direction, calculates the ratio of brightness change amplitude between frames, eliminates regions with a change ratio greater than the set moving light source disturbance threshold, retains unidirectional slowly changing regions, and generates fixed obstacle stability analysis results.

[0097] Based on the consistency marking results of brightness change direction, all regions marked as fluctuation sequences in the sequence are obtained. The absolute values ​​of the differences between the three frames' average reflection intensity values ​​are calculated sequentially. For example, if the three-frame average is [301.5, 296.2, 303.1], then the differences between the two sets are |301.5-296.2|=5.3 and |303.1-296.2|=6.9. The ratio of the maximum and minimum values ​​of these two sets is... This value is used to determine the amplitude of light source interference. If the ratio exceeds the set moving light source disturbance threshold, the area is excluded. The disturbance threshold is set to 1.25, which is based on the median value of the maximum amplitude ratio interval distribution of moving vehicle headlight areas in 100 manually sampled night images. The median + 20% deviation is used as the threshold standard. All areas with a ratio greater than 1.25 are marked and removed, and the remaining areas are retained as reliable brightness trend areas. All remaining areas are summarized again to output the stable brightness trend area, and finally the fixed obstacle stability analysis results are generated.

[0098] Please see Figure 2 and Figure 5 The path offset extraction module includes:

[0099] The boundary coordinate extraction submodule acquires image frames with time labels consistent with the fixed obstacle stability analysis results, extracts the abscissa and ordinate sequences of the left and right boundary line pixels from the start point to the end point, calculates the abscissa values ​​of the intersection points of the left and right boundary lines with the bottom row of the image on the bottom cross section, uses the abscissa value of the center pixel of the corresponding row image width as the reference center point, calculates the distance difference between the center point and the intersection point of the two boundary lines, uses the sign of the difference as the displacement direction mark, and generates a record of the lane relative displacement direction.

[0100] To acquire image frames with time labels consistent with the stability analysis results of the fixed obstacle, first read the time labels recorded in the analysis results, locate the video frame number corresponding to that time label, extract the complete image matrix of that image frame, and identify the pixel coordinate sets corresponding to the left and right boundary lines in the image frame. Then, obtain the pixel positions of the bottom row of the boundary lines in the vertical direction and calculate the x-coordinate value of the left boundary line in that row. L x-coordinate of the right boundary line R Determine the horizontal center position x at the bottom of the image. C Divide the image width by 2. For example, if the image width is 1280 pixels, then x C =640, calculate the relative position difference between the image center point and the lane center as . If Δx > 0, it means the image center is biased to the right; if Δx < 0, it means biased to the left. In the example, if x L =480, x R =880, then the center of the lane is If the image center is 640, then Δx = 640 - 680 = -40. Therefore, the current relative displacement direction can be determined to be leftward. This offset direction mark is used as the direction recognition result to obtain the lane relative displacement direction record.

[0101] The centerline offset determination submodule obtains the relative displacement direction of the lane center at consecutive image positions based on the lane relative displacement direction record, calculates the difference sequence between the relative displacement directions of each frame in turn, and performs a continuous direction determination based on whether the absolute value of the direction change is continuously non-zero and has the same sign, marks the continuous offset state, and obtains the path continuous offset determination result.

[0102] Based on the lane relative displacement direction record, the sign value of the lane center relative displacement direction in five frames before and after the current frame is obtained. The difference sequence of offset direction values ​​between adjacent frames is calculated. For example, if the offset direction values ​​of the five frames are [-40, -42, -43, -46, -47], the difference sequence is [-2, -1, -3, -1]. It is determined whether there is a sign change in the difference sequence. The sign set of the difference sequence is calculated to see if they are consistent. If all the differences are negative, the direction is consistent. Then, the absolute value set of the difference sequence is calculated to see if it is all greater than or equal to 2 pixels, which is used as the basis for judging continuous offset. The baseline value of the offset direction change amplitude is set to 2 pixels. This value comes from the 90th percentile distribution value in the test results of the lane boundary pixel smoothing trend and image annotation error range. If the five frames of offset direction value change are continuous and the direction is consistent, and the absolute value of the change in each frame is greater than 2, then the state of the five frames is marked as continuous offset state. This judgment method uses the difference sequence direction and amplitude together to participate in the judgment rule to obtain the path continuous offset judgment result.

[0103] The path trend recording submodule extracts the sequence of horizontal coordinate displacement values ​​of the lane center point in each consecutive frame based on the path continuous offset determination result, records the positions of the maximum and minimum values ​​in the sequence, calculates the slope of the trend line of the sequence, and if the slope is greater than the set threshold, it is determined to be a state of significant change in path trend, and generates a path center line offset trend record.

[0104] Based on the path offset determination results, the horizontal coordinate offset values ​​of the lane center point are extracted from five consecutive frames of images within the same time period, forming a sequence X = {640, 642, 645, 648, 651}. In this sequence, the maximum value is recorded as 651, corresponding to frame number 5, and the minimum value is recorded as 640, corresponding to frame number 1. The trend slope of this sequence is then calculated. The algorithm determines whether the trend slope exceeds a set slope threshold, which is set to 0.2 pixels per frame. This value is determined based on the statistical results of the 95th percentile rate of change in the average lane center change speed experiment. If the trend slope is greater than 0.2 or less than -0.2, the sequence is judged to have a significant unidirectional trend change. In the example above, the slope is 2.75, which is much greater than 0.2, so it is marked as a path with a significant upward trend. The current frame number and trend status are combined and recorded in a trend change table to obtain the path centerline offset trend record.

[0105] Please see Figure 2 and Figure 6 The trend judgment module includes:

[0106] The angular velocity threshold extraction submodule records the corresponding time period based on the offset trend of the path centerline, obtains the original angular velocity sampling value sequence of the vehicle steering wheel in the corresponding time interval, filters all effective angular velocity data points whose absolute value is not less than the set threshold, calculates the average value and records the maximum value, and generates effective angular velocity amplitude interval data.

[0107] Based on the time period corresponding to the path centerline offset trend record, a continuous raw angular velocity data sequence from the steering wheel angle sensor in the vehicle CAN bus is obtained. The time interval is set to the start and end time range marked by the trend record. Assuming the start time is t0 = 8.00 seconds and the end time is t1 = 13.00 seconds, the sampling time length is 5 seconds. The sampling frequency of the steering wheel angular velocity sensor is set to 10Hz, that is, 10 sets of angular velocity values ​​are collected per second. A total of 50 sets of angular velocity values ​​are obtained within this interval. The unit of angular velocity is ° / s. The absolute value of each of these 50 values ​​is taken, and a subset that meets the "effective angular velocity" standard defined in ISO19365:2016 is selected. The selection condition is that the absolute value of the angular velocity is ≥50° / s. For example, the raw data sequence contains the following values: {-45.3, -52}. The values ​​.1, -60.7, -30.0, 55.4, 49.9, 70.2, -62.6, 31.5, 48.0...} are converted to absolute values ​​as {45.3, 52.1, 60.7, 30.0, 55.4, 49.9, 70.2, 62.6, 31.5, 48.0...}. Among these, five data points satisfy the condition ≥50° / s: {52.1, 60.7, 55.4, 70.2, 62.6}. Averaging these values ​​yields an effective angular velocity mean of (52.1 + 60.7 + 55.4 + 70.2 + 62.6) / 5 = 60.2° / s. The maximum value recorded is 70.2° / s. This value reflects the instantaneous intensity of steering wheel operation within this time period, ultimately generating the effective angular velocity amplitude range data.

[0108] The steering wheel angular velocity trend judgment submodule uses the effective angular velocity amplitude range data and compares it with the offset direction information recorded in the path centerline offset trend record to determine whether the direction of the time point corresponding to the maximum value in the steering wheel angular velocity sequence is opposite to the offset direction, and determines it as a delayed correction or no correction action, and obtains the steering wheel angular velocity trend opposite relationship judgment result.

[0109] Based on the effective angular velocity amplitude range data, and comparing it with the offset direction recorded in the path centerline offset trend record, it is determined whether there is an opposite sign relationship between the steering wheel operation direction and the offset direction. Assuming the current offset trend is "leftward deviation," it is necessary to find items in the steering wheel angular velocity that have a positive value greater than 50° / s to represent a rightward correction action. If the maximum effective angular velocity is 70.2° / s and its value is positive, the direction is determined to be "right turn," which is opposite to leftward deviation, and is recorded as a successful match for the straightening direction. Further, the timestamp corresponding to this maximum angular velocity data point is extracted. t2 = 10.4 seconds, the starting time of the offset trend is t0 = 8.0 seconds, the time difference between the two is calculated as t2 - t0 = 2.4 seconds, compared with the correction delay threshold setting of 2.0 seconds. This setting is based on the 95th percentile of the average human-machine control response time of the vehicle at a speed of 30km / h in the experiment. If the current difference is greater than 2.0 seconds, it is marked as a delay correction state. If the direction of the maximum angular velocity is not opposite to the offset trend, it is marked as no correction state. In this example, the direction is opposite and the lag is more than 2 seconds, which is recorded as a delay correction. Finally, the result of the opposite relationship of the direction return to the center is obtained.

[0110] The control consistency identification submodule determines the result based on the opposite relationship between the direction and the return to the correct position. It marks the time period without correction or with delayed correction as the deviation period of control behavior. It extracts the start frame and end frame number in each deviation period of control behavior, establishes the correction delay amount in the segment as the frame difference between the offset trend occurrence point and the angular velocity peak point, calculates the delay duration and combines it with the maximum angular velocity amplitude to generate the control correction consistency identification result.

[0111] Based on the result of the opposite direction correction relationship, all recording segments in the uncorrected or delayed correction state are filtered out. Let the frame range of a certain delayed correction state be from frame 200 to frame 310, with the corresponding offset trend starting frame being frame 200 and the maximum angular velocity point appearing at frame 262. Calculated at a frame rate of 25 frames / second, the delay time between the two is (262-200) / 25 = 2.48 seconds. This delay duration is recorded as the correction lag. The maximum angular velocity value within this segment is then read and set to 68.5° / s. A set of delay behavior entries is constructed as {starting frame 200, corrected frame 262, delay duration 2.48 seconds, angular velocity amplitude 68.5° / s}. This entry is added to the offset control record table and marked as a deviation from the control behavior period. Corresponding data segments are extracted cyclically from all segments, and the offset duration and correction amplitude of all segments are summarized to complete the classification of abnormal behavior periods. Finally, a control correction consistency identification result is generated.

[0112] Please see Figure 2 and Figure 7 The risk linkage early warning module includes:

[0113] The joint state determination submodule matches the frame number based on the time tag in the control correction consistency identification result and the fixed obstacle stability analysis result, extracts whether the delayed correction or no correction state and the obstacle area stability boundary identification result exist simultaneously in the same image frame number, marks it as a high-risk frame, and generates a high-risk synchronization state record.

[0114] Based on the time stamps in the control correction consistency identification results and the fixed obstacle stability analysis results, frame number matching is performed. First, all time segments marked "no correction" or "delayed correction" in the control correction results are extracted, and their start and end frame number sequences are recorded. For example, the first correction anomaly segment is frame number 200 to 310, and the second segment is frame number 455 to 492. Then, all frame number intervals marked "stable high reflectivity" in the fixed obstacle stability results are extracted, such as frame numbers 295 to 320 and 470 to 486. The frame number ranges of all time periods from both sources are then retrieved for further analysis. The intersection judgment is used to determine whether there is an overlapping area of ​​frame numbers. If there is an overlap, it means that the same image frame has both the "obstacle stable" and "correction abnormal" states. The comparison method adopts the interval overlap judgment rule. When any frame number of the obstacle stable segment falls into the frame number interval of the correction abnormal segment, it is considered to meet the joint condition. For example, the first correction abnormal interval is 200 to 310, and the obstacle interval is 295 to 320. Then, frames 295 to 310 are the joint satisfying frame interval. Finally, all frame segments that meet the dual state conditions are selected, and the corresponding frame number and state identifier are recorded to generate a high-risk synchronization state record.

[0115] The warning instruction output submodule, based on the high-risk synchronization status record, triggers the sound and light output control module according to the time point corresponding to each frame number, writes the signal trigger instruction and instruction type, constructs a frame number-signal lookup dictionary, and obtains the sound and light signal trigger instruction data.

[0116] Based on the high-risk synchronization status record, each corresponding frame number in the status value is parsed into a millisecond-level timestamp. Assuming the current frame rate is 25 frames / second, the time corresponding to the 300th frame is... Multiply the number of seconds by 1000 to get 12000 milliseconds. Set the signal trigger response time window to a range of ±100 milliseconds. Output the sound and light command at 12000 milliseconds and write it into the control signal field. Set the voltage value to 5 volts and the output duration to 200 milliseconds. Set the signal type field to 1 for buzzer and 2 for red warning light. The encoding rule is the combination number "1+2=3", that is, set the combination output type to "3". Construct dictionary entries with frame number as key and signal type and output parameter as value, for example, {frame number 300: signal type 3, voltage 5V, duration 200ms}. This process iterates through all marked frames to generate a mapping table, completes the construction of the signal command data structure, and finally obtains the sound and light signal trigger command data.

[0117] The image annotation generation submodule reads the original pixel matrix of the image frame according to the image frame number marked in the audio-visual signal trigger command data, overlays the image annotation information, and generates image frame warning annotation data by marking the area as the bounding box of the obstacle boundary.

[0118] Based on the image frame number marked in the audio-visual signal trigger command data, the corresponding frame file in the image cache is read sequentially, the pixel matrix data corresponding to the image frame is extracted, the boundary rectangle information of the obstacle area is called, and the coordinates of its upper left and lower right corners are extracted, for example, the upper left corner is (520, 410) and the lower right corner is (760, 530). On the basis of the original boundary box, the horizontal expansion is 10 pixels and the vertical expansion is 10 pixels to obtain a new annotation box from (510, 400) to (770, 540). This area is drawn as a red border, the RGB value is set to (255, 0, 0), the border line width is 2 pixels, and the transparency value is set to 0.7. The text "High Risk Uncorrected" is drawn in the upper left corner of the boundary box, using a font size of 16 pixels, the text color is set to red, and the transparency is the same as the border at 0.7. After the annotation is completed, the image frame is saved as a JPEG format, and the file naming rule is the original frame name plus the suffix "_warning", such as "frame300_warning.jpg". Finally, the image frame warning annotation data is generated.

[0119] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A comprehensive safety driving warning system for mining rubber-tired vehicles, characterized in that, The system includes: The image reflection monitoring module acquires image frame data from the front-facing camera of the mining rubber-tired vehicle, extracts the average reflection intensity of the center pixel and the average reflection intensity of the edge pixel within the detection area, compares whether the difference between the two matches the retroreflection coefficient, confirms the outline boundary of the reflective obstacle, and generates a static obstacle boundary recognition result. Based on the static obstacle boundary recognition results, the brightness gradient recognition module extracts the brightness level distribution values ​​of the corresponding region in three consecutive frames, eliminates interference signals from moving light sources through gradient change detection, and generates the stability analysis results of the fixed obstacle. The path offset extraction module obtains the pixel coordinates of the left and right boundary lines in the image frame corresponding to the time label of the fixed obstacle stability analysis result, calculates the displacement direction of the horizontal center point of the image relative to the center position of the lane, extracts the offset direction marker and determines whether continuous offset has occurred, and generates a path center line offset trend record. The control trend judgment module obtains the steering wheel angular velocity threshold data for the corresponding time period based on the path centerline offset trend record, determines whether the steering wheel return trend has an opposite correction relationship with the path offset direction, and generates a control correction consistency recognition result.

2. The all-around safety driving early warning system for mining rubber-tired vehicles according to claim 1, characterized in that, The static obstacle boundary recognition results include the location of reflective obstacle boundaries, distribution data of abnormal reflective intensity areas, and edge contour feature curvature. The fixed obstacle stability analysis results include records of brightness direction gradient change trends, gradient change continuity indicators, and light source interference elimination information. The path centerline offset trend records include the displacement direction of the image center point, records of path centerline change trends, and offset duration. The control correction consistency recognition results include the relationship between the steering wheel return direction and the offset direction, the return action delay status, and the degree to which the control behavior deviates from the expectation.

3. The all-around safety driving early warning system for mining rubber-tired vehicles according to claim 1, characterized in that, The retroreflection coefficient specifically refers to the minimum brightness requirement for reflective markings, with a mining standard of ≥300 mcd·lx. -1 ·m -2 The reflective barrier contour boundary is determined using Canny edge detection; the gradient change detection is performed using directional gradient histogram feature matching.

4. The all-around safety driving early warning system for mining rubber-tired vehicles according to claim 1, characterized in that, The image reflectance monitoring module includes: The image frame acquisition submodule acquires image frame data from the front-facing camera of the mining rubber-tired vehicle in the underground roadway environment, extracts the center pixels and edge pixels within the detection area, and obtains the pixel set of the image detection area; The reflection intensity calculation submodule calculates the average reflection intensity of the pixels in the central region and the average reflection intensity of the pixels in the edge region based on the pixel set of the image detection area, determines whether the difference between the two meets the minimum brightness requirement threshold of the reflective mark, calculates the composite intensity value of the brightness difference, filters out the image area features that meet the reflective intensity benchmark conditions, and generates the regional reflection difference measurement result. Based on the regional reflection difference measurement results, the boundary recognition submodule performs Gaussian filtering on the image frame, extracts the edge pixel gradient values, constructs an edge intensity map, determines whether the closed strong edge meets the lane departure warning standard conditions, obtains the edge morphology of the continuous region, and generates static obstacle boundary recognition results.

5. The all-around safety driving early warning system for mining rubber-tired vehicles according to claim 1, characterized in that, The brightness gradient recognition module includes: The brightness distribution extraction submodule, based on the static obstacle boundary recognition results, locates the set of pixel coordinates of the same region in three consecutive frames of images, collects the set of pixel reflection intensity values ​​in the region in each frame, calculates the average reflection intensity value of the corresponding region in each frame, establishes the inter-frame brightness sequence distribution of the region, and generates the inter-frame brightness sequence of the region. The gradient direction detection submodule, based on the inter-frame brightness sequence of the region, obtains the change direction between the first and second frames and the change direction between the second and third frames according to the average reflection intensity value of adjacent frames in the three frames. It judges the two change directions. If the two directions are consistent, it is marked as a unidirectional change sequence; otherwise, it is marked as a fluctuating sequence, and the brightness change direction consistency marking result is obtained. The light source interference elimination submodule obtains all regions marked as fluctuation sequences based on the brightness change direction consistency marking results, calculates the ratio of the brightness change amplitude between frames, eliminates regions with a change ratio greater than the set moving light source disturbance threshold, retains unidirectional slowly changing regions, and generates fixed obstacle stability analysis results.

6. The all-around safety driving early warning system for mining rubber-tired vehicles according to claim 1, characterized in that, The path offset extraction module includes: The boundary coordinate extraction submodule acquires image frames with time labels consistent with the fixed obstacle stability analysis results, extracts the abscissa and ordinate sequences of the left and right boundary line pixels from the start point to the end point, calculates the abscissa values ​​of the intersection points of the left and right boundary lines with the bottom row of the image on the bottom cross section of the image, uses the abscissa value of the center pixel of the corresponding row image width as the reference center point, calculates the distance difference between the center point and the intersection point of the two boundary lines, uses the sign of the difference as the displacement direction mark, and generates a record of the relative displacement direction of the lane. The centerline offset determination submodule obtains the relative displacement direction of the lane center at consecutive image positions based on the lane relative displacement direction record, calculates the difference sequence between the relative displacement directions of each frame in turn, and performs a continuous direction determination based on whether the absolute value of the direction change is continuously non-zero and has the same sign, marks the continuous offset state, and obtains the path continuous offset determination result. The path trend recording submodule extracts the sequence of horizontal coordinate displacement values ​​of the lane center point in each consecutive frame based on the path continuous offset determination result, records the positions of the maximum and minimum values ​​in the sequence, calculates the slope of the trend line of the sequence, and if the slope is greater than a set threshold, it is determined to be a state of significant change in path trend, and generates a path center line offset trend record.

7. The all-around safety driving early warning system for mining rubber-tired vehicles according to claim 1, characterized in that, The manipulation trend judgment module includes: The angular velocity threshold extraction submodule records the corresponding time period according to the offset trend of the path centerline, obtains the original angular velocity sampling value sequence of the vehicle steering wheel in the corresponding time interval, filters all effective angular velocity data points whose absolute value is not less than the set threshold, calculates the average value and records the maximum value, and generates effective angular velocity amplitude interval data. The steering wheel angular velocity trend judgment submodule, based on the effective angular velocity amplitude range data, compares the offset direction information recorded in the path centerline offset trend record to determine whether the direction of the time point corresponding to the maximum value in the steering wheel angular velocity sequence is opposite to the offset direction, and determines it as a delayed correction or no correction action, and obtains the steering wheel angular velocity trend opposite relationship judgment result. The control consistency identification submodule determines the result based on the opposite direction return relationship, marks the time period without correction or with delayed correction as the deviation control behavior period, extracts the start frame and end frame number in each deviation control behavior period, establishes the correction delay amount in the segment as the frame difference between the offset trend occurrence point and the angular velocity peak point, calculates the delay duration and combines it with the maximum angular velocity amplitude to generate the control correction consistency identification result.

8. The all-around safety driving early warning system for mining rubber-tired vehicles according to claim 1, characterized in that, The system also includes: The risk linkage early warning module determines whether the current obstacle area is accompanied by uncorrected path behavior based on the control correction consistency recognition result and the fixed obstacle stability analysis result. If both conditions are met, it is marked as a high-risk state, and outputs the sound and light module trigger command and the current image frame early warning label data to generate all-round safe driving early warning information for mining rubber-tired vehicles.

9. The all-around safety driving early warning system for mining rubber-tired vehicles according to claim 8, characterized in that, The comprehensive safety driving warning information for mining rubber-tired vehicles includes high-risk status marking records, sound and light module trigger commands, and image frame warning annotation data.

10. The all-around safety driving early warning system for mining rubber-tired vehicles according to claim 8, characterized in that, The risk linkage early warning module includes: The joint state determination submodule matches the frame number based on the time tag in the operation correction consistency identification result and the fixed obstacle stability analysis result, extracts whether the delay correction or no correction state and the obstacle area stability boundary identification result exist simultaneously in the same image frame number, marks it as a high-risk frame, and generates a high-risk synchronization state record. The warning instruction output submodule, based on the high-risk synchronization status record, triggers the sound and light output control module according to the time point corresponding to each frame number, writes the signal trigger instruction and the instruction type, constructs a frame number-signal lookup dictionary, and obtains the sound and light signal trigger instruction data. The image annotation generation submodule reads the original pixel matrix of the image frame according to the image frame number marked in the audio-visual signal trigger command data, overlays the image annotation information, and generates image frame warning annotation data by marking the area as the outer bounding box of the obstacle boundary.

Images