Dangerous area intrusion early warning system and method based on vision and radar fusion

Abstract

The invention relates to the technical field of safety monitoring, and discloses a dangerous area intrusion early warning system and method based on vision and radar fusion, and the method comprises the steps: recognizing field personnel and a dangerous area marker through employing an RGB-D depth camera, and generating an initial three-dimensional point cloud under a camera coordinate system; the initial point cloud is transformed and fused to a radar coordinate system serving as a global reference by applying a rotation matrix and a translation vector; constructing a virtual electronic fence according to the transformed marker point cloud, and calculating the geometric center of the personnel point cloud in real time as the position coordinate of the personnel point cloud; and continuously comparing the spatial distance between the personnel position and the electronic fence, judging invasion once the spatial distance is smaller than a preset radius, and triggering a high-decibel voice alarm. By fusing the accuracy of visual identification and the stability of a radar space coordinate system, the problems that a single sensor is easily interfered by the environment and the target identification capability is insufficient are solved, and the detection reliability and early warning initiative of the system in a complex industrial environment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of safety monitoring, in particular to a dangerous area intrusion early warning system and method based on fusion of vision and radar. BACKGROUND

[0002] In modern industrial production environments, high-voltage operating areas pose a significant safety risk due to their inherent danger. Unauthorized personnel who mistakenly enter such areas can easily cause serious safety accidents such as electric shock, resulting in immeasurable losses to both human life and enterprise property. Therefore, real-time and effective monitoring and management of high-voltage dangerous areas is a core link in ensuring production safety.

[0003] Currently, safety management in high-voltage operating areas relies on physical isolation, manual inspection, and traditional video monitoring techniques. Physical isolation and warning signs serve as basic protection by setting up fences and warning signs to restrict personnel entry. Manual inspection involves assigning safety officers to conduct on-site patrols. Traditional video monitoring systems are used for picture recording for subsequent review and analysis.

[0004] However, these existing technical means have limitations in practical application. First, whether it is a physical fence or manual inspection, both essentially belong to passive protection or delayed intervention, lacking active early warning capabilities. The system cannot automatically identify and immediately alert at the moment of intrusion behavior, and its safety protection highly depends on the self-consciousness of personnel or the accidental discovery of inspection personnel, resulting in a serious lag in response and failing to meet the stringent requirements of high-voltage dangerous environments for immediacy.

[0005] Secondly, the sensing accuracy and reliability of existing technologies are difficult to guarantee. Traditional video monitoring systems, as a visual solution, are highly susceptible to interference from complex and variable environmental factors such as light and shadow in industrial sites, leading to a decrease in image recognition accuracy. Other single sensing technologies, such as simple infrared or laser beams, often cannot identify the attributes of the target, making it difficult to effectively distinguish between intruders and other objects (such as tool carts, floating objects, etc.), resulting in a high risk of false positives or false negatives in the monitoring system, and making it difficult to achieve stable and accurate spatial positioning and target recognition in complex scenarios.

[0006] In addition, the existing technology's early warning method is inefficient and has limited effectiveness. Even if an intrusion is discovered through manual or monitoring, the information transmission chain from the discoverer to the on-site intruder is too long, often requiring layer-by-layer notification through intercoms or phones. This indirect warning method can delay valuable intervention time. The inherent warning signs or simple sound and light alarms in the field have weak warning effects in noisy industrial environments, making it difficult to have a strong and direct impact on intruders, and unable to ensure that dangerous behavior is immediately and effectively stopped. SUMMARY

[0007] In view of the deficiencies of the prior art, the application provides a dangerous area intrusion early warning system and method based on fusion of vision and radar, which solves the problems of low detection accuracy and poor reliability of a single sensor in a complex industrial environment due to environmental interference or inability to identify target properties, and insufficient on-site warning effect of the prior art relying on passive protection and delayed response.

[0008] To solve the above technical problems, the application provides a dangerous area intrusion early warning system and method based on fusion of vision and radar.

[0009] The first aspect of the application provides a dangerous area intrusion early warning system based on fusion of vision and radar. The system includes a target detection and point cloud processing module, a coordinate transformation and electronic fence monitoring module, and a voice alarm module.

[0010] The target detection and point cloud processing module is used to synchronously collect color images and depth images of the on-site environment by using an RGB-D depth camera. By processing the color images, personnel on site or markers used to define the boundary of the dangerous area are identified and two-dimensionally positioned. Then, the two-dimensional positioning results are converted into initial three-dimensional point cloud data in the camera coordinate system by combining the depth information of the corresponding pixel points in the depth image, and preliminary three-dimensional perception of the target is completed.

[0011] The coordinate transformation and electronic fence monitoring module is a processing unit for realizing the core function of the technical solution. First, the coordinate system of the laser radar is defined as a globally unique reference coordinate system to ensure the uniformity and stability of the spatial position measurement. By using the rotation matrix and translation vector obtained by pre-calibration, the initial three-dimensional point cloud data generated by the target detection and point cloud processing module in the camera coordinate system is rigidly transformed to unify the coordinate system into the global reference coordinate system.

[0012] In the global reference coordinate system, the coordinate transformation and electronic fence monitoring module performs two parallel tasks: One is the construction of a virtual electronic fence. The coordinate transformation and electronic fence monitoring module processes the three-dimensional point cloud data of the dangerous area markers transformed into the global reference coordinate system, determines the center coordinate vector representing the geometric center of the dangerous area by averaging the coordinate values of all points in the three-dimensional point cloud data. Then, the horizontal Euclidean distance of each point in the marker point cloud and the center coordinate vector in the X-Y plane of the global reference coordinate system is calculated, and the maximum value of all calculated horizontal Euclidean distances is selected to determine the radius of the virtual electronic fence.

[0013] Secondly, real-time monitoring of the on-site personnel. The coordinate transformation and electronic fence monitoring module processes the three-dimensional point cloud data of the on-site personnel transformed into the global reference coordinate system, and determines the real-time position coordinate vector accurately representing the current position of the personnel by averaging the coordinate values of all points in the point cloud data.

[0014] Finally, the coordinate transformation and electronic fence monitoring module calculates the horizontal Euclidean distance between the real-time position coordinate vector and the center coordinate vector, and compares the horizontal Euclidean distance with the determined virtual electronic fence radius in real time. If the calculated distance is less than or equal to the fence radius, it is determined that an intrusion event occurs, and an intrusion trigger signal is generated; otherwise, if the distance is greater than the fence radius, it is determined that the state is safe, and an intrusion state release signal is generated.

[0015] The voice alarm module, as the execution terminal of the system, functions to respond to the signals generated by the coordinate transformation and electronic fence monitoring module. When receiving the intrusion trigger signal, the voice alarm module drives the high-decibel voice broadcaster to cyclically broadcast the preset warning voice; when receiving the intrusion state release signal, the voice alarm module immediately stops broadcasting.

[0016] The second aspect of the present application provides a dangerous area intrusion early warning method based on fusion of vision and radar. The method comprises the following steps: First, target detection and point cloud acquisition are performed. Color images and depth images are collected by using an RGB-D depth camera, two-dimensional boundary calibration of on-site personnel and dangerous area markers is completed by analyzing the color images, and initial three-dimensional point cloud data in the camera coordinate system is generated in combination with the depth images.

[0017] Subsequently, point cloud coordinate transformation is performed. A rotation matrix and a translation vector are applied to uniformly transform the initial three-dimensional point cloud data in the camera coordinate system generated in the previous step into the radar coordinate system as the global reference.

[0018] Then, virtual electronic fence construction is performed. The center coordinate vector and the radius of the virtual electronic fence are calculated based on the three-dimensional point cloud data of the dangerous area markers transformed into the radar coordinate system, so as to construct an accurate virtual warning boundary at the data level.

[0019] Then, intrusion detection and monitoring are performed. The real-time position coordinate vector representing the position of the on-site personnel is calculated in real time based on the three-dimensional point cloud data of the on-site personnel transformed into the radar coordinate system, and the real-time position coordinate vector is compared with the constructed virtual electronic fence in terms of spatial position, so as to determine whether an intrusion event occurs.

[0020] Finally, the voice alarm is executed, when the current step is determined as the intrusion occurrence, a high-decibel voice broadcaster is triggered to issue a warning; and when the intrusion state is determined to be released, the broadcast of the warning voice is stopped.

[0021] The application realizes complementary advantages by combining the visual recognition capability of the RGB-D depth camera with the stable space coordinate system of the laser radar. On the one hand, the visual information is used to accurately identify the monitoring target (person), avoiding the misjudgment of the traditional sensor to non-target objects; on the other hand, all the sensing data are processed and judged in the stable radar coordinate system, eliminating the problem that the visual sensor is easily disturbed by environmental light and other factors, and improving the detection accuracy and operation reliability of the system in a complex industrial environment. At the same time, the electronic fence and the positioning personnel are constructed through three-dimensional point cloud data, realizing high-precision spatial relationship judgment, so that the early warning is more timely and accurate.

[0022] The application provides a dangerous area intrusion early warning system and method based on fusion of vision and radar. The application has the following beneficial effects: 1. The application combines the visual recognition capability of deep learning with the stable space ranging capability of the laser radar, and unifies the visual recognition result through coordinate transformation and fusion into the radar coordinate system for unified analysis, overcoming the defect that a single visual scheme is easily disturbed by environmental factors such as light and shadow, and also solving the problem that a single radar scheme cannot identify target attributes, so that stable and accurate spatial positioning of personnel and dangerous areas can be realized in a complex industrial environment.

[0023] 2. In the fused three-dimensional point cloud space, the actual spatial distance between the personnel and the boundary of the dangerous area is calculated in real time and accurately, and is continuously compared with the preset safety threshold. Once the distance reaches the threshold, the dangerous area intrusion early warning system can trigger an alarm immediately, realizing early intervention and intervention of potential danger, rather than waiting for the accident to happen and then responding, so as to preposition the safety line and prevent the occurrence of safety accidents.

[0024] 3. After the dangerous area intrusion early warning system of the application determines that the intrusion occurs, a high-decibel voice broadcaster is triggered to issue a clear and loud voice warning to the on-site personnel. Compared with the mode of relying only on light or remote notification, this powerful acoustic warning can quickly attract the attention of the intruder and the surrounding personnel, and forcibly convey the danger information, so as to stop the continuation of the dangerous behavior and gain the key reaction time for avoiding risks. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The figure is a functional module structure diagram of the dangerous area intrusion early warning system of the embodiment of the application. Figure 2 The figure is a flow chart of the dangerous area intrusion early warning method of the embodiment of the application. Figure 3 A schematic diagram of the coordinate system transformation principle of the embodiment of the present application; Figure 4 A schematic diagram of the virtual electronic fence construction principle of the embodiment of the present application; Figure 5 A schematic diagram of the intrusion detection principle of the embodiment of the present application; Figure 6 A schematic diagram of the target detection and point cloud acquisition of the embodiment of the present application; Figure 7 A schematic diagram of the personnel intrusion process of the embodiment of the present application.

[0026] Among them, 10, target detection and point cloud processing module; 20, coordinate transformation and electronic fence monitoring module; 30, voice alarm module. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the specification of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0028] Referring to the drawings Figure 1 The present application provides a dangerous area intrusion early warning system based on fusion of vision and radar, which can include: an RGB-D depth camera, a laser radar, and a high-decibel voice broadcaster.

[0029] The RGB-D depth camera is used to collect color images and depth images of the high-voltage dangerous area and its surroundings in real time. The color images are used for subsequent target detection algorithm (such as YOLOv7-tiny model) processing to identify on-site personnel and dangerous area markers (such as red stakes in the embodiment); the depth images are used to provide depth information corresponding to the color image pixels, which is the basis for generating target three-dimensional point cloud data.

[0030] The laser radar is used to obtain high-precision three-dimensional point cloud data of the environment. In an embodiment of the present application, the coordinate system of the laser radar is set as the global reference coordinate system of the dangerous area intrusion early warning system.

[0031] The high-decibel voice broadcaster is used to receive a trigger signal and immediately broadcast a preset voice warning content when the dangerous area intrusion early warning system determines that an intrusion event has occurred. The voice warning content is, for example, "High voltage danger, please do not approach!" or "Warning! You have entered a high-voltage danger area, please evacuate immediately!".

[0032] In the deployment of the dangerous area intrusion early warning system, the RGB-D depth camera and the laser radar are installed near the high-voltage experimental area (for example, the circular range surrounded by red stakes). The installation positions and the field angles of view of the two are configured to ensure that the dangerous area itself and the passageway of the personnel into the area are within the effective detection range thereof, so that the information of the personnel and the dangerous area markers (red stakes) can be acquired at the same time.

[0033] The high-decibel voice broadcaster is installed on the site, and the selection of the installation position ensures that when it broadcasts the voice warning, the sound can be clearly received by the personnel near the high-voltage dangerous area and the personnel who have entered the area, so as to achieve the purpose of timely reminding.

[0034] The dangerous area intrusion early warning system provided by the embodiment of the application can be divided into the following modules in terms of software functions: The dangerous area intrusion early warning system comprises a target detection and point cloud processing module 10, which is used for real-time processing of color images collected by an RGB-D depth camera. The processing is based on a convolutional neural network, for example, a YOLOv7-tiny model, to identify and calibrate the target positions and sizes of the site personnel and the dangerous area (for example, the red stakes in the embodiment) in the images. The target detection and point cloud processing module 10 is also used to generate initial three-dimensional point cloud data of the personnel and the dangerous area in the camera coordinate system in combination with the target positions and the synchronously acquired depth images.

[0035] The dangerous area intrusion early warning system further comprises a coordinate transformation and electronic fence monitoring module 20, which is used for receiving the three-dimensional point cloud data in the camera coordinate system output by the target detection and point cloud processing module 10.

[0036] The coordinate transformation and electronic fence monitoring module 20 is used for converting the three-dimensional point cloud data in the camera coordinate system into the coordinate system of the laser radar by using a TF (Transform) coordinate system transformation technology and applying the external parameters (specifically, a rotation matrix and a translation vector) between the camera and the radar which are pre-calibrated.

[0037] The coordinate transformation and electronic fence monitoring module 20 is also used for constructing a virtual electronic fence in the radar coordinate system with the point cloud center point of the dangerous area as the reference and according to the fence radius pre-set by a user.

[0038] The coordinate transformation and electronic fence monitoring module 20 is also used for detecting and tracking the point cloud data of the personnel in the radar coordinate system in real time, calculating the relative distance between the personnel point cloud and the center point of the electronic fence, and judging whether there is an intrusion behavior by comparing the relative distance between the personnel point cloud and the center point of the electronic fence with the fence radius in real time.

[0039] The dangerous area intrusion early warning system further comprises a voice alarm module 30 connected with the coordinate transformation and electronic fence monitoring module 20. When the coordinate transformation and electronic fence monitoring module 20 judges that a person enters the electronic fence, that is, an intrusion behavior exists, the voice alarm module 30 is triggered, and a high-decibel voice broadcaster is driven to issue a preset voice reminder, such as "high-voltage danger, please do not approach!" or "warning! You have entered a high-voltage danger area, please evacuate immediately!".

[0040] Referring to the drawings Figure 2 and the drawings Figure 6 The dangerous area intrusion early warning method of the present application starts from step S1: target detection. In a specific embodiment, this step is performed by the target detection and point cloud processing module 10.

[0041] Specifically, in step S1, the dangerous area intrusion early warning system first uses an RGB-D depth camera to collect a color image of the working area, and the color image of the working area is transmitted to the target detection and point cloud processing module 10.

[0042] The target detection and point cloud processing module 10 internally deploys a lightweight target detection model YOLOv7-tiny, which uses the YOLOv7-tiny model to perform real-time processing on the color image. The YOLOv7-tiny model is selected because it has the characteristic of low computational resource requirements, making it suitable for meeting real-time processing requirements in industrial environments.

[0043] The target detection model YOLOv7-tiny performs forward inference on the input color image to detect and calibrate the target in the image, providing a basis for subsequent identification of the spatial position and size of the target through processing of the image data.

[0044] After the YOLOv7-tiny model processes the color image, the target monitoring model outputs the recognition result of the preset key target.

[0045] The key target specifically includes two categories: the first category is the on-site personnel, that is, the working personnel or irrelevant personnel entering the monitoring area; the second category is the dangerous area or the dangerous area marker used to define the boundary of the dangerous area.

[0046] In a specific embodiment, the high-voltage experimental area is surrounded by red stakes in a circular range, and at this time the dangerous area marker is the red stake.

[0047] The recognition result output by the YOLOv7-tiny model is a two-dimensional bounding box. The two-dimensional bounding box is used to calibrate the pixel coordinate position and the area range (i.e., size) occupied by the on-site personnel or the dangerous area marker (such as the red stake) in the color image.

[0048] This two-dimensional bounding box information is the basis for extracting the target three-dimensional point cloud in the subsequent step S1.

[0049] In the target detection of step S1, after obtaining the two-dimensional bounding box (i.e. the pixel coordinate position and area range of the target in the color image), the target detection and point cloud processing module 10 performs the acquisition of the target point cloud.

[0050] The dangerous area intrusion early warning system uses the RGB-D depth camera to synchronously acquire the corresponding depth image while collecting the color image, and the depth image contains depth information aligned with the pixels of the color image.

[0051] The target detection and point cloud processing module 10 extracts the three-dimensional coordinate information corresponding to all the pixels inside the bounding box in combination with the two-dimensional bounding box and the depth image.

[0052] The collection of these three-dimensional coordinate information constitutes the initial three-dimensional point cloud data of the key target (i.e. the on-site personnel or the dangerous area marker) in the camera coordinate system of the RGB-D depth camera.

[0053] The three-dimensional point cloud data in the camera coordinate system is a collection containing multiple three-dimensional coordinate points. The three-dimensional coordinate vector of any point in this collection can be expressed as : ; wherein is the three-dimensional coordinate vector of any point in the target point cloud in the camera coordinate system; , and are the coordinate values of the point on the X-axis, Y-axis and Z-axis of the camera coordinate system, respectively.

[0054] This three-dimensional point cloud data in the camera coordinate system will be used as the input of step S2: point cloud coordinate transformation.

[0055] Referring to the accompanying drawings, Figure 3 after step S1, the dangerous area intrusion early warning method enters step S2. This step is performed by the coordinate transformation and electronic fence monitoring module 20 in one specific embodiment.

[0056] The principle of coordinate system one is that the initial three-dimensional point cloud data (such as the personnel point cloud and the dangerous area marker point cloud) generated by the target detection and point cloud processing module 10 in step S1 is in the camera coordinate system of the RGB-D depth camera.

[0057] However, the dangerous area intrusion early warning system needs a unified global reference coordinate system to perform subsequent spatial position analysis and distance calculation. In the embodiments of the present application, the coordinate system of the laser radar is selected as the global reference coordinate system.

[0058] Therefore, in order to determine the center point of the danger zone in the global reference coordinate system (i.e. the radar coordinate system) in the subsequent step S3, and to be able to calculate the spatial relative distance between the person and the center point in step S4, coordinate system transformation must be performed.

[0059] The purpose of step S2 is to convert the three-dimensional point cloud data in the camera coordinate system to the radar coordinate system, so that the danger zone intrusion warning system can uniformly use the radar coordinate system for subsequent electronic fence construction, intrusion judgment and real-time monitoring.

[0060] In step S2, the coordinate transformation and electronic fence monitoring module 20 applies the TF coordinate system transformation technology to realize the unification of the coordinate system.

[0061] The implementation of the TF coordinate system transformation relies on the pre-acquired external calibration parameters, which describe the rigid spatial relationship between the camera coordinate system of the RGB-D depth camera and the radar coordinate system of the laser radar.

[0062] The external calibration parameters specifically include a rotation matrix and a translation vector The rotation matrix and the translation vector are determined in advance by an offline calibration method and stored in the danger zone intrusion warning system.

[0063] The coordinate transformation and electronic fence monitoring module 20 receives the initial three-dimensional point cloud data in the camera coordinate system from step S1.

[0064] The coordinate transformation and electronic fence monitoring module 20 applies the following transformation formula to each point in the three-dimensional point cloud data, transforming it from the three-dimensional coordinate vector in the camera coordinate system to the corresponding three-dimensional coordinate vector in the radar coordinate system: ; wherein, is the rotation matrix of , representing the rotation relationship from the camera coordinate system to the radar coordinate system; is the translation vector of , representing the translation amount of the origin of the camera coordinate system in the radar coordinate system; is the three-dimensional coordinate vector of the point in the target point cloud in the camera coordinate system; is the three-dimensional coordinate vector of the point in the target point cloud in the radar coordinate system.

[0065] which can be represented as: ; wherein, , and are the coordinate values of the point in the radar coordinate system respectively.

[0066] may be expressed as: ; wherein, , and are the translation components of the camera coordinate system origin on the X-axis, Y-axis and Z-axis of the radar coordinate system respectively.

[0067] By performing this transformation operation on both the three-dimensional point cloud of the on-site personnel and the three-dimensional point cloud of the dangerous area marker, the dangerous area intrusion early warning system unifies all the position information required for subsequent processing into the radar coordinate system.

[0068] Referring to the accompanying drawings, Figure 4 , the dangerous area intrusion early warning method enters step S3: virtual electronic fence construction after step S2. In one specific embodiment, this step is performed by the coordinate transformation and electronic fence monitoring module 20.

[0069] In step S3, the determination of the fence center is first performed.

[0070] The coordinate transformation and electronic fence monitoring module 20 obtains the three-dimensional point cloud data of the dangerous area marker that has been transformed into the radar coordinate system in step S2. In one embodiment, the dangerous area marker is a plurality of red stakes used to define the high-voltage experiment area, and the three-dimensional point cloud data is a collection of all three-dimensional coordinate points of these stakes in the radar coordinate system.

[0071] The coordinate transformation and electronic fence monitoring module 20 processes the three-dimensional point cloud data of the dangerous area marker, calculates the geometric center point (i.e. the centroid) of the point cloud collection, and determines the three-dimensional coordinate vector of the geometric center point in the radar coordinate system as .

[0072] The coordinate calculation formula of the center coordinate vector is as follows: ; is the three-dimensional coordinate vector of the center point of the virtual electronic fence in the radar coordinate system; , and are the X-axis, Y-axis and Z-axis coordinate values of the center point in the radar coordinate system.

[0073] The coordinate calculation formula of the center coordinate vector The coordinate components of each point are calculated by the following formula: ; ; ; wherein, is the total number of three-dimensional coordinate points contained in the three-dimensional point cloud data of the dangerous area marker; is a summation operation that adds all the terms from 1 to ; , and are the coordinate values of the X-axis, Y-axis and Z-axis of the th point in the three-dimensional point cloud data of the dangerous area marker in the radar coordinate system.

[0074] The center coordinate vector calculated by the above formula is determined by the dangerous area intrusion early warning system as a reference point, which is used for subsequent construction of the electronic fence and calculation of the intrusion distance.

[0075] After determining the center coordinate vector of the virtual electronic fence, the coordinate transformation and electronic fence monitoring module 20 continues to perform step S3 to set the radius of the virtual electronic fence.

[0076] The setting of the radius of the virtual electronic fence is based on the center coordinate vector and the three-dimensional point cloud data of the dangerous area marker that has been transformed into the radar coordinate system in step S2.

[0077] Specifically, the coordinate transformation and electronic fence monitoring module 20 calculates the Euclidean distance between each point in the three-dimensional point cloud data of the dangerous area marker and the horizontal projection (i.e., (x, y)) of the center coordinate vector on the X-Y plane (i.e., the horizontal plane) of the radar coordinate system.

[0078] Subsequently, the coordinate transformation and electronic fence monitoring module 20 selects the maximum value among all the calculated horizontal distances and determines this maximum value as the radius of the virtual electronic fence.

[0079] The radius of the virtual electronic fence is determined by the following formula: ; wherein, is the radius value of the virtual electronic fence; is a maximum value operation that represents the maximum value among all the calculated horizontal distances.​​ from 1 to the maximum value of the results of the expression inside the curly braces in the process of the total number of three-dimensional coordinate points contained in the three-dimensional point cloud data of the dangerous area marker; and are the X-axis and Y-axis coordinate values of the point in the three-dimensional point cloud data of the dangerous area marker in the radar coordinate system; and are the X-axis and Y-axis coordinate components of the center coordinate vector .

[0080] The radius of the virtual electronic fence and the center coordinate vector together define the spatial range of the virtual electronic fence and serve as the basis for performing intrusion judgment in step S4.

[0081] After determining the center coordinate vector of the virtual electronic fence and the radius of the virtual electronic fence, the coordinate transformation and electronic fence monitoring module 20 completes the construction of the virtual electronic fence.

[0082] The construction method of the virtual electronic fence is to take the projection point ( ) of the center coordinate vector on the X-Y plane (horizontal plane) as the center and the radius of the virtual electronic fence as the radius to determine a circular horizontal area on the X-Y plane.

[0083] This circular horizontal area is the dangerous area range defined by the virtual electronic fence.

[0084] The boundary of the virtual electronic fence is defined on the X-Y plane of the radar coordinate system by the following equation: ; Therefore, the dangerous area covered by the virtual electronic fence is defined as the set of points that satisfy the following inequality: ; In the above formula and inequality, and are the X-axis and Y-axis coordinate values of any point on the X-Y plane of the radar coordinate system; and are the X-axis and Y-axis coordinate components of the center coordinate vector ; is the radius value of the virtual electronic fence.

[0085] The coordinate transformation and electronic fence monitoring module 20 takes this constructed virtual electronic fence (i.e. the circular horizontal area defined by and as the reference geometric range for performing intrusion judgment in the subsequent step S4.

[0086] Referring to the attached Figure 5 , after step S3, the dangerous area intrusion early warning method enters step S4: intrusion detection and monitoring. This step is executed in real time and cyclically by the coordinate transformation and electronic fence monitoring module 20 in one specific embodiment.

[0087] In step S4, real-time tracking of the on-site personnel position is first performed.

[0088] The coordinate transformation and electronic fence monitoring module 20 obtains the three-dimensional point cloud data of the on-site personnel in real time and continuously, which is the latest data detected by the target detection and point cloud processing module 10 (step S1) and unified to the radar coordinate system via TF coordinate system transformation (step S2).

[0089] In order to determine the current instantaneous position of the on-site personnel in the radar coordinate system, the coordinate transformation and electronic fence monitoring module 20 processes the three-dimensional point cloud data (a set of three-dimensional coordinate points) of the on-site personnel in the current frame, and calculates the geometric center point (i.e. the centroid) of the point cloud set.

[0090] The three-dimensional coordinate vector of the geometric center point in the radar coordinate system is determined as the real-time position coordinate vector of the on-site personnel.

[0091] The calculation formula of the real-time position coordinate vector is as follows: ; wherein, is the real-time position coordinate vector of the on-site personnel in the radar coordinate system at the current time; , and are the X-axis, Y-axis and Z-axis coordinate components of the real-time position coordinate vector , respectively.

[0092] Each coordinate component of the real-time position coordinate vector is calculated by the following formula: ; ; ; wherein, is a summation operation that increments from 1 to all the terms are added together; is the total number of three-dimensional coordinate points contained in the live personnel three-dimensional point cloud data of the current frame, which is used as the termination index of accumulation in the summation operation; , and are the X-axis, Y-axis and Z-axis coordinate values of the th point in the live personnel three-dimensional point cloud data of the current frame in the radar coordinate system.

[0093] The coordinate transformation and electronic fence monitoring module 20 continuously updates the real-time position coordinate vector and uses the real-time position coordinate vector as the basis for intrusion judgment for subsequent spatial position comparison with the virtual electronic fence.

[0094] In step S4, the real-time position coordinate vector of the live personnel is obtained After that, the coordinate transformation and electronic fence monitoring module 20 then executes the intrusion judgment logic, which compares the real-time horizontal position of the live personnel (represented by the horizontal coordinate components , of ) with the circular horizontal region defined by the virtual electronic fence constructed in step S3. The coordinate transformation and electronic fence monitoring module 20 calculates the following inequality in real time: wherein and are the X-axis and Y-axis coordinate components of the real-time position coordinate vector of the live personnel at the current time; and are the X-axis and Y-axis coordinate components of the center coordinate vector of the virtual electronic fence; is the radius value of the virtual electronic fence.

[0095] The coordinate transformation and electronic fence monitoring module 20 executes the following judgment based on the calculation result of the inequality: If the result of the above inequality is true (i.e. established), the dangerous area intrusion warning system determines that the live personnel has intruded into the dangerous area defined by the virtual electronic fence.

[0096] If the result of the above inequality is false (i.e. not established), the dangerous area intrusion warning system determines that the live personnel is located outside the dangerous area and no intrusion has occurred.

[0097] When the result of the determination is true, the coordinate transformation and electronic fence monitoring module 20 triggers the subsequent step S5. If the result of the determination is false, the coordinate transformation and electronic fence monitoring module 20 returns to the beginning of step S4, continues to obtain the position of the next frame of personnel, and repeatedly performs the intrusion determination, thereby realizing real-time monitoring.

[0098] After step S4, when the coordinate transformation and electronic fence monitoring module 20 determines that the result of the inequality is true in the intrusion determination logic, the dangerous area intrusion early warning method enters step S5 and performs early warning. In a specific embodiment, step S5 is performed by the voice alarm module 30.

[0099] The alarm triggering mechanism in this step is specifically based on the Boolean result of the intrusion determination logic in step S4.

[0100] The alarm triggering mechanism includes a start condition and a stop condition.

[0101] The start condition is that, in step S4, the coordinate transformation and electronic fence monitoring module 20 determines that the real-time position of the on-site personnel satisfies the following inequality: ; When the start condition is met (i.e., the inequality result is “true”), the coordinate transformation and electronic fence monitoring module 20 immediately sends an intrusion trigger signal to the voice alarm module 30.

[0102] After receiving the intrusion trigger signal, the voice alarm module 30 immediately activates and drives the voice alarm device (for example, a high-decibel voice broadcaster) to start cyclically broadcasting a pre-set warning voice, such as “high-voltage danger, please do not approach!” or “warning! You have entered a high-voltage danger area, please evacuate immediately!”.

[0103] The stop condition of the alarm triggering mechanism is that, in the subsequent real-time cyclic execution of step S4, the coordinate transformation and electronic fence monitoring module 20 determines that the real-time position of the on-site personnel no longer satisfies the above inequality (i.e., the inequality result changes to false).

[0104] When the stop condition is met, the coordinate transformation and electronic fence monitoring module 20 sends an intrusion state release signal to the voice alarm module 30.

[0105] After receiving the release signal, the voice alarm module 30 immediately stops the warning voice broadcast of the voice alarm device.

[0106] In step S5, after the voice alarm module 30 receives the intrusion trigger signal from the coordinate transformation and electronic fence monitoring module 20, the voice alarm module 30 immediately performs the alarm operation.

[0107] The content of the alarm is a pre-recorded and stored warning voice in the voice alarm module 30 or its associated storage unit. In one embodiment, the content of the warning voice is set as a clear and explicit Chinese instruction, such as "High voltage danger, please do not approach!" or "Warning! You have entered the high voltage danger area, please evacuate immediately!".

[0108] The alarm is in the form of the voice alarm module 30 immediately retrieving the audio data of the warning voice from the storage when receiving the intrusion trigger signal.

[0109] The voice alarm module 30 converts the audio data into an electrical signal and outputs it to an external voice alarm device. In one embodiment, the voice alarm device is a high-decibel voice broadcaster deployed in the monitoring site to ensure that the site personnel can clearly receive the warning voice.

[0110] The voice alarm module 30 performs a loop broadcast of the warning voice, i.e. during the entire period when the intrusion trigger signal is continuously valid (no release signal is received), the high-decibel voice broadcaster continuously and repeatedly plays the voice content of "High voltage danger, please do not approach!" or "Warning! You have entered the high voltage danger area, please evacuate immediately!".

[0111] When the voice alarm module 30 receives the intrusion state release signal from the coordinate transformation and electronic fence monitoring module 20, the voice alarm module 30 immediately stops outputting the audio signal to the high-decibel voice broadcaster, and the warning voice broadcast is suspended.

[0112] During the execution of step S5, the alarm release mechanism of the voice alarm module 30 is based on the intrusion determination logic continuously executed by the coordinate transformation and electronic fence monitoring module 20 in step S4.

[0113] The trigger condition of the alarm release mechanism, i.e. the stop condition of the alarm, is that the coordinate transformation and electronic fence monitoring module 20 determines in the real-time monitoring loop that the real-time position of the site personnel has moved out of the dangerous area defined by the virtual electronic fence.

[0114] Specifically, the stop condition is that the coordinate transformation and electronic fence monitoring module 20 determines in the subsequent real-time loop execution of step S4 that the real-time position coordinates of the site personnel satisfy the following inequality: ; wherein and are the X-axis and Y-axis coordinate components of the real-time position coordinate vector of the site personnel at the current time; and respectively the center coordinate vector of the virtual electronic fence the X-axis and Y-axis coordinate components of respectively the radius value of the virtual electronic fence.

[0115] When the stop condition is met (i.e. the above inequality result is true), the coordinate transformation and electronic fence monitoring module 20 immediately sends an intrusion state release signal to the voice alarm module 30.

[0116] After receiving the intrusion state release signal, the voice alarm module 30 immediately stops playing the warning voice, i.e. stops outputting the audio signal to the voice alarm device (such as a high-decibel voice player).

[0117] Thereafter, the dangerous area intrusion early warning method returns to the real-time monitoring cycle of step S4, and the voice alarm module 30 remains in the standby state until the next time an intrusion trigger signal is received.

[0118] Referring to the accompanying drawings Figure 7 , the figure directly shows the intrusion detection and monitoring effect of step S4.

[0119] Before intrusion, the point cloud of the on-site personnel is located outside the constructed virtual electronic fence. At this time, the horizontal distance calculated by the system is greater than the radius of the virtual electronic fence , the coordinate transformation and electronic fence monitoring module 20 determines that it is a safe state, and no intrusion trigger signal is generated.

[0120] When the on-site personnel moves and crosses the electronic fence, the geometric center (i.e. real-time position coordinate vector ) of the point cloud enters the inside of the virtual electronic fence. At this time, the horizontal distance calculated by the system is less than or equal to , step S4 determines that intrusion has occurred, and the coordinate transformation and electronic fence monitoring module 20 immediately generates an intrusion trigger signal.

[0121] After step S4, the dangerous area intrusion early warning method enters step S5. After receiving the intrusion trigger signal, the voice alarm module 30 immediately drives the high-decibel voice player to play the warning voice, and when the on-site personnel leaves, the system generates an intrusion state release signal to stop playing.

[0122] Physical deployment of the dangerous area intrusion early warning method according to an embodiment of the present application in the specific application scenario of a high-pressure laboratory.

[0123] In this embodiment, the high-pressure experimental area is preliminarily defined in terms of physical boundaries on the ground by setting multiple dangerous area markers (such as physical red insulating warning stakes).

[0124] The target detection and point cloud processing module 10 is embodied as a three-dimensional laser radar sensor in this embodiment. The three-dimensional laser radar sensor is fixedly installed at a high position on one side of the high-pressure experiment area, such as the top of a wall or a dedicated stand.

[0125] The deployment position is selected so that the scanning field of view of the three-dimensional laser radar sensor can completely cover the entire ground area enclosed by the red warning stakes from top to bottom, and the monitoring blind area caused by equipment obstruction is minimized. The three-dimensional laser radar sensor is responsible for real-time scanning and generating three-dimensional point cloud data of the red warning stakes and the on-site personnel in the access area.

[0126] The coordinate transformation and electronic fence monitoring module 20 is embodied as an industrial control computer deployed in a security monitoring room or operation table in this embodiment. The industrial control computer maintains real-time data communication with the three-dimensional laser radar sensor through a high-bandwidth data cable, such as an industrial Ethernet.

[0127] The industrial control computer is the core computing unit that performs steps S2, S3, and S4 of the dangerous area intrusion early warning method of the present application.

[0128] The voice alarm module 30 is embodied as one or more high-decibel voice speakers in this embodiment. The high-decibel voice speakers are installed at the edge or entrance of the high-pressure experiment area, and their installation position ensures that the warning voice they emit can be clearly received by the on-site personnel, sufficient to suppress the background noise on site.

[0129] The high-decibel voice speakers are connected to the industrial control computer through an audio signal cable for receiving the intrusion trigger signal and intrusion state release signal issued by the computer in step S5, and accordingly executing the broadcast or stop of the warning voice.

[0130] In this deployment scenario, the on-site personnel do not need to wear any special active positioning beacon or reflective marker, and the three-dimensional laser radar sensor directly detects the physical form of the personnel through scanning to obtain their three-dimensional point cloud data.

[0131] After the dangerous area intrusion early warning system is started, the industrial control computer first performs step S3 to automatically calculate and construct a virtual electronic fence (i.e., determine and ) using the point cloud data of the red warning stakes. Then, the industrial control computer enters the real-time loop monitoring of step S4, continuously tracking the point cloud centroid of the on-site personnel and comparing it with the virtual electronic fence in spatial position to achieve automatic detection of intrusion and real-time voice alarm.

[0132] In one specific embodiment, an operational process drill was conducted for the aforementioned hazardous area intrusion early warning system deployed in a high-pressure laboratory scenario. This process is divided into two stages after the system is powered on: the initialization (electronic fence construction) stage and the real-time monitoring stage.

[0133] During the initialization phase, the industrial control computer first constructs the virtual electronic fence. The industrial control computer instructs the 3D LiDAR sensor to perform a static scan to acquire complete 3D point cloud data of all red warning posts (i.e., danger zone markers) on the ground of the high-voltage test area.

[0134] The industrial control computer receives the point cloud data of the red warning post and unifies it to the radar coordinate system in step S2. Subsequently, the industrial control computer strictly executes the calculation in step S3: First, based on the point cloud data (total number of points...), The coordinates of all points in ) , , The center coordinate vector is determined by calculating the average value formula. and its horizontal components ( ).

[0135] Next, the industrial control computer traverses all points in the point cloud, calculating the distance between each point and the horizontal center (…). The horizontal distance between them, and through The radius of the virtual electronic fence is determined by calculation (taking the maximum value). . and The parameters are stored in the memory of the industrial control computer and used as a benchmark for intrusion detection. Once the initialization phase is complete, the system automatically enters the real-time monitoring phase.

[0136] During the real-time monitoring phase, the industrial control computer continuously instructs the 3D LiDAR sensor to scan the monitored area in real time at a high frequency (e.g., 10 frames per second) and continues to execute step S4.

[0137] When a person enters the radar sensor's field of view, but their physical location is still outside the circular horizontal area defined by the virtual electronic fence, the 3D lidar sensor captures the person's real-time 3D point cloud data (total number of points). And send it to the industrial control computer.

[0138] The industrial control computer immediately performs real-time personnel position tracking and calculation in step S4 on the point cloud of personnel on site, and obtains the real-time position coordinate vector in the radar coordinate system through the average value formula. And extract its horizontal coordinates ( , ).

[0139] The industrial control computer then executes the intrusion decision logic of step S4, i.e. calculates the following inequality in real time: ; Since the field personnel is outside the virtual electronic fence, the result of the above inequality is false. Therefore, the industrial control computer does not execute step S5, and does not send the intrusion trigger signal to the high-decibel voice broadcaster. The system remains silent, and immediately returns to the start of step S4 to process the point cloud data of the next frame.

[0140] Subsequently, if the field personnel continues to move and crosses the boundary defined by and enters the interior of the virtual electronic fence. In the data processing of the next frame, the industrial control computer calculates the result of the above inequality , which makes the result of the above inequality true.

[0141] At this time, the industrial control computer immediately determines that an intrusion has occurred, and immediately executes step S5 to send the intrusion trigger signal to the high-decibel voice broadcaster.

[0142] Upon receiving the trigger signal, the high-decibel voice broadcaster is immediately activated, and starts to broadcast the pre-set warning voice in a loop, such as “High voltage danger, please do not approach!” or “Warning! You have entered a high voltage danger zone, please evacuate immediately!”.

[0143] During the broadcast of the warning voice, the industrial control computer and the three-dimensional laser radar sensor continue to operate, and track the position of the field personnel in real time .

[0144] When the field personnel responds to the warning and moves in the reverse direction, so that the position of the field personnel is outside the circular area of the virtual electronic fence, in the subsequent determination, the industrial control computer calculates the result of the above inequality which again becomes false (i.e. the condition of the alarm cancellation mechanism is met).

[0145] The industrial control computer immediately sends the intrusion state cancellation signal to the high-decibel voice broadcaster, and upon receiving the cancellation signal, the high-decibel voice broadcaster immediately stops broadcasting the warning voice. The high voltage danger zone intrusion early warning system returns to the silent monitoring state, and continues to execute step S4 to wait for the occurrence of the next intrusion event.

Claims

1. A dangerous area intrusion early warning system and method based on vision and radar fusion, characterized in that, include: The target detection and point cloud processing module is used to use color images and depth images acquired by an RGB-D depth camera to complete the visual perception of on-site personnel and hazardous area markers, and generate the results of the visual perception into initial three-dimensional point cloud data in the camera coordinate system. The coordinate transformation and electronic fence monitoring module uses the coordinate system of the lidar as the global reference coordinate system. By applying rotation matrices and translation vectors, it transforms and fuses the initial 3D point cloud data in the camera coordinate system generated by the target detection and point cloud processing module into the global reference coordinate system. In the global reference coordinate system, it constructs a virtual electronic fence based on the 3D point cloud data of the danger zone markers. At the same time, it calculates the real-time position coordinate vector based on the 3D point cloud data of the on-site personnel. By comparing the real-time position coordinate vector with the virtual electronic fence, it determines whether an intrusion has occurred and generates an intrusion trigger signal or an intrusion status cancellation signal. The voice alarm module is used to drive a high-decibel voice broadcaster to broadcast a warning voice when the intrusion trigger signal is received, and to stop broadcasting the warning voice when the intrusion status is cleared signal is received.

2. The dangerous area intrusion early warning system and method based on vision and radar fusion according to claim 1, characterized in that, The target detection and point cloud processing module is used to process the color image using the YOLOv7-tiny model, and to mark the pixel coordinates and area range of on-site personnel or hazardous area markers in the color image, forming a two-dimensional bounding box.

3. The dangerous area intrusion early warning system and method based on vision and radar fusion according to claim 2, characterized in that, The target detection and point cloud processing module is also used to extract the three-dimensional coordinate information corresponding to all pixels inside the two-dimensional bounding box by using the depth image synchronously acquired by the RGB-D depth camera and combining it with the two-dimensional bounding box, thereby forming the initial three-dimensional point cloud data in the camera coordinate system.

4. The dangerous area intrusion early warning system and method based on vision and radar fusion according to claim 1, characterized in that, The coordinate transformation and electronic fence monitoring module is used to apply a rotation transformation defined by the rotation matrix and a translation transformation defined by the translation vector to each point in the initial three-dimensional point cloud data in the camera coordinate system, transforming each point in the initial three-dimensional point cloud data in the camera coordinate system into a corresponding three-dimensional coordinate vector in the global reference coordinate system.

5. The dangerous area intrusion early warning system and method based on vision and radar fusion according to claim 1, characterized in that, The coordinate transformation and electronic fence monitoring module processes the three-dimensional point cloud data of the hazardous area markers transformed to the global reference coordinate system. By averaging the coordinate values ​​of all points in the three-dimensional point cloud data, the center coordinate vector is determined.

6. The dangerous area intrusion early warning system and method based on vision and radar fusion according to claim 5, characterized in that, The coordinate transformation and electronic fence monitoring module is also used to determine the radius of the virtual electronic fence by calculating the horizontal Euclidean distance between each point in the three-dimensional point cloud data and the center coordinate vector on the global reference coordinate system XY plane, based on the three-dimensional point cloud data of the hazardous area marker and the center coordinate vector, and selecting the maximum value among all calculated horizontal Euclidean distances.

7. The dangerous area intrusion early warning system and method based on vision and radar fusion according to claim 5, characterized in that, The coordinate transformation and electronic fence monitoring module processes the three-dimensional point cloud data of the on-site personnel transformed to the global reference coordinate system. By averaging the coordinate values ​​of all points in the three-dimensional point cloud data, the real-time position coordinate vector is determined.

8. The dangerous area intrusion early warning system and method based on vision and radar fusion according to claim 7, characterized in that, The coordinate transformation and electronic fence monitoring module calculates the horizontal Euclidean distance between the real-time location coordinate vector and the center coordinate vector, and compares the calculated horizontal Euclidean distance with the radius of the virtual electronic fence to determine whether an intrusion has occurred.

9. The dangerous area intrusion early warning system and method based on vision and radar fusion according to claim 1, characterized in that, After receiving the intrusion trigger signal, the voice alarm module retrieves the preset warning voice from the storage unit and executes it in a loop. After receiving the intrusion status cancellation signal, it stops broadcasting.

10. A method for early warning of intrusion into dangerous areas based on the fusion of vision and radar, characterized in that, Includes the following steps: S1. Use an RGB-D depth camera to acquire color images and depth images, process the color images to mark two-dimensional bounding boxes of on-site personnel and hazardous area markers, and combine the two-dimensional bounding boxes with the depth images to generate initial three-dimensional point cloud data in the camera coordinate system. S2. Apply rotation matrix and translation vector to uniformly transform the initial three-dimensional point cloud data in the camera coordinate system to the radar coordinate system; S3. Based on the three-dimensional point cloud data of the hazardous area markers transformed to the radar coordinate system, calculate the center coordinate vector and the radius of the virtual electronic fence, and construct the virtual electronic fence; S4. Based on the three-dimensional point cloud data of the personnel on site transformed to the radar coordinate system, calculate the real-time position coordinate vector of the personnel on site in real time, and compare the real-time position coordinate vector with the virtual electronic fence to determine whether an intrusion has occurred. S5. When step S4 determines that an intrusion has occurred, a high-decibel voice broadcaster is triggered to broadcast a warning voice. When step S4 determines that the intrusion status has been resolved, the broadcasting of the warning voice stops.

Images