Neural network binocular ranging opening fan switch detection positioning method and system

Abstract

The invention provides a neural network binocular distance measurement opening fan switch detection positioning method and system. The system comprises at least one pair of camera arrays installed at the bottom of a building; the data processing unit is used for receiving the data collected by the camera array and executing intelligent analysis, and the analysis process comprises the step of using a target detection algorithm to recognize an opening fan in an opening state; the matching module is used for calculating matching scores of the same opening sash in the two images with different view angles based on geometric constraints and determining the opening sash; the positioning module is used for determining the position coordinates of the opening sash in a three-dimensional space by utilizing a triangulation method and combining camera calibration parameters, and determining the floor and the number of households where the opening sash is located; and the alarm feedback module is used for triggering an alarm and providing position information for a manager when the unlocked opening sash is detected. The invention provides an efficient, accurate and intelligent opening sash opening and closing state detection and positioning scheme, the safety management level of the building is greatly improved, and the technical innovation and development of the building industry are promoted.

Description

Technical Field

[0001] The invention relates to the technical field of building safety, and in particular to a neural network binocular ranging opening fan switch detection and positioning method and system. Background Art

[0002] With the development of intelligent building management and automated inspection technology, higher requirements are placed on the safety and management efficiency of buildings. Traditional manual inspection methods are gradually unable to meet the requirements of modern buildings for safety and management efficiency due to their low efficiency, susceptibility to human factors, and inability to achieve efficient and continuous monitoring. Especially in extreme weather, open doors that are not closed or locked may cause serious safety hazards, such as water leakage, air ingress, and even safety accidents.

[0003] Existing methods based on deep learning and image data can be used to detect unlocked sashes, but they are usually limited to detection and cannot provide precise positioning capabilities; this limitation is particularly obvious in complex building curtain wall facade environments, because different sashes may look very similar, and it is difficult to accurately distinguish their specific locations through images from a single perspective; in addition, some existing solutions rely on manual inspections, which not only increases labor costs, but also reduces the speed and frequency of inspections, thereby affecting the safety management level of the building.

[0004] Deficiencies of existing technology: 1. Low efficiency of manual inspection: Traditional manual inspection methods require a lot of human resources, and the inspection process is easily affected by human factors, such as negligence or fatigue, resulting in inconsistent inspection results or missing important information.

[0005] 2. Lack of precise positioning capability: Although existing deep learning-based methods can detect unlocked opening panels to a certain extent, they cannot provide the specific locations of these opening panels, especially on complex building curtain wall facades with almost identical appearances, which makes subsequent processing difficult.

[0006] 3. Poor adaptability: Current technical solutions are often designed for specific types of opening sashes or specific environments. When faced with different types of curtain wall opening sashes (such as upward rotation, flat opening, etc.) and complex backgrounds, their performance may drop significantly, and consistent high-precision detection effects cannot be guaranteed.

[0007] 4. Insufficient real-time and continuity: Some existing automatic inspection systems cannot achieve true real-time monitoring, or even if they can achieve real-time monitoring, it is difficult to maintain long-term stable operation due to limited computing resources, thus affecting the continuity and timeliness of the inspection.

[0008] Therefore, the prior art has deficiencies and needs further improvement. Summary of the invention

[0009] In view of the problems existing in the prior art, the present invention provides a neural network binocular ranging fan switch detection and positioning method and system.

[0010] To achieve the above object, the specific scheme of the present invention is as follows: The present invention provides a neural network binocular ranging fan switch detection and positioning method and system, the system comprising: at least one pair of camera arrays mounted at the base of the building to capture data including images of the windows; A data processing unit, including a cloud server, is used to receive data collected by the camera array and perform intelligent analysis, wherein the analysis process includes using a target detection algorithm to identify an open fan in an open state; The matching module calculates the matching scores of the same opening fan in two images with different viewing angles based on geometric constraints, and selects the opening fan with the highest score as the successfully matched opening fan; The positioning module uses triangulation combined with camera calibration parameters to determine the position coordinates (X, Y, Z) of the opening door in three-dimensional space, and further determines the floor and number of households on which it is located; The alarm feedback module triggers an alarm when an unlocked open door is detected and provides location information to management personnel.

[0011] Furthermore, the camera array captures images at fixed time intervals and transmits the images to a cloud server of a data processing unit via the Internet of Things for processing.

[0012] Furthermore, the data processing unit also includes a data enhancement module for performing perspective transformation, random cropping, rotation and color adjustment operations on the original image to enrich the training set and improve the generalization ability of the model.

[0013] Furthermore, a deep neural network model is deployed in the data processing unit for selecting or adaptively adjusting according to different curtain wall building site conditions to improve the opening door detection effect.

[0014] Furthermore, the alarm feedback module adopts a real-time monitoring method or a timed monitoring method to regularly summarize the analysis results and send reports, thereby reducing the real-time computing burden and operating cost of the system.

[0015] The present invention also provides a neural network binocular ranging fan switch detection and positioning method. Based on the above system, the method includes the following steps: S1, install at least one pair of camera arrays at the bottom of the glass curtain wall building to ensure that all windows to be monitored are within the monitoring range and take images at set time intervals; S2, uploads the image data to the data processing unit through the Internet of Things, and processes and analyzes the image data in the cloud, including perspective transformation, annotation, data enhancement, and application of target detection algorithm to identify the open fan in the open state; S3, calculate the matching score of the same opening fan between two images with different viewing angles, integrate the confidence score, vertical coordinate offset and intersection over union (IoU), and select the opening fans with successful matching; S4, using the triangulation method of binocular vision to calculate the depth Z of the opening sash, and combining the calibration parameters of the camera to infer the specific position (X, Y, Z) of the opening sash in the world coordinate system, and determine the floor and number of households where it is located; S5, when it is detected that the opening fan is in the open state, an alarm is triggered and feedback including the opening fan position information is sent to the building management personnel.

[0016] Furthermore, in step S2, the processing and analysis further includes preprocessing the labeled image data set to ensure that images taken at different angles can be correctly parsed, thereby improving the accuracy of subsequent detection.

[0017] Furthermore, in step S2, the target detection algorithms include YOLO and Faster R-CNN, and the most suitable model is flexibly selected according to the actual application scenario to generate the Bounding Box and confidence of each target to optimize the detection efficiency and accuracy.

[0018] Furthermore, in step S3, based on the geometric constraints, namely the vertical coordinate offset and the IoU value, the matching score of the opening fan between the two images is calculated, and the opening fan pairs with successful matching are screened out, so as to more accurately confirm the identity of the opening fan.

[0019] Furthermore, in step S5, in addition to real-time feedback, a timed monitoring mode is also set to reduce the real-time computing load of the system and allow the system to process and analyze data during non-peak hours, thereby reducing operating costs.

[0020] The technical solution of the present invention has the following beneficial effects: 1. Improve inspection efficiency and accuracy: Through intelligent detection and positioning technology, automatic monitoring of the switch status of the open fan is achieved, eliminating the need for manual inspections and greatly improving the efficiency of inspections.

[0021] By combining deep learning algorithms with binocular vision technology, not only can the state of the opening door (such as closed or open) be accurately identified, but its position can also be precisely located, ensuring the high accuracy of the detection results.

[0022] 2. Enhance building safety: When an unlocked or abnormally open window is detected, the system will immediately trigger an alarm and send feedback information to the management staff, including the specific floor and number of households, so that the management staff can take prompt measures to prevent potential safety hazards (such as water leakage, air ingress, etc.).

[0023] It realizes all-weather and all-round monitoring, especially in extreme weather conditions, and can effectively prevent safety accidents caused by opening fan problems.

[0024] 3. Reduce operating costs: It reduces the dependence on manpower and reduces the time and economic cost required for manual inspections.

[0025] The real-time and automated nature of the system reduces the need for scheduled maintenance, further reducing long-term operating costs.

[0026] 4. Promote the modernization of construction management: The present invention provides a new solution for the intelligent management and maintenance of buildings, and helps to promote the development of the construction industry towards digitalization and intelligence.

[0027] The data recording and analysis functions provided can help managers better understand building usage and optimize daily management and emergency response strategies.

[0028] 5. Strong adaptability: The system can be flexibly adjusted according to different types of curtain wall opening panels (such as upward rotation, flat opening, etc.) and complex background environments, ensuring consistent and high-precision detection effects in various situations.

[0029] Supports scheduled monitoring mode, allowing the system to perform data processing and analysis during off-peak hours, thereby optimizing resource utilization and reducing the real-time computing burden of the system.

[0030] 6. Technical support and scalability: The installation position of the camera can be flexibly adjusted according to the building layout to ensure that all windows are within the monitoring range.

[0031] Data collection and enhancement methods, neural network models, feedback mechanisms, etc. can all be upgraded or replaced according to actual conditions and technological progress to maintain the system's advancement and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a camera array monitoring schematic diagram of the present invention; Figure 2 is the image perspective transformation of the present invention--the image before the viewing angle transformation; Figure 3 It is the image perspective transformation of the present invention - the image after the viewing angle transformation; Figure 4 It is the neural network of the present invention that automatically recognizes the image of the window; Figure 5 It is the result of the automatic recognition of windows by the neural network of the present invention; Figure 6 It is a system principle block diagram of the present invention; Figure 7 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments; it can be understood that the specific embodiments described herein are only used to explain the present invention rather than to limit the present invention; it should also be noted that, for ease of description, only the parts related to the present invention rather than all are shown in the accompanying drawings.

[0034] Combination Figure 1-Figure 7 As shown, the present invention provides a neural network binocular ranging fan switch detection and positioning system, the system comprising: at least one pair of camera arrays mounted at the base of the building to capture data including images of the windows; A data processing unit, including a cloud server, is used to receive data collected by the camera array and perform intelligent analysis, wherein the analysis process includes using a target detection algorithm to identify an open fan in an open state; The matching module calculates the matching scores of the same opening fan in two images with different viewing angles based on geometric constraints, and selects the opening fan with the highest score as the successfully matched opening fan; The positioning module uses triangulation combined with camera calibration parameters to determine the position coordinates (X, Y, Z) of the opening door in three-dimensional space, and further determines the floor and number of households on which it is located; The alarm feedback module triggers an alarm when an unlocked open door is detected and provides location information to management personnel.

[0035] The camera array captures images at fixed time intervals and transmits the images to a cloud server of a data processing unit via the Internet of Things for processing.

[0036] The data processing unit also includes a data enhancement module for performing perspective transformation, random cropping, rotation and color adjustment operations on the original image to enrich the training set and improve the generalization ability of the model.

[0037] A deep neural network model is deployed in the data processing unit, which is used to select or adaptively adjust according to different curtain wall building site conditions to improve the opening door detection effect.

[0038] The alarm feedback module adopts a real-time monitoring method or a timed monitoring method to regularly summarize the analysis results and send reports, thereby reducing the real-time computing burden and operating cost of the system.

[0039] The present invention also provides a neural network binocular ranging fan switch detection and positioning method. Based on the above system, the method includes the following steps: S1, install at least one pair of camera arrays at the bottom of the glass curtain wall building to ensure that all windows to be monitored are within the monitoring range and take images at set time intervals; S2, uploads the image data to the data processing unit through the Internet of Things, and processes and analyzes the image data in the cloud, including perspective transformation, annotation, data enhancement, and application of target detection algorithm to identify the open fan in the open state; S3, calculate the matching score of the same opening fan between two images with different viewing angles, integrate the confidence score, vertical coordinate offset and intersection over union (IoU), and select the opening fans with successful matching; S4, using the triangulation method of binocular vision to calculate the depth Z of the opening sash, and combining the calibration parameters of the camera to infer the specific position (X, Y, Z) of the opening sash in the world coordinate system, and determine the floor and number of households where it is located; S5, when it is detected that the opening fan is in the open state, an alarm is triggered and feedback including the opening fan position information is sent to the building management personnel.

[0040] In step S2, the processing and analysis further includes preprocessing the labeled image data set to ensure that images taken at different angles can be correctly parsed and improve the accuracy of subsequent detection.

[0041] In step S2, the target detection algorithms include YOLO and Faster R-CNN, and the most suitable model is flexibly selected according to the actual application scenario to generate the Bounding Box and confidence of each target to optimize the detection efficiency and accuracy.

[0042] In step S3, based on the geometric constraints, namely the vertical coordinate offset and the IoU value, the matching score of the opening fan between the two images is calculated, and the opening fan pairs with successful matching are screened out, so as to more accurately confirm the identity of the opening fan.

[0043] In step S5, in addition to real-time feedback, a timed monitoring mode is also set to reduce the real-time computing load of the system and allow the system to process and analyze data during off-peak hours, thereby reducing operating costs.

[0044] The system components of the present invention mainly include the following key modules: Camera array: Installed at the bottom of the building, used to capture data including window images. In order to ensure that all windows are within the monitoring range and to accurately determine the position of the open window, at least 2 pairs of cameras are required to be distributed in an array.

[0045] Data acquisition and preprocessing module: Including camera calibration steps, using Zhang Zhengyou et al.'s method to calibrate the camera array.

[0046] The original images are preprocessed, such as perspective transformation, random cropping, rotation, color adjustment, etc. to enhance the diversity of the dataset.

[0047] Mark the opening fan edge and switch status in the image.

[0048] Data processing unit: including cloud server: used to receive image data uploaded from the camera array through the Internet of Things and perform intelligent analysis on it, including target detection algorithms of deep neural networks (such as YOLO, Faster R-CNN, etc.) to identify the status of the open fan.

[0049] Opening fan matching module: Based on geometric constraints, the matching scores of the same opening fan between two images with different viewing angles are calculated, and the successfully matched opening fan pairs are screened out to determine that they are the same opening fan.

[0050] Binocular positioning module: Using triangulation combined with the camera's calibration parameters, the coordinates of the object in three-dimensional space are inferred based on the matching image pixel coordinates, that is, the specific position of the open door (X, Y, Z), and further determine the floor and number of households where it is located.

[0051] Alarm and feedback mechanism: When the sash is detected to be in the open state, the module will trigger an alarm and send feedback information to the building management personnel, including the floor and number of households where the sash was detected. In addition, the system will save images and videos of the detection results for subsequent verification and processing.

[0052] These components work together to achieve automatic detection and precise positioning of the switch status of open doors in glass curtain wall buildings, as well as real-time or scheduled monitoring and early warning functions.

[0053] The method of the present invention comprises the following key steps, which are used to realize automatic detection and precise positioning of the switch status of the open sash in a glass curtain wall building: 1. Install cameras and monitor area settings: Install a fixed camera array at the bottom of the glass curtain wall building to ensure that all windows are within the monitoring range. At least 2 pairs of cameras are required to be distributed in an array to cover all windows to be monitored.

[0054] The camera captures images at set time intervals and transmits the image data to the cloud server of the data processing unit through the Internet of Things.

[0055] 2. Data collection and preprocessing: Use camera calibration methods (such as Zhang Zhengyou et al.) to calibrate the camera array to ensure the accuracy of subsequent image analysis.

[0056] Collect image data containing various fan states (closed, open), covering different angles, lighting conditions and background environments.

[0057] 3. Preprocess the image, such as perspective transformation, converting oblique images into normal images, and using tools such as Labelme to mark the edge of the open fan and the switch status.

[0058] Data augmentation operations, such as random cropping, rotation, color adjustment, etc., increase the diversity of the training data set.

[0059] 4. Automatic window recognition using neural network: Apply the target detection algorithm (such as YOLO, Faster R-CNN, etc.) to detect the open fans in the open state in the two images and generate the bounding box and confidence score of each target.

[0060] 5. Opening fan matching based on geometric constraints: Calculate the matching scores of the same opening fan in two images with different viewpoints, taking into account factors such as confidence score, vertical coordinate constraints, and Intersection over Union (IoU).

[0061] The successfully matched opening fan pairs are screened out, that is, they are considered to represent the opening fans in the same physical world.

[0062] 6. Opening sash positioning based on dual purpose: The triangulation method of binocular vision is used in combination with the calibration parameters of the camera to infer the coordinates (X, Y, Z) of the object in three-dimensional space based on the matching image pixel coordinates.

[0063] Based on the calculated depth information Z, the height Y of the open door, as well as the floor L or number of households on which it is located are further determined.

[0064] 7. Alarm and feedback mechanism: When the system detects that a window is open, it triggers an alarm and sends feedback information to the building manager, providing the specific location of the open window (floor and number of households).

[0065] The system saves images and videos of the test results for management personnel to verify and subsequently process.

[0066] The above steps together constitute a complete process, from hardware deployment, data collection and preprocessing, to intelligent analysis and final early warning feedback, which realizes the automatic detection and positioning of the fan switch status, and improves the safety and efficiency of building management.

[0067] Working principle of the present invention The present invention describes a method and system for detecting and locating the switch state of an open fan based on a deep neural network and binocular ranging. The working principle of the method and system mainly includes the following key steps: Camera installation and monitoring area settings: At least one pair of camera arrays is installed at the bottom of the glass curtain wall building to ensure that all windows to be monitored are within the monitoring range. These cameras take images at fixed time intervals and transmit the image data to the cloud server of the data processing unit through the Internet of Things for processing.

[0068] Data collection and preprocessing: Use camera calibration methods (such as Zhang Zhengyou et al.) to calibrate the camera array to ensure the accuracy of subsequent image analysis.

[0069] Collect image data containing various fan states (closed, open), covering different angles, lighting conditions and background environments.

[0070] Perform preprocessing operations on the original image, such as perspective transformation, to convert the obliquely shot image into an orthographic one to ensure that images shot at different angles can be correctly interpreted.

[0071] Use annotation tools (such as Labelme) to annotate the edges of the open fan and the switch status in the image, and perform data enhancement through random cropping, rotation, color adjustment, etc. to increase the diversity of the training set.

[0072] Automatic window recognition with neural networks: Apply the target detection algorithm (such as YOLO, Faster R-CNN, etc.) to detect the open fans in the open state in the two images and generate the bounding box and confidence score of each target.

[0073] The detection process not only relies on the image content, but also combines background modeling and motion detection algorithms to accurately determine whether doors and windows are unlocked.

[0074] Opening fan matching based on geometric constraints: Calculate the matching score of the same opening fan between two images with different viewpoints, taking into account factors such as confidence score, vertical coordinate offset, and intersection over union (IoU).

[0075] The successfully matched opening fan pairs are selected, that is, they are considered to represent the opening fans in the same physical world. By traversing each Bounding Box in the two images, the box pair with the highest matching score in each pair of images is retained.

[0076] Opening sash positioning based on dual purpose: The triangulation method of binocular vision is used in combination with the calibration parameters of the camera to infer the coordinates (X, Y, Z) of the object in three-dimensional space based on the matching image pixel coordinates.

[0077] According to the calculated depth information Z, the height Y of the open sash, and the floor L or number of households where the sash is located are further determined. For a pair of cameras placed horizontally, the number of households where the windows are located can be determined.

[0078] Alarm and feedback mechanism: When the system detects that a window is open, it triggers an alarm and sends feedback information to the building manager, providing the specific location of the open window (floor and number of households).

[0079] The system saves images and videos of the test results for management personnel to verify and follow up, ensuring that any abnormal situations can be responded to in a timely manner.

[0080] Example 1 1. Application Scenarios The invention is applicable to any glass curtain wall building. Through a fixed camera installed at the bottom of the building, the status of each window can be monitored in real time. The system recognizes the outline of the window based on a deep neural network, and combines background modeling and motion detection algorithms to accurately determine whether the doors and windows are unlocked.

[0081] 2. Process (1) Install cameras and set up monitoring areas: Install fixed cameras at the bottom of glass curtain wall buildings to ensure that the monitoring range covers all windows. Figure 1 As shown in the figure, in order to determine the position of the open door, the cameras need to be arrayed, that is, at least 2 pairs of cameras are required. The cameras take images at fixed time intervals and then transmit them to the cloud through the Internet of Things for intelligent analysis.

[0082] (2) Data collection and processing: Before collecting data, the camera array is calibrated using the camera calibration method of Zhang Zhengyou et al. Then, image data containing various curtain wall opening states (closed and open) are collected, covering different angles, lighting conditions, and background environments. Labeling tools such as Labelme are used to annotate the dataset, marking the edges of the opening sashes in the open state and the switch state. The dataset needs to cover various types of curtain wall opening sashes (such as upward rotation, horizontal opening, etc.) and complex backgrounds. Data enhancement is then performed through random cropping, rotation, color adjustment, and other operations.

[0083] In addition, the image needs to be perspective transformed before recognition, such as Figure 2-3 As shown, the obliquely shot image is converted into a normal image.

[0084]

[0085] (3) Automatic window recognition using neural network: First, use the target detection algorithm (such as YOLO, Faster R-CNN, etc.) to detect the open fans in the two images and generate the bounding box and confidence of each target. Taking a pair of cameras placed vertically as an example, the detection output result is: Image 1:

[0086] Image 2:

[0087] in, and Represents the first i and j A target box, that is, the open state open fan detected by the neural network. and Represents the pixel coordinates of the four corner points of the bounding box, , are confidence scores respectively. Figure 4-5 The following is the detection result of one of the cameras.

[0088]

[0089] (4) Opening fan matching based on geometric constraints: Let’s take a pair of cameras placed vertically as an example. In binocular vision, the two cameras are aligned vertically, and the parallax mainly occurs in the vertical coordinate direction (assuming that the stereo camera calibration is complete). Therefore, the horizontal coordinates of the same object in the two images are almost the same, but the vertical coordinates will be offset.

[0090] Taking the vertical coordinate constraint as a filtering condition, the center coordinate of the bounding box in the image is , set the vertical constraint condition to .

[0091] If the positions and shapes of the detected opening fans in the two images are similar, their IoU is calculated and used as another filtering criterion.

[0092] Considering the confidence score, vertical coordinate constraint and IoU constraint, the matching score is set as:

[0093] in, w 1 , w 2 , w 3 It is the weight coefficient, which is adjusted according to the actual situation.

[0094] Finally, traverse each bounding box in the two images and retain the box pair with the highest matching score in each pair of images. This box pair is the same opening fan.

[0095] (5) Opening door positioning based on dual purpose: After the matching is completed, the positioning of the opening fan mainly relies on the triangulation method of binocular vision, that is, the coordinates of the object in three-dimensional space are inferred based on the matched image pixel coordinates and the calibration parameters of the camera, and then the height of the opening fan is obtained from it. First, it is necessary to obtain the depth Z from the camera to the opening fan, and calculate the depth Z through disparity, that is, the horizontal offset between the two images is determined by the center point of the bounding box of the same opening fan image taken by two cameras. The calculation formula is as follows:

[0096]

[0097] In the formula, and They are the center points of the bounding box of the same opening fan in the two images. B is the baseline distance between cameras (horizontal distance or vertical distance), f is the focal length of the camera.

[0098] Get the depth distance of the opening fan Z Then, the position of the target in the world coordinate system is calculated through the camera's intrinsic parameter matrix and depth information ( X, Y, Z ). The conversion formula from image coordinates to camera coordinates is:

[0099]

[0100] In the formula, x and y is the coordinate of any pixel in the bounding box, c x and c y is the pixel coordinate of the camera's optical center, f x andf y is the focal length of the camera. Taking a pair of cameras placed vertically as an example, the height Y of the opening sash can be determined by the above formula. Furthermore, the floor L can be calculated based on the floor height H:

[0101] Similarly, based on a pair of horizontally placed cameras, the number of households where the windows are located can be determined.

[0102] (6) Alarm and feedback mechanism: When a window is detected to be open, the system will immediately trigger an alarm and send feedback information to the building management personnel, including the floor and number of households where the window is open. The system can save images and videos of the detection results for subsequent verification and processing by the management personnel.

[0103] The present invention solves the problems of low efficiency, high cost and poor accuracy of existing manual inspections through intelligent detection and positioning technology. Compared with traditional methods, the present invention significantly improves inspection efficiency and safety, reduces labor costs, can accurately locate the position of the open door in the open state, realizes the digitalization and intelligence of building management, and promotes the modernization of building maintenance and management.

[0104] The key point of the present invention is to use a method combining deep neural network and binocular positioning to realize automatic detection and positioning of the open sash status in glass curtain wall buildings. The protection point is to detect and locate the windows of the glass curtain wall through neural network and binocular vision. First, the same open sash in two images is detected using image coordinates, confidence scores and intersection over union (IoU) constraints, and then the building and household position of the open sash is located through binocular positioning. Finally, an alarm is triggered to feedback the open sash information to the user.

[0105] Example 2: Vertically placed binocular camera array detects and locates the opening door System deployment Camera installation: A pair of vertically aligned cameras are installed at the bottom of a high-rise building to ensure that all windows are within the monitoring range. The pair of cameras capture images at fixed time intervals (for example, every 5 minutes) and transmit the image data to the cloud server of the data processing unit through the Internet of Things.

[0106] Data collection and preprocessing: The pair of cameras was calibrated using Zhang Zhengyou’s camera calibration method to ensure the accuracy of subsequent image analysis.

[0107] Collect image data containing various fan states (closed, open), covering different angles, lighting conditions and background environments.

[0108] A perspective transformation is performed on the original image to convert the obliquely shot image into an orthographic one, ensuring that images shot at different angles can be correctly interpreted.

[0109] Detection and matching Neural network recognition: Apply the YOLOv5 target detection algorithm to detect the open fans in the two images and generate the bounding box and confidence score of each target.

[0110] Opening fan matching based on geometric constraints: Calculate the matching score of the same opening fan between two images with different viewpoints, taking into account the confidence score, vertical coordinate offset and intersection over union (IoU). Filter out the opening fan pairs that have been successfully matched, that is, they are considered to represent the opening fans in the same physical world.

[0111] Positioning and feedback Binocular vision positioning: Using triangulation combined with the camera's calibration parameters, the coordinates of the object in three-dimensional space (X, Y, Z) are calculated based on the matching image pixel coordinates to determine the height Y of the open door and the floor L on which it is located.

[0112] Alarm and feedback mechanism: When the system detects that a window is open, it triggers an alarm and sends feedback information to the building manager, providing the specific location of the open window (floor and number of households). The system saves images and videos of the detection results for management personnel to verify and follow up.

[0113] Example 3: Horizontally placed binocular camera array detects and locates the opening door System deployment Camera installation: A pair of horizontally aligned cameras are installed at different locations on the same floor to ensure that all windows on the floor are within the monitoring range. The pair of cameras also capture images at fixed time intervals and upload them to the cloud server of the data processing unit through the Internet of Things.

[0114] Data collection and preprocessing: The pair of cameras was calibrated using Zhang Zhengyou’s camera calibration method to ensure the accuracy of image analysis.

[0115] Collect image data containing various fan states (closed, open), covering different angles, lighting conditions and background environments.

[0116] Perform preprocessing operations on the original image, such as perspective transformation, to ensure that the image can be parsed correctly.

[0117] Detection and matching Neural network recognition: The Faster R-CNN target detection algorithm is applied to detect the open fans in the two images and generate the bounding box and confidence score of each target.

[0118] Opening fan matching based on geometric constraints: Calculate the matching score of the same opening fan between two images with different viewpoints, taking into account the confidence score, horizontal coordinate offset and intersection over union (IoU). Filter out the opening fan pairs that have been successfully matched, that is, they are considered to represent the opening fans in the same physical world.

[0119] Positioning and feedback Binocular vision positioning: Using triangulation combined with the camera's calibration parameters, the coordinates of the object in three-dimensional space (X, Y, Z) are calculated based on the matching image pixel coordinates to determine the horizontal position X of the open door and the number of households it is located in.

[0120] Alarm and feedback mechanism: When the system detects that a window is open, it triggers an alarm and sends feedback information to the building manager, providing the specific location of the open window (floor and number of households). The system saves images and videos of the detection results for management personnel to verify and follow up.

[0121] These two embodiments demonstrate how the technology of the present invention can flexibly adjust the arrangement of cameras according to the needs of different scenarios, thereby achieving efficient and accurate detection and positioning of the switch status of the open sash. Whether the camera array is placed vertically or horizontally, it can ensure that all windows to be monitored are within the monitoring range and can accurately provide the position information of the open sash, thereby improving the safety management level of the building.

[0122] The above description is only a preferred embodiment of the present invention, and does not limit the scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the protection scope of the present invention.

Claims

1. Neural network binocular ranging fan switch detection and positioning system, characterized in that: The system includes: at least one pair of camera arrays mounted at the base of the building to capture data including images of the windows; A data processing unit, including a cloud server, is used to receive data collected by the camera array and perform intelligent analysis, wherein the analysis process includes using a target detection algorithm to identify an open fan in an open state; The matching module calculates the matching scores of the same opening fan in two images with different viewing angles based on geometric constraints, and selects the opening fan with the highest score as the successfully matched opening fan; The positioning module uses triangulation combined with camera calibration parameters to determine the position coordinates of the opening door in three-dimensional space, and further determines the floor and number of households on which it is located; The alarm feedback module triggers an alarm when an unlocked open door is detected and provides location information to management personnel.

2. The system according to claim 1, characterized in that: The camera array captures images at fixed time intervals and transmits the images to a cloud server of a data processing unit via the Internet of Things for processing.

3. The system according to claim 1, characterized in that: The data processing unit also includes a data enhancement module for performing perspective transformation, random cropping, rotation and color adjustment operations on the original image to enrich the training set and improve the generalization ability of the model.

4. The system according to claim 1, characterized in that: A deep neural network model is deployed in the data processing unit, which is used to select or adaptively adjust according to different curtain wall building site conditions to improve the opening door detection effect.

5. The system according to claim 1, characterized in that: The alarm feedback module adopts a real-time monitoring method or a timed monitoring method to regularly summarize the analysis results and send reports, thereby reducing the real-time computing burden and operating cost of the system.

6. A neural network binocular ranging method for detecting and locating a fan switch, based on the system described in any one of claims 1 to 5, characterized in that: The method comprises the following steps: S1, install at least one pair of camera arrays at the bottom of the glass curtain wall building to ensure that all windows to be monitored are within the monitoring range and take images at set time intervals; S2, uploads the image data to the data processing unit through the Internet of Things, and processes and analyzes the image data in the cloud, including perspective transformation, annotation, data enhancement, and application of target detection algorithm to identify the open fan in the open state; S3, calculate the matching score of the same opening fan between two images with different viewing angles, integrate the confidence score, vertical coordinate offset and intersection over union (IoU), and select the opening fans with successful matching; S4, using the triangulation method of binocular vision to calculate the depth Z of the opening sash, and combining the calibration parameters of the camera to infer the specific position (X, Y, Z) of the opening sash in the world coordinate system, and determine the floor and number of households where it is located; S5, when it is detected that the opening fan is in the open state, an alarm is triggered and feedback including the opening fan position information is sent to the building management personnel.

7. The method according to claim 6, characterized in that: In step S2, the processing and analysis further includes preprocessing the labeled image data set to ensure that images taken at different angles can be correctly parsed and improve the accuracy of subsequent detection.

8. The method according to claim 6, characterized in that: In step S2, the target detection algorithms include YOLO and Faster R-CNN, and the most suitable model is flexibly selected according to the actual application scenario to generate the Bounding Box and confidence of each target to optimize the detection efficiency and accuracy.

9. The method according to claim 6, characterized in that: In step S3, based on the geometric constraints, namely the vertical coordinate offset and the IoU value, the matching score of the opening fan between the two images is calculated, and the opening fan pairs with successful matching are screened out, so as to more accurately confirm the identity of the opening fan.

10. The method according to claim 6, characterized in that: In step S5, in addition to real-time feedback, a timed monitoring mode is also set to reduce the real-time computing load of the system and allow the system to process and analyze data during off-peak hours, thereby reducing operating costs.

Images