Building construction site edge hole identification method based on visual deep learning

Abstract

The invention provides a building construction site edge hole recognition method based on visual deep learning, relates to the field of image recognition and intelligent building equipment, and solves the limitation problems of insufficient research of edge hole recognition functions, lack of means and the like. The method comprises the following steps: deploying multiple types of sensors on an intelligent trolley, and obtaining two-dimensional image data, infrared thermal image data and three-dimensional point cloud data of a building construction site; features of the three types of data are extracted respectively, feature fusion is carried out, and a comprehensive feature vector is constructed; using the training data set to carry out limb hole recognition training on the visual deep learning algorithm model to obtain a limb hole recognition model; the comprehensive feature vector is input into a limb hole recognition model, and the model outputs a limb hole recognition result in the moving process of the intelligent trolley; according to the identified different types of limb holes and the distribution information thereof, the control decision of the intelligent trolley is changed; according to the method, the edge hole recognition effect of the intelligent trolley in a dynamic and complex environment can be improved.

Description

Technical Field

[0001] The present invention relates to the fields of image recognition and intelligent building equipment, and is applied to the navigation process of a portable intelligent vehicle. Specifically, it relates to a method for identifying the edges and openings at the construction site of a building based on visual deep learning. Background Art

[0002] In an unfinished construction site, the existence of edges and openings poses a major safety hazard to construction workers and equipment. When a transportation intelligent vehicle travels in such an environment, if it cannot effectively identify and avoid unprotected edges and openings, it may lead to serious safety accidents and cause losses of human and material resources. Currently, the existing technologies mainly focus on the design of edge and opening protection devices, while the research on how to effectively detect and identify edges and openings in the intelligent vehicle navigation system is relatively scarce.

[0003] The construction site environment is complex and changeable, which poses challenges to the perception ability of the intelligent vehicle. Adverse factors such as light changes, dust, and noise will interfere with the operation of the sensor and reduce its detection accuracy. In order to ensure that the intelligent vehicle can maintain a high recognition accuracy in such a harsh environment, effective measures must be taken to enhance the robustness and adaptability of the system. This not only involves the selection and optimization of hardware, but also includes the design and improvement of software algorithms.

[0004] The safety hazards at the construction site need to be discovered and feedback processed in a timely manner. Therefore, the recognition system of the intelligent vehicle must have the ability to respond quickly and be able to provide information on the position and state of the opening in real time. This is crucial for ensuring construction safety. This means that the algorithms used must not only be able to efficiently process a large amount of data, but also, on the basis of ensuring accuracy, minimize the calculation time and resource consumption to meet the real-time requirements.

[0005] In addition, considering that the research and development, production, and maintenance costs of intelligent vehicles are relatively high, construction units often weigh the cost-benefit when choosing whether to adopt this technology. Therefore, on the premise of ensuring the recognition accuracy, how to reduce costs through technological innovation and design optimization is one of the key factors to promote the popularization and application of this technology. At the same time, the application of any new technology must comply with relevant laws, regulations, and industry standards. Ensuring that the operation of the intelligent vehicle in identifying edges and openings complies with these regulations is also an important point that cannot be ignored.

[0006] In summary, although intelligent vehicles can play advantageous roles such as material transportation and environmental detection in construction, and their function of identifying edge and opening areas in construction sites will have broad application prospects, to achieve this, it is necessary to overcome challenges in multiple aspects, including poor environmental adaptability, high real-time requirements, high algorithm complexity, cost issues, and compliance with laws and regulations. To address these problems, it is necessary to conduct in-depth technological innovation, optimize existing algorithms, explore more cost-effective identification solutions, and establish a complete maintenance and update mechanism to ensure the application of intelligent vehicles in a safe and efficient environment. Summary of the Invention

[0007] The purpose of the present invention is to solve the problems of insufficient research on the function of identifying edge and opening areas in construction sites, lack of identification and response means, and unsatisfactory efficiency and cost when an intelligent vehicle is navigating. Therefore, a method for identifying edge and opening areas in construction sites based on visual deep learning is proposed. Traditional identification methods only perform contrast detection based on sensor data, such as size boundaries and length and depth, without utilizing the advantages of computer image recognition and deep learning in this regard. The present invention combines visual data on the basis of traditional sensors and analyzes and identifies them through deep learning algorithms, improving the identification effect of edge and opening areas by the intelligent vehicle in a dynamic and complex environment.

[0008] The present invention adopts the following technical solutions to achieve the purpose:

[0009] A method for identifying edge and opening areas in construction sites based on visual deep learning, the method comprising the following steps:

[0010] S1. Deploy multiple types of sensors on the intelligent vehicle. As the intelligent vehicle moves while performing transportation tasks, obtain two-dimensional image data, infrared thermal image data, and three-dimensional point cloud data of the construction site.

[0011] S2. After respectively extracting the corresponding features of the two-dimensional image data, infrared thermal image data, and three-dimensional point cloud data, perform feature fusion to construct a comprehensive feature vector.

[0012] S3. Obtain a pre-trained visual deep learning algorithm model, and after performing edge and opening area identification training on the basis of a pre-constructed training data set, obtain an edge and opening area identification model.

[0013] S4. Input the constructed comprehensive feature vector into the trained edge and opening area identification model, and the model outputs the edge and opening area identification results corresponding to the movement process of the intelligent vehicle.

[0014] S5. The intelligent vehicle navigation control system receives and, based on the different types of edge and opening areas and their distribution information in the edge and opening area identification results, changes its own control decision and adjusts the navigation movement path.

[0015] Specifically, in step S1, an industrial camera, an infrared sensor, and a lidar are deployed on the intelligent vehicle; the industrial camera is used to obtain two-dimensional image data, the infrared sensor is used to obtain infrared thermal image data, and the lidar is used to obtain three-dimensional point cloud data; the industrial camera, the infrared sensor, and the lidar all adopt timestamp synchronization technology to collect matching two-dimensional image data, infrared thermal image data, and three-dimensional point cloud data at the same timestamp.

[0016] Preferably, multiple industrial cameras are deployed on the intelligent vehicle; the horizontal field of view angles of the multiple industrial cameras cover the range of 360 degrees in the peripheral direction of the intelligent vehicle after being superimposed, and there is an overlapping interval between the horizontal field of view angles of two adjacent industrial cameras; the distance between the ground position corresponding to the lower limit of the vertical field of view angle of each industrial camera and the ground position where the intelligent vehicle itself is located is less than a preset distance threshold.

[0017] Preferably, multiple infrared sensors are deployed on the intelligent vehicle, and the number of infrared sensors is the same as that of the industrial cameras and the deployment positions match one by one; each infrared sensor obtains the infrared thermal image data corresponding to the two-dimensional image data of its matching industrial camera.

[0018] Preferably, in step S2, the two-dimensional image data, infrared thermal image data, and three-dimensional point cloud data obtained by multiple types of sensors are transmitted to the construction site monitoring center in real time, and the cloud server of the construction site monitoring center performs data preprocessing operations, and then extracts corresponding features after completing the data preprocessing operations; the data preprocessing operations are as follows:

[0019] For the two-dimensional image data and the infrared thermal image data, denoising, enhancement, and correction processing are all performed; for the continuous two-dimensional image data at different angles, after completing the correction processing, a stitching operation is performed to convert the corresponding images into the same image coordinate system; for the three-dimensional point cloud data, filtering, segmentation, and registration processing are performed.

[0020] Further, after completing the data preprocessing operations, feature data including color, texture, and edges are extracted from the two-dimensional image data, feature data of temperature distribution are extracted from the infrared thermal image data, and feature data of spatial position and shape are extracted from the three-dimensional point cloud data; the extraction of the feature data adopts general image processing methods or deep learning-based methods; the three types of extracted feature data are subjected to feature fusion, and the feature fusion methods include early fusion, intermediate fusion, and late fusion. After the fusion is completed, a comprehensive feature vector is constructed.

[0021] Further, in step S3, the pre-trained visual deep learning algorithm model includes any one of a convolutional neural network model, a recurrent neural network model, or a long short-term memory network model; the pre-constructed training data set includes training feature vectors of the same type as the comprehensive feature vectors constructed in step S2; when training the visual deep learning algorithm model, a combination of template matching, feature matching, and classifier detection is adopted, and based on the training feature vectors, the visual deep learning algorithm model is made to identify the edge and opening; the training strategy adopted includes a strategy that combines region proposal and classifier.

[0022] Specifically, in step S4, the trained edge and opening recognition model is deployed in the cloud server of the construction site monitoring center. The cloud server inputs the comprehensive feature vectors into the edge and opening recognition model. The processing method of the edge and opening recognition model for the comprehensive feature vectors is the same as the method for processing the training feature vectors during training. After identifying the edge and opening, the model outputs the edge and opening recognition result; in the edge and opening recognition result, the types of edge and opening include one or a combination of multiple types among floor edge, ramp edge, stair opening, elevator opening, and reserved opening.

[0023] Preferably, in step S3, when training the visual deep learning algorithm model, the training feature vectors also correspond to two types of situations where there are protective measures and where there are no protective measures for the edge and opening.

[0024] In step S4, when the trained edge and opening recognition model identifies the edge and opening based on the comprehensive feature vectors, information on whether there are corresponding protective measures for one or more types of edge and opening is added to the output edge and opening recognition result.

[0025] Further, in step S5, one or more types of edge and opening in the edge and opening recognition result output by the model are visually presented in the three-dimensional environment model of the construction site monitoring center; the three-dimensional environment model is constructed based on two-dimensional image data and three-dimensional point cloud data.

[0026] When the edge and opening recognition result indicates that there are protective measures for the corresponding type of edge and opening, the intelligent vehicle navigation control system moves according to the corresponding edge and opening distribution information in the three-dimensional environment model along the preset navigation path, maintaining a first safety distance threshold from the edge and opening.

[0027] When the edge and opening recognition result indicates that there are no protective measures for the corresponding type of edge and opening, the construction site monitoring center issues an external alarm. At the same time, the intelligent vehicle navigation control system modifies the preset navigation path according to the corresponding edge and opening distribution information in the three-dimensional environment model and moves along the modified navigation path, maintaining a second safety distance threshold from the edge and opening; the second safety distance threshold is greater than the first safety distance threshold.

[0028] In summary, due to the adoption of this technical solution, the beneficial effects of the present invention are as follows:

[0029] The method of the present invention integrates an industrial camera, an infrared sensor, and a lidar in image acquisition, significantly improving the accuracy and reliability of the identification of edge and opening areas at construction sites. The industrial camera captures subtle features with its high resolution and sensitivity, the infrared sensor senses the temperature distribution, and the lidar provides precise distance measurement. The combination of the three enables high-precision detection of the position and size of the openings, serving as high-quality input data for the image recognition algorithm. This method supports round-the-clock operation. Through the complementarity of different sensing data, the navigation process is less restricted by lighting conditions. The combination of the two-dimensional image of the industrial camera and the three-dimensional point cloud of the lidar enables high-quality three-dimensional reconstruction of the edge and opening areas, providing data support for the feedback and warning of the intelligent vehicle, and ultimately improving the identification and detection ability of the edge and opening areas.

[0030] The method of the present invention applies a deep learning image processing algorithm such as a convolutional neural network, which can effectively improve the recognition accuracy of the intelligent vehicle for the edge and opening areas at construction sites. By training a large number of data sets, this algorithm can effectively learn and extract complex features of the edge and opening areas, achieving high-accuracy recognition. On the basis of multi-sensor image acquisition, the image processing technology further optimizes the input data and enhances its performance, enabling the method of the present invention to exhibit good robustness. When facing perspective changes or interference environments such as occlusion at the construction site and the movement of the intelligent vehicle, it can still maintain a high recognition accuracy. Through the real-time nature of the identification and detection of the edge and opening areas, while meeting the navigation of the intelligent vehicle, it also facilitates the management personnel to timely grasp and handle safety hazards of the edge and opening areas at the construction site.

[0031] The method of the present invention also improves the identification and decision-making of the intelligent vehicle for the edge and opening areas. Based on the image recognition results, it can optimize the overall optimization and resource allocation for the intelligent vehicle itself or the deployment plan of the intelligent vehicle at the construction site. For example, different warning threshold rules and vehicle navigation plans can be determined based on whether the identified edge and opening areas are protected, etc., so as to meet the specific needs of the movement of the intelligent vehicle at the construction site. This method greatly reduces the labor cost required for manual response to different conditions and improves the work efficiency of the intelligent vehicle in the scenario of transporting materials. Combining the remote monitoring management and data sharing and collaboration functions of the intelligent vehicle, the high-accuracy identification data of the edge and opening areas can be remotely transmitted and online monitored. The recognition results can assist the decision-making of the management level and have a clear traceable path, ultimately improving the intelligent process of safety management at the construction site, facilitating the transportation of materials while reducing the risk of accidents at the edge and opening areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the overall process of the method of the present invention. Detailed implementation manners

[0033] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations.

[0034] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0035] Embodiment

[0036] A method for identifying edges and openings at a construction site based on visual deep learning, Figure 1 The overall process of the method is shown and can be referred to synchronously. The steps of the method can be summarized as follows:

[0037] S1. Deploy multiple types of sensors on the intelligent trolley. As the intelligent trolley moves during the transportation task, obtain two-dimensional image data, infrared thermal image data, and three-dimensional point cloud data of the construction site;

[0038] S2. After respectively extracting the corresponding features of the two-dimensional image data, infrared thermal image data, and three-dimensional point cloud data, perform feature fusion to construct a comprehensive feature vector;

[0039] S3. Obtain a pre-trained visual deep learning algorithm model, and perform edge and opening recognition training on the basis of a pre-constructed training data set to obtain an edge and opening recognition model;

[0040] S4. Input the constructed comprehensive feature vector into the trained edge and opening recognition model, and the model outputs the edge and opening recognition results corresponding to the movement process of the intelligent trolley;

[0041] S5. The intelligent trolley navigation control system receives and changes its own control decision and adjusts the navigation movement path according to different types of edges and openings and their distribution information in the edge and opening recognition results.

[0042] This embodiment will introduce the details of each step in the order of the above steps. This embodiment is a solution to a series of problems brought about by the existence of edge openings during the navigation of an intelligent vehicle used at a construction site. The primary solution is to efficiently and accurately identify the edge openings, and then corresponding safety measures can be taken, such as alerting the management personnel and guiding the vehicle to avoid them.

[0043] The intelligent vehicle is about to be popularized and applied at the construction site. The biggest feature is its intelligent and automated transportation of on-site materials. Therefore, corresponding navigation and recognition control technologies are required. The intelligent vehicle in this embodiment will collect and monitor real-time images of its surrounding environment by carrying a variety of sensors, and then identify the edge openings through image processing and recognition technologies. Based on the recognition results, corresponding safety measures can be taken, such as issuing warnings and avoiding.

[0044] First, in step S1 of the method of this embodiment, an industrial camera, an infrared sensor, and a lidar are deployed on the intelligent vehicle. The industrial camera is used to obtain two-dimensional image data. It can be selected as an industrial camera with high resolution, high sensitivity, and wide dynamic range, and at the same time, relevant functions such as automatic exposure and white balance are configured to capture clear images of the construction site and form two-dimensional image data.

[0045] The infrared sensor is used to obtain infrared thermal image data. An infrared sensor with high spatial resolution and wide temperature measurement range can be selected to obtain the temperature distribution information of the construction site and form infrared thermal image data. The lidar is used to obtain three-dimensional point cloud data. A lidar with high measurement accuracy and large measurement range can be selected to make the three-dimensional point cloud data cover a certain range of space in the area where the intelligent vehicle is located.

[0046] In this embodiment, the industrial camera, the infrared sensor, and the lidar all adopt timestamp synchronization technology to collect matching two-dimensional image data, infrared thermal image data, and three-dimensional point cloud data at the same timestamp. Here, it can be achieved by means of synchronous trigger signals in terms of hardware, and in terms of software, existing mature timestamp calibration tools can be used, such as Precision Time Protocol, PTP protocol software, and other tools.

[0047] In this step, as a preference of this embodiment, multiple industrial cameras are deployed on the intelligent vehicle; the horizontal field of view angles of the multiple industrial cameras cover the range of 360 degrees in the circumferential direction of the intelligent vehicle after being superimposed, and there is an overlapping interval between the horizontal field of view angles of two adjacent industrial cameras; the distance between the ground position corresponding to the lower limit of the vertical field of view angle of each industrial camera and the ground position where the intelligent vehicle itself is located is less than a preset distance threshold. This design ensures the integrity of the two-dimensional image data collection and is conducive to the complete three-dimensional reconstruction process of the construction site environment.

[0048] Preferably, in the present embodiment, a plurality of infrared sensors are deployed on the intelligent trolley. The number of infrared sensors is the same as that of industrial cameras and their deployment positions match one by one. Each infrared sensor acquires the infrared thermal image data corresponding to the two-dimensional image data of its matched industrial camera. The matching characteristics of the two-dimensional image data and the infrared thermal image data ensure image registration and can be used as auxiliary means for each other to ensure effective information of the construction site can be obtained under different lighting conditions.

[0049] In step S2, the two-dimensional image data, infrared thermal image data, and three-dimensional point cloud data acquired by multiple types of sensors are transmitted to the construction site monitoring center in real time. The cloud server of the construction site monitoring center performs data preprocessing operations, and corresponding feature extraction is carried out after the data preprocessing operations are completed. The data preprocessing operations reduce the background interference of the corresponding data in various ways as follows:

[0050] Denoising, enhancement, and correction processing are performed on the two-dimensional image data. Sharpening, contrast adjustment, etc. may also be included during the denoising and enhancement processes, aiming to improve the image quality. The same denoising, enhancement, and correction processing are performed on the infrared thermal image data to enhance the presentation quality of its temperature distribution. Moreover, for continuous two-dimensional image data at different angles, stitching operations are performed after the correction processing to eliminate the distortion of the industrial camera lens, thereby converting the corresponding images into the same image coordinate system. For the three-dimensional point cloud data, filtering, segmentation, and registration processing are performed to effectively improve the usability of the three-dimensional point cloud data.

[0051] After the data preprocessing operations are completed in this embodiment, feature data including color, texture, and edges are extracted from the two-dimensional image data, feature data of the temperature distribution is extracted from the infrared thermal image data, and feature data of the spatial position and shape is extracted from the three-dimensional point cloud data. General image processing methods such as SIFT, SURF, etc. can be used for feature extraction; or feature extraction can be implemented using deep learning-based methods, such as using the convolutional neural network CNN, etc. The features extracted and fused in the subsequent process can be used for the identification and positioning of the edge openings.

[0052] In this embodiment, the three types of extracted feature data are subjected to feature fusion. The ways of feature fusion include early fusion, mid-term fusion, and late fusion. After the fusion is completed, a comprehensive feature vector is constructed. Each of the three fusion methods has its own characteristics and can be selected for processing according to the different main environmental characteristics of the construction site as needed. In early fusion, data from different modalities are combined at the very beginning, and then the fused data is processed. Its advantage is that the target model will learn the potential relationships between different modalities, and its disadvantage is that it may lead to the curse of dimensionality, that is, the feature space is too large, increasing the computational complexity and introducing noise.

[0053] Mid-term fusion in the field of image recognition refers to the fusion that occurs after feature extraction but before decision-making. The data of each modality first extracts features through their respective network layers, and then these features are combined in a certain way to continue subsequent processing; it can be regarded as a compromise between early fusion and late fusion, attempting to find a balance between information retention and computational efficiency.

[0054] Late fusion is the integration of results after each modality has been independently processed (such as classification or other preliminary prediction tasks); the advantages are that it can maintain the information integrity of each modality, reduce the influence of noise, and is easy to implement and debug; the disadvantage is that it ignores the interaction between modalities, which may lead to the loss of important cross-modal information. Generally speaking, the comprehensive feature vector obtained by the method of this embodiment through the feature fusion of three types of data will provide rich information for the subsequent recognition of edge and opening.

[0055] In step S3, the pre-trained visual deep learning algorithm model includes any one of the convolutional neural network model CNN, the recurrent neural network model RNN, or the long short-term memory network model LSTM. By training these models, different types of edge and opening in the construction site environment can be recognized in real time.

[0056] When training the model in this embodiment, the pre-constructed training dataset includes training feature vectors of the same type as the comprehensive feature vectors constructed in step S2. During the training process, a combination of template matching, feature matching, and classifier detection is adopted. Based on the training feature vectors, the visual deep learning algorithm model performs the recognition of edge and opening, and the specific training strategy can include the strategy of combining region proposal and classifier. The final finished model can be in the form of Faster R-CNN and YOLO, etc., which are also relatively mature in the field of image recognition.

[0057] In step S4, the trained edge and opening recognition model is deployed in the cloud server of the construction site monitoring center. The cloud server inputs the comprehensive feature vector into the edge and opening recognition model. The edge and opening recognition model processes the comprehensive feature vector in the same way as it processes the training feature vectors during training. After recognizing the edge and opening, the model outputs the edge and opening recognition result.

[0058] In this embodiment, the types of edge and opening included in the typical edge and opening recognition results are: floor edge, ramp edge, stair opening, elevator opening, and reserved opening; during the movement of the intelligent vehicle, it may detect and recognize one or more combinations of the above-mentioned edge and opening.

[0059] Here, as a preference of this embodiment, in step S3, when training the visual deep learning algorithm model, the training feature vectors also correspond to two types of edge and opening with protective measures and without protective measures; in step S4, when the trained edge and opening recognition model recognizes the edge and opening based on the comprehensive feature vectors, information on whether one or more types of edge and opening have corresponding protective measures is added to the output edge and opening recognition result.

[0060] The reason for this design is that for the dynamic changes of the edge and opening on the construction site, there may be two different states of temporary removal of protection and rectification and installation, but in terms of the manifestation form, it is only divided into whether there is protection. The edge and opening without protection are major safety hazards on the construction site of a building. Therefore, it is necessary to accurately identify and take corresponding measures in a timely manner.

[0061] In step S5, one or more types of edge and opening in the edge and opening recognition result output by the model are visually presented in the three-dimensional environment model of the construction site monitoring center; the three-dimensional environment model is constructed based on two-dimensional image data and three-dimensional point cloud data;

[0062] When the edge and opening recognition result indicates that the corresponding type of edge and opening has protective measures, the intelligent vehicle navigation control system moves according to the corresponding edge and opening distribution information in the three-dimensional environment model and maintains the first safety distance threshold from the edge and opening according to the preset navigation path;

[0063] When the edge and opening recognition result indicates that the corresponding type of edge and opening lacks protective measures, the construction site monitoring center gives an external alarm. At the same time, the intelligent vehicle navigation control system modifies the preset navigation path according to the corresponding edge and opening distribution information in the three-dimensional environment model and moves according to the modified navigation path, maintaining the second safety distance threshold from the edge and opening; the second safety distance threshold is greater than the first safety distance threshold.

[0064] In this embodiment, the decision rule strategy of the intelligent vehicle itself will be predefined based on the safety specifications and actual requirements of the construction site, but at the same time, it covers the recognition and processing of the edge and opening, that is, it can involve dangerous situations related to the edge and opening, such as too large an opening, deformation of the structure around the opening, and people approaching the opening, etc., and all can be alarmed through the construction site monitoring center.

[0065] The intelligent vehicle navigation control system can, based on the above decision rules, preset corresponding control strategies, enabling the intelligent vehicle to perform corresponding operations such as on-site acoustic and light warnings, automatic evacuation from dangerous areas, and behavior recording through its own devices and equipment. In addition, in addition to the deployment of the edge and opening recognition model, the construction site monitoring center can also additionally build an intelligent decision-making model supported by a cloud server to make more specific operations that are more in line with the recognition and disposal of edges and openings. This model can use machine learning methods such as support vector machines and random forests, or it can be a rule-based expert system, etc. When the intelligent decision-making model detects a dangerous situation, it can automatically execute corresponding safety measures according to the preset strategy, such as reminding on-site personnel to pay attention to the safety of edges and openings through an acoustic and light alarm, etc.

[0066] The method of this embodiment for the recognition of edges and openings also enables the relevant monitoring data and recognition results to be well recorded and interacted with users. The visualization display method is easy to implement in the construction site monitoring center. The real-time monitoring results, historical data, alarm information, etc. can be displayed to the construction site management personnel through visualization tools, so that they can timely understand the safety status of the construction site and make artificial intervention decisions when necessary. By designing a user interaction interface, the construction site management personnel can conveniently query the real-time data, historical records, alarm information, etc. of the edge and opening recognition, and at the same time can adjust the decision-making strategy configuration of the intelligent vehicle according to the requirements.

[0067] Finally, during the process of edge and opening recognition, both the data obtained by the initial multi-type sensors and the recognition results, etc., can be recorded and stored in detail. This process stores the image data, recognition results, alarm information, and personnel operation records for subsequent analysis and traceability. Cloud storage or local storage methods can be used to ensure the security and reliability of the data. By establishing a perfect data management system, including operations such as data entry, query, deletion, backup, etc., the data management of the above-mentioned edge and opening recognition process can be effectively realized; safe data management and application can be carried out through a database management system such as MySQL, MongoDB, etc.

Claims

1. A construction site edge opening recognition method based on visual deep learning, characterized in that: The method comprises the following steps: S1. Deploy multiple types of sensors on the smart car to obtain two-dimensional image data, infrared thermal image data and three-dimensional point cloud data of the construction site as the smart car moves during transportation tasks; S2, respectively extracting corresponding features of the two-dimensional image data, infrared thermal image data and three-dimensional point cloud data, and then performing feature fusion to construct a comprehensive feature vector; S3, obtaining a pre-trained visual deep learning algorithm model, and performing edge opening recognition training on the basis of a pre-built training data set to obtain an edge opening recognition model; S4, inputting the constructed comprehensive feature vector into the trained edge hole recognition model, and the model outputs the edge hole recognition result corresponding to the moving process of the intelligent car; S5. The intelligent car navigation control system receives and changes its own control decision and adjusts the navigation movement path according to the different types of edge openings and their distribution information in the edge opening identification results.

2. The method for identifying edge openings at a construction site according to claim 1, characterized in that: In step S1, industrial cameras, infrared sensors and laser radars are deployed on the smart car; the industrial camera is used to obtain two-dimensional image data, the infrared sensor is used to obtain infrared thermal imaging data, and the laser radar is used to obtain three-dimensional point cloud data; the industrial camera, infrared sensor and laser radar all use timestamp synchronization technology to collect mutually matching two-dimensional image data, infrared thermal imaging data and three-dimensional point cloud data at the same timestamp.

3. The method for identifying edge openings at a construction site according to claim 2, characterized in that: Multiple industrial cameras are deployed on the smart car. The horizontal field of view angles of multiple industrial cameras are superimposed to cover the 360-degree range of the smart car's periphery, and the horizontal field of view angles of two adjacent industrial cameras have an overlapping interval. The distance between the ground position corresponding to the lower limit of the vertical field of view angle of each industrial camera and the ground position of the smart car itself is less than the preset distance threshold.

4. The method for identifying edge openings at a construction site according to claim 3, characterized in that: Multiple infrared sensors are deployed on the smart car. The number of infrared sensors is the same as the number of industrial cameras and the deployment positions match one by one. Each infrared sensor obtains infrared thermal imaging data corresponding to the two-dimensional image data of the matching industrial camera.

5. The method for identifying edge openings at a construction site according to claim 1, characterized in that: In step S2, the two-dimensional image data, infrared thermal image data and three-dimensional point cloud data acquired by multiple types of sensors are transmitted to the construction site monitoring center in real time, and the cloud server of the construction site monitoring center performs data preprocessing operations, and then extracts corresponding features after the data preprocessing operations are completed; The data preprocessing operations are: For two-dimensional image data and infrared thermal imaging data, denoising, enhancement and correction processing are performed; for continuous two-dimensional image data at different angles, stitching operation is performed after correction processing is completed, and the corresponding images are converted to the same image coordinate system; for three-dimensional point cloud data, filtering, segmentation and registration processing are performed.

6. The method for identifying edge openings at a construction site according to claim 5, characterized in that: After completing the data preprocessing operation, feature data including color, texture and edge are extracted from the two-dimensional image data, feature data of temperature distribution is extracted from the infrared thermal imaging data, and feature data of spatial position and shape is extracted from the three-dimensional point cloud data; general image processing methods or deep learning-based methods are used to extract feature data; the three types of feature data extracted are subjected to feature fusion, and the feature fusion methods include early fusion, mid-term fusion and late fusion. After the fusion is completed, a comprehensive feature vector is constructed.

7. The method for identifying edge openings at a construction site according to claim 1, characterized in that: In step S3, the pre-trained visual deep learning algorithm model includes any one of a convolutional neural network model, a recurrent neural network model or a long short-term memory network model; the pre-constructed training data set includes a training feature vector of the same type as the comprehensive feature vector constructed in step S2; when training the visual deep learning algorithm model, a combination of template matching, feature matching and classifier detection is adopted to enable the visual deep learning algorithm model to identify edge openings based on the training feature vector; the training strategy adopted includes a strategy of combining region proposal with classifier.

8. The method for identifying edge openings at a construction site according to claim 7, characterized in that: In step S4, the trained edge hole recognition model is deployed in the cloud server of the construction site monitoring center. The cloud server inputs the comprehensive feature vector into the edge hole recognition model. The edge hole recognition model processes the comprehensive feature vector in the same way as the training feature vector during training. After identifying the edge hole, the model outputs the edge hole recognition result; in the edge hole recognition result, the types of edge holes include one or more combinations of floor edges, ramp edges, stair openings, elevator openings and reserved openings.

9. The method for identifying edge openings at a construction site according to claim 8, characterized in that: In step S3, when the visual deep learning algorithm model is trained, the training feature vectors also correspond to two types of edge openings, namely, the presence of protective measures and the absence of protective measures; In step S4, when the trained edge opening recognition model recognizes edge openings based on the comprehensive feature vector, information on whether corresponding protective measures exist for one or more types of edge openings is added to the output edge opening recognition result.

10. The method for identifying edge openings at a construction site according to claim 9, characterized in that: In step S5, one or more types of edge openings in the edge opening recognition results output by the model are visualized in a three-dimensional environment model of a construction site monitoring center; the three-dimensional environment model is constructed based on the two-dimensional image data and the three-dimensional point cloud data; When the adjacent hole identification result indicates that there are protective measures for the corresponding adjacent hole, the intelligent vehicle navigation control system moves according to the preset navigation path based on the corresponding adjacent hole distribution information in the three-dimensional environment model, and maintains the first safety distance threshold between the adjacent hole and the intelligent vehicle; When the identification result of the adjacent hole indicates that the corresponding type of adjacent hole lacks protective measures, the construction site monitoring center will issue an external alarm, and at the same time, the intelligent car navigation control system will modify the preset navigation path according to the corresponding adjacent hole distribution information in the three-dimensional environment model, and move according to the modified navigation path to maintain the second safety distance threshold between the adjacent hole and the adjacent hole; The second safety distance threshold is greater than the first safety distance threshold.

Images