Construction site inspection system

Abstract

The invention discloses a construction site routing inspection system, which is applied to the technical field of construction site routing inspection and comprises a construction site exterior scene acquisition module, a routing inspection module and a routing inspection module, the construction site indoor scene acquisition module is used for acquiring construction site indoor scene image data; the potential safety hazard detection module is used for carrying out potential safety hazard detection on the construction site image data based on a potential safety hazard detection model; the potential safety hazard detection model is composed of a feature extraction model based on a Transform encoding and decoding algorithm and a potential safety hazard detection model based on a prediction head array; the three-dimensional construction site model construction module is used for constructing a construction site three-dimensional model based on the construction site image data and the construction site potential safety hazard detection result, and three-dimensionally presenting the interior and exterior scene conditions of the construction site and the potential safety hazard marking condition; the three-dimensional model of the construction site is constructed based on the matching relationship between the images. The construction site inspection quality and efficiency are effectively improved.

Description

Technical Field

[0001] The invention relates to the technical field of construction site inspection, and in particular to a construction site inspection system. Background Art

[0002] With the continuous improvement of construction safety requirements at construction sites, site inspections have become one of the important means to effectively correct and prevent related safety hazards. How to effectively improve the quality and efficiency of site inspections has always been a hot topic.

[0003] At present, construction site inspection mainly relies on the cooperation of the front-end and the back-end. The front-end is used to obtain the image data of the construction site, and the back-end is used to construct a three-dimensional model based on the image data of the construction site to present the construction site scene in three dimensions. For the acquisition of image data at the front-end, human intervention is required to ensure that the adjacent image data collected have certain common points to ensure the quality of subsequent three-dimensional model splicing. For the three-dimensional model construction at the back-end, while it is necessary to further improve the efficiency of model construction, it is also necessary to ensure the quality of model construction, so that the inspection personnel can accurately correct and prevent safety hazards. It can be seen that the existing construction site inspection has the following problems: in the process of collecting front-end image data, since it is necessary to ensure that the adjacent image data collected have certain common points, the professional requirements of personnel are high, the cost of construction site inspection is increased, and the collection efficiency is low; the construction efficiency of the back-end three-dimensional model needs to be further improved, and the accuracy of safety hazard identification by subsequent inspection personnel cannot be guaranteed, and the identification efficiency is low.

[0004] Therefore, how to provide a construction site inspection system that can effectively solve the above-mentioned problems existing in the existing construction site inspections is an issue that needs to be urgently resolved by those skilled in the art. Summary of the invention

[0005] In view of this, the present invention proposes a construction site inspection system.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A construction site inspection system, comprising:

[0008] Construction site exterior scene acquisition module: used to acquire construction site exterior scene image data;

[0009] Construction site interior scene acquisition module: used to acquire construction site interior scene image data;

[0010] Safety hazard detection module: used to perform safety hazard detection on the acquired construction site exterior image data and construction site interior image data based on the safety hazard detection models of the construction site exterior and construction site interior respectively; wherein the safety hazard detection models of the construction site exterior and construction site interior are both composed of a feature extraction model based on the Transformer encoding and decoding algorithm and a safety hazard detection model based on a prediction head array;

[0011] Three-dimensional construction site model building module: used to build a three-dimensional model of the construction site exterior and a three-dimensional model of the construction site interior based on the construction site exterior image data and the construction site exterior safety hazard detection results as well as the construction site interior image data and the construction site interior safety hazard detection results, respectively, to present the interior and exterior conditions of the construction site and the marking of safety hazards of the construction site interior and exterior in three dimensions; among which, the three-dimensional model of the construction site exterior and the three-dimensional model of the construction site interior are both built based on the matching relationship between images.

[0012] Optionally, before training the feature extraction model based on the Transformer encoding and decoding algorithm and the safety hazard detection model based on the prediction head array, it also includes: training set data enhancement, and using the automatic learning data enhancement strategy to learn the best training set data enhancement method, specifically:

[0013] For each training set data augmentation operation, an execution probability is set as a hyperparameter that can be updated iteratively; the training set data augmentation operation includes but is not limited to: large-scale jitter and mixed copy and paste, four pictures are stitched into a large picture, and the pixels of one picture are superimposed on another picture;

[0014] The genetic search operation is used to search within a limited number of steps to obtain the optimal execution probability of the training set data enhancement operation, and the best training set data enhancement strategy is obtained as follows:

[0015] Initialize multiple hyperparameter combinations and determine the fitness function; the fitness function uses the average accuracy of identifying various safety hazards when the IoU threshold is 0.5 as the evaluation indicator;

[0016] Eliminate some of the hyperparameter combinations with the worst performance, leaving some of the best hyperparameter combinations, and cross-sample two of the remaining hyperparameter combinations to swap some parameters; the sampling probability is set according to the value of the fitness function;

[0017] Its value is modified on the new parameter combinations formed by crossover with a probability of 80% and a variance of 0.04.

[0018] Optional feature extraction model based on Transformer encoding and decoding algorithm, specifically:

[0019] The input image is preprocessed by using a preset convolutional backbone neural network to reduce the dimension, and C two-dimensional features with width W and height H are obtained. The C two-dimensional feature matrices are flattened to obtain C one-dimensional flattened vectors of H·W.

[0020] Add the spatial position encoding vector corresponding to each one-dimensional flattened vector to obtain each new feature vector, and connect every two new feature vectors end to end to obtain serialized data; wherein the spatial position encoding vector is predefined;

[0021] The serialized data is input into the Transformer encoder, and the Transformer's attention mechanism is used for feature extraction encoding to extract the intrinsic semantic connection between image pixels and pixel contexts. The Transformer decoder then performs feature output decoding to obtain extracted features for safety hazard detection.

[0022] Optionally, a safety hazard detection model based on a prediction head array is as follows:

[0023] The prediction head array is a plurality of prediction heads, each of which is composed of a predefined feedforward neural network. One prediction head corresponds to one feedforward neural network. The number of prediction heads is set according to the number of safety hazard types. When multiple prediction heads output together, at least one safety hazard prediction box is obtained.

[0024] Optionally, based on the matching relationship between the images, a three-dimensional model of the exterior scene of the construction site and a three-dimensional model of the interior scene of the construction site are constructed, specifically:

[0025] According to the matching relationship between the scene images, a matching relationship model is constructed, and based on the edge element weights in the matching relationship model, the matching relationship model is divided into multiple sub-models to obtain corresponding multiple image groups;

[0026] Based on the number of cross-sub-model edge elements between the sub-models, the matching degree between the sub-models is determined, and based on the matching degree, the adjacent sub-models of the sub-model are determined, and according to the matching degree of the adjacent sub-models, the sub-models are sequentially expanded to obtain a plurality of expanded image groups;

[0027] Based on the number of cross-sub-model edge elements between the expanded sub-models, the matching degree between the expanded sub-models is determined, and based on the matching degree, a minimum height path tree is established to determine the merging order of the local scenes;

[0028] The SFM algorithm is used to construct the local three-dimensional scene of the expanded multiple image groups, and based on the local scene merging order, the local three-dimensional scenes are merged in sequence to obtain a complete three-dimensional scene model.

[0029] Optionally, obtain the matching relationship between scene images, specifically:

[0030] Calculate the feature points of the scene image; feature points are pixels with large gradient changes in the image, including but not limited to: corner points and edge points;

[0031] The feature points are assigned main directions, and rectangular area blocks along the main directions are taken around the feature points. The Harr wavelet features of 25 pixels in the horizontal and vertical directions are counted for each rectangular area block. The sum of the horizontal direction value, the sum of the vertical direction value, the sum of the horizontal absolute value, and the sum of the vertical absolute value are used as the feature descriptor corresponding to each feature point. The Euclidean distance between the feature descriptors of the scene image is calculated. When the Euclidean distance is less than the distance threshold, the feature point is taken as a matching point pair, and the number of matching point pairs is taken as the matching relationship between the scene images.

[0032] Optionally, a matching relationship model is constructed according to the matching relationship between scene images, and the matching relationship model is divided into multiple sub-models based on the edge element weights in the matching relationship model to obtain corresponding multiple image groups, specifically:

[0033] According to the matching relationship between the scene images, a matching relationship model is constructed; wherein the vertex elements of the matching relationship model represent the scene images, and the edge elements of the matching relationship model represent the matching relationship between the scene images;

[0034] Based on the weights of edge elements in the matching relationship model, the matching relationship model is divided into multiple sub-models using a normalized graph cut algorithm to obtain corresponding multiple image groups.

[0035] Optionally, the matching degree between sub-models is determined based on the number of cross-sub-model edge elements between sub-models, and the adjacent sub-models of the sub-model are determined based on the matching degree. The sub-models are sequentially expanded according to the matching degree of the adjacent sub-models to obtain multiple expanded image groups, specifically:

[0036] Based on the number of cross-sub-model edge elements between sub-models, the matching degree between sub-models is determined, and the sub-models with matching degrees greater than a preset matching threshold are regarded as adjacent sub-models;

[0037] The adjacent sub-models are sorted according to their matching degrees, and the sub-models to be expanded are expanded in turn; wherein the expansion is specifically as follows: according to the weights of the cross-sub-model edge elements between the sub-model to be expanded and the adjacent sub-models, the cross-sub-model edge elements are sorted from large to small, and the vertex elements connected to the cross-sub-model edge elements are copied to the sub-model to be expanded; wherein the vertex elements are sub-models.

[0038] Optionally, based on the number of cross-sub-model edge elements between the expanded sub-models, the matching degree between the expanded sub-models is determined, and based on the matching degree, a minimum height path tree is established to determine the order of merging the local scenes, specifically:

[0039] Determine the matching degree between the expanded sub-models based on the number of cross-sub-model edge elements between the expanded sub-models;

[0040] Select any expanded sub-model as the parent node, add the sub-model whose matching degree with the parent node is greater than the preset matching degree threshold as the child node of the parent node, and use the child node with the lowest matching degree with the parent node as the new parent node, and use the sub-model that has not been added as a node and whose matching degree with the new parent node is greater than the preset matching degree threshold as the child node of the new parent node, until all sub-models are added to the path tree;

[0041] The Kruskal algorithm is used to find the minimum height path tree among all path trees; among them, the minimum height path tree is the one with the smallest sum of paths from all child nodes to parent nodes among all possible rooted trees, which can merge all local three-dimensional scenes with the least number of transformations.

[0042] Optionally, based on the order of merging the local scenes, the local 3D scenes are merged in sequence to obtain a complete 3D scene model, specifically:

[0043] Based on the merging order of the local scenes, one of the two local 3D scenes is transferred to the coordinate system of the other local 3D scene according to the rotation and translation transformation parameters, and the local 3D scenes are merged in sequence to obtain a complete 3D scene model;

[0044] The method for obtaining the rotation and translation transformation parameters is as follows:

[0045] Obtain the pose parameters of at least two common points between two local three-dimensional scenes, and obtain the rotation and translation transformation parameters between the two local three-dimensional scenes based on the pose parameters, as follows:

[0046]

[0047]

[0048] Among them, R 12 is the rotation matrix of the common point between the two local 3D scene coordinate systems; R2 is the rotation matrix of the local 3D scene coordinate system as the target of coordinate system transfer; R1 is the rotation matrix of the local 3D scene coordinate system of the transferred coordinate system; t 12 is the translation matrix of the common point between the two local 3D scene coordinate systems; t 12 is the translation matrix between two local three-dimensional scene coordinate systems; t2 is the translation matrix of the local three-dimensional scene coordinate system as the target of coordinate system transfer; t1 is the translation matrix of the local three-dimensional scene coordinate system of the transferred coordinate system.

[0049] It can be seen from the above technical solutions that compared with the prior art, the present invention proposes a construction site inspection system. By improving the construction process of the three-dimensional model of the construction site inspection backend, the quality and efficiency of the construction site inspection are effectively improved, as follows: when identifying safety hazards, a safety hazard detection model composed of a feature extraction model based on the Transformer encoding and decoding algorithm and a safety hazard detection model based on a prediction head array is used instead of manual identification of safety hazards at construction sites, which effectively improves the efficiency and accuracy of safety hazard identification at construction sites; when constructing a three-dimensional model, a matching relationship model is constructed based on the matching relationship between images, and the matching relationship model is divided into multiple images based on the weights of the edge elements in the matching relationship model. sub-models, further, based on the number of cross-sub-model edge elements between the sub-models, the matching degree between the sub-models is determined, and based on the matching degree, the adjacent sub-models of the sub-model are determined, and according to the matching degree of the adjacent sub-models, the sub-models are sequentially expanded to obtain a plurality of image groups after expansion, and based on the number of cross-sub-model edge elements between the expanded sub-models, the matching degree between the expanded sub-models is determined, and based on the matching degree, a minimum height path tree is established to determine the order of merging local scenes, and the local three-dimensional scenes of the expanded multiple image groups are constructed by using the SFM algorithm, and based on the order of merging local scenes, the local three-dimensional scenes are sequentially transformed according to the rotation and translation transformation parameters. The 3D scene is merged to obtain a complete 3D scene model; wherein, based on the number of cross-sub-model edge elements between the sub-models, the matching degree between the sub-models is determined, and based on the matching degree, the adjacent sub-models of the sub-model are determined, and the sub-models are expanded in sequence according to the matching degree of the adjacent sub-models, which effectively solves the problem of front-end image data acquisition requiring human intervention to ensure that the collected adjacent image data have certain common points to ensure the high quality requirements of subsequent 3D model stitching quality. The expanded sub-models can make the adjacent scene images contain a certain number of common points, reducing the difficulty of image data acquisition; and based on the matching degree between images The matching relationship and the weight of the edge elements in the matching relationship model are used to split the construction site scene into local scenes, and the local scene models are constructed separately. The matching degree is determined based on the number of cross-sub-model edge elements between models, and a minimum height path tree is established to determine the merging order of local scenes, that is, to obtain the local scene merging order that can merge all local three-dimensional scenes with the least number of transformations, and the three-dimensional model construction method that merges the local three-dimensional scenes in sequence according to the rotation and translation transformation parameters based on this local scene merging order with the least number of transformations also effectively improves the efficiency of three-dimensional model construction and reduces the memory and computing resource burden of the construction equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0051] Figure 1 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Embodiment 1:

[0054] Embodiment 1 of the present invention discloses a construction site inspection system, such as Figure 1 As shown, including:

[0055] Construction site exterior scene acquisition module: used to acquire construction site exterior scene image data.

[0056] Construction site interior scene acquisition module: used to acquire construction site interior scene image data.

[0057] Safety hazard detection module: used to perform safety hazard detection on the acquired construction site exterior image data and construction site interior image data based on the safety hazard detection models of the construction site exterior and the construction site interior respectively; among which, the safety hazard detection models of the construction site exterior and the construction site interior are both composed of a feature extraction model based on the Transformer encoding and decoding algorithm and a safety hazard detection model based on a prediction head array.

[0058] Considering that when obtaining the training set, the construction site environment is complex, the lighting and obstructions change from time to time, and there are large differences between different construction sites, which poses a challenge to the generalization performance of the construction site safety hazard detection model. Therefore, before training the feature extraction model based on the Transformer encoding and decoding algorithm and the safety hazard detection model based on the prediction head array, it also includes: training set data enhancement, and using the automatic learning data enhancement strategy to learn the best training set data enhancement method to improve the training effect and enhance the model generalization performance and recognition performance, specifically:

[0059] For each training set data augmentation operation, an execution probability is set as a hyperparameter that can be updated iteratively; the training set data augmentation operation includes but is not limited to: large-scale jitter and mixed copy and paste, four pictures are stitched into a large picture, and the pixels of one picture are superimposed on another picture;

[0060] The genetic search operation is used to search within a limited number of steps to obtain the optimal execution probability of the training set data enhancement operation, and the best training set data enhancement strategy is obtained as follows:

[0061] Initialize multiple hyperparameter combinations and determine the fitness function; the fitness function uses the average accuracy of identifying various safety hazards when the IoU threshold is 0.5 as the evaluation indicator;

[0062] Eliminate some of the hyperparameter combinations with the worst performance, leaving some of the best hyperparameter combinations, and cross-sample two of the remaining hyperparameter combinations to swap some parameters; the sampling probability is set according to the value of the fitness function;

[0063] Its value is modified on the new parameter combinations formed by crossover with a probability of 80% and a variance of 0.04.

[0064] The feature extraction model based on the Transformer encoding and decoding algorithm is as follows:

[0065] The input image is preprocessed with a preset convolutional backbone neural network to obtain C two-dimensional features with width W and height H to reduce the overall complexity of the algorithm, and the C two-dimensional feature matrices are flattened to obtain C one-dimensional flattened vectors of H·W; C, W, and H can be set accordingly according to the actual application scenario;

[0066] Add the spatial position encoding vector corresponding to each one-dimensional flattened vector to obtain each new feature vector, and connect every two new feature vectors end to end to obtain serialized data; wherein the spatial position encoding vector is predefined;

[0067] The serialized data is input into the Transformer encoder, and the Transformer's attention mechanism is used for feature extraction encoding to extract the intrinsic semantic connection between image pixels and pixel contexts. The Transformer decoder then performs feature output decoding to obtain extracted features for safety hazard detection.

[0068] The safety hazard detection model based on the prediction head array is as follows:

[0069] The prediction head array is a plurality of prediction heads, each of which is composed of a predefined feedforward neural network. One prediction head corresponds to one feedforward neural network. The number of prediction heads is set according to the number of safety hazard types. When multiple prediction heads output together, at least one safety hazard prediction box is obtained.

[0070] Three-dimensional construction site model building module: used to build a three-dimensional model of the construction site exterior and a three-dimensional model of the construction site interior based on the construction site exterior image data and the construction site exterior safety hazard detection results as well as the construction site interior image data and the construction site interior safety hazard detection results, respectively, to present the interior and exterior conditions of the construction site and the marking of safety hazards of the construction site interior and exterior in three dimensions; among which, the three-dimensional model of the construction site exterior and the three-dimensional model of the construction site interior are both built based on the matching relationship between images.

[0071] Based on the matching relationship between the images, a 3D model of the exterior scene of the construction site and a 3D model of the interior scene of the construction site are constructed, specifically:

[0072] In order to split the construction site scene into local scenes and build local scene models respectively, effectively improve the efficiency of 3D model construction, while ensuring that the similarity within each image group is high enough and the similarity between different image groups is low enough, a matching relationship model is constructed according to the matching relationship between scene images, and based on the edge element weights in the matching relationship model, the matching relationship model is divided into multiple sub-models to obtain corresponding multiple image groups;

[0073] Based on the number of cross-submodel edge elements between submodels, the matching degree between submodels is determined, and based on the matching degree, the adjacent submodels of the submodel are determined. According to the matching degree of the adjacent submodels, the submodels are expanded in sequence to obtain multiple extended image groups. The extended submodels can make the adjacent scene images contain a certain number of common points, which effectively solves the problem that the front-end image data acquisition requires human intervention to ensure that the collected adjacent image data have certain common points to ensure the high quality requirements of the subsequent three-dimensional model splicing quality, effectively reduces the difficulty of image data acquisition, and ensures the model accuracy;

[0074] Based on the number of cross-sub-model edge elements between the expanded sub-models, the matching degree between the expanded sub-models is determined, and based on the matching degree, a minimum height path tree is established to determine the merging order of the local scenes;

[0075] The SFM algorithm is used to construct the local 3D scene of the expanded multiple image groups, and based on the local scene merging order, the local 3D scenes are merged in sequence to obtain a complete 3D scene model. The SFM algorithm is a prior art and will not be described in detail here.

[0076] Get the matching relationship between scene images, specifically:

[0077] Calculate the feature points of the scene image; feature points are pixels with large gradient changes in the image, including but not limited to: corner points and edge points;

[0078] The feature points are assigned main directions, and rectangular area blocks along the main directions are taken around the feature points. The Harr wavelet features of 25 pixels in the horizontal and vertical directions are counted for each rectangular area block. The sum of the horizontal direction value, the sum of the vertical direction value, the sum of the horizontal absolute value, and the sum of the vertical absolute value are used as the feature descriptor corresponding to each feature point. The Euclidean distance between the feature descriptors of the scene image is calculated. When the Euclidean distance is less than the distance threshold, the feature point is taken as a matching point pair, and the number of matching point pairs is taken as the matching relationship between the scene images.

[0079] According to the matching relationship between scene images, a matching relationship model is constructed, and based on the edge element weights in the matching relationship model, the matching relationship model is divided into multiple sub-models to obtain corresponding multiple image groups, specifically:

[0080] According to the matching relationship between the scene images, a matching relationship model is constructed; wherein the vertex elements of the matching relationship model represent the scene images, and the edge elements of the matching relationship model represent the matching relationship between the scene images;

[0081] Based on the weights of the edge elements in the matching relationship model, the matching relationship model is divided into multiple sub-models using a normalized graph cut algorithm to obtain multiple corresponding image groups. The normalized graph cut algorithm is a prior art and will not be described in detail here.

[0082] Based on the number of cross-sub-model edge elements between sub-models, the matching degree between sub-models is determined, and based on the matching degree, the adjacent sub-models of the sub-model are determined. According to the matching degree of the adjacent sub-models, the sub-models are expanded in sequence to obtain multiple image groups after expansion, specifically:

[0083] Based on the number of cross-sub-model edge elements between sub-models, the matching degree between sub-models is determined, and the sub-models with matching degrees greater than a preset matching threshold are regarded as adjacent sub-models;

[0084] Sort by the matching degree of the adjacent sub-models, and expand the sub-models to be expanded in turn; wherein the expansion is specifically: sort the cross-sub-model edge elements from large to small according to the weight of the cross-sub-model edge elements between the sub-model to be expanded and the adjacent sub-models, and copy the vertex elements connected to the cross-sub-model edge elements to the sub-model to be expanded; wherein the vertex elements are sub-models. In the process of copying the vertex elements, whether the expansion is completed can be determined by whether the ratio of the number of copied vertex elements to the total number of copied vertex elements reaches a preset threshold;

[0085] Based on the number of cross-sub-model edge elements between the expanded sub-models, the matching degree between the expanded sub-models is determined, and a minimum height path tree is established based on the matching degree to determine the order of merging local scenes, specifically:

[0086] Determine the matching degree between the expanded sub-models based on the number of cross-sub-model edge elements between the expanded sub-models;

[0087] Select any expanded sub-model as the parent node, add the sub-model whose matching degree with the parent node is greater than the preset matching degree threshold as the child node of the parent node, and use the child node with the lowest matching degree with the parent node as the new parent node, and use the sub-model that has not been added as a node and whose matching degree with the new parent node is greater than the preset matching degree threshold as the child node of the new parent node, until all sub-models are added to the path tree;

[0088] The Kruska l algorithm is used to find the minimum height path tree among all path trees; wherein the minimum height path tree is the path tree with the smallest sum of all child nodes to parent nodes among all possible rooted trees, and can realize merging all local three-dimensional scenes with the least number of transformations. wherein the Kruska l algorithm is a prior art and will not be described in detail here.

[0089] Based on the local scene merging order, the local 3D scenes are merged in sequence to obtain a complete 3D scene model, specifically:

[0090] Based on the merging order of the local scenes, one of the two local 3D scenes is transferred to the coordinate system of the other local 3D scene according to the rotation and translation transformation parameters, and the local 3D scenes are merged in sequence to obtain a complete 3D scene model;

[0091] The method for obtaining the rotation and translation transformation parameters is as follows:

[0092] Obtain the pose parameters of at least two common points between two local three-dimensional scenes, and obtain the rotation and translation transformation parameters between the two local three-dimensional scenes based on the pose parameters, as follows:

[0093]

[0094] Among them, R 12 is the rotation matrix of the common point between the two local 3D scene coordinate systems; R2 is the rotation matrix of the local 3D scene coordinate system as the target of coordinate system transfer; R1 is the rotation matrix of the local 3D scene coordinate system of the transferred coordinate system; t 12 is the translation matrix of the common point between the two local 3D scene coordinate systems; t 12is the translation matrix between two local three-dimensional scene coordinate systems; t2 is the translation matrix of the local three-dimensional scene coordinate system as the target of coordinate system transfer; t1 is the translation matrix of the local three-dimensional scene coordinate system of the transferred coordinate system.

[0095] The embodiment of the present invention discloses a construction site inspection system. By improving the construction process of the back-end three-dimensional model of the construction site inspection, the quality and efficiency of the construction site inspection are effectively improved, as follows: when identifying safety hazards, a safety hazard detection model composed of a feature extraction model based on the Transformer encoding and decoding algorithm and a safety hazard detection model based on a prediction head array is used to replace manual identification of safety hazards at the construction site, thereby effectively improving the efficiency and accuracy of safety hazard identification at the construction site; when constructing a three-dimensional model, a matching relationship model is constructed based on the matching relationship between images, and the matching relationship model is divided into multiple images based on the edge element weights in the matching relationship model. sub-models, further, based on the number of cross-sub-model edge elements between the sub-models, the matching degree between the sub-models is determined, and based on the matching degree, the adjacent sub-models of the sub-model are determined, and according to the matching degree of the adjacent sub-models, the sub-models are sequentially expanded to obtain a plurality of image groups after expansion, and based on the number of cross-sub-model edge elements between the expanded sub-models, the matching degree between the expanded sub-models is determined, and based on the matching degree, a minimum height path tree is established to determine the order of merging local scenes, and the local three-dimensional scenes of the expanded multiple image groups are constructed by using the SFM algorithm, and based on the order of merging local scenes, the local three-dimensional scenes are sequentially transformed according to the rotation and translation transformation parameters. The 3D scene is merged to obtain a complete 3D scene model; wherein, based on the number of cross-sub-model edge elements between the sub-models, the matching degree between the sub-models is determined, and based on the matching degree, the adjacent sub-models of the sub-model are determined, and the sub-models are expanded in sequence according to the matching degree of the adjacent sub-models, which effectively solves the problem of front-end image data acquisition requiring human intervention to ensure that the collected adjacent image data have certain common points to ensure the high quality requirements of subsequent 3D model stitching quality. The expanded sub-models can make the adjacent scene images contain a certain number of common points, reducing the difficulty of image data acquisition; and based on the matching degree between images The matching relationship and the weight of the edge elements in the matching relationship model are used to split the construction site scene into local scenes, and the local scene models are constructed separately. The matching degree is determined based on the number of cross-sub-model edge elements between models, and a minimum height path tree is established to determine the merging order of local scenes, that is, to obtain the local scene merging order that can merge all local three-dimensional scenes with the least number of transformations, and the three-dimensional model construction method that merges the local three-dimensional scenes in sequence according to the rotation and translation transformation parameters based on this local scene merging order with the least number of transformations also effectively improves the efficiency of three-dimensional model construction and reduces the memory and computing resource burden of the construction equipment.

[0096] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0097] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A construction site inspection system, characterized in that: include: Construction site exterior scene acquisition module: used to acquire construction site exterior scene image data; Construction site interior scene acquisition module: used to acquire construction site interior scene image data; Safety hazard detection module: used to perform safety hazard detection on the acquired construction site exterior image data and construction site interior image data based on the safety hazard detection models of the construction site exterior and construction site interior respectively; wherein the safety hazard detection models of the construction site exterior and construction site interior are both composed of a feature extraction model based on the Transformer encoding and decoding algorithm and a safety hazard detection model based on a prediction head array; Three-dimensional construction site model building module: used to build a three-dimensional model of the construction site exterior and a three-dimensional model of the construction site interior based on the construction site exterior image data and the construction site exterior safety hazard detection results as well as the construction site interior image data and the construction site interior safety hazard detection results, respectively, to present the interior and exterior conditions of the construction site and the marking of safety hazards of the construction site interior and exterior in three dimensions; wherein the three-dimensional model of the construction site exterior and the three-dimensional model of the construction site interior are both built based on the matching relationship between images.

2. A construction site inspection system according to claim 1, characterized in that: Before training the feature extraction model based on the Transformer encoding and decoding algorithm and the safety hazard detection model based on the prediction head array, it also includes: training set data enhancement, and using the automatic learning data enhancement strategy to learn the best training set data enhancement method, specifically: For each training set data augmentation operation, an execution probability is set as a hyperparameter that can be updated iteratively; wherein the training set data augmentation operation includes but is not limited to: large-scale jitter and mixed copy and paste, four pictures are stitched into a large picture, and pixels of one picture are superimposed on another picture; The optimal execution probability of the training set data enhancement operation is obtained by using the genetic search operation within a limited number of steps, and the best training set data enhancement strategy is obtained as follows: Initialize multiple hyperparameter combinations and determine a fitness function; wherein the fitness function uses the average accuracy of identifying various safety hazards when the IoU threshold is 0.5 as an evaluation indicator; Eliminate some of the hyperparameter combinations with the worst performance, leaving some of the hyperparameter combinations with the best effects, and cross-sample two by two in the remaining hyperparameter combinations to exchange some parameters; wherein the sampling probability is set according to the value of the fitness function; Its value is modified on the new parameter combinations formed by crossover with a probability of 80% and a variance of 0.

04.

3. A construction site inspection system according to claim 1, characterized in that: The feature extraction model based on the Transformer encoding and decoding algorithm is specifically: The input image is preprocessed by using a preset convolutional backbone neural network to reduce the dimensionality, and C two-dimensional features with width W and height H are obtained. The C two-dimensional feature matrices are flattened to obtain C one-dimensional flattened vectors of H·W. Adding the spatial position coding vector corresponding to each one-dimensional flattened vector to obtain each new feature vector, and connecting every two new feature vectors end to end to obtain serialized data; wherein the spatial position coding vector is predefined; The serialized data is input into the Transformer encoder, and feature extraction encoding is performed using the Transformer's attention mechanism to extract the intrinsic semantic connection between image pixels and pixel contexts. The Transformer decoder then performs feature output decoding to obtain extracted features for safety hazard detection.

4. A construction site inspection system according to claim 1, characterized in that: The potential safety hazard detection model based on the prediction head array is specifically: The prediction head array is a plurality of prediction heads, each prediction head is composed of a predefined feedforward neural network, one prediction head corresponds to one feedforward neural network, the number of the prediction heads is set according to the number of safety hazard types, and when the plurality of prediction heads output together, at least one safety hazard prediction box is obtained.

5. A construction site inspection system according to claim 1, characterized in that: Based on the matching relationship between the images, the three-dimensional model of the exterior scene of the construction site and the three-dimensional model of the interior scene of the construction site are constructed, specifically: According to the matching relationship between the scene images, a matching relationship model is constructed, and based on the edge element weights in the matching relationship model, the matching relationship model is divided into a plurality of sub-models to obtain a plurality of corresponding image groups; Determine the matching degree between the sub-models based on the number of cross-sub-model edge elements between the sub-models, determine the adjacent sub-models of the sub-model based on the matching degree, and expand the sub-models in sequence according to the matching degree of the adjacent sub-models to obtain a plurality of expanded image groups; Determine the matching degree between the expanded sub-models based on the number of cross-sub-model edge elements between the expanded sub-models, establish a minimum height path tree based on the matching degree, and determine the order of merging the local scenes; The local three-dimensional scenes of the expanded multiple image groups are constructed using the SFM algorithm, and based on the local scene merging order, the local three-dimensional scenes are merged in sequence to obtain a complete three-dimensional scene model.

6. A construction site inspection system according to claim 5, characterized in that: Get the matching relationship between scene images, specifically: Calculate feature points of the scene image; the feature points are pixels with large gradient changes in the image, including but not limited to corner points and edge points; The feature points are assigned a main direction, and rectangular area blocks along the main direction are taken around the feature points. The Harr wavelet features of 25 pixels in the horizontal and vertical directions are counted for each rectangular area block. The sum of the horizontal direction values, the sum of the vertical direction values, the sum of the horizontal direction absolute values, and the sum of the vertical direction absolute values ​​are used as the feature descriptors corresponding to each feature point. The Euclidean distance between the feature descriptors of the scene image is calculated. When the Euclidean distance is less than the distance threshold, the feature points are used as matching point pairs, and the number of the matching point pairs is used as the matching relationship between the scene images.

7. A construction site inspection system according to claim 5, characterized in that: According to the matching relationship between scene images, a matching relationship model is constructed, and based on the edge element weights in the matching relationship model, the matching relationship model is divided into multiple sub-models to obtain corresponding multiple image groups, specifically: According to the matching relationship between the scene images, a matching relationship model is constructed; wherein the vertex elements of the matching relationship model represent the scene images, and the edge elements of the matching relationship model represent the matching relationship between the scene images; Based on the edge element weights in the matching relationship model, the matching relationship model is segmented into a plurality of sub-models using a normalized graph cut algorithm to obtain a plurality of corresponding image groups.

8. A construction site inspection system according to claim 5, characterized in that: Based on the number of cross-sub-model edge elements between sub-models, the matching degree between the sub-models is determined, and based on the matching degree, the adjacent sub-model of the sub-model is determined, and according to the matching degree of the adjacent sub-model, the sub-model is sequentially expanded to obtain multiple image groups after expansion, specifically: Determine the matching degree between the sub-models based on the number of cross-sub-model edge elements between the sub-models, and take the sub-models whose matching degree is greater than a preset matching threshold as adjacent sub-models; The adjacent sub-models are sorted according to their matching degrees, and the sub-models to be expanded are expanded in sequence; wherein the expansion is specifically as follows: according to the weights of the cross-sub-model edge elements between the sub-model to be expanded and the adjacent sub-models, the cross-sub-model edge elements are sorted from large to small, and the vertex elements connected to the cross-sub-model edge elements are copied to the sub-model to be expanded; wherein the vertex elements are sub-models.

9. A construction site inspection system according to claim 5, characterized in that: Based on the number of cross-sub-model edge elements between the expanded sub-models, the matching degree between the expanded sub-models is determined, and based on the matching degree, a minimum height path tree is established to determine the order of merging the local scenes, specifically: Determining the matching degree between the expanded sub-models based on the number of cross-sub-model edge elements between the expanded sub-models; Select any of the expanded sub-models as the parent node, add the sub-model whose matching degree with the parent node is greater than a preset matching degree threshold as the child node of the parent node, and use the child node with the lowest matching degree with the parent node as the new parent node, and use the sub-models that have not been added as nodes and whose matching degree with the new parent node is greater than the preset matching degree threshold as the child nodes of the new parent node, until all the sub-models are added to the path tree; The Kruskal algorithm is used to find the minimum height path tree among all path trees; wherein the minimum height path tree is the one with the smallest sum of paths from all child nodes to parent nodes among all possible rooted trees, and can merge all local three-dimensional scenes with the least number of transformations.

10. A construction site inspection system according to claim 5, characterized in that: Based on the local scene merging order, the local three-dimensional scenes are merged in sequence to obtain a complete three-dimensional scene model, specifically: Based on the merging order of the local scenes, one of the two local three-dimensional scenes is transferred to the coordinate system of the other local three-dimensional scene according to the rotation and translation transformation parameters, and the local three-dimensional scenes are merged in sequence to obtain a complete three-dimensional scene model; The rotation and translation transformation parameters are obtained as follows: Obtain the pose parameters of at least two common points between two local three-dimensional scenes, and obtain the rotation and translation transformation parameters between the two local three-dimensional scenes based on the pose parameters, as follows: Among them, R 12 is the rotation matrix of the common point between the two local 3D scene coordinate systems; R2 is the rotation matrix of the local 3D scene coordinate system as the target of coordinate system transfer; R1 is the rotation matrix of the local 3D scene coordinate system of the transferred coordinate system; t 12 is the translation matrix of the common point between the two local 3D scene coordinate systems; t 12 is the translation matrix between two local three-dimensional scene coordinate systems; t2 is the translation matrix of the local three-dimensional scene coordinate system as the target of coordinate system transfer; t1 is the translation matrix of the local three-dimensional scene coordinate system of the transferred coordinate system.

Images