Self-supervised monocular depth estimation transfer learning method and system for construction safety
By employing a transfer learning method based on a self-supervised monocular depth estimation model, combined with an encoder-decoder structure and quantitative evaluation metrics, the problem of lacking real depth data in construction scenarios was solved, enabling model transfer from autonomous driving to construction scenarios and improving depth estimation accuracy.
Patent Information
- Application Number
- CN202210924807.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-02
- Publication Date
- 2025-10-21
- Estimated Expiration
- 2042-08-02
AI Technical Summary
Existing self-supervised monocular depth estimation methods suffer from a lack of real depth data when migrating from autonomous driving scenarios to building construction scenarios, leading to model domain drift and making them difficult to apply effectively to construction safety tasks.
A self-supervised monocular depth estimation model is adopted, combined with an encoder-decoder structure, and the scene depth is hierarchically distinguished. By constructing a transfer dataset and quantitative evaluation indicators, the loss function is optimized to realize the transfer of the model from autonomous driving scenarios to construction scenarios. Rectangular bounding box annotation and depth order loss function are used to evaluate the model performance.
It improves the depth estimation accuracy in construction scenarios, solves the model domain drift problem, and realizes the effective application of the self-supervised monocular depth estimation model in construction scenarios.
Smart Images
Figure CN115170971B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a self-supervised monocular depth estimation transfer learning method and system for construction safety. Background Art
[0002] Estimating the depth of construction scenes from a single RGB image is a key prerequisite for a variety of applications, including workspace safety, localization, productivity analysis, activity recognition, and scene understanding. Self-supervised monocular depth estimation methods leverage large amounts of unlabeled data to train depth representations and even outperform some supervised methods.
[0003] Currently, such methods have unavoidable limitations in their practical application on construction sites. In the field of self-supervised monocular depth estimation, the main datasets used for training and evaluation, such as the KITTI dataset and the DDAD dataset, are both used for autonomous driving tasks. They include real depth data for training, validating, and testing models. Due to the differences in feature space and data distribution between autonomous driving and construction scene analysis, knowledge transfer between them will improve representation learning in downstream tasks. However, although we can easily solve the problem of transfer learning between autonomous driving tasks, such as model transfer from the KITTI dataset to the DDAD dataset, migrating tasks to construction scenarios remains a huge challenge. Because real depth values like lidar data are not available on construction sites, migrating and evaluating models is a challenging problem.
[0004] Therefore, how to research and design a self-supervised monocular depth estimation transfer learning method and system for construction safety that can overcome the above-mentioned defects is an issue that we urgently need to solve. Summary of the Invention
[0005] To address the deficiencies in the prior art, the present invention aims to provide a self-supervised monocular depth estimation transfer learning method and system for construction safety. By combining the codec structure characteristics of the self-supervised monocular depth estimation model and utilizing the hierarchical differentiation of scene depth, the self-supervised monocular depth estimation network model can be migrated from scenarios such as autonomous driving to construction scenarios, and reliable evaluation indicators are proposed to evaluate the effect of the migration.
[0006] The above technical objectives of the present invention are achieved through the following technical solutions:
[0007] First, a self-supervised monocular depth estimation transfer learning method for construction safety is provided, which includes the following steps:
[0008] Collect construction site images containing construction equipment and workers, annotate and segment all construction site images, and build a migration dataset;
[0009] Build a self-supervised monocular depth estimation network model and load the pre-trained parameters of the monocular depth estimation network model;
[0010] The monocular depth estimation network model is transferred and trained based on the migration dataset, and the training process is constrained by the loss function.
[0011] Quantitative evaluation indicators are used to evaluate the performance of the monocular depth estimation network model after each training, and the optimal monocular depth estimation network model is screened.
[0012] Furthermore, the annotation box in the construction site image is annotated using a rectangular bounding box;
[0013] A rectangular bounding box is an area marked at any depth with distinct levels and equal depth inside;
[0014] The labels of the rectangular bounding boxes are the serial numbers of all selected rectangular bounding boxes from small to large according to depth.
[0015] Furthermore, the depth loss calculated by the loss function includes:
[0016] The loss caused by the deep deviation between the random variable and the corresponding mean in each annotation box;
[0017] And the loss corresponding to the depth order between all annotation boxes.
[0018] Furthermore, the calculation formula for the loss caused by the depth deviation is specifically as follows:
[0019]
[0020] Among them, L dd Indicates the loss caused by depth deviation; N represents the number of annotation boxes in the current image; D′ i It represents the D′ of the i-th annotation box area in the current image; D′ represents the normalized depth map of the current image corresponding to the depth map predicted by the depth estimation network; σ(D′ i ) represents the random variable D′ i The standard deviation of
[0021] The calculation formula of the depth map is as follows:
[0022]
[0023] Where D represents the depth map corresponding to the current image predicted by the depth estimation network; D max 、D min represent the maximum and minimum values in D respectively.
[0024] Furthermore, the loss calculation formula corresponding to the depth order is specifically:
[0025]
[0026] Among them, L do Indicates the loss corresponding to the depth order; N represents the number of annotation boxes in the current image; V ij Represents the estimated loss of the distance between the i-th standard box and the j-th standard box; i and j are both integers, representing the serial number of the annotated box in the image. The larger the serial number, the greater the average depth value of the annotated box.
[0027] The calculation formula for distance estimation loss is as follows:
[0028]
[0029] Where μ(·) represents the function for finding the mean value of the corresponding random variable; This is to avoid the perturbations introduced by trivial solutions.
[0030] Furthermore, the migration training process of the monocular depth estimation network model is specifically as follows:
[0031] Load the PackNet network model pre-trained on the Kitti autonomous driving dataset and freeze the Encoder module of the model;
[0032] Load the training images and labels of the dataset;
[0033] Train the PackNet model and use the transfer loss function to constrain the parameters of the Decoder module of the model;
[0034] The model after each round of training is tested using the validation set and evaluation metrics of the dataset.
[0035] Furthermore, the performance evaluation process of the monocular depth estimation network model is specifically as follows:
[0036] Comprehensive evaluation is performed based on the relative average error of depth deviation and the sequential accuracy of depth order;
[0037] If the relative average error is lower and the sequential accuracy is higher, the performance of the monocular depth estimation network model is good.
[0038] Secondly, we provide a self-supervised monocular depth estimation transfer learning system for construction safety, including:
[0039] The data acquisition module is used to collect construction site images including construction equipment and workers, and to label and divide all construction site images to build a migration dataset;
[0040] The model building module is used to build a self-supervised monocular depth estimation network model and load the pre-trained parameters of the monocular depth estimation network model;
[0041] The transfer training module is used to perform transfer training on the monocular depth estimation network model based on the transfer dataset, and the training process is constrained by the loss function;
[0042] The evaluation and screening module is used to evaluate the performance of the monocular depth estimation network model after each training using quantitative evaluation indicators, and to screen out the optimal monocular depth estimation network model.
[0043] In a third aspect, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the self-supervised monocular depth estimation transfer learning method for construction safety as described in any one of the first aspects is implemented.
[0044] In a fourth aspect, a computer-readable medium is provided, on which a computer program is stored. The computer program is executed by a processor to implement the self-supervised monocular depth estimation transfer learning method for construction safety as described in any one of the first aspects.
[0045] Compared with the prior art, the present invention has the following beneficial effects:
[0046] 1. The self-supervised monocular depth estimation transfer learning method for construction safety provided by this invention combines the encoder-decoder structure characteristics of the self-supervised monocular depth estimation model and uses the hierarchical differentiation of scene depth to achieve the migration of the self-supervised monocular depth estimation network model from scenarios such as autonomous driving to construction scenarios. Reliable evaluation indicators are also proposed to evaluate the effect of the migration.
[0047] 2. The present invention solves the model domain drift problem of the self-supervised monocular depth prediction model caused by the lack of real depth data in the construction scene by constructing a new learning paradigm, including a self-supervised monocular depth estimation model migration method, a loss function, an evaluation index and a depth labeling method, thereby improving the depth estimation accuracy of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:
[0049] Figure 1 is a flow chart in an embodiment of the present invention;
[0050] Figure 2 It is a system block diagram in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples and drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.
[0052] Example 1: Self-supervised monocular depth estimation transfer learning method for construction safety, such as Figure 1 As shown, the following steps are included:
[0053] S1: Collect construction site images containing construction equipment and workers, annotate and divide all construction site images, and build a migration dataset;
[0054] S2: Build a self-supervised monocular depth estimation network model and load the pre-trained parameters of the monocular depth estimation network model;
[0055] S3: Perform transfer training on the monocular depth estimation network model based on the transfer dataset, and use loss function constraints in the training process;
[0056] S4: Use quantitative evaluation indicators to evaluate the performance of the monocular depth estimation network model after each training, and screen out the optimal monocular depth estimation network model.
[0057] The images in the dataset are annotated using a method similar to the rectangular bounding box annotation method commonly used in the field of target detection. The difference is that the rectangular bounding boxes are annotated in areas with distinct layers and equal depth at any depth; the labels of the rectangular bounding boxes are the serial numbers of all selected rectangular bounding boxes in ascending order of depth.
[0058] The transferred loss function does not directly calculate the depth loss of each pixel, but compares the depth deviation of the random variable in each annotation box from its mean value, as well as the depth order between all annotation boxes, which can be directly applied to the depth loss.
[0059] The calculation formula of the loss function is as follows:
[0060] L=αL do +L dd
[0061] Among them, L represents the depth loss calculated by the loss function; L dd is the loss caused by depth deviation; L d o is the depth-order loss; α is a weight adjustment parameter, α = 1 / 3.
[0062] The calculation formula for the loss caused by depth deviation is as follows:
[0063]
[0064] Among them, L dd Indicates the loss caused by depth deviation; N represents the number of annotation boxes in the current image; D′ i It represents the D′ of the i-th annotation box area in the current image; D′ represents the normalized depth map of the current image corresponding to the depth map predicted by the depth estimation network; σ(D′ i ) represents the random variable D′ i The standard deviation of .
[0065] The calculation formula of the depth map is as follows:
[0066]
[0067] Where D represents the depth map corresponding to the current image predicted by the depth estimation network; D max 、D min represent the maximum and minimum values in D respectively.
[0068] The loss calculation formula corresponding to the depth order is:
[0069]
[0070] Among them, L do Indicates the loss corresponding to the depth order; N represents the number of annotation boxes in the current image; V ij It represents the estimated loss of the inter-box distance between the i-th standard box and the j-th standard box; i and j are both integers, representing the serial number of the annotated box in the image. The larger the serial number, the larger the average depth value of the annotated box.
[0071] The calculation formula for distance estimation loss is as follows:
[0072]
[0073] Where μ(·) represents the function for finding the mean value of the corresponding random variable; This is to avoid the perturbations introduced by trivial solutions.
[0074] The specific transfer training process for the monocular depth estimation network model is as follows: load the PackNet network model pre-trained on the Kitti autonomous driving dataset and freeze the encoder module of the model; load the training images and labels of the dataset; train the PackNet model and use the transfer loss function to constrain the parameters of the decoder module of the model; and test the model after each round of training using the validation set and evaluation indicators of the dataset.
[0075] The evaluation metrics include two accuracy metrics, which evaluate each test model based on two losses: depth bias and depth order. The first metric is the relative mean error (RME), which is defined as:
[0076]
[0077] Among them, d j is the annotation box Y in the corresponding predicted depth image i Pixels of the area; μX i It is the comment box Y i The average depth of the area; N is the annotation box Y i The number of pixels in the region.
[0078] Another indicator is the Order Accuracy (OAcc) indicator, which is defined as:
[0079]
[0080] Where K is the total number of annotation boxes in the image.
[0081] δ ij The calculation formula is:
[0082]
[0083]
[0084] in, Indicates the true depth relationship between the two annotation boxes (IO indicates ascending order, DO indicates descending order); represents the normal difference distribution of two random variables; X i Represents the depth value distribution of the i-th annotation box area; R(X i ,X j ) represents the depth relationship function between two annotation boxes calculated based on the predicted depth.
[0085] The calculation formula for Label(x) is:
[0086]
[0087] When evaluating depth estimation models, models with higher OAcc and lower RME are better.
[0088] Example 2: A self-supervised monocular depth estimation transfer learning system for construction safety. The self-supervised monocular depth estimation transfer learning system is used to implement the self-supervised monocular depth estimation transfer learning method described in Example 1, such as Figure 2As shown, it includes data acquisition module, model building module, transfer training module and evaluation screening module.
[0089] Among them, the data acquisition module is used to collect construction site images including construction equipment and workers, and to label and divide all construction site images to construct a migration dataset; the model construction module is used to establish a self-supervised monocular depth estimation network model and load the pre-training parameters of the monocular depth estimation network model; the migration training module is used to perform migration training on the monocular depth estimation network model based on the migration dataset, and the training process adopts loss function constraints; the evaluation and screening module is used to use quantitative evaluation indicators to evaluate the performance of the monocular depth estimation network model after each training, and screen out the optimal monocular depth estimation network model.
[0090] Working principle: This invention combines the encoder-decoder structure characteristics of the self-supervised monocular depth estimation model and uses the hierarchical distinction of scene depth to realize the migration of the self-supervised monocular depth estimation network model from scenarios such as autonomous driving to construction scenarios, and proposes reliable evaluation indicators to evaluate the effect of migration; by constructing a new learning paradigm, including a self-supervised monocular depth estimation model migration method, loss function, evaluation index and depth labeling method, it solves the model domain drift problem caused by the lack of real depth data in the construction scene of the self-supervised monocular depth prediction model, and improves the depth estimation accuracy of downstream tasks.
[0091] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0092] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0093] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.
[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.
[0095] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A self-supervised monocular depth estimation transfer learning method for construction safety, characterized by: The following steps are involved: Collect construction site images containing construction equipment and workers, annotate and segment all construction site images, and build a migration dataset; Build a self-supervised monocular depth estimation network model and load the pre-trained parameters of the monocular depth estimation network model; The monocular depth estimation network model is transferred and trained based on the migration dataset, and the training process is constrained by the loss function. Use quantitative evaluation indicators to evaluate the performance of the monocular depth estimation network model after each training, and screen out the optimal monocular depth estimation network model; The depth loss calculated by the loss function includes: The loss caused by the deep deviation between the random variables in each annotation box and the corresponding mean; And, the loss corresponding to the depth order between all annotation boxes; The calculation formula for the loss caused by the depth deviation is specifically: in, Indicates the loss caused by depth deviation; Indicates the number of annotation boxes in the current image; It means the current image Marking box area ; Represents the normalized depth map of the current image corresponding to the depth map predicted by the depth estimation network; represents a random variable The standard deviation of The calculation formula of the depth map is as follows: in, Represents the depth map corresponding to the current image predicted by the depth estimation network; 、 Respectively The maximum and minimum values in ; The loss calculation formula corresponding to the depth order is specifically: in, Indicates the loss corresponding to the depth order; Indicates the number of annotation boxes in the current image; Indicates the The annotation box and The inter-box distance estimation loss between the labeled boxes; and They are all integers, representing the serial number of the annotation box in the image. The larger the serial number, the greater the average depth value of the annotation box. The calculation formula for distance estimation loss is as follows: in, Represents a function that finds the mean value of the corresponding random variable; This is to avoid the perturbations introduced by trivial solutions.
2. The self-supervised monocular depth estimation transfer learning method for construction safety according to claim 1 is characterized in that: The annotation box in the construction site image is annotated using a rectangular bounding box; A rectangular bounding box is an area marked at any depth with distinct levels and equal depth inside; The labels of the rectangular bounding boxes are the serial numbers of all selected rectangular bounding boxes from small to large according to depth.
3. The self-supervised monocular depth estimation transfer learning method for construction safety according to any one of claims 1-2, characterized in that: The migration training process of the monocular depth estimation network model is specifically as follows: Load the PackNet network model pre-trained on the Kitti autonomous driving dataset and freeze the Encoder module of the model; Load the training images and labels of the dataset; Train the PackNet model and use the transfer loss function to constrain the parameters of the Decoder module of the model; The model after each round of training is tested using the validation set and evaluation metrics of the dataset.
4. The self-supervised monocular depth estimation transfer learning method for construction safety according to any one of claims 1-2, characterized in that: The performance evaluation process of the monocular depth estimation network model is specifically as follows: Comprehensive evaluation is performed based on the relative average error of depth deviation and the sequential accuracy of depth order; If the relative average error is lower and the sequential accuracy is higher, the performance of the monocular depth estimation network model is good.
5. A self-supervised monocular depth estimation transfer learning system for construction safety, characterized by: A method for implementing a self-supervised monocular depth estimation transfer learning method for construction safety according to any one of claims 1 to 4, comprising: The data acquisition module is used to collect construction site images including construction equipment and workers, and to label and divide all construction site images to build a migration dataset; The model building module is used to build a self-supervised monocular depth estimation network model and load the pre-trained parameters of the monocular depth estimation network model; The transfer training module is used to perform transfer training on the monocular depth estimation network model based on the transfer dataset, and the training process is constrained by the loss function; The evaluation and screening module is used to evaluate the performance of the monocular depth estimation network model after each training using quantitative evaluation indicators, and to screen out the optimal monocular depth estimation network model.
6. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the self-supervised monocular depth estimation transfer learning method for construction safety as described in any one of claims 1 to 4.
7. A computer-readable medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the self-supervised monocular depth estimation transfer learning method for construction safety as described in any one of claims 1 to 4.