Detection Method and Device for Covering Bare Soil, and Training Method and Device for Recognition Model

Abstract

The present application relates to the field of artificial intelligence technology, and discloses a method for detecting the covering of bare soil, including: obtaining a plurality of images to be detected reflecting different perspectives of the construction site according to a preset overlapping rate; inputting the plurality of images to be detected into a preset recognition model to obtain a recognition result corresponding to each image to be detected, wherein the recognition result includes a detection frame of bare soil and the category of the bare soil in the detection frame; performing a duplicate removal operation on the detection frames of the plurality of bare soils to obtain at least one target detection frame; and generating and sending a corresponding alarm message when the category of the bare soil in any of the target detection frames is uncovered. The present application also discloses a method for training a recognition model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, for example, to a method and device for detecting the covering of bare soil, and a method and device for training an identification model. Background Art

[0002] Currently, in the field of environmental protection, pollution source tracing is crucial for environmental protection. Pollution source tracing involves a large number of scenarios and the application of many technical means. Among them, dust is one of the most common situations of environmental pollution, and the covering of bare soil at the construction site is an important link to prevent dust. In order to improve the monitoring accuracy of the covering of muck trucks, a method for detecting the covering of muck trucks in related technologies includes: obtaining a video image frame to be detected; detecting whether the covering of the muck truck is closed according to the video image frame to be detected; and if the covering of the muck truck is not closed, sending a prompt message.

[0003] In the process of implementing the embodiments of the present disclosure, it is found that at least the following problems exist in the related technologies:

[0004] The detection method of the prior art can only be used to detect whether the covering of the muck truck is closed, and cannot identify the covering situation of the bare soil at the construction site. Currently, the covering situation of the bare soil at the construction site is still determined by manual inspection by staff. However, the manual inspection method has very large instability and subjectivity on the one hand, and on the other hand, it is also easy to be covered up by the construction site personnel targeted, resulting in ineffective inspection. Moreover, the manual inspection method is difficult to achieve non-blind spot monitoring and cannot detect all day long. Summary of the Invention

[0005] To have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. The summary is not a general review, nor is it intended to identify key / important constituent elements or delineate the protection scope of these embodiments, but rather serves as a preface to the subsequent detailed description.

[0006] The embodiments of the present disclosure provide a method and device for detecting the covering of bare soil, and a method and device for training an identification model, and realize the artificial intelligence detection of the covering of bare soil at the construction site through the wide application of deep learning in the field of object detection.

[0007] In some embodiments, the method for detecting the covering of bare soil includes:

[0008] Obtaining a plurality of images to be detected reflecting different perspectives of the construction site according to a preset overlap rate;

[0009] Inputting the plurality of images to be detected into a preset identification model to obtain an identification result corresponding to each image to be detected, where the identification result includes a detection frame of the bare soil and the category of the bare soil in the detection frame;

[0010] Perform a duplicate removal operation on the detection frames of multiple pieces of bare soil to obtain at least one target detection frame;

[0011] When the category of the bare soil in any of the target detection frames is uncovered, generate and send a corresponding alarm message

[0012] In some embodiments, the training method of the recognition model includes:

[0013] Obtain training samples and label training labels, where the training samples include multiple construction site images containing bare soil, and the training labels include the position information of at least one reference frame corresponding to each construction site image and the category of the bare soil in the reference frame;

[0014] Train a pre-trained DERT model according to the training samples and training labels;

[0015] Construct a weighted loss function, and continuously train the DERT model by performing weighted backpropagation on the weighted loss function to obtain the recognition model.

[0016] In some embodiments, the detection device for bare soil covering includes:

[0017] A camera control module, configured to obtain a plurality of images to be detected reflecting different perspectives of the construction site according to a preset overlap rate;

[0018] A detection module, configured to input the plurality of images to be detected into a preset recognition model to obtain a recognition result corresponding to each image to be detected, where the recognition result includes a detection frame of the bare soil and the category of the bare soil in the detection frame;

[0019] An identification module, configured to perform a duplicate removal operation on the detection frames of multiple pieces of bare soil to obtain at least one target detection frame;

[0020] An alarm module, configured to generate and send a corresponding alarm message when the category of the bare soil in any of the target detection frames is uncovered.

[0021] In some embodiments, the training device of the recognition model includes:

[0022] A sample label acquisition module, configured to obtain training samples and label training labels, where the training samples include multiple construction site images containing bare soil, and the training labels include the position information of at least one reference frame corresponding to each construction site image and the category of the bare soil in the reference frame;

[0023] A model training module, configured to train a pre-trained DERT model according to the training samples and training labels;

[0024] A model feedback adjustment module, configured to construct a weighted loss function and continuously train the DERT model by performing weighted backpropagation on the weighted loss function to obtain the recognition model.

[0025] The bare soil covering detection method and device, and the recognition model training method and device provided by the embodiments of the present disclosure can achieve the following technical effects:

[0026] By combining deep learning technology with the factory safety mechanism, this application acquires multiple images to be detected reflecting different perspectives of the construction site, and uses the recognition model to perform target recognition on the images to be detected for whether the bare soil is covered, realizing all-weather real-time monitoring of the covering situation of the bare soil at the construction site and sending out alarm information, solving problems such as instability and subjectivity brought by manual inspection, and being of great significance for pollution traceability and ecological environment protection.

[0027] The above general description and the following description are only exemplary and explanatory, and are not used to limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] One or more embodiments are exemplarily illustrated by corresponding drawings. These exemplary illustrations and the drawings do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation, and among them:

[0029] Figure 1 is a schematic diagram of a bare soil covering detection method provided by an embodiment of the present disclosure;

[0030] Figure 2 is a schematic diagram of another bare soil covering detection method provided by an embodiment of the present disclosure;

[0031] Figure 3 is a schematic diagram of another bare soil covering detection method provided by an embodiment of the present disclosure;

[0032] Figure 4 is a schematic diagram of another bare soil covering detection method provided by an embodiment of the present disclosure;

[0033] Figure 5 is a schematic diagram of a recognition model training method provided by an embodiment of the present disclosure;

[0034] Figure 6 is a schematic diagram of another recognition model training method provided by an embodiment of the present disclosure;

[0035] Figure 7 is a schematic diagram of another recognition model training method provided by an embodiment of the present disclosure;

[0036] Figure 8 It is a schematic diagram of a training method for another recognition model provided by an embodiment of the present disclosure;

[0037] Figure 9 It is a schematic diagram of a detection device for uncovering bare soil and a training device for a recognition model provided by an embodiment of the present disclosure;

[0038] Figure 10 It is a schematic framework diagram of a computing device provided by an embodiment of the present disclosure. Detailed implementation manners

[0039] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The attached drawings are for reference and illustration only and are not used to limit the embodiments of the present disclosure. In the following technical descriptions, for the sake of explanation, numerous details are provided to give a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and bare soil covering recognition models may be shown in a simplified manner to simplify the drawings.

[0040] In the embodiments of the present disclosure, terms such as "first" and "second" in the specification, claims, and the above-mentioned drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so as to implement the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion.

[0041] Unless otherwise specified, the term "plurality" means two or more.

[0042] In the embodiments of the present disclosure, the character " / " indicates that the objects before and after are in an "or" relationship. For example, A / B means: A or B.

[0043] The term "and / or" is a description of the association relationship of an object and indicates that three relationships can exist. For example, A and / or B means: A or B, or, the three relationships of A and B.

[0044] The term "corresponding" may refer to an association relationship or a binding relationship. A corresponding to B means that there is an association relationship or a binding relationship between A and B.

[0045] In the embodiments of the present disclosure, a terminal device refers to an electronic device with wireless connection capabilities. The terminal device can communicate with the intelligent home appliance devices as described above by connecting to the Internet, or directly communicate with the intelligent home appliance devices as described above through means such as Bluetooth and Wi-Fi. In some embodiments, the terminal device can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, or other hardware devices with various operating systems, touch screens, and / or display screens.

[0046] Currently, the covering of bare soil at construction sites is determined by manual inspection by workers. This method is completed by human subjective behavior. On the one hand, there is a very high degree of instability and subjectivity. On the other hand, it is also easy to be covered up purposefully by construction site personnel. At the same time, the manual method is difficult to monitor in real time without dead spots and cannot monitor all day long.

[0047] Deep learning is playing an increasingly important role in the field of object detection. It uses advanced convolutional neural theory, deep belief networks, neural networks, etc. to construct a multi-level learning system, which can achieve feature detection, object detection, etc., improving the accuracy, reliability, and advancement of machine learning.

[0048] The embodiments of the present disclosure relate to the field of artificial intelligence technology, specifically to technical fields such as computer vision and deep learning, and can be applied to intelligent clouds and construction sites, improving the efficiency of ecological traceability and playing a positive role in protecting the ecological environment.

[0049] Among them, artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence.

[0050] Deep learning is to learn the internal laws and representation levels of sample data. The information obtained during these learning processes is very helpful for the interpretation of data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have the ability to analyze and learn like humans, and be able to recognize data such as text, images, and sounds.

[0051] Computer vision refers to using cameras and computers to replace human eyes for object recognition, tracking, measurement, and other machine vision, and further performing graphics processing to make the computer-processed images more suitable for human eye observation or transmission to instruments for detection.

[0052] DETR (Detection Transformer, the vision version of Transformer) model: Using a common backbone network (such as Residual Network ResNet, etc.), through feature extraction of video frames, generating a feature map, and then inputting the feature map into a model with Transformer as the basic structure for encoding-decoding, finally outputting the detection box and the category of the target object in the detection box.

[0053] Combined with Figure 1 As shown, the embodiments of the present disclosure provide a method for detecting the covering of bare soil, including:

[0054] Step 101: Obtain a plurality of images to be detected reflecting different perspectives of the construction site according to a preset overlap rate.

[0055] In the embodiments of the present application, the camera control module of the present application can, according to the preset overlap rate, continuously rotate the cameras deployed at the construction site and take pictures through a plurality of cameras, and obtain a plurality of images to be detected reflecting different perspectives of the construction site.

[0056] Step 102: Input the plurality of images to be detected into a preset recognition model to obtain the recognition result corresponding to each image to be detected, where the recognition result includes the detection box of the bare soil and the category of the bare soil in the detection box.

[0057] In the embodiments of the present application, the present application performs object detection on a plurality of images to be detected through a pre-trained recognition model to obtain the recognition result corresponding to each image to be detected, where the recognition result includes the detection box of the bare soil, the category of the bare soil in the detection box (uncovered or covered) and the predicted category probability, etc.

[0058] Step 103: Perform a duplicate removal operation on the detection boxes of the plurality of bare soils to obtain at least one target detection box.

[0059] In the embodiments of the present application, due to the overlap during the shooting of the cameras of the present application, there will be a large overlap between the detection boxes. At this time, in order to reduce unnecessary calculations, it is necessary to perform a duplicate removal operation on the detection boxes of the plurality of bare soils, delete the repeatedly detected areas, and obtain at least one target detection box.

[0060] Step 104: Generate and send a corresponding alarm message when the category of the bare soil in any of the target detection boxes is uncovered.

[0061] In the embodiments of the present application, when the category of the bare soil in any of the target detection boxes is uncovered, that is, there is a situation where there is bare soil that is not covered, an alarm process is performed.

[0062] By adopting the detection method for covering bare soil provided by the embodiments of the present disclosure, through combining deep learning technology with the factory safety mechanism, by acquiring multiple images to be detected reflecting different perspectives of the construction site, and through using an identification model to perform target identification on the images to be detected for whether the bare soil is covered, it realizes all-weather real-time monitoring of the covering situation of the bare soil at the construction site and sends out alarm information, solves problems such as instability and subjectivity brought by manual inspection, realizes all-weather and non-blind-spot monitoring and detection, and is of great significance for pollution traceability and ecological environment protection.

[0063] Optionally, in combination with Figure 2 As shown, the obtaining of multiple images to be detected reflecting different perspectives of the construction site according to a preset overlap rate includes:

[0064] Step 201: Calculate the horizontal size of the image to be detected according to the focal length of the shooting device, the size of the shooting device in the horizontal direction, and the horizontal width of the construction site.

[0065] In the embodiments of the present application, by querying the focal length f of the shooting device such as a camera and the size S of the shooting device in the horizontal direction w , combined with the horizontal width B of the construction site obtained by on-site measurement w , according to the following formula, calculate the horizontal size w of the image to be detected:

[0066]

[0067] Step 202: Calculate the required number of rotation steps of the shooting device and the corresponding rotation angle for each step according to the total angle required for the shooting device to shoot the panoramic view of the construction site and the horizontal size of the image to be detected.

[0068] In the embodiments of the present application, according to the total angle A required for the shooting device to shoot the panoramic view of the construction site obtained by testing the pan-tilt of the on-site rotating camera, and the horizontal size w of the image to be detected, calculate the required number of rotation steps t of the shooting device and the corresponding rotation angle ω for each step:

[0069]

[0070] Wherein, represents rounding x downwards, and t is the quotient of the total angle A and the corresponding rotation angle ω for each step.

[0071] Step 203: Starting from the extreme position of the shooting device, control the shooting device to rotate according to the required number of rotation steps of the shooting device and the corresponding rotation angle for each step, so that the shooting device shoots the construction site after each rotation, and obtain multiple images to be detected reflecting different perspectives of the construction site.

[0072] In an embodiment of the present application, starting from the extreme position of the shooting device, such as the leftmost extreme position or the rightmost extreme position of the camera pan-tilt, the shooting device rotates according to the rotation angle ω and takes pictures until the number of steps corresponding to the rotation steps t (usually a single-digit number, such as 3 steps, 4 steps, or 5 steps) is completed, and a plurality of images to be detected reflecting different perspectives of the construction site are obtained. It should be noted that in the case where a plurality of shooting devices are arranged at the construction site, multiple sets of images to be detected will be obtained.

[0073] In this way, in the present application, the camera control module continuously rotates the camera and takes pictures according to the calculated rotation steps and angles, and then pushes them to the recognition model, so as to realize the information collection of the bare soil covering situation at the construction site without manual participation.

[0074] Optionally, as shown in Figure 3 performing a duplicate removal operation on the detection frames of the plurality of bare soils to obtain at least one target detection frame, including:

[0075] Step 301: Obtain the detection frames of any two bare soils, and calculate the overlapping area of the two detection frames.

[0076] In an embodiment of the present application, the n detection frames output by the recognition model are:

[0077]

[0078] Among them, x represents the abscissa of the upper left corner of the detection frame, y represents the ordinate of the detection frame, w represents the width of the detection frame, h represents the height of the detection frame, c represents the predicted category (covered or uncovered), and p represents the probability of the predicted category being c.

[0079] In the present application, any two detection frames of bare soil are obtained from the n detection frames, and the overlapping area Inter of the two detection frames is calculated through a function.

[0080] Step 302: Calculate the intersection over union of the two detection frames according to the overlapping area of the two detection frames.

[0081] In an embodiment of the present application, further according to the overlapping area of the two detection frames, the intersection over union IoU of the two detection frames is calculated through a function.

[0082] Step 303: When the intersection over union of the two detection frames is greater than or equal to a preset intersection over union threshold and the categories of the bare soil in the two detection frames are the same, delete the detection frame with a smaller predicted category probability in the two detection frames to obtain at least one target detection frame of the bare soil.

[0083] In an embodiment of the present application, a predefined intersection over union threshold τ is set. If the intersection over union (IoU) of two detection boxes is greater than or equal to the intersection over union threshold τ, and the classes of the bare soil in the two detection boxes are the same (both covered or uncovered), then compare the predicted class probabilities in the two detection boxes, and select the detection box with the smaller predicted class probability Delete.

[0084] In this way, the detection boxes with a higher degree of overlap can be more accurately determined, and the detection box with the most judgment value can be reasonably retained.

[0085] Optionally, as shown in Figure 4 the method for detecting the covering of bare soil in the present application further includes:

[0086] Step 401: Bind multiple images to be detected, at least one target detection box of bare soil, the class of the bare soil in each target detection box, and the predicted class probability of each target detection box to the construction site information of the construction site to which the photographing device belongs, to obtain construction site bare soil covering detection information.

[0087] Step 402: Store the construction site bare soil covering detection information in a database.

[0088] Step 403: Provide a user query function interface, and query the construction site bare soil covering detection information according to the retrieval conditions input by the user.

[0089] In an embodiment of the present application, the present application binds multiple images to be detected, at least one target detection box of bare soil, the class of the bare soil in each target detection box, and the predicted class probability of each target detection box to the construction site information of the construction site to which the photographing device belongs, to obtain construction site bare soil covering detection information, and then stores the construction site bare soil covering detection information in a database, and at the same time provides a user query function, and can query the construction site covering detection information according to time, location, and whether it is covered, etc.

[0090] In this way, the efficiency of ecological traceability can be improved, and evidence and files can be retained for subsequent supervision and rectification.

[0091] As shown in Figure 5 an embodiment of the present disclosure provides a method for training an identification model, including:

[0092] Step 501: Obtain training samples and label training labels, where the training samples include multiple construction site images containing bare soil, and the training labels include the position information of at least one reference box corresponding to each construction site image and the class of the bare soil in the reference box.

[0093] In the embodiments of the present application, the present application can obtain image data through a security camera, select video and image data under different construction sites, different lighting conditions, and different weather conditions, and then transmit the obtained video and picture data to this system.

[0094] During the video frame extraction process, this system needs to perform a frame skipping operation on the obtained video. To cover different situations as much as possible and improve the accuracy of the present invention, when performing the frame skipping process, frames with obvious differences in lighting are selected from different construction sites. Some image scenes and lighting are very similar, and the images are blurred, which do not play a positive role in model training and are discarded. Images with different scenes, diverse lighting, and comprehensive bare soil shooting are retained.

[0095] The video can be an online collected video. For example, through web crawler technology, an online video to be detected can be collected. Or, the video can also be an offline collected video. Or, the video can also be a real-time collected video stream. Or, the video can also be an artificially synthesized video, etc. The embodiments of the present disclosure do not limit this.

[0096] For example, more than 2000 images including covered or uncovered bare soil are collected for label annotation, mainly relying on manual labor through relevant annotation software. Both covered and uncovered bare soil are annotated. During the annotation process, for some bare soil that is incomplete in the image, the visible part in the image is annotated. For bare soil with an edge less than 50 pixels, no annotation is performed. After the data annotation is completed, it is divided into a training set, a validation set, a test set, etc., to obtain m reference bounding boxes σ i (x i ,y i ,w i ,h i ,l i ), where i = 1, 2,..., m, and the five components are the abscissa, ordinate, width, height, and label of the upper left corner of the region respectively. The label l i = 1 indicates covered, and l i = 0 indicates uncovered.

[0097] Step 502: Train the pre-trained DERT model according to the training samples and training labels.

[0098] In the embodiments of the present application, the present application uses a pre-trained DERT model for training. Among them, the front-end network structure until the FFNFFN (Feed-Forward Network) layer is the same as the DETR model. After reaching the FFN, n prediction bounding boxes are obtained where j = 1, 2,..., n, the predicted category indicates covered, indicates uncovered, indicates that the predicted category is probability

[0099] Step 503: Construct a weighted loss function, and continuously train the DERT model by performing weighted backpropagation on the weighted loss function to obtain the recognition model.

[0100] Using the training method of the recognition model provided by the embodiments of the present disclosure, for the situation that a certain instance cannot be found or the position is completely misdetected in the object detection method based on the DETR model, the problem of undetected or misdetected partial target regions is solved by constructing a weighted loss function for reverse weighted propagation, and the accuracy of the model is improved.

[0101] Optionally, in combination with Figure 6 As shown, the constructing a weighted loss function and continuously training the DERT model by performing weighted backpropagation on the weighted loss function includes:

[0102] Step 601: Obtain a plurality of predicted bounding boxes output by the DERT model.

[0103] Step 602: Traverse and calculate the intersection over union (IoU) of each predicted bounding box with all the reference bounding boxes, and determine the target reference bounding box corresponding to the minimum IoU of each predicted bounding box.

[0104] Step 603: Traverse and calculate the loss value of each predicted bounding box with the target reference bounding box to obtain the overall loss value corresponding to all the predicted bounding boxes.

[0105] Step 604: Perform backpropagation according to the overall loss value to continuously train the DERT model.

[0106] In the embodiments of the present application, the constructed weighted loss function of the present application includes two-layer loops. The outer loop is the predicted bounding box loop, and the inner loop is the loop of the labeled reference bounding boxes. Among them, in the inner loop, the calculation is performed simultaneously, and the reference bounding box with the minimum IoU with the current predicted bounding box is found simultaneously. Finally, the center distance, the predicted bounding box loss, and the weighted loss are calculated in sequence using this found reference bounding box, and finally the weighted loss is added to the overall loss. After all the predicted bounding boxes in the outer layer are calculated, backpropagation is performed using the accumulated overall loss.

[0107] In this way, it can be realized that during the feedback adjustment process of model training, the predicted bounding boxes are randomly traversed and selected, a target reference bounding box with high confidence is obtained in each iteration process, and all the predicted bounding boxes are weighted according to the IoU with the target reference bounding box, so that each predicted bounding box contributes to the training.

[0108] Optionally, in combination with Figure 7 As shown, the traversing and calculating the intersection over union (IoU) of each predicted bounding box with all the reference bounding boxes and determining the target reference bounding box corresponding to the minimum IoU of each predicted bounding box includes:

[0109] Step 701: Calculate the overlapping area between the current prediction box and the current reference box.

[0110] In the embodiments of the present application, the overlapping area between any current reference box and any current prediction box is calculated by the following formula:

[0111]

[0112] wherein, it includes m reference boxes σ i (x i , y i , w i , h i , l i ), and n prediction boxes to obtain the corresponding overlapping area Inter.

[0113] Step 702: Calculate the intersection-over-union (IoU) between the current prediction box and the current reference box according to the overlapping area between the current prediction box and the current reference box.

[0114] In the embodiments of the present application, the intersection-over-union (IoU) between any current reference box and any current prediction box is calculated by the following formula:

[0115]

[0116] Step 703: In the case where the intersection-over-union (IoU) between the current prediction box and the current reference box is less than the minimum value of the cached intersection-over-union (IoU), cache the intersection-over-union (IoU) between the current prediction box and the current reference box as the minimum value of the intersection-over-union (IoU).

[0117] In the embodiments of the present application, the closest region is solved by the following formula:

[0118]

[0119] Step 704: Repeat the above steps until all reference boxes are traversed to determine the target reference box corresponding to the current prediction box.

[0120] In the embodiments of the present application, it is judged whether there are still uncalculated reference boxes. If so, repeat steps 701 to 703 to determine the target reference box corresponding to the current prediction box in a loop manner.

[0121] Step 705: Repeat the above steps until all prediction boxes are traversed to determine the target reference box corresponding to each prediction box.

[0122] In an embodiment of the present application, it is determined whether there are still uncalculated prediction boxes. If so, steps 701 to 704 are repeated to determine the target reference box corresponding to each prediction box in a loop manner.

[0123] In this way, by means of an internal loop, the target reference box with the intersection over union (IoU) closest to each prediction box is determined, improving the training effect of the model.

[0124] Optionally, as shown in Figure 8 The traversal calculates the loss value between each prediction box and the target reference box to obtain the overall loss value corresponding to all prediction boxes, including:

[0125] Step 801: Calculate the center distance between each prediction box and the corresponding target reference box.

[0126] In an embodiment of the present application, the center distance between each prediction box and the corresponding target reference box is calculated by the following formula:

[0127]

[0128] Step 802: Calculate the coordinate loss function of each prediction box according to the center distance between each prediction box and the corresponding target reference box, the IoU between each prediction box and the corresponding target reference box, and the IoU-position weight coefficient.

[0129] In an embodiment of the present application, the coordinate loss function of each prediction box is calculated by the following formula:

[0130]

[0131] Among them, the IoU-position weight coefficient needs to meet the following conditions:

[0132] λ IoU +λ d =1,λ IoU >0,λ d >0

[0133] Step 803: Calculate the weighted loss function according to the IoU between each prediction box and the corresponding target reference box, the indicator function, and the coordinate loss function of each prediction box to obtain the overall loss value corresponding to all prediction boxes.

[0134] In an embodiment of the present application, the weighted loss function is calculated by the following formula:

[0135]

[0136] Among them, the definition of the indicator function I is:

[0137]

[0138] In this way, the present application performs backpropagation according to the weighted loss function and continuously trains to obtain the final model to implement the outer loop.

[0139] Combined with Figure 9 As shown, an embodiment of the present disclosure provides a detection device for uncovering bare soil, including:

[0140] A camera control module 901, configured to obtain a plurality of images to be detected reflecting different perspectives of the construction site according to a preset overlap rate;

[0141] A detection module 902, configured to input the plurality of images to be detected into a preset recognition model to obtain a recognition result corresponding to each image to be detected, where the recognition result includes a detection frame of the bare soil and the category of the bare soil in the detection frame;

[0142] A recognition module 903, configured to perform a duplicate removal operation on a plurality of detection frames of the bare soil to obtain at least one target detection frame;

[0143] An alarm module 904, configured to generate and send a corresponding alarm message when the category of the bare soil in any one of the target detection frames is uncovered.

[0144] Optionally, the camera control module 901 is specifically configured to:

[0145] Calculate the horizontal size of the image to be detected according to the focal length of the shooting device, the horizontal size of the shooting device along the horizontal direction, and the horizontal width of the construction site;

[0146] Calculate the number of rotation steps required for the shooting device and the rotation angle corresponding to each step according to the total angle required for the shooting device to capture the panoramic view of the construction site and the horizontal size of the image to be detected;

[0147] Start from the extreme position of the shooting device, and control the shooting device to rotate according to the number of rotation steps required for the shooting device and the rotation angle corresponding to each step, so that the shooting device captures the construction site after each rotation to obtain a plurality of images to be detected reflecting different perspectives of the construction site.

[0148] Optionally, the recognition module 903 is specifically configured to:

[0149] Obtain any two detection frames of the bare soil and calculate the overlapping area of the two detection frames;

[0150] Calculate the intersection over union of the two detection frames according to the overlapping area of the two detection frames;

[0151] When the intersection over union (IoU) of the two detection boxes is greater than or equal to a preset IoU threshold and the classes of the bare soil in the two detection boxes are the same, delete the detection box with a smaller predicted class probability among the two detection boxes to obtain at least one target detection box for the bare soil.

[0152] Optionally, the device further includes a storage and query module, configured to:

[0153] Bind multiple images to be detected, at least one target detection box for the bare soil, the class of the bare soil in each target detection box, and the predicted class probability of each target detection box to the construction site information of the construction site to which the photographing device belongs to obtain construction site bare soil covering detection information;

[0154] Store the construction site bare soil covering detection information in a database;

[0155] Provide a user query function interface to query the construction site bare soil covering detection information according to the retrieval conditions input by the user.

[0156] In this application, by combining deep learning technology with the factory safety mechanism, by obtaining multiple images to be detected reflecting different perspectives of the construction site, and through the recognition model to perform target recognition on whether the bare soil in the images to be detected is covered, it realizes all-weather real-time monitoring of the covering situation of the bare soil at the construction site and sends out alarm information, solving the problems of instability and subjectivity brought by manual inspection, and is of great significance for pollution traceability and ecological environment protection.

[0157] Combined Figure 9 As shown, the embodiments of the present disclosure provide a training device for an identification model, including:

[0158] A sample label acquisition module 905, configured to acquire training samples and label training labels, where the training samples include multiple construction site images containing bare soil, and the training labels include the position information of at least one reference box corresponding to each construction site image and the class of the bare soil in the reference box;

[0159] A model training module 906, configured to train a pre-trained DERT model according to the training samples and training labels;

[0160] A model feedback adjustment module 907, configured to construct a weighted loss function, and continuously train the DERT model by performing weighted backpropagation on the weighted loss function to obtain the identification model.

[0161] Optionally, the model feedback adjustment module 907 is specifically configured to:

[0162] Obtain multiple prediction boxes output by the DERT model;

[0163] Traverse and calculate the intersection over union (IoU) of each predicted bounding box with all the ground truth bounding boxes, and determine the target ground truth bounding box corresponding to the minimum IoU for each predicted bounding box;

[0164] Traverse and calculate the loss value of each predicted bounding box with the target ground truth bounding box, and obtain the overall loss value corresponding to all the predicted bounding boxes;

[0165] Perform backpropagation based on the overall loss value to continuously train the DERT model.

[0166] Optionally, the model feedback adjustment module 907 is specifically configured to:

[0167] Calculate the overlapping area between the current predicted bounding box and the current ground truth bounding box;

[0168] Calculate the intersection over union (IoU) of the current predicted bounding box and the current ground truth bounding box based on the overlapping area between the current predicted bounding box and the current ground truth bounding box;

[0169] In the case where the IoU of the current predicted bounding box and the current ground truth bounding box is less than the cached minimum IoU value, cache the IoU of the current predicted bounding box and the current ground truth bounding box as the minimum IoU value;

[0170] Repeat the above steps until all the ground truth bounding boxes are traversed to determine the target ground truth bounding box corresponding to the current predicted bounding box;

[0171] Repeat the above steps until all the predicted bounding boxes are traversed to determine the target ground truth bounding box corresponding to each predicted bounding box.

[0172] Optionally, the model feedback adjustment module 907 is specifically configured to:

[0173] Calculate the center distance between each predicted bounding box and the corresponding target ground truth bounding box;

[0174] Calculate the coordinate loss function of each predicted bounding box based on the center distance between each predicted bounding box and the corresponding target ground truth bounding box, the IoU of each predicted bounding box and the corresponding target ground truth bounding box, and the IoU - position weight coefficient;

[0175] Calculate the weighted loss function based on the IoU of each predicted bounding box and the corresponding target ground truth bounding box, the indicator function, and the coordinate loss function of each predicted bounding box, and obtain the overall loss value corresponding to all the predicted bounding boxes.

[0176] In view of the situation that the object detection method based on the DETR model may not find a certain instance or completely mislocate it, this application constructs a weighted loss function for backpropagation with weights to solve the problem that some target regions cannot be detected or are detected incorrectly, and improves the accuracy of the model.

[0177] Combined with Figure 10As shown in the figure, an embodiment of the present disclosure provides a computing device, including a processor 100 and a memory 101. Optionally, the computing device may further include a communication interface 102 and a bus 103. Among them, the processor 100, the communication interface 102, and the memory 101 can complete communication with each other through the bus 103. The communication interface 102 can be used for information transmission. The processor 100 can call the logical instructions in the memory 101 to execute the detection method of the bare soil covering and the training method of the recognition model in the above embodiments.

[0178] In addition, when the logical instructions in the above-mentioned memory 101 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0179] The memory 101, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of the present disclosure. The processor 100 executes functional applications and data processing by running the program instructions / modules stored in the memory 101, that is, implements the detection method of the bare soil covering and the training method of the recognition model in the above embodiments.

[0180] The memory 101 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 101 may include a high-speed random access memory and may also include a non-volatile memory.

[0181] An embodiment of the present disclosure provides a storage medium storing computer-executable instructions, and the computer-executable instructions are set to execute the training method of the recognition model and the detection method of the bare soil covering.

[0182] The above storage medium can be a transient computer-readable storage medium or a non-transient computer-readable storage medium.

[0183] The technical solution of the embodiments of the present disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present disclosure. The foregoing storage medium may be a non-transitory storage medium, including: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, or it may also be a transitory storage medium.

[0184] The above description and the drawings fully illustrate the embodiments of the present disclosure, enabling those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The embodiments only represent possible variations. Unless explicitly required, separate components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terms used in this application are only for describing the embodiments and do not limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to also include the plural forms. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations including one or more of the associated listed items. Additionally, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising", etc. mean the presence of the stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groupings of these. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of another identical element in the process, method, or device including the element. In this article, each embodiment may focus on the differences from other embodiments, and the same or similar parts between the embodiments may be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, the relevant parts may refer to the description of the method part.

[0185] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner can depend on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the embodiments of the present disclosure. The skilled person can clearly understand that for the convenience and conciseness of description, the specific working processes of the system, the bare soil covering identification model, and the units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0186] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to the bare soil covering identification model, devices, etc.) can be implemented in other ways. For example, the embodiments of the bare soil covering identification model described above are merely illustrative. For example, the division of the units can be merely a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Additionally, the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the bare soil covering identification model or units can be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to implement this embodiment. Additionally, in the embodiments of the present disclosure, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the block may occur in an order different from that noted in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. Each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A detection method for covering bare soil, characterized in that, Including: Obtain multiple images to be detected that reflect different perspectives of the construction site according to a preset overlap rate; Input the multiple images to be detected into a preset recognition model to obtain a recognition result corresponding to each image to be detected, where the recognition result includes a detection frame of bare soil and the category of the bare soil in the detection frame; Perform a duplicate removal operation on the detection frames of the multiple bare soils to obtain at least one target detection frame; When the category of the bare soil in any of the target detection frames is uncovered, generate and send a corresponding alarm message; The performing a duplicate removal operation on the detection frames of the multiple bare soils to obtain at least one target detection frame includes: obtaining any two detection frames of bare soil, calculating the overlapping area of the two detection frames; calculating the intersection-over-union ratio of the two detection frames according to the overlapping area of the two detection frames; when the intersection-over-union ratio of the two detection frames is greater than or equal to a preset intersection-over-union ratio threshold and the categories of the bare soil in the two detection frames are the same, delete the detection frame with a smaller predicted category probability among the two detection frames to obtain at least one target detection frame of bare soil; The training method of the recognition model includes: obtaining training samples and annotating training labels, where the training samples include multiple construction site images containing bare soil, and the training labels include the position information of at least one reference frame corresponding to each construction site image and the category of the bare soil in the reference frame; training a pre-trained DERT model according to the training samples and training labels; constructing a weighted loss function, and continuously training the DERT model by performing weighted backpropagation on the weighted loss function to obtain the recognition model; the constructing a weighted loss function and continuously training the DERT model by performing weighted backpropagation on the weighted loss function includes: obtaining multiple prediction frames output by the DERT model; traversing and calculating the intersection-over-union ratio of each prediction frame and all reference frames, and determining the target reference frame with the smallest intersection-over-union ratio corresponding to each prediction frame; traversing and calculating the loss value of each prediction frame and the target reference frame to obtain the overall loss value corresponding to all prediction frames; performing backpropagation according to the overall loss value to continuously train the DERT model.

2. The detection method according to claim 1, wherein The obtaining multiple images to be detected that reflect different perspectives of the construction site according to a preset overlap rate includes: Calculate the horizontal size of the image to be detected according to the focal length of the imaging device, the horizontal size of the imaging device along the horizontal direction, and the horizontal width of the construction site; Calculate the number of rotation steps required for the imaging device and the rotation angle corresponding to each step according to the total angle required for the imaging device to capture the panoramic view of the construction site and the horizontal size of the image to be detected; Start from the extreme position of the imaging device, and control the imaging device to rotate according to the number of rotation steps required for the imaging device and the rotation angle corresponding to each step, so that the imaging device captures the construction site after each rotation to obtain multiple images to be detected that reflect different perspectives of the construction site.

3. The detection method according to claim 1, wherein Also including: Bind multiple images to be detected, at least one target detection box of bare soil, the category of bare soil in each target detection box, and the predicted category probability of each target detection box to the construction site information of the construction site to which the shooting device belongs to obtain the construction site bare soil covering detection information; Store the construction site bare soil covering detection information in a database; Provide a user query function interface to query the construction site bare soil covering detection information according to the retrieval conditions input by the user.

4. The detection method according to claim 1, characterized in that, The traversing and calculating the intersection over union of each prediction box and all reference boxes to determine the target reference box with the smallest intersection over union corresponding to each prediction box includes: Calculate the overlapping area between the current prediction box and the current reference box; Calculate the intersection over union of the current prediction box and the current reference box according to the overlapping area between the current prediction box and the current reference box; When the intersection over union of the current prediction box and the current reference box is less than the cached minimum intersection over union value, cache the intersection over union of the current prediction box and the current reference box as the minimum intersection over union value; Repeat the above steps until all reference boxes are traversed to determine the target reference box corresponding to the current prediction box; Repeat the above steps until all prediction boxes are traversed to determine the target reference box corresponding to each prediction box.

5. The detection method according to claim 1, wherein The traversing and calculating the loss value of each prediction box and the target reference box to obtain the overall loss value corresponding to all prediction boxes includes: Calculate the center distance between each prediction box and the corresponding target reference box; Calculate the coordinate loss function of each prediction box according to the center distance between each prediction box and the corresponding target reference box, the intersection over union of each prediction box and the corresponding target reference box, and the intersection over union-position weight coefficient; Calculate the weighted loss function according to the intersection over union of each prediction box and the corresponding target reference box, the indicator function, and the coordinate loss function of each prediction box to obtain the overall loss value corresponding to all prediction boxes.

6. A detection system for covering bare soil, characterized in that, Includes: The detection device for bare soil covering and the training device for the recognition model; The detection device for bare soil covering includes: A camera control module configured to obtain multiple images to be detected reflecting different perspectives of the construction site according to a preset overlap rate; A detection module configured to input the multiple images to be detected into a preset recognition model to obtain the recognition result corresponding to each image to be detected, where the recognition result includes the detection box of bare soil and the category of bare soil in the detection box; An identification module configured to perform a duplicate removal operation on multiple detection boxes of bare soil to obtain at least one target detection box; An alarm module configured to generate and send a corresponding alarm message when the category of bare soil in any of the target detection boxes is uncovered; The identification module is specifically configured to: obtain any two detection boxes of bare soil, calculate the overlapping area of the two detection boxes; calculate the intersection over union of the two detection boxes according to the overlapping area of the two detection boxes; when the intersection over union of the two detection boxes is greater than or equal to a preset intersection over union threshold and the categories of bare soil in the two detection boxes are the same, delete the detection box with a smaller predicted category probability among the two detection boxes to obtain at least one target detection box of bare soil; The training device for the recognition model includes: A sample label acquisition module, configured to acquire training samples and label training labels, wherein the training samples include a plurality of construction site images containing bare soil, and the training labels include position information of at least one reference box corresponding to each construction site image and the category of bare soil in the reference box; A model training module, configured to train a pre-trained DERT model according to the training samples and training labels; A model feedback adjustment module, configured to construct a weighted loss function, and continuously train the DERT model by performing weighted backpropagation on the weighted loss function to obtain the recognition model; Specifically, the model feedback adjustment module is configured to: acquire a plurality of predicted boxes output by the DERT model; traverse and calculate the intersection over union of each predicted box and all reference boxes, and determine the target reference box with the smallest intersection over union corresponding to each predicted box; traverse and calculate the loss value between each predicted box and the target reference box to obtain the overall loss value corresponding to all predicted boxes; and perform backpropagation according to the overall loss value to continuously train the DERT model.

Images