Construction scene object recognition method and device, equipment and medium
By introducing multi-level feature fusion and attention modules into the object detection model, the problem that the prior art cannot capture the nonlinear relationships in complex construction scenarios is solved, and the recognition accuracy and adaptability of construction scenario objects are improved.
Patent Information
- Application Number
- CN202410997821.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-23
AI Technical Summary
Existing object detection methods cannot effectively capture the nonlinear relationships of multiple categories of objects in complex construction scene images, resulting in insufficient feature representation and indiscriminateness, affecting the recognition accuracy of construction scene objects.
A construction scene object recognition method is adopted. By introducing backbone network, neck network and detection network into the object detection model, the backbone network includes a convolution processing module, a multi-level hourglass module, a multi-level depth convolution module and a spatial pyramid pool. The multi-level feature fusion module and multi-level attention module are set up in the neck network to perform multi-head self-attention processing to capture nonlinear relationships.
The recognition accuracy of construction scene objects is improved, especially the detection accuracy of targets of different sizes, and the adaptability of the backbone network to objects of different sizes and the feature fusion ability of the neck network is enhanced.
Smart Images

Figure CN119068408A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the technical field of construction scene image understanding, and in particular, to a construction scene object recognition method, device, equipment and medium. Background Art
[0002] Video surveillance systems are often deployed at construction sites of hydropower and civil engineering projects to make up for the shortcomings of manual monitoring and improve security monitoring efficiency.
[0003] As a basic computer vision analysis task, the target detection task based on deep learning technology can realize the recognition and positioning of multiple categories of objects such as various construction machinery and workers in the construction scene, provide key inputs for related objects in construction safety management, construction progress management, etc., and support the gradual realization of all-weather online monitoring and analysis based on video surveillance systems. Therefore, the recognition accuracy of the object directly affects the effectiveness of subsequent analysis tasks and engineering management goals.
[0004] In practical applications, the shapes, textures, and contextual information of multi-category objects in complex construction scene images are often nonlinear. However, current object detection methods cannot effectively capture these nonlinear relationships, resulting in insufficient and non-discriminative feature representation, which affects the recognition accuracy of construction scene objects. Summary of the invention
[0005] An embodiment of the present application provides a method for identifying objects in a construction scene, which can improve the recognition accuracy of objects in a construction scene.
[0006] Correspondingly, an embodiment of the present application also provides a construction scene object recognition device, an electronic device and a machine-readable medium to ensure the implementation and application of the above method.
[0007] In order to solve the above problems, the present application discloses a construction scene object recognition method, including:
[0008] Collect construction images;
[0009] According to the target detection model, the construction image is recognized to obtain a recognition result; the recognition result includes: the category of the target object and the position information of the target object in the construction image;
[0010] The target detection model includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network includes: a convolution processing module, a multi-level hourglass module, a multi-level deep convolution module and a spatial pyramid pooling module; the neck network includes: a multi-level feature fusion module and a multi-level attention module;
[0011] The identifying the construction image according to the target detection model includes:
[0012] The backbone network determines a plurality of first image features corresponding to the construction image;
[0013] A multi-level feature fusion module in the neck network performs feature fusion processing according to the multiple first image features to obtain multiple fused image features;
[0014] The multi-level attention module in the neck network performs multi-head self-attention processing on the multiple fused image features respectively to obtain multiple multi-head self-attention image features;
[0015] The detection network determines the recognition result based on the multiple multi-head self-attention image features.
[0016] The present application also discloses a construction scene object recognition device, the device comprising:
[0017] A collection module, used for collecting construction images;
[0018] An object recognition module is used to recognize the construction image according to the target detection model to obtain a recognition result; the recognition result includes: the category of the target object and the location information of the target object in the construction image;
[0019] The target detection model includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network includes: a convolution processing module, a multi-level hourglass module, a multi-level deep convolution module and a spatial pyramid pooling module; the neck network includes: a multi-level feature fusion module and a multi-level attention module;
[0020] The object recognition module comprises:
[0021] A feature extraction module, used to determine a plurality of first image features corresponding to the construction image using a backbone network;
[0022] A feature fusion module, used to utilize the multi-level feature fusion module in the neck network to perform feature fusion processing according to the multiple first image features to obtain multiple fused image features;
[0023] An attention processing module, used for performing multi-head self-attention processing on the multiple fused image features respectively using the multi-level attention modules in the neck network to obtain multiple multi-head self-attention image features;
[0024] The recognition result determination module is used to determine the recognition result based on the multiple multi-head self-attention image features using the detection network.
[0025] Optionally, the multi-level attention module in the neck network performs multi-head self-attention processing on the multiple fused image features respectively, including:
[0026] Performing weighted processing on the fused image features to obtain a query matrix, a key matrix and a value matrix;
[0027] Adding the horizontal position weight matrix and the vertical position weight matrix corresponding to the fused image features to obtain a position weight matrix;
[0028] Multiplying the query matrix and the keyword matrix to obtain a first product result;
[0029] Multiplying the query matrix and the position weight matrix to obtain a second product result;
[0030] Adding the first multiplication result and the second multiplication result to obtain an addition result;
[0031] Performing activation processing on the addition result to obtain an activation result;
[0032] The activation result is multiplied by the numerical matrix to obtain the multi-head self-attention image features.
[0033] Optionally, the backbone network includes: n+1 level hourglass modules, n level deep convolutional modules; the first level hourglass module includes: at least one hourglass module; the first level deep convolutional module includes: at least one deep convolutional module;
[0034] Wherein, the i-th level hourglass module is connected in front of the i-th level depthwise convolutional module; n is a positive integer greater than 1, and i is a positive integer not greater than n;
[0035] The n+1th level hourglass module is connected behind the nth level depthwise convolutional module and in front of the spatial pyramid pooling module.
[0036] Optionally, the second-level hourglass module is connected to the neck network and outputs the first image feature A to the neck network; the third-level hourglass module is connected to the neck network and outputs the first image feature B to the neck network; and the spatial pyramid pooling module is connected to the neck network and outputs the first image feature C to the neck network.
[0037] Optionally, the hourglass module includes: a first lightweight convolution module, a second lightweight convolution module, a first splicing module, a third convolution module and a fourth convolution module;
[0038] Among them, the first lightweight convolution module and the second lightweight convolution model correspond to the same network structure; the first lightweight convolution module includes: a first convolution module and a fourth depth-separable convolution module; the second lightweight convolution module includes: a second convolution module and a fifth depth-separable convolution module;
[0039] The output of the first lightweight convolution module is respectively input into the second lightweight convolution module and the first splicing module; the output of the second lightweight convolution module is input into the first splicing module; the output of the first splicing module is input into the third convolution module; the output of the third convolution module is input into the fourth convolution module.
[0040] Optionally, the spatial pyramid pooling module includes: a convolution module A, p first maximum pooling modules, a second splicing module and a convolution module B; p is a positive integer greater than 1.
[0041] Optionally, the convolution processing module includes: a fifth convolution module, a sixth convolution module, a seventh convolution module, a second maximum pooling module, a third splicing module, an eighth convolution module and a ninth convolution module;
[0042] Among them, the fifth convolution module receives input features and provides outputs to the sixth convolution module and the second maximum pooling module respectively; the output of the sixth convolution module is input to the seventh convolution module; the third splicing module receives the outputs of the second maximum pooling module and the seventh convolution module respectively, and provides outputs to the eighth convolution module; the ninth convolution module is connected to the eighth convolution module.
[0043] Optionally, the multi-level feature fusion module includes: a first-level feature fusion module, a second-level feature fusion module and a third-level feature fusion module; the multi-level attention module includes: a first attention module, a second attention module and a third attention module;
[0044] The first-level feature fusion module includes: a first upsampling module, a fourth splicing module, a first channel to pixel module, a second upsampling module, a fifth splicing module, and a second channel to pixel module connected in series; the output of the second channel to pixel module is input into the first attention module;
[0045] The second-level feature fusion module includes: a tenth convolution module, a sixth splicing module and a third channel to pixel module connected in series; the input of the third channel to pixel module is input into the second attention module; the output of the first attention module is input into the tenth convolution module;
[0046] The third-level feature fusion module includes: an eleventh convolution module, a seventh splicing module and a fourth channel-to-pixel module connected in series; the output of the fourth channel-to-pixel module is input into the third attention module; the output of the second attention module is input into the eleventh convolution module;
[0047] Among them, the fifth splicing module receives the first image feature A output by the backbone network, the fourth splicing module receives the first image feature B output by the backbone network, the first upsampling module and the seventh splicing module respectively receive the first image feature C output by the backbone network; the first channel to pixel module is also connected to the sixth splicing module; the first attention module outputs multi-head self-attention image feature A, the second attention module outputs multi-head self-attention image feature B, and the third attention module outputs multi-head self-attention image feature C.
[0048] Optionally, the training process of the target detection model includes:
[0049] Input the construction image sample into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: prediction box information corresponding to the target object category; and real box information corresponding to the construction image sample;
[0050] Determine loss information corresponding to the real frame information and the predicted frame information; the loss information includes: a cross entropy loss corresponding to the predicted frame category, an intersection-over-union loss corresponding to the predicted frame position, and a focal loss corresponding to the predicted frame position;
[0051] According to the loss information, the parameters of the target detection model are updated.
[0052] Optionally, the process of acquiring the construction image sample includes:
[0053] Randomly read four original construction images from the construction image collection;
[0054] Perform random enhancement operations on the four original construction images respectively to obtain four enhanced construction images;
[0055] The four enhanced construction images are fused into one fused image;
[0056] Annotate a fused image with the true box information to obtain a training sample.
[0057] The embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method described in the embodiment of the present application.
[0058] The embodiment of the present application further discloses a machine-readable medium on which executable code is stored. When the executable code is executed, a processor executes the method described in the embodiment of the present application.
[0059] The embodiments of the present application include the following advantages:
[0060] In the technical solution of the embodiment of the present application, an attention module is set in the neck network of the target detection model, and the attention module is used to perform multi-head self-attention processing on the above-mentioned multiple fused image features to obtain multiple multi-head self-attention image features. Among them, the above-mentioned multi-head self-attention processing can capture the complex dependencies between elements in the sequence. Specifically to the embodiment of the present application, the sequence is specifically a fused image feature, and the elements in the sequence can represent image features such as shape, texture and context information of multiple categories of objects; in this way, the multi-head self-attention processing of the embodiment of the present application can effectively capture the nonlinear relationship between the image features of multiple categories of objects in the fused image features, and perform cross-channel information interaction. The information interaction corresponding to the above nonlinear relationship can improve the feature fusion and feature processing capabilities of the neck network. On the basis of improving the feature fusion and feature processing capabilities of the neck network, the three layers of fused image features with different resolutions output by the neck network can simultaneously improve the detection accuracy of larger, medium and smaller size targets in the construction scene image. Therefore, the embodiment of the present application can improve the recognition accuracy of construction scene objects.
[0061] Moreover, the backbone network of the embodiment of the present application specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level depth convolution module and a spatial pyramid pooling module. The above-mentioned multi-level hourglass module and the multi-level depth convolution module can extract the depth feature information of different levels of the construction scene image through repeated calculations, capture the multi-scale image semantic features, and enhance the adaptability of the backbone network to construction scene objects of different sizes. Moreover, the above-mentioned structure of the backbone network merges the operations of dimensionality increase and downsampling into one module, and can maintain high-level semantic information and low-level detail information in the deep network. In this way, the embodiment of the present application can improve the recognition accuracy of construction scene objects of different sizes, especially, can improve the recognition accuracy of small-sized construction scene objects. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 is a schematic diagram of an application environment of a construction scene object recognition method according to an embodiment of the present application;
[0063] Figure 2 This is a schematic diagram of the steps of a construction scene object recognition method according to an embodiment of the present application;
[0064] Figure 3 is a schematic diagram of the structure of a target detection model according to an embodiment of the present application;
[0065] Figure 4 is a schematic structural diagram of an hourglass module according to an embodiment of the present application;
[0066] Figure 5 is a schematic diagram of the structure of a deep convolutional module according to an embodiment of the present application;
[0067] Figure 6 It is a structural diagram of a spatial pyramid pooling module according to an embodiment of the present application;
[0068] Figure 7 is a structural diagram of a convolution processing module of an embodiment of the present application;
[0069] Figure 8 It is a schematic diagram of the structure of a backbone network of an embodiment of the present application;
[0070] Fig. 9 is a schematic diagram of the structure of a channel-to-pixel module according to an embodiment of the present application;
[0071] Fig.10 is a processing flow chart of an attention module according to an embodiment of the present application;
[0072] Fig.11 is a schematic diagram of the structure of a neck network according to an embodiment of the present application;
[0073] Fig.12 is a schematic diagram of the structure of a construction scene object recognition device according to an embodiment of the present application;
[0074] Fig.13 It is a schematic diagram of the structure of a device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0075] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0076] The embodiments of the present application can be applied to engineering industries such as hydropower and civil engineering to improve the recognition accuracy of objects in construction scenes.
[0077] The present application provides a method for identifying objects in a construction scene, which specifically includes the following steps: collecting a construction image; identifying the construction image according to a target detection model to obtain a recognition result; the recognition result specifically includes: the category of the target object and the location information of the target object in the construction image;
[0078] The target detection model specifically includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level deep convolution module and a spatial pyramid pooling module; the neck network specifically includes: a multi-level feature fusion module and a multi-level attention module;
[0079] The process of identifying the construction image according to the target detection model specifically includes:
[0080] The backbone network determines a plurality of first image features corresponding to the construction image;
[0081] The multi-level feature fusion module in the neck network performs feature fusion processing according to the multiple first image features to obtain multiple fused image features;
[0082] The multi-level attention module in the neck network performs multi-head self-attention processing on the above-mentioned multiple fusion image features to obtain multiple multi-head self-attention image features;
[0083] The detection network determines the recognition result based on the above-mentioned multi-head self-attention image features.
[0084] In the embodiment of the present application, an attention module is set in the neck network of the target detection model, and the attention module is used to perform multi-head self-attention processing on the above-mentioned multiple fused image features to obtain multiple multi-head self-attention image features. Among them, the above-mentioned multi-head self-attention processing can capture the complex dependencies between elements in the sequence. Specifically to the embodiment of the present application, the sequence is specifically a fused image feature, and the elements in the sequence can represent image features such as the shape, texture and context information of multiple categories of objects; in this way, the multi-head self-attention processing of the embodiment of the present application can effectively capture the nonlinear relationship between the image features of multiple categories of objects in the fused image features, and perform cross-channel information interaction. The information interaction corresponding to the above nonlinear relationship can improve the feature fusion and feature processing capabilities of the neck network. On the basis of improving the feature fusion and feature processing capabilities of the neck network, the three layers of fused image features with different resolutions output by the neck network can simultaneously improve the detection accuracy of larger, medium and smaller size targets in the construction scene image. Therefore, the embodiment of the present application can improve the recognition accuracy of construction scene objects.
[0085] Moreover, the backbone network of the embodiment of the present application specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level depth convolution module and a spatial pyramid pooling module. The above-mentioned multi-level hourglass module and the multi-level depth convolution module can extract the depth feature information of different levels of the construction scene image through repeated calculations, capture the multi-scale image semantic features, and enhance the adaptability of the backbone network to construction scene objects of different sizes. Moreover, the backbone network of the embodiment of the present application merges the operations of dimensionality increase and downsampling into one module, and can maintain high-level semantic information and low-level detail information in the deep network, so that the recognition accuracy of construction scene objects of different sizes can be improved, especially, the recognition accuracy of small-sized construction scene objects can be improved.
[0086] Reference Figure 1 , showing a schematic diagram of an application environment of a construction scene object recognition method according to an embodiment of the present application, wherein an image acquisition terminal 101 and a service terminal 102 can perform data exchange based on a wireless network or a wired network.
[0087] In practical applications, the image acquisition terminal 101 may be equipped with an image acquisition device having an image acquisition function, such as an image sensor. The image acquisition device can acquire construction images, and the image acquisition terminal 101 can send the construction images to the server 102 according to a preset time period. Of course, the image acquisition device can acquire construction videos, and the image acquisition terminal 101 can send the construction videos to the server 102 according to a preset time period; in this case, the server 102 can parse the construction images from the construction videos.
[0088] After receiving the construction image sent by the image acquisition end 101, the server end 102 can use the method of the embodiment of the present application to identify the above construction image to obtain a recognition result.
[0089] The above identification results can help improve the safety, quality and progress control level of hydropower project construction, and play an important role in promoting the intelligent development of hydropower construction.
[0090] For example, the above recognition results can be used to determine various states in the construction process, such as whether the safety helmet is worn, whether the machinery operation is standardized, whether the materials are stacked reasonably, etc.; in this way, the workload of manual monitoring can be reduced and the accuracy and efficiency of monitoring can be improved.
[0091] For example, based on the identification of excavators and dump trucks in the construction scene, real-time monitoring of the excavation volume in the construction scene can be carried out, which is helpful for controlling the construction progress and rationally allocating resources.
[0092] Method Example 1
[0093] refer to Figure 2, shows a schematic flow chart of the steps of a construction scene object recognition method according to an embodiment of the present application, the method specifically comprises the following steps:
[0094] Step 201: Collect construction images;
[0095] Step 202: Recognize the construction image according to the target detection model to obtain a recognition result; the recognition result specifically includes: the category of the target object and the location information of the target object in the construction image;
[0096] The target detection model specifically includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level deep convolution module and a spatial pyramid pooling module; the neck network specifically includes: a multi-level feature fusion module and a multi-level attention module;
[0097] The process of step 202 identifying the construction image according to the target detection model specifically includes:
[0098] Step 221, the backbone network determines a plurality of first image features corresponding to the construction image;
[0099] Step 222: The multi-level feature fusion module in the neck network performs feature fusion processing according to the multiple first image features to obtain multiple fused image features;
[0100] Step 223, the multi-level attention module in the neck network performs multi-head self-attention processing on the above-mentioned multiple fusion image features respectively to obtain multiple multi-head self-attention image features;
[0101] Step 224: The detection network determines the recognition result based on the above-mentioned multiple multi-head self-attention image features.
[0102] Figure 2 The steps included in the method embodiment shown can be executed by the server, which can take advantage of its rich computing resources to quickly process the construction image. Figure 2 The specific execution subject of the illustrated method embodiment is not limited.
[0103] In step 201, the server may receive the construction images sent by the image acquisition terminal according to a preset time period. The image acquisition terminal may be set at the construction site to collect the construction images of the construction site in real time and send the construction images to the server according to the preset time period.
[0104] In step 202, the construction image is identified according to the target detection model, and the identification result obtained specifically includes: the category of the target object and the location information of the target object in the construction image. The category of the target object specifically includes at least one of the following categories: human category, construction machinery category, safety protection facility wearing category and safety sign category.
[0105] Examples of the human category may include: construction workers. Examples of the construction machinery category may include: backhoe excavators, tower cranes, dump trucks, truck cranes, loaders, hooks, pump trucks, smooth rollers, concrete mixers, pile drivers, etc. The safety protection facility wearing category specifically includes: safety protection facility wearing category, or safety protection facility not wearing category. The safety sign category specifically includes: safety slogans, etc.
[0106] Reference Figure 3 , shows a schematic diagram of the structure of a target detection model of an embodiment of the present application, which specifically includes: a backbone network 301, a neck network 302 and a detection network 303 connected in sequence.
[0107] The backbone network 301 is used to extract features from the input image to obtain a plurality of first image features. In the training phase, the input image may be a construction image sample. In the image recognition phase, the input image may be a real-time construction image.
[0108] The neck network 302 is used to perform feature fusion processing and multi-head attention processing on multiple first image features, and output multiple multi-head self-attention image features to the detection head network 303.
[0109] The detection network 303 is used to determine the recognition result based on the above-mentioned multiple multi-head self-attention image features.
[0110] The backbone network of the embodiment of the present application specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level deep convolution module and a spatial pyramid pooling module. The multi-level hourglass module and the multi-level deep convolution module can capture multi-scale object features and enhance the adaptability of the backbone network to construction scene objects of different sizes.
[0111] Reference Figure 4 , shows a schematic diagram of the structure of an hourglass module of an embodiment of the present application, the hourglass module specifically includes: a first lightweight convolution module 401, a second lightweight convolution module 402, a first splicing module 403, a third convolution module 404 and a fourth convolution module 405;
[0112] Among them, the first lightweight convolution module 401 and the second lightweight convolution model 402 correspond to the same network structure; the first lightweight convolution module 401 includes: a first convolution module and a fourth depth-separable convolution module; the second lightweight convolution module 402 includes: a second convolution module and a fifth depth-separable convolution module;
[0113] The output of the first lightweight convolution module 401 is input into the second lightweight convolution module 402 and the first splicing module 403 respectively; the output of the first splicing module 403 is input into the third convolution module 404; the output of the third convolution module 404 is input into the fourth convolution module 405.
[0114] In practical applications, the hourglass module may include more than two lightweight convolution modules. It is understood that the embodiment of the present application does not limit the specific number of lightweight convolution modules included in the hourglass module.
[0115] In the structure of the hourglass module, the outputs of multiple lightweight convolution modules are merged through the first splicing module, which can not only solve the problem of gradient disappearance, but also capture feature information at different levels. Then, the merged result output by the first splicing module is subjected to two convolution operations with a convolution kernel of 1×1, which can compress the feature channel and reduce the number of parameters.
[0116] In addition, the structure of the above-mentioned hourglass module can capture multi-scale features, enhance the adaptability of the backbone network to construction scene objects of different sizes, and at the same time, through its unique topological structure, it can maintain high-level semantic information and low-level detail information in the deep network, which is particularly helpful for the recognition of construction scene objects of different sizes, especially small-sized construction scene objects.
[0117] Reference Figure 5 , shows a schematic diagram of the structure of a deep convolution module of an embodiment of the present application. The deep convolution module can use filters to perform deep convolution operations on each channel of the input feature, extract features while maintaining the spatial dimension of the input feature, and fuse the output features of different channels through point-by-point convolution to reduce the number of channels, thereby achieving deep learning of local spatial features while effectively reducing the parameters required for convolution calculation and improving the calculation speed of the backbone network.
[0118] The spatial pyramid pooling module space is used to perform spatial pyramid pooling processing on the input features. The spatial pyramid pooling processing may include: convolution operation and maximum pooling operation, which can achieve deep fusion of input features. In particular, the input features of the spatial pyramid pooling module are multi-scale features output by the hourglass module, so the spatial pyramid pooling module can achieve deep fusion of multi-scale features.
[0119] Reference Figure 6 , shows a schematic diagram of the structure of a spatial pyramid pooling module of an embodiment of the present application, the spatial pyramid pooling module specifically includes: a convolution module A601, p first maximum pooling modules 602, a second splicing module 603 and a convolution module B604. p can be a positive integer greater than 1, and the value of p in the figure is 3.
[0120] Among them, the convolution module A601 is used to perform convolution processing on the input features to obtain the convolution result A.
[0121] The p second maximum pooling modules 602 are respectively used to perform maximum pooling processing on the input features to obtain corresponding p maximum pooling processing results.
[0122] The convolution result A and the p maximum pooling processing results are respectively input into the second splicing module 603, and the second splicing module 603 performs deep fusion on the convolution result A and the p maximum pooling processing results to obtain a corresponding deep fusion result.
[0123] The convolution module B604 is used to perform convolution processing on the depth fusion result to obtain a convolution result B. The convolution result B is provided to the neck network.
[0124] In a specific implementation, the backbone network specifically includes: n+1 level hourglass modules, n level deep convolutional modules; the first level hourglass module may include: at least one hourglass module; the first level deep convolutional module may include: at least one deep convolutional module;
[0125] Wherein, the i-th level hourglass module is connected in front of the i-th level depthwise convolutional module; n is a positive integer greater than 1, and i is a positive integer not greater than n;
[0126] The n+1th level hourglass module is connected behind the nth level depthwise convolutional module and in front of the spatial pyramid pooling module.
[0127] The value n may be determined by those skilled in the art according to actual application requirements. For example, the value n may be 3.
[0128] Optionally, the second-level hourglass module is connected to the neck network and outputs the first image feature A to the neck network; the third-level hourglass module is connected to the neck network and outputs the first image feature B to the neck network; the spatial pyramid pooling module is connected to the neck network and outputs the first image feature C to the neck network. Thus, the backbone network can output three first image features to the neck network.
[0129] Reference Figure 7, shows a schematic diagram of the structure of a convolution processing module of an embodiment of the present application, the convolution processing module specifically includes: a fifth convolution module 701, a sixth convolution module 702, a seventh convolution module 703, a second maximum pooling module 704, a third splicing module 705, an eighth convolution module 706 and a ninth convolution module 707;
[0130] Among them, the fifth convolution module 701 receives input features and provides outputs to the sixth convolution module 702 and the second maximum pooling module 704 respectively; the output of the sixth convolution module 702 is input to the seventh convolution module 703; the third splicing module 705 receives the outputs of the second maximum pooling module 704 and the seventh convolution module 703 respectively, and provides outputs to the eighth convolution module 706; the ninth convolution module 707 is connected to the eighth convolution module 706.
[0131] The convolution processing module can perform feature extraction on the construction image, extract image features, and reduce the size of image features by upscaling and downsampling operations. Figure 7 As shown, the construction image first passes through the fifth convolution module 701. The fifth convolution module 701 can use a 3×3 convolution operation and use a linear rectification function for activation calculation, and then use the obtained features as the input of two parallel branches. One branch enters the second maximum pooling module 704, which performs a 2×2 maximum pooling operation, and the other branch passes through the sixth convolution module 702 and the seventh convolution module 703 continuously. The convolution kernels of the sixth convolution module 702 and the seventh convolution module 703 are both 2×2. The features calculated by the two parallel branches are channel-merged through the splicing operation of the third splicing module 705, and the merged results are continuously passed through the eighth convolution module 706 with a convolution kernel of 3×3 and the ninth convolution module 707 with a convolution kernel of 1×1, and the extracted features with 48 output channels are output.
[0132] Reference Figure 8, shows a schematic diagram of the structure of a backbone network of an embodiment of the present application, the backbone network specifically includes: 1 convolution processing module, 4-level hourglass modules and 3-level depth-separable convolution modules. The above-mentioned backbone network specifically includes: a convolution processing module 801, a first hourglass module 802, a first depth-separable convolution module 803, a second hourglass module 804, a second depth-separable convolution module 805, at least one third hourglass module 806, a third depth-separable convolution module 807, a fourth hourglass module 808 and a spatial pyramid pooling module 809. The first-level hourglass module includes: a first hourglass module. The second-level hourglass module includes: a second hourglass module. The third-level hourglass module includes: multiple third hourglass modules. The fourth hourglass module includes: a fourth hourglass module. The first-level depth-separable convolution module includes: a first depth-separable convolution module. The second-level depth-separable convolution module includes: a second depth-separable convolution module. The third-level depth-separable convolution module includes: a third depth-separable convolution module.
[0133] The second hourglass module 804 may output the first image feature A to the neck network. The last third hourglass module 806 may output the first image feature B to the neck network. The spatial pyramid pooling module may output the first image feature C to the neck network.
[0134] In step 221, for example Figure 8 The backbone network of the structure shown can determine a variety of first image features corresponding to the above-mentioned construction image.
[0135] In step 222, the multi-level feature fusion module in the neck network may perform feature fusion processing according to the above-mentioned multiple first image features to obtain multiple fused image features.
[0136] The multi-level fusion image features specifically include:
[0137] The first-level fused image features with the highest resolution are used to better detect smaller target objects in construction scene images;
[0138] The second-level fused image features with higher resolution are used to better detect medium-sized target objects in construction scene images;
[0139] The third-level fused image features with the smallest resolution are used to better detect target objects with larger sizes in construction scene images.
[0140] Optionally, the feature fusion module specifically includes: an upsampling module, a splicing module and a channel to pixel module.
[0141] The up-sampling module can make the size of the first image feature twice as large as the original one.
[0142] The splicing module is used to merge the upsampling result with the first image feature output by the backbone network.
[0143] Reference Fig. 9 , shows a schematic diagram of the structure of a channel-to-pixel module of an embodiment of the present application, the channel-to-pixel module specifically includes: a twelfth convolution module 901, a segmentation processing module 902, M bottleneck modules 903, an eighth splicing module 904, and a thirteenth convolution module 905. M can be a positive integer greater than 1.
[0144] Among them, the twelfth convolution module 901 is used to perform a first convolution process on the input features to obtain a twelfth convolution result.
[0145] The segmentation processing module 902 is used to segment the twelfth convolution result into two parts with the same number of channels. The two parts may include: a first part of features and a second part of features. Assuming that the number of channels before segmentation is the third number of channels, the number of channels after segmentation may be the fourth number of channels, and the third number of channels may be twice the fourth number of channels.
[0146] After the first part of the features are processed by M bottleneck modules 903, the bottleneck processed features obtained enter the eighth splicing module 904. The twelfth convolution result and the second part of the features also enter the eighth splicing module 904. The eighth splicing module 904 is used to perform a channel splicing operation on the twelfth convolution result, the second part of the features and the bottleneck processed features to obtain a spliced feature.
[0147] The spliced features enter the thirteenth convolution module 905, and the thirteenth convolution module 905 restores the number of channels for the spliced features, for example, restores the number of channels from the fourth number of channels to the third number of channels.
[0148] It should be noted that the first convolution module, the second convolution module and other convolution modules of the embodiment of the present application all belong to the category of convolution structure. In one example, the above convolution structure specifically includes: at least one convolution layer, at least one batch normalization layer, and at least one activation function. It can be understood that those skilled in the art can adopt the required convolution structure according to the actual application requirements, and the embodiment of the present application does not limit the specific convolution structure.
[0149] In a specific implementation, the bottleneck module specifically includes: a fourteenth convolution module and a fifteenth convolution module.
[0150] Among them, the fourteenth convolution module is used to reduce the number of channels of the input features to half of the original number to obtain the fourteenth convolution result.
[0151] The fifteenth convolution module is used to increase the number of channels of the fourteenth convolution result to twice the original number to obtain the fifteenth convolution processing result. The fifteenth convolution processing result has the same number of channels as the input feature. The fifteenth convolution processing result is fused with the input feature to obtain the output feature.
[0152] In the embodiment of the present application, a self-attention module can be set after the primary channel to the pixel module to enhance the modeling of the nonlinear relationship between different positions in the fused image features, and perform cross-channel information interaction, thereby improving the feature fusion and feature processing capabilities of the neck network and improving the detection accuracy of target objects of different sizes.
[0153] In a specific implementation, the multi-level attention module in the neck network performs multi-head self-attention processing on the multiple fused image features, which specifically includes the following steps:
[0154] Step A1, performing weighted processing on the fused image features to obtain a query matrix, a key matrix and a value matrix;
[0155] Step A2, adding the horizontal position weight matrix and the vertical position weight matrix corresponding to the fused image features to obtain a position weight matrix;
[0156] Step A3, multiplying the query matrix and the key matrix to obtain a first product result;
[0157] Step A4: multiplying the query matrix and the position weight matrix to obtain a second product result;
[0158] The embodiment of the present application introduces a position weight matrix, which can take into account the spatial position information of image features. The target detection model not only needs to identify the target category in the image, but also needs to accurately locate the target. The position weight matrix helps the target detection model better understand the specific position of the target object in the image, thereby improving the positioning accuracy of the bounding box.
[0159] Step A5, adding the first product result and the second product result to obtain an addition result;
[0160] Step A6, performing activation processing on the addition result to obtain an activation result;
[0161] Step A7: multiply the activation result by the numerical matrix to obtain the multi-head self-attention image feature.
[0162] Reference Fig.10 , showing a processing flow chart of the attention module of an embodiment of the present application.
[0163] First, using the query weight matrix W q , key weight matrix W k, value weight matrix W v , the fused features are weighted to obtain the query matrix (Query), key matrix (Key), and value matrix (Value).
[0164] Then, using the position embedding technology, the lateral position weight matrix Rel of the fusion processing feature is transformed into w , vertical position weight matrix Rel h Add them together to get the position weight matrix.
[0165] Next, the position weight matrix is multiplied by the query matrix to obtain a content-position matrix, which is called the second product result.
[0166] Then, the attention weight is calculated according to the query matrix (Query) and the key matrix (Key) to obtain the first product result.
[0167] Next, the first multiplication result and the second multiplication result are added to obtain an addition result.
[0168] Furthermore, an activation function may be used to perform activation processing on the addition result to obtain an activation result.
[0169] Finally, the activation result can be multiplied by the numerical matrix to obtain the multi-head self-attention image features.
[0170] The calculation process of the query matrix (Query), key matrix (Key), and value matrix (Value) is shown in formula (1).
[0171] Query = XW q +b q
[0172] Key=XW k +b k (1)
[0173] Value = XW v +b v
[0174] Among them, X represents the fused image features, W q is a randomly generated query weight matrix, b q is the bias of the query mapping; W k is a randomly generated key weight matrix, b k is the offset of the key mapping; W v is a randomly generated value weight matrix, b v Bias for the value mapping.
[0175] The process of calculating the attention weight is performed according to the query matrix (Query) and the key matrix (Key), as shown in formula (2).
[0176]
[0177] Where D is the vector length of Query, Key, and Value. The Softmax function is a commonly used function in machine learning and deep learning. It is mainly used to compress (map) a K-dimensional vector containing arbitrary real numbers into another K-dimensional real vector whose element values are between (0,1) and the sum of all elements is 1.
[0178] The overall calculation process of the attention module is shown in formula (3):
[0179] MHSA(X)=Softmax((Rel h +Rel w )Query T +SA(X))×Value(3)
[0180] In a specific implementation, the multi-level feature fusion module may include: a first-level feature fusion module, a second-level feature fusion module, and a third-level feature fusion module; the multi-level attention module may include: a first attention module, a second attention module, and a third attention module;
[0181] The first-level feature fusion module includes: a first upsampling module, a fourth splicing module, a first channel to pixel module, a second upsampling module, a fifth splicing module, and a second channel to pixel module connected in series; the output of the second channel to pixel module is input into the first attention module;
[0182] The second-level feature fusion module includes: a tenth convolution module, a sixth splicing module and a third channel to pixel module connected in series; the input of the third channel to pixel module is input into the second attention module; the output of the first attention module is input into the tenth convolution module;
[0183] The third-level feature fusion module includes: an eleventh convolution module, a seventh splicing module and a fourth channel to pixel module connected in series in sequence; the output of the fourth channel to pixel module is input into the third attention module; the output of the second attention module is input into the eleventh convolution module.
[0184] Reference Fig.11, shows a schematic diagram of the structure of a neck network of an embodiment of the present application, the neck network specifically includes: a first upsampling module 1101, a fourth splicing module 1102, a first channel to pixel module 1103, a second upsampling module 1104, a fifth splicing module 1105, a second channel to pixel module 1106, a first attention module 1107, a tenth convolution module 1108, a sixth splicing module 1109, a third channel to pixel module 1110, a second attention module 1111, an eleventh convolution module 1112, a seventh splicing module 1113, a fourth channel to pixel module 1114, and a third attention module 1115, which are connected in series in sequence;
[0185] Among them, the fifth splicing module 1105 receives the first image feature A output by the backbone network, the fourth splicing module 1102 receives the first image feature B output by the backbone network, the first upsampling module 1101 and the seventh splicing module 1113 respectively receive the first image feature C output by the backbone network; the first channel to pixel module 1103 is also connected to the sixth splicing module 1109; the first attention module 1107 outputs multi-head self-attention image feature A, the second attention module 1111 outputs multi-head self-attention image feature B, and the third attention module 1115 outputs multi-head self-attention image feature C.
[0186] In step 224, the detection network determines the recognition result based on the above-mentioned multiple multi-head self-attention image features.
[0187] The detection network may include: at least one detection module. The detection module may be used to perform classification and regression calculations based on a variety of multi-head self-attention image features to obtain the category of the target object in the construction image and the location information of the target object in the construction image.
[0188] In summary, the construction scene object recognition method of the embodiment of the present application is provided with an attention module in the neck network of the target detection model, and the attention module is used to perform multi-head self-attention processing on the above-mentioned multiple fused image features to obtain multiple multi-head self-attention image features. Among them, the above-mentioned multi-head self-attention processing can capture the complex dependencies between elements in the sequence. Specifically to the embodiment of the present application, the sequence is specifically a fused image feature, and the elements in the sequence can represent image features such as the shape, texture and context information of multiple categories of objects; in this way, the multi-head self-attention processing of the embodiment of the present application can effectively capture the nonlinear relationship between the image features of multiple categories of objects in the fused image features, and perform cross-channel information interaction. The information interaction corresponding to the above-mentioned nonlinear relationship can improve the feature fusion and feature processing capabilities of the neck network. On the basis of improving the feature fusion and feature processing capabilities of the neck network, the three layers of fused image features with different resolutions output by the neck network can simultaneously improve the detection accuracy of larger, medium and smaller size targets in the construction scene image. Therefore, the embodiment of the present application can improve the recognition accuracy of construction scene objects.
[0189] Moreover, the backbone network of the embodiment of the present application specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level depth convolution module and a spatial pyramid pooling module. The above-mentioned multi-level hourglass module and the multi-level depth convolution module can extract the depth feature information of different levels of the construction scene image through repeated calculations, capture multi-scale object features, and enhance the adaptability of the backbone network to construction scene objects of different sizes. Moreover, the above-mentioned structure of the backbone network of the embodiment of the present application combines the operations of dimensionality increase and downsampling into one module, and can maintain high-level semantic information and low-level detail information in the deep network, so that the recognition accuracy of construction scene objects of different sizes can be improved, especially, the recognition accuracy of small-sized construction scene objects can be improved.
[0190] Method Example 2
[0191] The embodiments of the present application illustrate the training and experimental results of the target detection model.
[0192] In an embodiment of the present application, the training process of the target detection model may include: forward propagation and back propagation.
[0193] Among them, forward propagation can calculate the prediction information of the recognition result in sequence from the backbone network to the detection network according to the parameters of the target detection model. The prediction information is used to determine the loss information.
[0194] Backward Propagation can calculate and update the parameters of the target detection model in sequence according to the loss information, in the order from the detection network to the backbone network. The target detection model usually adopts the structure of a neural network, and the parameters of the target detection model may include: parameters such as the weights of the neural network. Among them, during the back propagation process, the gradient information of the parameters of the target detection model can be determined, and the parameters of the target detection model can be updated using the gradient information. For example, back propagation can calculate and store the gradient information of the parameters of the target detection model in sequence from the detection network to the neck network and then to the backbone network according to the chain rule in calculus.
[0195] In one embodiment, the training process of the target detection model includes:
[0196] Step B1: input the construction image sample into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: prediction box information corresponding to the target object category; and real box information corresponding to the construction image sample;
[0197] Step B2, determining loss information corresponding to the real box information and the predicted box information; the loss information includes: cross entropy loss corresponding to the predicted box category, intersection-over-union loss corresponding to the predicted box position, and focus loss corresponding to the predicted box position; the predicted box represents a predicted bounding box;
[0198] Step B3: Update the parameters of the target detection model according to the loss information.
[0199] The process of obtaining the above construction image samples specifically includes:
[0200] Step C1, randomly reading four original construction images from the construction image set;
[0201] Step C2, performing random enhancement operations on the four original construction images respectively to obtain four enhanced construction images;
[0202] Step C3, fusing the four enhanced construction images into a fused image;
[0203] Step C4: annotate a pair of fused images with real frame information to obtain a training sample.
[0204] The random enhancement operation specifically includes at least one of the following operations: a flip operation, a random scaling operation, a random tone transformation operation, and a random perspective transformation operation.
[0205] Step C3 is a process of fusing the four enhanced construction images into a fused image, specifically including: placing the four enhanced construction images on an intermediate image according to the offsets of [0,0], [0,243], [320,320], [320,0]; cropping the part of the intermediate image that exceeds the size range, and shrinking the range of the real frame to prevent crossing the boundary, and finally obtaining a fused image. The size range specifically includes: the coordinate range corresponding to [0,0] to [320,320].
[0206] Referring to formula (4), the process of determining the loss information corresponding to the real box information and the predicted box information is shown.
[0207] L total =λ1L BCE +λ2L CIoU +λ3L DFL (4)
[0208] Where, L BCE Represents the cross entropy loss corresponding to the predicted box category; L CIoU represents the intersection-over-union loss corresponding to the predicted box position; L DFL Represents the focal loss corresponding to the predicted box position; λ1, λ2, and λ3 are the weight proportions of the three losses respectively.
[0209] L BCE The loss function calculation process is shown in formula (5):
[0210]
[0211] Where N is the number of construction image samples, and the predicted y i is the true category label corresponding to the bounding box of the target object in the i-th construction image sample, p i is the predicted probability of the category corresponding to the i-th construction image sample.
[0212] L CIoU The loss function calculation process is shown in formula (6):
[0213]
[0214] Where IoU, d, and c are the area intersection and union ratio of the real box and the predicted box of the target object, the center point distance, and the diagonal distance of the minimum circumscribed rectangle, respectively. is a correction factor to take into account the influence of bounding box shape and orientation, w g 、w p 、h g 、h p They are the width and height of the real box and predicted box of the target object respectively.
[0215] LDFL The loss function calculation process is shown in formula (7):
[0216] L DFL =-((y i+1 -y)log(S i )+(yy i )log(S i+1 ))(7)
[0217] In the formula, y i ,y i+1 are the two integers on both sides of the real coordinate value y of the detection box and closest to y, S i , S i+1 They are respectively the predicted values of the target detection model for the distance from the true coordinate value to the nearest integer on both sides.
[0218] The benefits of using the above three loss functions in combination are:
[0219] Comprehensive optimization: Each loss function optimizes different aspects of the object detection task. The cross entropy loss focuses on category accuracy, the intersection-over-union loss focuses on location accuracy, and the focal loss focuses on learning difficult samples.
[0220] Balancing class imbalance: Focal loss helps deal with the class imbalance problem, while cross entropy loss and IoU loss focus on optimizing class classification and location accuracy, respectively.
[0221] Improved detection performance: By comprehensively considering category classification, location accuracy, and hard sample learning, the model is able to improve the accuracy and robustness of object detection in various scenarios.
[0222] In summary, the use of multiple loss functions can make the target detection model more comprehensive and balanced in the target detection task, thereby improving the detection performance.
[0223] In order to further verify the effect of the target detection model in the embodiment of the present application, the embodiment of the present application conducted the following simulation experiments:
[0224] The embodiment of the present application first compares the detection accuracy of the existing target detection model and the method proposed in the present application on a construction scene target detection benchmark dataset.
[0225] Specifically, the 19404 labeled training set images in the construction scene target detection benchmark dataset are used as the training dataset, and the 4000 validation set images are used as the validation dataset. Each labeled image contains the sample category, the upper left corner coordinate x of the real box, and the min and min , the width and height of the frame.
[0226] Definition of AP (Average Precision), AP 50 ,AP 75 AP is an evaluation indicator. It is the area under the precision-recall curve, which is used to measure the detection accuracy of the target detection model under different thresholds. The larger the AP value, the higher the model detection accuracy. 50 The AP value when the intersection-over-union ratio threshold is 0.5 is used to determine the degree to which the model balances precision and recall under loose conditions; AP 75 The AP value when the intersection-over-union ratio threshold is 0.75 is used to determine the degree to which the model balances the precision and recall under strict conditions, so as to determine whether the target detection model has good positioning accuracy.
[0227] In the training process of the target detection model, the training set of construction scene images was first scaled and the resolution was uniformly adjusted to 640 × 640 to ensure the consistency of the input image size. The initial learning rate of the training was set to 0.01, the final learning rate was set to 0.0001, and the momentum of the stochastic gradient descent was set to 0.937.
[0228] The simulation experimental results of different target detection models on the construction scene target detection benchmark dataset are shown in Table 1. It can be seen that the construction scene target detection method proposed in this application has better detection accuracy on the construction scene target detection benchmark dataset than the existing technology.
[0229] Taking the n-parameter scale and AP index as an example, the detection accuracy of the target detection model of this application is 56.7%, which is 18.4% higher than that of the YOLOv8n model. YOLOv8 is the 8th version of the YOLO (You Only Look Once) target detection model series. The YOLOv8n model is the n-parameter scale of YOLOv8.
[0230] According to the different parameter quantities, the parameter scale of the target detection model can be divided into:
[0231] n-parameter scale (Nano): The n version usually refers to a very small model designed to run on devices with very limited computing resources, such as mobile devices or embedded systems. This version of the model usually has the least parameters and computation, so it is very fast, but may sacrifice detection accuracy.
[0232] s parameter scale (Small): The s version refers to a small model, which is slightly larger than the n version, but still relatively lightweight. This version of the model provides a better balance between speed and accuracy and is suitable for application scenarios that require a compromise between speed and performance.
[0233] m parameter scale (Medium): The m version refers to a medium-sized model, which is larger than the s version and has more parameters and computation. This version of the model usually provides higher detection accuracy while maintaining a relatively fast detection speed, and is suitable for applications that have certain performance requirements but still need to consider speed.
[0234] Moreover, before the training converges, as the training rounds proceed, the object detection model of the present application improves in accuracy, recall, AP, and 50 In terms of indicators, the results are significantly better than the YOLOv8n model, and can achieve higher detection accuracy for thirteen types of target objects in construction scene images, such as dump trucks, backhoe excavators, concrete tankers, etc., proving the effectiveness of the target detection model of this application.
[0235] Table 1
[0236]
[0237] The backbone network proposed in the embodiment of the present application can effectively reduce the number of parameters and improve the accuracy of target detection by merging the operations of dimensionality upscaling and downsampling into an hourglass module and capturing features of different scales and abstraction levels.
[0238] In addition, the embodiment of the present application sets an attention module in the neck network, which can model the nonlinear relationship between different positions in the fused image features and perform cross-channel information interaction, so as to better model the shape, texture and contextual information of the target object, enhance the target detection model's perception and distinction capabilities of the target object, and further improve the accuracy of target detection.
[0239] Method Example 3
[0240] This embodiment can not only identify the excavators and dump trucks in the construction scene, but also conduct real-time monitoring of the excavation volume in the construction scene, which is helpful for controlling the construction progress and rationally allocating resources.
[0241] The present invention provides a method for real-time monitoring of excavation volume, which specifically includes the following steps:
[0242] Step S1, collecting video stream images of the construction area to be counted; the video stream images can be a specific implementation of the construction images.
[0243] Step S2, performing target detection on the video stream image to obtain first target information corresponding to the excavating equipment contained in the video stream image, and second target information corresponding to the self-unloading equipment contained in the video stream image;
[0244] Step S3, determining N target areas in the video stream image according to the first target information and the second target information; N is a positive integer;
[0245] Step S4, for a current target area among the N target areas, determining the excavation operation state of the current target area according to the operation state of the excavation equipment in the current target area, whether the current target area includes a self-unloading equipment, and the positional relationship between the excavation equipment and the self-unloading equipment in the current target area; the excavation operation state includes: one of: an unexcavated state, an excavating state, and a loading state;
[0246] Step S5: When the excavation operation state of the current target area is updated from the loading state to the unexcavated state or the excavation state, it is considered that the current target area has completed the current loading, and the excavation volume corresponding to the current loading of the current target area is determined according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and the cumulative excavation volume of the current target area is updated;
[0247] Step S6: Determine the total excavation volume corresponding to the construction area to be counted according to the cumulative excavation volume of all current target areas among the above N target areas.
[0248] The embodiment of the present application first collects video stream images of the construction area to be counted; then, target detection is performed on the above video stream images to obtain target information corresponding to targets such as excavating equipment and dump equipment in the video stream images; then, based on the above target information, N target areas in the above video stream images are determined; then, based on the objective rules between the excavation operation status of the target area and the operation status of the excavation equipment, whether there is a dump equipment, and the position of the excavation equipment and the dump equipment, the excavation volume corresponding to the current loading of the current target area is determined; and further, based on the cumulative excavation volume of all current target areas among the above N target areas, the excavation volume corresponding to the video stream image is determined.
[0249] Since the embodiment of the present application determines N target areas in the video stream image according to the target information corresponding to the targets such as the excavation equipment and the self-unloading equipment in the video stream image, and determines the excavation amount corresponding to the current loading of the current target area according to the objective law between the excavation operation state of the target area and the operation state of the excavation equipment, whether there is a self-unloading equipment, and the position of the excavation equipment and the self-unloading equipment; in this way, the embodiment of the present application can continuously determine the target area existing in the video stream image and the excavation operation state corresponding to the target area according to the continuously updated video stream image; on this basis, the embodiment of the present application can continuously determine the excavation amount corresponding to the current loading of the current target area in the video stream image, and the cumulative excavation amount of all current target areas in the video stream image. Therefore, the embodiment of the present application can realize the real-time monitoring of the earthwork excavation amount at a lower cost and faster speed based on the analysis of the video stream image. In other words, the embodiment of the present application can save the labor cost, data collection cost, and data processing time cost consumed by using drone oblique photography or three-dimensional laser scanning technology, and can improve the processing efficiency and real-time performance of the earthwork excavation amount.
[0250] In a hydropower engineering scenario, multiple video sources may be set, and different video sources may correspond to different video stream images. Among them, the video source may continuously provide continuous video stream images. The embodiment of the present application may process the continuous video stream images provided by any video source to obtain the excavation amount corresponding to the video source. The construction area to be counted may be a construction area represented by a certain video source.
[0251] The recognition result of the embodiment of the present application specifically includes: first target information corresponding to the excavating equipment contained in the video stream image, and second target information corresponding to the dump truck contained in the above video stream image.
[0252] The first target information and the second target information belong to the category of target information. The target information may correspond to category information. For example, the category corresponding to the first target information is the excavation equipment category, and the category corresponding to the second target information is the dump equipment category.
[0253] The target information specifically includes: the bounding box corresponding to the target. The bounding box is used to indicate the position and range of the target in the video stream image. The bounding box is usually a rectangular box whose four sides are aligned with the outermost edges of the target, thereby surrounding the entire target. The bounding box information may include: the coordinates of the upper left corner of the rectangular box, the width and the height; or, the bounding box information may include: the coordinates of the center point of the rectangular box, the width and the height.
[0254] The target area may be an area in the video stream image that contains an excavating device. According to whether the target area contains a dumping device, the target area may be divided into a first target area and a second target area. The first target area contains an excavating device and does not contain a dumping device. The second target area contains an excavating device and a dumping device.
[0255] In one implementation of the present application, the first target information includes: a first bounding box, and the second target information includes: a second bounding box;
[0256] The process of determining the N target areas in the video stream image according to the first target information and the second target information specifically includes:
[0257] Step D1, determining the number N of target areas according to the number N of first bounding boxes;
[0258] Step D2, performing one-to-one matching on the first bounding box and the second bounding box to obtain a corresponding matching result;
[0259] Step D3: If the first bounding box has a second bounding box that matches it, the minimum enclosing rectangular area that contains both the first bounding box and the second bounding box is used as the target area;
[0260] Step D4: If there is no second bounding box matching the first bounding box, the rectangular area corresponding to the first bounding box is used as the target area.
[0261] In step D1 , the number N of target regions is equal to the number N of first bounding boxes. In other words, the number of target regions included in one video stream image is equal to the number of first bounding boxes included in one video stream image.
[0262] In step D2, a one-to-one matching is performed between the first bounding box and the second bounding box, which may be a distance matching between the first bounding box and the second bounding box.
[0263] In one implementation, step D2 is a process of performing a one-to-one matching between the first bounding box and the second bounding box, specifically comprising: determining a distance intersection-and-union index value between a first bounding box and a second bounding box; and selecting, from the multiple second bounding boxes, one with the largest distance intersection-and-union index value according to the distance intersection-and-union index value between the first bounding box and multiple second bounding boxes as the second bounding box that matches the first bounding box.
[0264] Among them, for the second bounding boxes of the M dump trucks in the video stream image, the distance intersection and union ratio index value is used as a criterion to perform one-to-one matching with the first bounding boxes in the video stream image one by one. mn The determination process of is shown in formula (8).
[0265]
[0266] Among them, A m represents the mth (m∈[1,M], and is a positive integer) second bounding box, B n Represents the nth (n∈[1,N], and is a positive integer) first bounding box. a m 、b n Represent the second bounding box A respectively m , first bounding box B n The center point, ρ mn The center point a m 、b n The Euclidean distance between them, d is the diagonal distance of the minimum enclosing area that contains both the first bounding box and the second bounding box.
[0267] The distance intersection and union ratio index value reflects the degree of overlap between the first bounding box and the second bounding box and the distance between the center points. Generally speaking, the larger the distance intersection and union ratio index value, the higher the degree of overlap between the two and the closer the distance between the center points.
[0268] For a first bounding box, there are M distance intersection and union index values between it and the M second bounding boxes. Then, a maximum distance intersection and union index value can be selected from the M distance intersection and union index values, and the second bounding box corresponding to the maximum distance intersection and union index value can be used as the second bounding box that matches the first bounding box.
[0269] Step D3 and step D4 may be performed in parallel.
[0270] In practical applications, the N first bounding boxes in the video stream image can be traversed in a first preset order to determine the target area corresponding to the first bounding box one by one. The first preset order can be: from left to right, from bottom to top, or from left to right, from top to bottom.
[0271] In the embodiment of the present application, the N first boundary boxes in the video stream image may be numbered according to the first preset order. The numbering of the N first boundary boxes may be the same as the numbering of the N target areas.
[0272] In the process of traversing the N first bounding boxes in the video stream image, it can be determined whether there is a second bounding box matching a first bounding box. If so, step A3 can be executed to use the minimum enclosed rectangular area containing both the first bounding box and the second bounding box as the target area. If not, step A4 can be executed to use the rectangular area corresponding to the first bounding box as the target area.
[0273] After the target area is determined based on step D3 and step D4, information such as the number of the target area, the position information of the rectangular box corresponding to the target area, and whether the second bounding box is included can be obtained.
[0274] In step S4, the excavation operation status of the current target area among the N target areas can be determined. Among them, the current target area can represent one target area among the N target areas. It can be understood that the embodiment of the present application can execute the processing of multiple current target areas among the N target areas in parallel or in series. The processing of the current target area may include: the processing of step S4 and step S5. The process of serially executing the processing of multiple current target areas among the N target areas specifically includes: traversing the N target areas in a second preset order of the target area numbers from small to large or from large to small to obtain a current target area among the N target areas, and serially executing the processing of multiple current target areas among the N target areas in a second preset order.
[0275] The excavation operation status of the present application specifically includes: one of the unexcavated state, the excavation state and the loading state; wherein, the unexcavated state means that the current target area is not undergoing excavation operation. The excavation state means that the current target area is undergoing excavation operation. The loading state means that the current target area is undergoing loading operation.
[0276] The embodiment of the present application can determine the operating status of the current target area according to the operating status of the excavating equipment in the current target area, whether the current target area includes a dump truck, and the positional relationship between the excavating equipment and the dump truck in the current target area.
[0277] The method for determining the excavation operation status of the current target area may specifically include the following steps:
[0278] Step E1, determining whether the excavation equipment included in the current target area is in a stationary state to obtain a first determination result;
[0279] If the first judgment result is yes, then executing step E2: determining that the excavation operation state of the current target area is an unexcavated state;
[0280] If the first judgment result is no, then executing step E3: judging whether the current target area includes a self-unloading device, so as to obtain a second judgment result;
[0281] If the second judgment result is no, then executing step E4: determining that the excavation operation state of the current target area is in the excavation state;
[0282] If the second judgment result is yes, then executing step E5: judging whether there is overlap between the self-unloading equipment and the excavating equipment in the current target area, so as to obtain a third judgment result;
[0283] If the third judgment result is yes, then executing step E6: determining that the excavation operation state of the current target area is in a loading state;
[0284] If the third judgment result is no, step E7 is executed: determining that the excavation operation state of the current target area is in the excavation state.
[0285] The present application embodiment provides the following determination rules for the excavation operation status:
[0286] The rule for determining the unexcavated state is that the excavation equipment is in a stationary state.
[0287] The judgment rules for the excavation state include: the excavator is in a non-stationary state, and the current target area does not include the dump equipment; or, the current target area includes the dump equipment, and there is no overlap between the dump equipment and the excavation equipment.
[0288] The judgment rules of the loading state include: the excavation equipment is in a non-stationary state, the current target area contains the dump equipment, and there is overlap between the dump equipment and the excavation equipment.
[0289] Step E1 is a process of determining whether the excavation equipment in the current target area is in a stationary state, specifically comprising:
[0290] In the i-th frame video stream image to the i+α-th frame video stream image, if the coordinate difference of the mining equipment in any two frames of video stream images is less than the second threshold, it is determined that the mining equipment contained in the current target area is in a stationary state; wherein i and α are positive integers, and α is greater than i.
[0291] The embodiment of the present application can select two video stream images to be compared from the α-frame video stream images corresponding to the i-th frame video stream image to the i+α-th frame video stream image, and determine whether the coordinate difference between the two video stream images to be compared is less than a second threshold.
[0292] Two video stream images to be compared are selected from the α frame video stream images, and the selection schemes may be P. If the coordinate differences corresponding to the P selection schemes are all less than the second threshold, it can be determined that the mining equipment included in the current target area is in a stationary state.
[0293] Considering the influence of signal transmission, camera shaking, etc., when the mining device is in a stationary state, the coordinates of its first bounding box may be offset to a certain extent. Assume that the upper left corner pixel coordinates of the first bounding box of the mining device in the i-th frame of the current target area are The pixel coordinates of the lower right corner are Assume that the second threshold corresponding to the coordinate offset is δ and the frame number threshold is α. If formula (9) holds, that is, the coordinate difference between any two frames from the i-th frame video stream image to the i+α-th frame video stream image is less than the second threshold δ, then it can be determined that the excavation equipment is in a stationary state.
[0294] Formula (9) calculates and determines the coordinate difference between the upper left corner pixel coordinates of the i-th frame video stream image and the (i+j)-th frame video stream image. Therefore, Represents the horizontal coordinate of the upper left corner pixel of the (i+j)th frame of the video stream image, Represents the vertical coordinate of the upper left corner pixel of the (i+j)th frame of the video stream image.
[0295]
[0296] It can be understood that formula (9) calculates and determines the coordinate difference between the upper left pixel coordinates of the i-th frame video stream image and the (i+j)-th frame video stream image, and is only an optional embodiment. In fact, the coordinate difference can be calculated and determined for the upper right pixel coordinates, lower left pixel coordinates, or lower right pixel coordinates of the i-th frame video stream image and the (i+j)-th frame video stream image.
[0297] After determining the target area, step S3 obtains the target area number, the position information of the rectangular box corresponding to the target area, whether the second bounding box is included, and other information. Therefore, step E3 can determine whether the current target area contains the self-unloading equipment according to whether the current target area contains the information corresponding to the second bounding box. Of course, target detection technology can be used to perform target detection on the area image corresponding to the current target area to determine whether the current target area contains the self-unloading equipment.
[0298] Step E5 is a process of determining whether there is overlap between the dump equipment and the excavation equipment in the current target area, specifically including:
[0299] Step F1, determining an intersection-and-union ratio index value between a second bounding box corresponding to the dump equipment in the current target area and a first bounding box corresponding to the mining equipment in the current target area;
[0300] Step F2: If the intersection-to-union ratio index value is greater than a first threshold, it is determined that there is overlap between the dump equipment and the excavation equipment in the current target area.
[0301] Formula (10) shows the calculation process of the intersection-over-union index value between the second bounding box and the first bounding box.
[0302]
[0303] Wherein, A represents the second bounding box, B represents the first bounding box, A∩B represents the intersection area between the second bounding box and the first bounding box, and A∪B represents the union area between the second bounding box and the first bounding box.
[0304] In practical applications, when the intersection-and-union ratio index is greater than 0, it can be considered that there is an overlap between the dump equipment and the excavation equipment in the current target area. Therefore, examples of the first threshold may include: 0 or a real number greater than 0.
[0305] In step S5, the change of the excavation operation state of the current target area can be monitored. If the excavation operation state is updated from the loading state to the unexcavated state or the excavation state, it can be considered that the current target area has completed the current loading, and the excavation amount corresponding to the current loading of the current target area is determined according to the rated loading capacity corresponding to the self-unloading equipment in the current target area, and the cumulative excavation amount of the current target area is updated. The cumulative excavation amount of the current target area can be the sum of the excavation amounts corresponding to multiple loadings of the current target area. The cumulative excavation amount of the current target area can be the sum of the excavation amounts corresponding to multiple loadings of the current target area.
[0306] The process of determining the rated loading capacity corresponding to the self-unloading equipment in the current target area specifically includes:
[0307] Step G1, using the self-unloading equipment model classification model, determining the target model corresponding to the self-unloading equipment in the current target area;
[0308] Step G2: According to the target model, search in the mapping relationship between the search model and the rated loading capacity to determine the rated loading capacity corresponding to the self-unloading equipment in the current target area.
[0309] The dump truck model classification model of the embodiment of the present application may have the ability to classify dump truck models. In other words, it may determine the target model corresponding to the dump truck in the area image based on the area image corresponding to the current target area.
[0310] In a specific implementation, the embodiment of the present application can label the model category for the dump truck image data set, and train the dump truck model classification model based on the dump truck image data set.
[0311] Here, a process for acquiring a dump equipment image dataset is provided. Specifically, based on a video surveillance system, images of dump equipment covering different construction backgrounds, different lighting conditions, different working postures, etc. can be collected. In addition, it is also necessary to collect a combined image of a dump equipment and an excavation equipment in cooperation with each other in excavation operations to reflect the actual situation of mutual occlusion of the two types of construction machinery. Therefore, the dump equipment image dataset may include: dump equipment images and combined images.
[0312] The embodiment of the present application does not limit the specific structure of the dump truck model classification model. For example, the dump truck model classification model may include: a convolutional neural network structure such as VGG (Visual Geometry Group).
[0313] The embodiment of the present application can pre-save the mapping relationship between the model and the rated loading capacity. In this way, according to the target model corresponding to the self-unloading equipment in the current target area, the rated loading capacity corresponding to the self-unloading equipment in the current target area can be searched in the above mapping relationship to obtain.
[0314] In the embodiment of the present application, when the excavation operation state of the current target area is monitored to be updated from the loading state to the unexcavated state or the excavation state, the current target area may be considered to have completed the current loading, and the excavation amount corresponding to the current loading of the current target area may be determined according to the rated loading capacity corresponding to the self-unloading equipment in the current target area. The excavation amount corresponding to the current loading may be equal to the rated loading capacity corresponding to the self-unloading equipment in the current target area.
[0315] In step S6, the accumulated excavation volumes of all current target areas among the N target areas may be summed up to obtain the total real-time excavation volume corresponding to the construction area to be counted.
[0316] In one example of the present application, the total real-time excavation volume corresponding to the construction area to be counted can be set to 0 at the beginning of a natural day. Subsequently, the total real-time excavation volume can be updated according to the completion of the current loading of the current target area to obtain the continuously accumulated total real-time excavation volume. Therefore, the real-time monitoring process of the excavation volume can be a process of continuously updating the total real-time excavation volume according to the completion of the current loading of N current target areas.
[0317] In practical applications, if there are multiple construction areas to be counted in a hydropower project, the total real-time excavation volume corresponding to each construction area to be counted can be monitored separately according to natural days. After a natural day, the total real-time excavation volume corresponding to multiple construction areas to be counted can be merged. It can be understood that the embodiment of the present application does not limit the specific fusion method.
[0318] In summary, the real-time monitoring method of excavation volume in the embodiment of the present application determines N target areas in the video stream image according to the target information corresponding to the targets such as the excavation equipment and the self-unloading equipment in the video stream image, and determines the excavation volume corresponding to the current loading of the current target area according to the objective law between the excavation operation state of the target area and the operation state of the excavation equipment, whether there is a self-unloading equipment, and the position of the excavation equipment and the self-unloading equipment; in this way, the embodiment of the present application can continuously determine the target area in the video stream image and the operation state corresponding to the target area according to the continuously updated video stream image; on this basis, the embodiment of the present application can continuously determine the excavation volume corresponding to the current loading of the current target area in the video stream image, and the cumulative excavation volume of all current target areas in the video stream image. Therefore, the embodiment of the present application can realize real-time monitoring of earthwork excavation volume at a relatively low cost and a relatively fast speed based on the analysis of the video stream image.
[0319] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.
[0320] Based on the above embodiment, this embodiment also provides a construction scene object recognition device, referring to Fig.12 The device may specifically include: a collection module 1201 and an object recognition module 1202.
[0321] The acquisition module 1201 is used to acquire construction images;
[0322] The object recognition module 1202 is used to recognize the construction image according to the target detection model to obtain a recognition result; the recognition result includes: the category of the target object and the location information of the target object in the construction image;
[0323] The target detection model specifically includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level deep convolution module and a spatial pyramid pooling module; the neck network specifically includes: a multi-level feature fusion module and a multi-level attention module;
[0324] The object recognition module 1202 specifically includes:
[0325] A feature extraction module 1221 is used to determine a plurality of first image features corresponding to the construction image using a backbone network;
[0326] A feature fusion module 1222 is used to utilize a multi-level feature fusion module in the neck network to perform feature fusion processing according to the above-mentioned multiple first image features to obtain multiple fused image features;
[0327] An attention processing module 1223 is used to use the multi-level attention module in the neck network to perform multi-head self-attention processing on the above-mentioned multiple fusion image features respectively to obtain multiple multi-head self-attention image features;
[0328] The recognition result determination module 1224 is used to use the detection network to determine the recognition result based on the above-mentioned multiple multi-head self-attention image features.
[0329] Optionally, the multi-level attention module in the neck network performs multi-head self-attention processing on the multiple fused image features, including:
[0330] Performing weighted processing on the above fused image features to obtain a query matrix, a key matrix and a value matrix;
[0331] Add the horizontal position weight matrix and the vertical position weight matrix corresponding to the above fused image features to obtain a position weight matrix;
[0332] Multiplying the query matrix and the key matrix to obtain a first product result;
[0333] Multiplying the query matrix and the position weight matrix to obtain a second product result;
[0334] Adding the first multiplication result and the second multiplication result to obtain an addition result;
[0335] Performing activation processing on the addition result to obtain an activation result;
[0336] The activation result is multiplied by the value matrix to obtain the multi-head self-attention image features.
[0337] Optionally, the backbone network includes: n+1 level hourglass modules, n level deep convolutional modules; the first level hourglass module includes: at least one hourglass module; the first level deep convolutional module includes: at least one deep convolutional module;
[0338] Wherein, the i-th level hourglass module is connected in front of the i-th level depthwise convolutional module; n is a positive integer greater than 1, and i is a positive integer not greater than n;
[0339] The n+1th level hourglass module is connected behind the nth level depthwise convolutional module and in front of the spatial pyramid pooling module.
[0340] Optionally, the second-level hourglass module is connected to the above-mentioned neck network, and outputs the first image feature A to the above-mentioned neck network; the third-level hourglass module is connected to the above-mentioned neck network, and outputs the first image feature B to the above-mentioned neck network; the spatial pyramid pooling module is connected to the above-mentioned neck network, and outputs the first image feature C to the above-mentioned neck network.
[0341] Optionally, the hourglass module includes: a first lightweight convolution module, a second lightweight convolution module, a first splicing module, a third convolution module and a fourth convolution module;
[0342] Among them, the above-mentioned first lightweight convolution module and the above-mentioned second lightweight convolution model correspond to the same network structure; the above-mentioned first lightweight convolution module includes: a first convolution module and a fourth depth-separable convolution module; the above-mentioned second lightweight convolution module includes: a second convolution module and a fifth depth-separable convolution module;
[0343] The outputs of the first lightweight convolution module are respectively input into the second lightweight convolution module and the first splicing module; the output of the second lightweight convolution module is input into the first splicing module; the output of the first splicing module is input into the third convolution module; and the output of the third convolution module is input into the fourth convolution module.
[0344] Optionally, the above-mentioned spatial pyramid pooling module includes: a convolution module A, p first maximum pooling modules, a second splicing module and a convolution module B; p is a positive integer greater than 1.
[0345] Optionally, the backbone network further includes: a convolution processing module connected before the first-stage hourglass module;
[0346] The convolution processing module includes: a fifth convolution module, a sixth convolution module, a seventh convolution module, a second maximum pooling module, a third splicing module, an eighth convolution module and a ninth convolution module;
[0347] Among them, the fifth convolution module receives input features and provides outputs to the sixth convolution module and the second maximum pooling module respectively; the output of the sixth convolution module is input to the seventh convolution module; the third splicing module receives the outputs of the second maximum pooling module and the seventh convolution module respectively, and provides outputs to the eighth convolution module; the ninth convolution module is connected to the eighth convolution module.
[0348] Optionally, the multi-level feature fusion module includes: a first-level feature fusion module, a second-level feature fusion module and a third-level feature fusion module; the multi-level attention module includes: a first attention module, a second attention module and a third attention module;
[0349] The first-level feature fusion module includes: a first upsampling module, a fourth splicing module, a first channel to pixel module, a second upsampling module, a fifth splicing module, and a second channel to pixel module connected in series; the output of the second channel to pixel module is input into the first attention module;
[0350] The second-level feature fusion module includes: a tenth convolution module, a sixth splicing module and a third channel to pixel module connected in series; the input of the third channel to pixel module is input into the second attention module; the output of the first attention module is input into the tenth convolution module;
[0351] The third-level feature fusion module includes: an eleventh convolution module, a seventh splicing module and a fourth channel-to-pixel module connected in series; the output of the fourth channel-to-pixel module is input into the third attention module; the output of the second attention module is input into the eleventh convolution module;
[0352] Among them, the fifth splicing module receives the first image feature A output by the backbone network, the fourth splicing module receives the first image feature B output by the backbone network, the first upsampling module and the seventh splicing module respectively receive the first image feature C output by the backbone network; the first channel to pixel module is also connected to the sixth splicing module; the first attention module outputs multi-head self-attention image feature A, the second attention module outputs multi-head self-attention image feature B, and the third attention module outputs multi-head self-attention image feature C.
[0353] Optionally, the training process of the above target detection model includes:
[0354] Input the construction image sample into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: the prediction box information corresponding to the target object category; the real box information corresponding to the construction image sample;
[0355] Determine loss information corresponding to the real frame information and the predicted frame information; the loss information includes: cross entropy loss corresponding to the predicted frame category, intersection-over-union loss corresponding to the predicted frame position, and focal loss corresponding to the predicted frame position;
[0356] According to the above loss information, the parameters of the above target detection model are updated.
[0357] Optionally, the process of acquiring the construction image sample includes:
[0358] Randomly read four original construction images from the construction image collection;
[0359] Perform random enhancement operations on the four original construction images respectively to obtain four enhanced construction images;
[0360] The four enhanced construction images are fused into one fused image;
[0361] Annotate a fused image with the true box information to obtain a training sample.
[0362] In summary, the construction scene object recognition device of the embodiment of the present application is provided with an attention module in the neck network of the target detection model, and the attention module is used to perform multi-head self-attention processing on the above-mentioned multiple fused image features to obtain multiple multi-head self-attention image features. Among them, the above-mentioned multi-head self-attention processing can capture the complex dependencies between elements in the sequence. Specifically to the embodiment of the present application, the sequence is specifically a fused image feature, and the elements in the sequence can represent image features such as the shape, texture and context information of multiple categories of objects; in this way, the multi-head self-attention processing of the embodiment of the present application can effectively capture the nonlinear relationship between the image features of multiple categories of objects in the fused image features, and perform cross-channel information interaction. The information interaction corresponding to the above nonlinear relationship can improve the feature fusion and feature processing capabilities of the neck network. On the basis of improving the feature fusion and feature processing capabilities of the neck network, the three layers of fused image features with different resolutions output by the neck network can simultaneously improve the detection accuracy of larger, medium and smaller size targets in the construction scene image. Therefore, the embodiment of the present application can improve the recognition accuracy of construction scene objects.
[0363] Moreover, the backbone network of the embodiment of the present application specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level depth convolution module and a spatial pyramid pooling module. The above-mentioned multi-level hourglass module and the multi-level depth convolution module can extract the depth feature information of different levels of the construction scene image through repeated calculations for many times, can capture multi-scale object features, and enhance the adaptability of the backbone network to construction scene objects of different sizes. Moreover, the backbone network of the embodiment of the present application merges the operations of dimensionality increase and downsampling into one module, and can maintain high-level semantic information and low-level detail information in the deep network, so that the recognition accuracy of construction scene objects of different sizes can be improved, especially, the recognition accuracy of small-sized construction scene objects can be improved.
[0364] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.
[0365] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).
[0366] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform a desired configuration, and the device may include: electronic devices such as terminal devices and servers (clusters). Fig.13 An exemplary apparatus 1300 that can be used to implement various embodiments described in this application is schematically shown.
[0367] For one embodiment, Fig.13 An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, an NVM (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.
[0368] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a terminal device, server (cluster), and other devices described in the embodiments of the present application.
[0369] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.
[0370] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .
[0371] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.
[0372] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, such as a suitable DRAM (Dynamic Random Access Memory). In some embodiments, the memory 1306 may include a double data rate type four synchronous dynamic random access memory.
[0373] For one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1308 and the input / output device(s) 1310 .
[0374] For example, non-volatile memory / storage 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile optical disk drives).
[0375] The non-volatile memory / storage device 1308 may include storage resources that are physically part of the device on which the apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, the non-volatile memory / storage device 1308 may be accessed via (one or more) input / output devices 1310 over a network.
[0376] (One or more) input / output devices 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, an audio component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2nd Generation Wireless Telephone Technology), 3G (3rd Generation Wireless Telephone Technology), 4G (4th Generation Wireless Telephone Technology), 5G (5th Generation Wireless Telephone Technology), etc., or a combination thereof for wireless communication.
[0377] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip.
[0378] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touch screen device, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, a dedicated integrated circuit, and a speaker.
[0379] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or non-volatile memory / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.
[0380] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0381] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0382] The present application embodiment is described with reference to the flowchart and / or block diagram of the method, terminal device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.
[0383] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0384] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0385] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0386] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0387] The above is a detailed introduction to a construction scene object recognition method and device, an electronic device and a machine-readable medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application sets. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A construction scene object recognition method, characterized in that: The method comprises: Collect construction images; According to the target detection model, the construction image is recognized to obtain a recognition result; the recognition result includes: the category of the target object and the position information of the target object in the construction image; The target detection model includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network includes: a convolution processing module, a multi-level hourglass module, a multi-level deep convolution module and a spatial pyramid pooling module; the neck network includes: a multi-level feature fusion module and a multi-level attention module; The identifying the construction image according to the target detection model includes: The backbone network determines a plurality of first image features corresponding to the construction image; A multi-level feature fusion module in the neck network performs feature fusion processing according to the multiple first image features to obtain multiple fused image features; The multi-level attention module in the neck network performs multi-head self-attention processing on the multiple fused image features respectively to obtain multiple multi-head self-attention image features; The detection network determines the recognition result based on the multiple multi-head self-attention image features.
2. The method according to claim 1, characterized in that The multi-level attention module in the neck network performs multi-head self-attention processing on the multiple fused image features respectively, including: Performing weighted processing on the fused image features to obtain a query matrix, a key matrix and a value matrix; Adding the horizontal position weight matrix and the vertical position weight matrix corresponding to the fused image features to obtain a position weight matrix; Multiplying the query matrix and the key matrix to obtain a first product result; Multiplying the query matrix and the position weight matrix to obtain a second product result; Adding the first multiplication result and the second multiplication result to obtain an addition result; Performing activation processing on the addition result to obtain an activation result; The activation result is multiplied by the value matrix to obtain the multi-head self-attention image features.
3. The method according to claim 1, characterized in that The backbone network includes: n+1 level hourglass modules, n level deep convolution modules; the first level hourglass module includes: at least one hourglass module; the first level deep convolution module includes: at least one deep convolution module; Wherein, the i-th level hourglass module is connected in front of the i-th level depthwise convolutional module; n is a positive integer greater than 1, and i is a positive integer not greater than n; The n+1th level hourglass module is connected to the back of the nth level depthwise convolutional module and to the front of the spatial pyramid pooling module.
4. The method according to claim 3, characterized in that The second-level hourglass module is connected to the neck network and outputs the first image feature A to the neck network; the third-level hourglass module is connected to the neck network and outputs the first image feature B to the neck network; the spatial pyramid pooling module is connected to the neck network and outputs the first image feature C to the neck network.
5. The method according to any one of claims 1 to 4, characterized in that: The hourglass module includes: a first lightweight convolution module, a second lightweight convolution module, a first splicing module, a third convolution module and a fourth convolution module; Among them, the first lightweight convolution module and the second lightweight convolution model correspond to the same network structure; the first lightweight convolution module includes: a first convolution module and a fourth depth-separable convolution module; the second lightweight convolution module includes: a second convolution module and a fifth depth-separable convolution module; The output of the first lightweight convolution module is respectively input into the second lightweight convolution module and the first splicing module; the output of the second lightweight convolution module is input into the first splicing module; the output of the first splicing module is input into the third convolution module; the output of the third convolution module is input into the fourth convolution module.
6. The method according to any one of claims 1 to 4, characterized in that: The spatial pyramid pooling module includes: a convolution module A, p first maximum pooling modules, a second splicing module and a convolution module B; p is a positive integer greater than 1.
7. The method according to any one of claims 1 to 4, characterized in that: The convolution processing module includes: a fifth convolution module, a sixth convolution module, a seventh convolution module, a second maximum pooling module, a third splicing module, an eighth convolution module and a ninth convolution module; Among them, the fifth convolution module receives input features and provides outputs to the sixth convolution module and the second maximum pooling module respectively; the output of the sixth convolution module is input to the seventh convolution module; the third splicing module receives the outputs of the second maximum pooling module and the seventh convolution module respectively, and provides outputs to the eighth convolution module; the ninth convolution module is connected to the eighth convolution module.
8. The method according to any one of claims 1 to 4, characterized in that: The multi-level feature fusion module includes: a first-level feature fusion module, a second-level feature fusion module and a third-level feature fusion module; the multi-level attention module includes: a first attention module, a second attention module and a third attention module; The first-level feature fusion module includes: a first upsampling module, a fourth splicing module, a first channel to pixel module, a second upsampling module, a fifth splicing module, and a second channel to pixel module connected in series; the output of the second channel to pixel module is input into the first attention module; The second-level feature fusion module includes: a tenth convolution module, a sixth splicing module and a third channel to pixel module connected in series; the input of the third channel to pixel module is input into the second attention module; the output of the first attention module is input into the tenth convolution module; The third-level feature fusion module includes: an eleventh convolution module, a seventh splicing module and a fourth channel-to-pixel module connected in series; the output of the fourth channel-to-pixel module is input into the third attention module; the output of the second attention module is input into the eleventh convolution module; Among them, the fifth splicing module receives the first image feature A output by the backbone network, the fourth splicing module receives the first image feature B output by the backbone network, the first upsampling module and the seventh splicing module respectively receive the first image feature C output by the backbone network; the first channel to pixel module is also connected to the sixth splicing module; the first attention module outputs multi-head self-attention image feature A, the second attention module outputs multi-head self-attention image feature B, and the third attention module outputs multi-head self-attention image feature C.
9. The method according to any one of claims 1 to 4, characterized in that: The training process of the target detection model includes: Input the construction image sample into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: prediction box information corresponding to the target object category; and real box information corresponding to the construction image sample; Determine loss information corresponding to the real frame information and the predicted frame information; the loss information includes: a cross entropy loss corresponding to the predicted frame category, an intersection-over-union loss corresponding to the predicted frame position, and a focal loss corresponding to the predicted frame position; According to the loss information, the parameters of the target detection model are updated.
10. The method according to claim 9, characterized in that The process of acquiring the construction image sample includes: Randomly read four original construction images from the construction image collection; Perform random enhancement operations on the four original construction images respectively to obtain four enhanced construction images; The four enhanced construction images are fused into one fused image; Annotate a fused image with the true box information to obtain a training sample.
11. A construction scene object recognition device, characterized in that: The device comprises: A collection module, used for collecting construction images; An object recognition module is used to recognize the construction image according to the target detection model to obtain a recognition result; the recognition result includes: the category of the target object and the location information of the target object in the construction image; The target detection model includes: a backbone network, a neck network and a detection network connected in sequence; the backbone network includes: a convolution processing module, a multi-level hourglass module, a multi-level deep convolution module and a spatial pyramid pooling module; the neck network includes: a multi-level feature fusion module and a multi-level attention module; The object recognition module comprises: A feature extraction module, used to determine a plurality of first image features corresponding to the construction image using a backbone network; A feature fusion module, used to utilize the multi-level feature fusion module in the neck network to perform feature fusion processing according to the multiple first image features to obtain multiple fused image features; An attention processing module, used for performing multi-head self-attention processing on the multiple fused image features respectively using the multi-level attention modules in the neck network to obtain multiple multi-head self-attention image features; The recognition result determination module is used to determine the recognition result based on the multiple multi-head self-attention image features using the detection network.
12. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method according to any one of claims 1 to 10.
13. A machine-readable medium having executable codes stored thereon, which, when executed, causes a processor to perform the method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Safety helmet wearing detection method in complex scene
CN114419659A
Multi-scale feature extraction fusion power distribution network equipment positioning identification method
CN115830302A
Traffic sign recognition method, device and equipment and storage medium
CN117671641A
Unmanned aerial vehicle aerial photography small target detection method based on multi-level feature fusion
CN118072146A
Cited By
Smart city target detection method and device based on improved YOLOv5, and storage medium
CN119672550A
