Construction scene object identification method, device, equipment and medium

Abstract

Embodiments of the present application provide a construction scene object identification method, device, equipment and medium, wherein the method specifically comprises: identifying a construction image according to a target detection model to obtain an identification result; the target detection model comprises a backbone network, a neck network and a detection network connected in sequence; the neck network comprises a multi-level feature fusion module and a multi-level attention module; the backbone network determines a plurality of first image features corresponding to the construction image; the multi-level feature fusion module in the neck network performs feature fusion processing according to the plurality of first image features to obtain a plurality of fused image features; the multi-level attention module in the neck network respectively performs multi-head self-attention processing on the plurality of fused image features to obtain a plurality of multi-head self-attention image features; and the detection network determines the identification result according to the plurality of multi-head self-attention image features. The embodiments of the present application can improve the identification accuracy of construction scene objects.

Description

Technical Field

[0001] This application relates to the field of construction scene image understanding technology, and in particular to a method, apparatus, equipment and medium for recognizing objects in a construction scene. Background Technology

[0002] Video surveillance systems are often installed at construction sites for hydropower and civil engineering projects to compensate for the shortcomings of manual inspection and improve the efficiency of security monitoring.

[0003] Object detection based on deep learning technology is a fundamental computer vision analysis task. It can identify and locate various objects, such as construction machinery and workers, in construction scenarios, providing key inputs for relevant objects in construction safety management and construction progress management, and supporting the gradual realization of all-weather online monitoring and analysis based on video surveillance systems. Therefore, the accuracy of object recognition directly affects the effectiveness of subsequent analysis tasks and project management objectives.

[0004] In practical applications, the shape, texture, and contextual information of multi-category objects in complex construction scene images are often non-linear. However, current object detection methods cannot effectively capture these non-linear relationships, resulting in insufficient and non-discriminatory feature representations, which in turn affects the recognition accuracy of construction scene objects. Summary of the Invention

[0005] This application provides a method for recognizing objects in a construction scene, which can improve the recognition accuracy of objects in a construction scene.

[0006] Accordingly, embodiments of this application also provide a construction scene object recognition device, an electronic device, and a machine-readable medium to ensure the implementation and application of the above methods.

[0007] To address the aforementioned problems, this application discloses a method for object recognition in a construction scene, comprising:

[0008] Acquire construction images;

[0009] The construction image is identified based on the target detection model to obtain the identification result; the identification result includes: the category of the target object and the location information of the target object in the construction image;

[0010] The target detection model comprises a backbone network, a neck network, and a detection network connected in sequence; the backbone network comprises a convolutional processing module, a multi-level hourglass module, a multi-level deep convolutional module, and a spatial pyramid pooling module; the neck network comprises a multi-level feature fusion module and a multi-level attention module.

[0011] The step of identifying the construction image based on the target detection model includes:

[0012] The backbone network determines multiple first image features corresponding to the construction images;

[0013] The multi-level feature fusion module in the neck network performs feature fusion processing based on the multiple first image features to obtain multiple fused image features;

[0014] The multi-level attention modules in the neck network perform multi-head self-attention processing on the various fused image features to obtain various multi-head self-attention image features;

[0015] The detection network determines the recognition result based on the various multi-head self-attention image features.

[0016] This application also discloses a construction scene object recognition device, the device comprising:

[0017] The acquisition module is used to acquire construction images;

[0018] The object recognition module is used to recognize the construction image according to the target detection model and obtain the recognition result; the recognition result includes: the category of the target object and the location information of the target object in the construction image;

[0019] The target detection model comprises a backbone network, a neck network, and a detection network connected in sequence; the backbone network comprises a convolutional processing module, a multi-level hourglass module, a multi-level deep convolutional module, and a spatial pyramid pooling module; the neck network comprises a multi-level feature fusion module and a multi-level attention module.

[0020] The object recognition module includes:

[0021] The feature extraction module is used to determine multiple first image features corresponding to the construction image using the backbone network;

[0022] The feature fusion module is used to perform feature fusion processing based on the multiple first image features using the multi-level feature fusion module in the neck network to obtain multiple fused image features;

[0023] An attention processing module is used to perform multi-head self-attention processing on the various fused image features using multi-level attention modules in the neck network, so as to obtain various multi-head self-attention image features.

[0024] The recognition result determination module is used to determine the recognition result based on the various multi-head self-attention image features using the detection network.

[0025] Optionally, the multi-level attention modules in the neck network perform multi-head self-attention processing on the various fused image features, including:

[0026] The fused image features are weighted to obtain a query matrix, a key matrix, and a value matrix;

[0027] The horizontal position weight matrix and the vertical position weight matrix corresponding to the fused image features are added together to obtain the position weight matrix;

[0028] Multiply the query matrix and the keyword matrix to obtain the first product result;

[0029] The query matrix and the position weight matrix are multiplied to obtain the second product result;

[0030] Add the first product result to the second product result to obtain the sum;

[0031] The addition result is activated to obtain the activated result;

[0032] The activation result is multiplied by the numerical matrix to obtain the multi-head self-attention image features.

[0033] Optionally, the backbone network includes: an n+1 level hourglass module and an n-level depthwise convolutional module; the first-level hourglass module includes: at least one hourglass module; the first-level depthwise convolutional module includes: at least one depthwise convolutional module.

[0034] Here, the i-th level hourglass module is connected in front of the i-th level depthwise convolutional module; n is a positive integer greater than 1, and i is a positive integer not greater than n;

[0035] The (n+1)th level hourglass module is connected after the nth level depthwise convolutional module and before the spatial pyramid pooling module.

[0036] Optionally, the second-level hourglass module is connected to the neck network and outputs a first image feature A to the neck network; the third-level hourglass module is connected to the neck network and outputs a first image feature B to the neck network; the spatial pyramid pooling module is connected to the neck network and outputs a first image feature C to the neck network.

[0037] Optionally, the hourglass module includes: a first lightweight convolution module, a second lightweight convolution module, a first splicing module, a third convolution module, and a fourth convolution module;

[0038] Wherein, the first lightweight convolutional module and the second lightweight convolutional model correspond to the same network structure; the first lightweight convolutional module includes: a first convolutional module and a fourth depthwise separable convolutional module; the second lightweight convolutional module includes: a second convolutional module and a fifth depthwise separable convolutional module;

[0039] The output of the first lightweight convolution module is input to the second lightweight convolution module and the first stitching module, respectively; the output of the second lightweight convolution module is input to the first stitching module; the output of the first stitching module is input to the third convolution module; and the output of the third convolution module is input to the fourth convolution module.

[0040] Optionally, the spatial pyramid pooling module includes: a convolution module A, p first max pooling modules, a second concatenation module, and a convolution module B; where p is a positive integer greater than 1.

[0041] Optionally, the convolution processing module includes: a fifth convolution module, a sixth convolution module, a seventh convolution module, a second max pooling module, a third concatenation module, an eighth convolution module, and a ninth convolution module;

[0042] The fifth convolutional module receives input features and provides outputs to the sixth convolutional module and the second max pooling module, respectively; the output of the sixth convolutional module is input to the seventh convolutional module; the third concatenation module receives the outputs of the second max pooling module and the seventh convolutional module, respectively, and provides outputs to the eighth convolutional module; the ninth convolutional module is connected to the eighth convolutional module.

[0043] Optionally, the multi-level feature fusion module includes: a first-level feature fusion module, a second-level feature fusion module, and a third-level feature fusion module; the multi-level attention module includes: a first attention module, a second attention module, and a third attention module;

[0044] The first-level feature fusion module includes: a first upsampling module, a fourth stitching module, a first channel-to-pixel module, a second upsampling module, a fifth stitching module, and a second channel-to-pixel module connected in series; the output of the second channel-to-pixel module is input to the first attention module.

[0045] The second-level feature fusion module includes: a tenth convolutional module, a sixth concatenation module, and a third channel-to-pixel module connected in series; the input of the third channel-to-pixel module is input to the second attention module; the output of the first attention module is input to the tenth convolutional module.

[0046] The third-level feature fusion module includes: an eleventh convolutional module, a seventh concatenation module, and a fourth channel-to-pixel module connected in series; the output of the fourth channel-to-pixel module is input to the third attention module; the output of the second attention module is input to the eleventh convolutional module.

[0047] The fifth stitching module receives the first image feature A output by the backbone network, the fourth stitching module receives the first image feature B output by the backbone network, and the first upsampling module and the seventh stitching module respectively receive the first image feature C output by the backbone network; the first channel to pixel module is also connected to the sixth stitching module; the first attention module outputs multi-head self-attention image feature A, the second attention module outputs multi-head self-attention image feature B, and the third attention module outputs multi-head self-attention image feature C.

[0048] Optionally, the training process of the object detection model includes:

[0049] A construction image sample is input into a target detection model, which outputs a prediction result corresponding to the construction image sample. The prediction result includes: predicted bounding box information corresponding to the target object category; and ground truth bounding box information corresponding to the construction image sample.

[0050] Determine the loss information corresponding to the ground truth bounding box information and the predicted bounding box information; the loss information includes: cross-entropy loss corresponding to the predicted bounding box category, cross-union ratio loss corresponding to the predicted bounding box position, and focus loss corresponding to the predicted bounding box position;

[0051] The parameters of the target detection model are updated based on the loss information.

[0052] Optionally, the process of acquiring the construction image samples includes:

[0053] Four original construction images are randomly selected from the collection of construction images;

[0054] Random enhancement operations were performed on the four original construction images to obtain four enhanced construction images;

[0055] Four enhanced construction images were merged into one merged image;

[0056] Annotate the ground truth bounding boxes on a fused image to obtain a training sample.

[0057] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform the method described in this application.

[0058] This application also discloses a machine-readable medium storing executable code thereon, which, when executed, causes a processor to perform the method described in this application.

[0059] The embodiments of this application have the following advantages:

[0060] In the technical solution of this application embodiment, an attention module is set in the neck network of the target detection model. This attention module is used to perform multi-head self-attention processing on the above-mentioned multiple fused image features to obtain multiple multi-head self-attention image features. The above-mentioned multi-head self-attention processing can capture complex dependencies between elements within a sequence. Specifically, in this application embodiment, the sequence is specifically fused image features, and the elements within the sequence can represent image features such as shape, texture, and contextual information of multiple categories of objects. Thus, the multi-head self-attention processing of this application embodiment can effectively capture the nonlinear relationships between image features of multiple categories of objects in the fused image features and perform cross-channel information interaction. The information interaction corresponding to the above-mentioned nonlinear relationships can improve the feature fusion and feature processing capabilities of the neck network. Based on improving the feature fusion and feature processing capabilities of the neck network, the three layers of fused image features with different resolutions output by the neck network can simultaneously improve the detection accuracy of large, medium, and small-sized targets in construction scene images. Therefore, this application embodiment can improve the recognition accuracy of objects in construction scenes.

[0061] Furthermore, the backbone network of this embodiment specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level depthwise convolutional module, and a spatial pyramid pooling module. The aforementioned multi-level hourglass module and multi-level depthwise convolutional module, through repeated computation, can extract depth feature information from different levels of the construction scene image, capture multi-scale image semantic features, and enhance the backbone network's adaptability to construction scene objects of different sizes. Moreover, the above-mentioned structure of the backbone network combines upsampling and downsampling operations into one module, which can maintain high-level semantic information and low-level detail information in deep networks. Thus, this embodiment can improve the recognition accuracy of construction scene objects of different sizes, especially improving the recognition accuracy of small-sized construction scene objects. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the application environment of a construction scene object recognition method according to an embodiment of this application;

[0063] Figure 2 This is a flowchart illustrating the steps of a construction scene object recognition method according to an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of the structure of an object detection model according to an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of the structure of an hourglass module according to an embodiment of this application;

[0066] Figure 5 This is a schematic diagram of the structure of a depthwise convolutional module according to an embodiment of this application;

[0067] Figure 6 This is a schematic diagram of the structure of a spatial pyramid pooling module according to an embodiment of this application;

[0068] Figure 7 This is a schematic diagram of the structure of a convolution processing module according to an embodiment of this application;

[0069] Figure 8 This is a schematic diagram of the backbone network structure according to an embodiment of this application;

[0070] Figure 9 This is a schematic diagram of the structure of a channel-to-pixel module according to an embodiment of this application;

[0071] Figure 10 This is a flowchart of the attention module processing according to an embodiment of this application;

[0072] Figure 11 This is a schematic diagram of the neck network structure according to an embodiment of this application;

[0073] Figure 12 This is a schematic diagram of the structure of a construction scene object recognition device according to an embodiment of this application;

[0074] Figure 13 This is a schematic diagram of the structure of an apparatus provided in one embodiment of this application. Detailed Implementation

[0075] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] The embodiments of this application can be applied to engineering industries such as hydropower and civil engineering to improve the recognition accuracy of objects in construction scenes.

[0077] This application provides a method for recognizing objects in a construction scene. The method specifically includes the following steps: acquiring construction images; recognizing the construction images according to a target detection model to obtain recognition results; the recognition results specifically include: the category of the target object and the location information of the target object in the construction image.

[0078] The aforementioned target detection model specifically includes: a backbone network, a neck network, and a detection network connected in sequence; the backbone network specifically includes: a convolutional processing module, a multi-level hourglass module, a multi-level deep convolutional module, and a spatial pyramid pooling module; the neck network specifically includes: a multi-level feature fusion module and a multi-level attention module.

[0079] The process of identifying the construction images based on the target detection model specifically includes:

[0080] The backbone network determines various first image features corresponding to the aforementioned construction images;

[0081] The multi-level feature fusion module in the neck network performs feature fusion processing based on the aforementioned multiple first image features to obtain multiple fused image features;

[0082] The multi-level attention modules in the neck network perform multi-head self-attention processing on the above-mentioned multiple fused image features to obtain multiple multi-head self-attention image features;

[0083] The detection network determines the recognition result based on the aforementioned multi-head self-attention image features.

[0084] This application embodiment incorporates an attention module within the neck network of the target detection model. This attention module performs multi-head self-attention processing on the aforementioned multiple fused image features to obtain various multi-head self-attention image features. Specifically, this multi-head self-attention processing captures complex dependencies between elements within a sequence. In this embodiment, the sequence refers to fused image features, and the elements within the sequence can represent image features such as shape, texture, and contextual information of multiple object categories. Thus, this multi-head self-attention processing effectively captures the nonlinear relationships between image features of multiple object categories within the fused image features and performs cross-channel information interaction. The information interaction corresponding to these nonlinear relationships enhances the feature fusion and processing capabilities of the neck network. Based on this improved feature fusion and processing capabilities, the three layers of fused image features with different resolutions output by the neck network can simultaneously improve the detection accuracy for large, medium, and small-sized targets in construction scene images. Therefore, this application embodiment can improve the recognition accuracy of objects in construction scenes.

[0085] Furthermore, the backbone network of this embodiment specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level depthwise convolutional module, and a spatial pyramid pooling module. The aforementioned multi-level hourglass module and multi-level depthwise convolutional module, through repeated computation, can extract depth feature information from different levels of the construction scene image, capture multi-scale image semantic features, and enhance the backbone network's adaptability to construction scene objects of different sizes. Moreover, the backbone network of this embodiment combines upsampling and downsampling operations into one module, which can maintain high-level semantic information and low-level detail information in the deep network. This improves the recognition accuracy of construction scene objects of different sizes, especially improving the recognition accuracy of small-sized construction scene objects.

[0086] Reference Figure 1 The diagram illustrates the application environment of a construction scene object recognition method according to an embodiment of this application, wherein the image acquisition terminal 101 and the server terminal 102 can interact with each other based on a wireless network or a wired network.

[0087] In practical applications, the image acquisition terminal 101 can be equipped with an image acquisition device such as an image sensor. If the image acquisition device can acquire construction images, the image acquisition terminal 101 can send these images to the server 102 according to a preset time period. Alternatively, if the image acquisition device can acquire construction video, the image acquisition terminal 101 can send the construction video to the server 102 according to a preset time period; in this case, the server 102 can parse the construction images from the construction video.

[0088] After receiving the construction image sent by the image acquisition terminal 101, the server 102 can use the method of this application embodiment to identify the construction image and obtain the identification result.

[0089] The above identification results can help improve the safety, quality, and schedule control of hydropower projects, and play an important role in promoting the intelligent development of hydropower construction.

[0090] For example, the above identification results can be used to determine various states during the construction process, such as whether safety helmets are worn, whether machinery is operated in accordance with regulations, and whether materials are stacked reasonably. This can reduce the workload of manual monitoring and improve the accuracy and efficiency of monitoring.

[0091] For example, in addition to identifying excavators and dump trucks in the construction scene, it is also possible to monitor the excavation volume in real time, which helps to control the construction progress and allocate resources rationally.

[0092] Method Example 1

[0093] refer to Figure 2The diagram illustrates a flowchart of a construction scene object recognition method according to an embodiment of this application. The method specifically includes the following steps:

[0094] Step 201: Acquire construction images;

[0095] Step 202: Based on the target detection model, the above construction image is identified to obtain the identification result; the identification result specifically includes: the category of the target object and the location information of the target object in the above construction image;

[0096] The aforementioned target detection model specifically includes: a backbone network, a neck network, and a detection network connected in sequence; the backbone network specifically includes: a convolutional processing module, a multi-level hourglass module, a multi-level deep convolutional module, and a spatial pyramid pooling module; the neck network specifically includes: a multi-level feature fusion module and a multi-level attention module.

[0097] Step 202 above, which involves recognizing the construction image based on the target detection model, specifically includes:

[0098] Step 221: The backbone network determines the various first image features corresponding to the above construction images;

[0099] Step 222: The multi-level feature fusion module in the neck network performs feature fusion processing based on the above-mentioned multiple first image features to obtain multiple fused image features;

[0100] Step 223: The multi-level attention module in the neck network performs multi-head self-attention processing on the above-mentioned multiple fused image features to obtain multiple multi-head self-attention image features;

[0101] Step 224: The detection network determines the recognition result based on the above-mentioned multi-head self-attention image features.

[0102] Figure 2 The steps included in the illustrated method embodiment can be executed by the server, which can leverage its abundant computing resources to process construction images quickly. It is understood that the embodiments of this application are for... Figure 2 The specific entity that performs the method embodiment shown is not limited.

[0103] In step 201, the server can receive construction images sent by the image acquisition terminal according to a preset time period. The image acquisition terminal can be located at the construction site to collect construction images in real time and send them to the server according to the preset time period.

[0104] In step 202, the construction image is identified according to the target detection model. The identification result specifically includes: the category of the target object and the location information of the target object in the construction image. The category of the target object specifically includes at least one of the following categories: human category, construction machinery category, safety protection equipment wearing category, and safety sign category.

[0105] Examples of categories for "humans" include: construction workers. Examples of categories for "construction machinery" include: backhoe excavators, tower cranes, dump trucks, truck cranes, loaders, hooks, pump trucks, smooth rollers, concrete mixer trucks, pile drivers, etc. The category for "wearing safety protective equipment" specifically includes: the category for "wearing safety protective equipment" or the category for "not wearing safety protective equipment." The category for "safety signs" specifically includes: safety slogans, etc.

[0106] Reference Figure 3 The diagram shows a structural schematic of a target detection model according to an embodiment of this application, which specifically includes a backbone network 301, a neck network 302 and a detection network 303 connected in sequence.

[0107] The backbone network 301 is used to extract features from the input image to obtain various primary image features. During the training phase, the input image can be a construction image sample. During the image recognition phase, the input image can be a real-time construction image.

[0108] The neck network 302 is used to perform feature fusion processing and multi-head attention processing on multiple first image features, and outputs multiple multi-head self-attention image features to the detection head network 303.

[0109] The detection network 303 is used to determine the recognition result based on the above-mentioned multi-head self-attention image features.

[0110] The backbone network of this embodiment specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level depthwise convolutional module, and a spatial pyramid pooling module. The aforementioned multi-level hourglass module and multi-level depthwise convolutional module can capture multi-scale object features, enhancing the backbone network's adaptability to construction scene objects of different sizes.

[0111] Reference Figure 4 The diagram shows a structural schematic of an hourglass module according to an embodiment of the present application. The hourglass module specifically includes: a first lightweight convolution module 401, a second lightweight convolution module 402, a first splicing module 403, a third convolution module 404, and a fourth convolution module 405.

[0112] The first lightweight convolution module 401 and the second lightweight convolution model 402 correspond to the same network structure; the first lightweight convolution module 401 includes: a first convolution module and a fourth depthwise separable convolution module; the second lightweight convolution module 402 includes: a second convolution module and a fifth depthwise separable convolution module.

[0113] The output of the first lightweight convolution module 401 is input to the second lightweight convolution module 402 and the first splicing module 403 respectively; the output of the first splicing module 403 is input to the third convolution module 404; and the output of the third convolution module 404 is input to the fourth convolution module 405.

[0114] In practical applications, the hourglass module may include two or more lightweight convolutional modules. It is understood that this application embodiment does not limit the specific number of lightweight convolutional modules included in the hourglass module.

[0115] In the structure of the hourglass module, the outputs of multiple lightweight convolutional modules are merged through the first concatenation module. This can solve the gradient vanishing problem and capture feature information from different levels. Then, the merged output of the first concatenation module is subjected to two convolution operations with 1×1 kernels, which can compress feature channels and reduce the number of parameters.

[0116] Furthermore, the structure of the aforementioned hourglass module can capture multi-scale features, enhancing the adaptability of the backbone network to construction scene objects of different sizes. At the same time, through its unique topology, it can maintain high-level semantic information and low-level detailed information in the deep network, which is particularly helpful for the identification of construction scene objects of different sizes, especially small-sized construction scene objects.

[0117] Reference Figure 5 The diagram shows a schematic of the structure of a deep convolutional module according to an embodiment of this application. The deep convolutional module can perform deep convolution operation on each channel of the input feature using filters. It can extract features while maintaining the spatial dimension of the input features, and fuse the output features of different channels through pointwise convolution to reduce the number of channels. This enables deep learning of local spatial features while effectively reducing the parameters required for convolution calculation and improving the computation speed of the backbone network.

[0118] The spatial pyramid pooling module is used to perform spatial pyramid pooling processing on the input features. Spatial pyramid pooling processing can include convolution and max pooling operations, enabling deep fusion of input features. Specifically, the input features of the spatial pyramid pooling module are multi-size features output by the hourglass module; therefore, the spatial pyramid pooling module can achieve deep fusion of multi-size features.

[0119] Reference Figure 6 The diagram illustrates the structure of a spatial pyramid pooling module according to an embodiment of this application. The spatial pyramid pooling module specifically includes: a convolution module A601, p first max pooling modules 602, a second concatenation module 603, and a convolution module B604. p can be a positive integer greater than 1; in the diagram, p is 3.

[0120] The convolution module A601 is used to perform convolution processing on the input features to obtain the convolution result A.

[0121] p second max pooling modules 602 are used to perform max pooling processing on the input features to obtain the corresponding p max pooling processing results.

[0122] The convolution result A and the p max pooling results are respectively input into the second splicing module 603, which performs deep fusion on the convolution result A and the p max pooling results to obtain the corresponding deep fusion result.

[0123] Convolutional module B604 is used to perform convolution processing on the deep fusion result to obtain convolutional result B. Convolutional result B is then provided to the neck network.

[0124] In the specific implementation, the backbone network includes: n+1 level hourglass modules and n level depthwise convolutional modules; the first-level hourglass modules may include: at least one hourglass module; the first-level depthwise convolutional modules may include: at least one depthwise convolutional module.

[0125] Here, the i-th level hourglass module is connected in front of the i-th level depthwise convolutional module; n is a positive integer greater than 1, and i is a positive integer not greater than n;

[0126] The (n+1)th level hourglass module is connected after the nth level depthwise convolutional module and before the spatial pyramid pooling module.

[0127] The value n can be determined by those skilled in the art based on the actual application requirements; for example, the value of n can be 3.

[0128] Optionally, the second-level hourglass module is connected to the neck network and outputs first image feature A to the neck network; the third-level hourglass module is connected to the neck network and outputs first image feature B to the neck network; the spatial pyramid pooling module is connected to the neck network and outputs first image feature C to the neck network. Thus, the backbone network can output three types of first image features to the neck network.

[0129] Reference Figure 7The diagram shows a structural schematic of a convolution processing module according to an embodiment of the present application. The convolution processing module specifically includes: a fifth convolution module 701, a sixth convolution module 702, a seventh convolution module 703, a second maximum pooling module 704, a third splicing module 705, an eighth convolution module 706, and a ninth convolution module 707.

[0130] The fifth convolutional module 701 receives input features and provides outputs to the sixth convolutional module 702 and the second max pooling module 704, respectively; the output of the sixth convolutional module 702 is input to the seventh convolutional module 703; the third concatenation module 705 receives the outputs of the second max pooling module 704 and the seventh convolutional module 703, respectively, and provides outputs to the eighth convolutional module 706; the ninth convolutional module 707 is connected to the eighth convolutional module 706.

[0131] The convolution processing module can extract features from construction images, reducing the size of these features through upsampling and downsampling operations. For example... Figure 7 As shown, the construction image first passes through the fifth convolutional module 701. The fifth convolutional module 701 uses a 3×3 convolution operation with a linear rectified function for activation calculation, and then uses the resulting features as input to two parallel branches. One branch enters the second max-pooling module 704, where a 2×2 max-pooling operation is performed. The other branch passes consecutively through the sixth convolutional module 702 and the seventh convolutional module 703. Both the sixth and seventh convolutional modules 702 and 703 have 2×2 kernels. The features calculated by the two parallel branches are then merged through a concatenation operation in the third stitching module 705. The merged result passes consecutively through the eighth convolutional module 706 (with a 3×3 kernel) and the ninth convolutional module 707 (with a 1×1 kernel), outputting 48 extracted features.

[0132] Reference Figure 8This diagram illustrates the structure of a backbone network according to an embodiment of this application. The backbone network specifically includes: one convolutional processing module, four-level hourglass modules, and three-level depthwise separable convolutional modules. Specifically, the backbone network includes: a convolutional processing module 801, a first hourglass module 802, a first depthwise separable convolutional module 803, a second hourglass module 804, a second depthwise separable convolutional module 805, at least one third hourglass module 806, a third depthwise separable convolutional module 807, a fourth hourglass module 808, and a spatial pyramid pooling module 809. The first-level hourglass module includes one first hourglass module. The second-level hourglass module includes one second hourglass module. The third-level hourglass module includes multiple third hourglass modules. The fourth hourglass module includes one fourth hourglass module. The first-level depthwise separable convolutional module includes one first depthwise separable convolutional module. The second-level depthwise separable convolutional module includes one second depthwise separable convolutional module. The third-level depthwise separable convolutional module includes one third depthwise separable convolutional module.

[0133] The second hourglass module 804 can output the first image feature A to the neck network. The last third hourglass module 806 can output the first image feature B to the neck network. The spatial pyramid pooling module can output the first image feature C to the neck network.

[0134] In step 221, for example Figure 8 The backbone network shown can determine various first image features corresponding to the above construction images.

[0135] In step 222, the multi-level feature fusion module in the neck network can perform feature fusion processing based on the above-mentioned multiple first image features to obtain multiple fused image features.

[0136] Multi-level fusion image features specifically include:

[0137] The highest resolution first-level fused image features are used to better detect small target objects in construction scene images;

[0138] High-resolution second-level fused image features are used to better detect medium-sized target objects in construction scene images;

[0139] The third-level fusion image features with the lowest resolution are used to better detect large target objects in construction scene images.

[0140] Optionally, the aforementioned feature fusion module specifically includes: an upsampling module, a stitching module, and a channel-to-pixel module.

[0141] The upsampling module can double the size of the first image feature.

[0142] The stitching module is used to merge the upsampling results with the first image features output by the backbone network.

[0143] Reference Figure 9 The diagram illustrates the structure of a channel-to-pixel module according to an embodiment of this application. The channel-to-pixel module specifically includes: a twelfth convolution module 901, a segmentation processing module 902, M bottleneck modules 903, an eighth stitching module 904, and a thirteenth convolution module 905. M can be a positive integer greater than 1.

[0144] The twelfth convolution module 901 is used to perform the first convolution process on the input features to obtain the twelfth convolution result.

[0145] The segmentation module 902 is used to segment the twelfth convolution result into two parts with the same number of channels. These two parts may include: a first part of features and a second part of features. Assuming that the number of channels before segmentation is the third number of channels, the number of channels after segmentation can be the fourth number of channels, and the third number of channels can be twice the fourth number of channels.

[0146] After the first part of the features is processed by M bottleneck modules 903, the resulting bottleneck-processed features enter the eighth stitching module 904. The twelfth convolution result and the second part of the features also enter the eighth stitching module 904. The eighth stitching module 904 is used to perform channel stitching operations on the twelfth convolution result, the second part of the features, and the bottleneck-processed features to obtain the stitched features.

[0147] The spliced ​​features are fed into the thirteenth convolutional module 905, which restores the number of channels for the spliced ​​features. For example, the number of channels is restored from the fourth channel to the third channel.

[0148] It should be noted that the first convolutional module, the second convolutional module, and other convolutional modules in the embodiments of this application all fall under the category of convolutional structures. In one example, the above-mentioned convolutional structure specifically includes: at least one convolutional layer, at least one batch normalization layer, and at least one activation function. It is understood that those skilled in the art can adopt the required convolutional structure according to actual application needs, and the embodiments of this application do not limit the specific convolutional structure.

[0149] In the specific implementation, the bottleneck modules include the fourteenth convolution module and the fifteenth convolution module.

[0150] The fourteenth convolution module is used to reduce the number of channels of the input features to half of the original number to obtain the fourteenth convolution result.

[0151] The fifteenth convolutional module doubles the number of channels in the result of the fourteenth convolution to obtain the fifteenth convolutional result. The fifteenth convolutional result has the same number of channels as the input features. The fifteenth convolutional result is then fused with the input features to obtain the output features.

[0152] In this embodiment, a self-attention module can be set after the first-level channel to pixel module to enhance the modeling of nonlinear relationships between different positions in the fused image features and to perform cross-channel information interaction, thereby improving the feature fusion and feature processing capabilities of the neck network and increasing the detection accuracy of target objects of different sizes.

[0153] In the specific implementation, the multi-level attention modules in the neck network perform multi-head self-attention processing on the various fused image features, which specifically includes the following steps:

[0154] Step A1: Perform weighted processing on the fused image features to obtain a query matrix, a key matrix, and a value matrix;

[0155] Step A2: Add the horizontal position weight matrix and the vertical position weight matrix corresponding to the fused image features to obtain the position weight matrix;

[0156] Step A3: Multiply the query matrix and the key matrix to obtain the first product result;

[0157] Step A4: Multiply the query matrix and the position weight matrix to obtain the second product result;

[0158] This application introduces a position weight matrix, which takes into account the spatial location information of image features. Object detection models not only need to identify the object category in the image, but also need to accurately locate the object's position. The position weight matrix helps the object detection model better understand the specific location of the target object in the image, thereby improving the localization accuracy of the bounding box.

[0159] Step A5: Add the first product result and the second product result to obtain the sum result;

[0160] Step A6: Activate the addition result to obtain the activated result;

[0161] Step A7: Multiply the activation result with the numerical matrix to obtain the multi-head self-attention image features.

[0162] Reference Figure 10 The diagram shows a processing flowchart of the attention module according to an embodiment of this application.

[0163] First, using the query weight matrix W q Key weight matrix W kValue weight matrix W v The fusion processing features are weighted to obtain the query matrix, key matrix, and value matrix.

[0164] Then, using position embedding techniques, the lateral position weight matrix Rel of the fused features is... w Vertical position weight matrix Rel h Add them together to obtain the position weight matrix.

[0165] Next, the position weight matrix is ​​multiplied by the query matrix to obtain the content-position matrix, which is called the second product result.

[0166] Then, attention weights are calculated based on the query matrix and key matrix to obtain the first product result.

[0167] Next, the first product result is added to the second product result to obtain the sum.

[0168] Furthermore, activation functions can be used to activate the addition result to obtain the activation result.

[0169] Finally, the activation results can be multiplied by the numerical matrix to obtain the multi-head self-attention image features.

[0170] The calculation process of the query matrix, key matrix, and value matrix is ​​shown in formula (1).

[0171] Query=XW q +b q

[0172] Key = XW k +b k (1)

[0173] Value = XW v +b v

[0174] Where X represents the fused image features, W q b is a randomly generated query weight matrix. q For the bias of the query mapping; W k b is a randomly generated key weight matrix. k For the bias of the key mapping; W v b is a randomly generated value weight matrix. v The bias for the value mapping.

[0175] The process of calculating attention weights based on the query matrix and key matrix is ​​shown in formula (2).

[0176]

[0177] Where D is the vector length of Query, Key, and Value. The Softmax function is a commonly used function in machine learning and deep learning. It is mainly used to compress (map) a K-dimensional vector containing arbitrary real numbers into another K-dimensional real vector, whose element values ​​are between (0,1) and the sum of all elements is 1.

[0178] The overall calculation process of the attention module is shown in formula (3):

[0179] MHSA(X)=Softmax((Rel h +Rel w )Query T +SA(X))×Value(3)

[0180] In a specific implementation, the multi-level feature fusion module may include: a first-level feature fusion module, a second-level feature fusion module, and a third-level feature fusion module; the multi-level attention module may include: a first attention module, a second attention module, and a third attention module.

[0181] The first-level feature fusion module includes: a first upsampling module, a fourth stitching module, a first channel-to-pixel module, a second upsampling module, a fifth stitching module, and a second channel-to-pixel module connected in series; the output of the second channel-to-pixel module is input to the first attention module.

[0182] The second-level feature fusion module includes: a tenth convolutional module, a sixth concatenation module, and a third channel-to-pixel module connected in series; the input of the third channel-to-pixel module is input to the second attention module; the output of the first attention module is input to the tenth convolutional module.

[0183] The third-level feature fusion module includes: an eleventh convolutional module, a seventh stitching module, and a fourth channel-to-pixel module connected in series; the output of the fourth channel-to-pixel module is input to the third attention module; and the output of the second attention module is input to the eleventh convolutional module.

[0184] Reference Figure 11The diagram illustrates the structure of a neck network according to an embodiment of this application. The neck network specifically includes: a first upsampling module 1101, a fourth stitching module 1102, a first channel-to-pixel module 1103, a second upsampling module 1104, a fifth stitching module 1105, a second channel-to-pixel module 1106, a first attention module 1107, a tenth convolution module 1108, a sixth stitching module 1109, a third channel-to-pixel module 1110, a second attention module 1111, an eleventh convolution module 1112, a seventh stitching module 1113, a fourth channel-to-pixel module 1114, and a third attention module 1115, connected in series.

[0185] The fifth stitching module 1105 receives the first image feature A output by the backbone network, the fourth stitching module 1102 receives the first image feature B output by the backbone network, the first upsampling module 1101 and the seventh stitching module 1113 respectively receive the first image feature C output by the backbone network; the first channel to pixel module 1103 is also connected to the sixth stitching module 1109; the first attention module 1107 outputs multi-head self-attention image feature A, the second attention module 1111 outputs multi-head self-attention image feature B, and the third attention module 1115 outputs multi-head self-attention image feature C.

[0186] In step 224, the detection network determines the recognition result based on the aforementioned multi-head self-attention image features.

[0187] The detection network may include at least one detection module. This detection module can be used to perform classification and regression calculations based on various multi-head self-attention image features to obtain the category of the target object in the construction image and the location information of the target object in the construction image.

[0188] In summary, the construction scene object recognition method of this application embodiment sets an attention module in the neck network of the target detection model. This attention module is used to perform multi-head self-attention processing on the above-mentioned multiple fused image features to obtain multiple multi-head self-attention image features. The above-mentioned multi-head self-attention processing can capture the complex dependencies between elements within a sequence. Specifically, in this application embodiment, the sequence is specifically fused image features, and the elements within the sequence can represent image features such as shape, texture, and contextual information of multiple categories of objects. Thus, the multi-head self-attention processing of this application embodiment can effectively capture the nonlinear relationships between image features of multiple categories of objects in the fused image features and perform cross-channel information interaction. The information interaction corresponding to the above-mentioned nonlinear relationships can improve the feature fusion and feature processing capabilities of the neck network. Based on improving the feature fusion and feature processing capabilities of the neck network, the three layers of fused image features with different resolutions output by the neck network can simultaneously improve the detection accuracy of large, medium, and small-sized targets in construction scene images. Therefore, this application embodiment can improve the recognition accuracy of construction scene objects.

[0189] Furthermore, the backbone network of this embodiment specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level depthwise convolutional module, and a spatial pyramid pooling module. The aforementioned multi-level hourglass module and multi-level depthwise convolutional module, through repeated computation, can extract depth feature information from different levels of the construction scene image, capture multi-scale object features, and enhance the backbone network's adaptability to construction scene objects of different sizes. Moreover, the above-described structure of the backbone network in this embodiment combines upsampling and downsampling operations into a single module, which can maintain high-level semantic information and low-level detail information in the deep network. This improves the recognition accuracy of construction scene objects of different sizes, especially improving the recognition accuracy of small-sized construction scene objects.

[0190] Method Example 2

[0191] This application illustrates the training and experimental results of the target detection model.

[0192] In this embodiment of the application, the training process of the target detection model may include forward propagation and backward propagation.

[0193] The forward propagation process calculates the predicted information for the final recognition result sequentially from the backbone network to the detection network, based on the parameters of the target detection model. This predicted information is used to determine the loss information.

[0194] Backpropagation, based on loss information, sequentially calculates and updates the parameters of an object detection model, proceeding from the detection network to the backbone network. Object detection models typically employ a neural network structure, and their parameters can include neural network weights and other parameters. During backpropagation, the gradient information of the object detection model's parameters is determined and used to update these parameters. For example, backpropagation can follow the chain rule in calculus, sequentially calculating and storing the gradient information of the object detection model's parameters from the detection network to the neck network and then to the backbone network.

[0195] In one implementation, the training process of the object detection model includes:

[0196] Step B1: Input the construction image sample into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: the predicted bounding box information corresponding to the target object category; and the ground truth bounding box information corresponding to the construction image sample.

[0197] Step B2: Determine the loss information corresponding to the ground truth bounding box information and the predicted bounding box information; the loss information includes: cross-entropy loss corresponding to the predicted bounding box category, cross-union ratio loss corresponding to the predicted bounding box position, and focus loss corresponding to the predicted bounding box position; the predicted bounding box represents the predicted bounding box.

[0198] Step B3: Update the parameters of the target detection model based on the loss information.

[0199] The process of acquiring the aforementioned construction image samples specifically includes:

[0200] Step C1: Randomly read four original construction images from the construction image set;

[0201] Step C2: Perform random enhancement operations on the four original construction images to obtain four enhanced construction images;

[0202] Step C3: Merge the four enhanced construction images into one merged image;

[0203] Step C4: Label the ground truth bounding boxes on a fused image to obtain a training sample.

[0204] The aforementioned random enhancement operations specifically include at least one of the following operations: flip operation, random scaling operation, random tone transformation operation, and random perspective transformation operation.

[0205] Step C3, which merges four enhanced construction images into a single fused image, specifically includes: placing the four enhanced construction images onto an intermediate image with offsets of [0,0], [0,243], [320,320], and [320,0]; cropping the portion of the intermediate image that exceeds the size range; and shrinking the bounding box to prevent it from going out of bounds, ultimately resulting in a fused image. The size range specifically includes the coordinate range corresponding to [0,0] to [320,320].

[0206] Referring to formula (4), the process of determining the loss information corresponding to the ground truth box information and the predicted box information is shown.

[0207] L total =λ1L BCE +λ2L CIoU +λ3L DFL (4)

[0208] In the formula, L BCE L represents the cross-entropy loss corresponding to the predicted box category; CIoU L represents the crossover ratio loss corresponding to the predicted bounding box location; DFL λ1 represents the focus loss corresponding to the predicted box position; λ2 and λ3 are the weight proportions of the three types of losses, respectively.

[0209] L BCE The loss function calculation process is shown in formula (5):

[0210]

[0211] In the formula, N is the number of construction image samples, and the predicted y i p represents the ground truth class label corresponding to the bounding box of the target object in the i-th construction image sample. i Let be the predicted probability of the category corresponding to the i-th construction image sample.

[0212] L CIoU The loss function calculation process is shown in formula (6):

[0213]

[0214] In the formula, IoU, d, and c are the area intersection-union ratio (IoU) of the ground truth bounding box and the predicted bounding box of the target object, the distance between their center points, and the diagonal distance of their minimum bounding rectangle, respectively. To account for the correction factor of the bounding box shape and orientation, w g w p h g h p These represent the width and height of the ground truth bounding box and the predicted bounding box of the target object, respectively.

[0215] LDFL The loss function calculation process is shown in formula (7):

[0216] L DFL =-((y) i+1 -y)log(S i )+(yy i )log(S i+1 ))(7)

[0217] In the formula, y i y i+1 Let S be the two integers located on either side of the actual coordinate value y of the detection box and closest to y. i S i+1 These are the predicted values ​​of the object detection model for the nearest integer distances from the true coordinates to the two sides.

[0218] The advantages of using the above three loss functions in combination are:

[0219] Comprehensive optimization: Each loss function is optimized for different aspects of the object detection task. Cross-entropy loss focuses on class accuracy, intersection-union loss focuses on location accuracy, and focus loss focuses on learning hard samples.

[0220] Balancing class imbalance: Focus loss helps to handle class imbalance, while cross-entropy loss and IoU loss focus on optimizing class classification and positional accuracy, respectively.

[0221] Improved detection performance: By comprehensively considering category classification, location accuracy, and learning from difficult samples, the model can improve the accuracy and robustness of target detection in various scenarios.

[0222] In summary, using multiple loss functions can enable object detection models to be optimized more comprehensively and in a more balanced way in object detection tasks, thereby improving detection performance.

[0223] To further verify the effectiveness of the target detection model in this application, the following simulation experiments were conducted in this application:

[0224] The embodiments of this application first compare the detection accuracy of existing target detection models and the method proposed in this application on a benchmark dataset for target detection in construction scenarios.

[0225] Specifically, 19,404 pre-annotated training images from the construction scene target detection benchmark dataset are used as the training dataset, and 4,000 validation images are used as the validation dataset. Each annotated image includes the sample category and the x-coordinate of the top-left corner of the ground truth bounding box. min and y min The width and height of the frame.

[0226] Define AP (Average Precision). 50 AP 75 As an evaluation metric, AP (Area Under the Precision-Recall Curve) measures the detection accuracy of an object detection model at different thresholds. A higher AP value indicates higher model detection accuracy. 50 AP is the AP value when the intersection-union ratio (IU) threshold is 0.5, used to determine the degree to which the model balances precision and recall under relaxed conditions; AP 75 The AP value is the intersection-union ratio (IU) threshold of 0.75, used to judge the degree to which the model balances precision and recall under relatively strict conditions, thereby judging whether the object detection model has good localization accuracy.

[0227] During the training of the object detection model, the training set of construction scene images was first scaled to a uniform resolution of 640×640 to ensure consistent input image size. The initial learning rate was set to 0.01, the final learning rate to 0.0001, and the momentum for stochastic gradient descent to 0.937.

[0228] The simulation results of different target detection models on the benchmark dataset for target detection in construction scenarios are shown in Table 1. It can be seen that the target detection method for construction scenarios proposed in this application has better detection accuracy than existing technologies on the benchmark dataset for target detection in construction scenarios.

[0229] Taking the n-parameter scale and AP metric as examples, the detection accuracy of the object detection model in this application is 56.7%, which is 18.4% higher than that of the YOLOv8n model. YOLOv8 is the 8th version of the YOLO (You Only Look Once) object detection model series. The YOLOv8n model is the n-parameter scale of YOLOv8.

[0230] Based on the number of parameters, the parameter scales of object detection models can be divided into:

[0231] n-parameter scale (Nano): The n-version typically refers to a very small model designed to run on devices with very limited computing resources, such as mobile devices or embedded systems. This version of the model usually has the fewest parameters and computational cost, so it is very fast, but may sacrifice detection accuracy.

[0232] s-parameter scaling (Small): The s-version refers to a small model, slightly larger than the n-version, but still relatively lightweight. This version of the model offers a better balance between speed and accuracy, making it suitable for applications where a trade-off between speed and performance is required.

[0233] Medium (m-parameter scale): The m-version refers to a medium-sized model, larger than the s-version, with more parameters and higher computational cost. This version of the model typically provides higher detection accuracy while maintaining relatively fast detection speed, making it suitable for applications that require some performance but still need to consider speed.

[0234] Furthermore, before training convergence, as the training rounds progress, the object detection model in this application shows improvements in precision, recall, and AP. 50 In terms of metrics, the results are significantly better than the YOLOv8n model, achieving higher detection accuracy for thirteen types of target objects in construction scene images, such as dump trucks, backhoe excavators, and concrete mixer trucks, proving the effectiveness of the target detection model in this application.

[0235] Table 1

[0236]

[0237] The backbone network proposed in this application combines the operations of dimensionality upsampling and downsampling into an hourglass module, and captures features at different scales and levels of abstraction. This can effectively reduce the number of parameters and improve the accuracy of target detection.

[0238] Furthermore, the embodiments of this application set an attention module in the neck network, which can model the nonlinear relationship between different positions in the fused image features and perform cross-channel information interaction, better model the shape, texture and context information of the target object, enhance the target detection model's perception and discrimination ability of the target object, and further improve the accuracy of target detection.

[0239] Method Example 3

[0240] In addition to identifying excavators and dump trucks in the construction scene, this embodiment can also monitor the excavation volume in real time, which helps to control the construction progress and allocate resources rationally.

[0241] This application provides a method for real-time monitoring of excavation volume, which specifically includes the following steps:

[0242] Step S1: Collect video stream images of the construction area to be analyzed; the video stream images can be a specific implementation of the construction images.

[0243] Step S2: Perform target detection on the above video stream image to obtain the first target information corresponding to the excavation equipment contained in the above video stream image and the second target information corresponding to the self-unloading equipment contained in the above video stream image.

[0244] Step S3: Based on the first target information and the second target information mentioned above, determine N target regions in the video stream image; N is a positive integer;

[0245] Step S4: For the current target area among the N target areas, determine the excavation operation status of the current target area based on the operating status of the excavating equipment in the current target area, whether the current target area contains dump equipment, and the positional relationship between the excavating equipment and the dump equipment in the current target area; the excavation operation status includes one of the following: unexcavated state, excavating state, and loading state;

[0246] Step S5: When the excavation operation status of the current target area is updated from loading to not excavated or excavating, it is considered that the current loading of the current target area has been completed. Based on the rated loading capacity of the dumping equipment in the current target area, the excavation volume corresponding to the current loading of the current target area is determined, and the cumulative excavation volume of the current target area is updated.

[0247] Step S6: Based on the cumulative excavation volume of all current target areas in the above N target areas, determine the corresponding total excavation volume of the construction area to be counted.

[0248] This application embodiment first acquires video stream images of the construction area to be statistically analyzed; then, target detection is performed on the video stream images to obtain target information corresponding to targets such as excavating equipment and dumping equipment in the video stream images; next, based on the target information, N target areas in the video stream images are determined; then, based on the objective laws between the excavation operation status and the excavating equipment operation status of the target areas, the presence of dumping equipment, and the positions of the excavating equipment and dumping equipment, the excavation volume corresponding to the current loading of the current target area is determined; furthermore, based on the cumulative excavation volume of all current target areas in the above N target areas, the excavation volume corresponding to the video stream image is determined.

[0249] This embodiment of the application determines N target areas in the video stream image based on target information corresponding to targets such as excavating equipment and dumping equipment. It then determines the excavation volume corresponding to the current loading of the current target area based on the objective laws governing the excavation operation status of the target area, the operation status of the excavating equipment, the presence of dumping equipment, and the positions of the excavating equipment and dumping equipment. Thus, this embodiment can continuously determine the target areas present in the video stream image and the corresponding excavation operation status based on the constantly updated video stream image. Furthermore, this embodiment can continuously determine the excavation volume corresponding to the current loading of the current target area in the video stream image, as well as the cumulative excavation volume of all current target areas in the video stream image. Therefore, this embodiment can achieve real-time monitoring of earthwork excavation volume with lower cost and faster speed based on video stream image analysis. In other words, this embodiment can save the manpower costs, data acquisition costs, and data processing time costs associated with using UAV oblique photography or 3D laser scanning technology, and can improve the processing efficiency and real-time performance of earthwork excavation volume.

[0250] In hydropower engineering scenarios, multiple video sources can be set up, each corresponding to a different video stream. These video sources can continuously provide a continuous video stream. This application embodiment can process the continuous video stream provided by any video source to obtain the excavation volume corresponding to that video source. The construction area to be statistically analyzed can be the construction area represented by a specific video source.

[0251] The identification results of this application embodiment specifically include: first target information corresponding to the excavation equipment contained in the video stream image, and second target information corresponding to the self-unloading equipment contained in the video stream image.

[0252] The first and second target information fall under the category of target information. Target information can correspond to category information. For example, the first target information corresponds to the category of excavating equipment, and the second target information corresponds to the category of dump trucks.

[0253] The target information specifically includes the bounding box corresponding to the target. The bounding box is used to represent the position and extent of the target in the video stream image. The bounding box is usually a rectangle whose four sides are aligned with the outermost edge of the target, thus enclosing the entire target. The information of the bounding box may include: the coordinates of the top-left corner of the rectangle, its width, and its height; or, the information of the bounding box may include: the coordinates of the center point of the rectangle, its width, and its height.

[0254] The target region can be a region in the video stream image that contains an excavating device. Based on whether the target region contains a self-unloading device, it can be divided into a first target region and a second target region. The first target region contains one excavating device but not a self-unloading device. The second target region contains both one excavating device and one self-unloading device.

[0255] In one implementation of this application, the first target information includes a first bounding box, and the second target information includes a second bounding box.

[0256] The process of determining N target regions in the video stream image based on the first target information and the second target information mentioned above specifically includes:

[0257] Step D1: Determine the number of target regions N based on the number of first bounding boxes N;

[0258] Step D2: Perform a one-to-one matching between the first bounding box and the second bounding box to obtain the corresponding matching results;

[0259] Step D3: If the first bounding box has a matching second bounding box, then the smallest closure rectangle region that simultaneously contains the first bounding box and the second bounding box is taken as the target region.

[0260] Step D4: If there is no matching second bounding box for the first bounding box, then the rectangular region corresponding to the first bounding box is taken as the target region.

[0261] In step D1, the number of target regions N is equal to the number of first bounding boxes N. In other words, the number of target regions contained in a video stream image is equal to the number of first bounding boxes contained in a video stream image.

[0262] In step D2, a one-to-one matching is performed between the first bounding box and the second bounding box, which can be a distance matching between the first bounding box and the second bounding box.

[0263] In one implementation, step D2, which involves a one-to-one matching process between the first bounding box and the second bounding box, specifically includes: determining the distance intersection-union ratio (DIR) between a first bounding box and a second bounding box; and selecting the second bounding box with the largest DIR from among the multiple second bounding boxes based on the DIR between the first bounding box and multiple second bounding boxes, as the second bounding box that matches the first bounding box.

[0264] Specifically, for the second bounding boxes of the M self-unloading devices in the video stream image, a one-to-one matching is performed with the first bounding boxes in the video stream image based on the Distance Intersection Over Union (DIoU) metric. mn The process of determining is shown in formula (8).

[0265]

[0266] Among them, A m B represents the m-th (m∈[1,M], and is a positive integer) second bounding box. n This represents the first bounding box of the nth element (n∈[1,N], and is a positive integer). m b n These represent the second bounding box A. m First bounding box B n The center point, ρ mn Center point a m b n The Euclidean distance between them is d, where d is the diagonal distance of the smallest closure region that simultaneously contains the first and second bounding boxes.

[0267] The Intersection over Union (IoU) ratio reflects the degree of overlap between the first and second bounding boxes and the distance between their center points. Generally, a higher IoU value indicates a greater degree of overlap and a closer proximity between their center points.

[0268] For a first bounding box, there are M distance intersection-union ratio (DIR) values ​​between it and M second bounding boxes. Then, the second bounding box with the largest DIR value can be selected from the M DIR values ​​and can be used as the second bounding box that matches the first bounding box.

[0269] Steps D3 and D4 can be executed in parallel.

[0270] In practical applications, the N first bounding boxes in the video stream image can be traversed according to a first preset order to determine the target region corresponding to each first bounding box. The first preset order can be: from left to right and from bottom to top, or from left to right and from top to bottom.

[0271] In this embodiment, the N first bounding boxes in the video stream image can also be numbered according to a first preset order. The numbers of the N first bounding boxes can be the same as the numbers of the N target regions.

[0272] During the traversal of the N first bounding boxes in the video stream image, it can be determined whether there is a second bounding box that matches a first bounding box. If so, step A3 can be executed, and the smallest closure rectangle region that simultaneously contains the first and second bounding boxes can be taken as the target region. If not, step A4 can be executed, and the rectangle region corresponding to the first bounding box can be taken as the target region.

[0273] After determining the target area based on steps D3 and D4, information such as the target area number, the position information of the rectangle corresponding to the target area, and whether it contains a second bounding box can be obtained.

[0274] In step S4, the excavation status of the current target area among the N target areas can be determined. The current target area can represent one of the N target areas. It is understood that the embodiments of this application can execute the processing of multiple current target areas among the N target areas in parallel or sequentially. The processing of the current target area can include the processing in steps S4 and S5. The process of sequentially executing the processing of multiple current target areas among the N target areas specifically includes: traversing the N target areas according to a second preset order of target area numbers from smallest to largest or from largest to smallest to obtain one current target area among the N target areas, and sequentially executing the processing of multiple current target areas among the N target areas according to the second preset order.

[0275] The excavation operation status in this application specifically includes one of the following: unexcavated status, excavating status, and loading status; wherein, the unexcavated status means that no excavation work is being carried out in the current target area; the excavating status means that excavation work is being carried out in the current target area; and the loading status means that loading work is being carried out in the current target area.

[0276] The embodiments of this application can determine the operating status of the current target area based on the operating status of the excavating equipment in the current target area, whether the current target area contains dump equipment, and the positional relationship between the excavating equipment and the dump equipment in the current target area.

[0277] The method for determining the current excavation status of the target area may specifically include the following steps:

[0278] Step E1: Determine whether the excavating equipment in the current target area is stationary to obtain the first determination result;

[0279] If the first judgment result is yes, then execute step E2: determine the current excavation operation status of the target area as an unexcavated state;

[0280] If the first judgment result is negative, then proceed to step E3: determine whether the current target area contains a self-unloading device, in order to obtain the second judgment result;

[0281] If the second judgment result is negative, then execute step E4: determine the current excavation operation status of the target area as excavation in progress;

[0282] If the second judgment result is yes, then execute step E5: determine whether there is an overlap between the dumping equipment and the excavating equipment in the current target area, so as to obtain the third judgment result;

[0283] If the third judgment result is yes, then execute step E6: determine the current excavation operation status of the target area as loading status;

[0284] If the third judgment result is negative, then proceed to step E7: determine the current excavation operation status of the target area as excavation in progress.

[0285] This application provides the following rules for determining the excavation operation status:

[0286] The rule for determining the unexcavated state is: the excavation equipment is in a stationary state.

[0287] The rules for determining the state during excavation include: the excavator is in a non-stationary state and the current target area does not contain dump equipment; or, the current target area contains dump equipment and there is no overlap between the dump equipment and the excavator.

[0288] The rules for determining the loading status include: the excavating equipment is in a non-stationary state, the current target area contains a dumping equipment, and there is an overlap between the dumping equipment and the excavating equipment.

[0289] Step E1, determining whether the excavating equipment within the current target area is stationary, specifically includes:

[0290] In the video stream from frame i to frame i+α, if the difference in coordinates of the excavating device between any two frames is less than the second threshold, then the excavating device contained in the current target area is determined to be in a stationary state; where i and α are positive integers, and α is greater than i.

[0291] In this embodiment, two video stream images to be compared can be selected from the α-frame video stream images corresponding to the i-th frame to the (i+α)-th frame video stream images, and it can be determined whether the coordinate difference between the two video stream images to be compared is less than a second threshold.

[0292] Two video stream images to be compared are selected from the α-frame video stream images. There can be P selection schemes. If the coordinate difference corresponding to the P selection schemes is less than the second threshold, it can be determined that the excavation equipment contained in the current target area is in a stationary state.

[0293] Considering the effects of signal transmission and camera shake, the coordinates of the first bounding box of the excavating equipment may be slightly offset when it is stationary. Assume the top-left pixel coordinates of the first bounding box of the excavating equipment in the i-th frame of the current target area video stream are... The pixel coordinates of the bottom right corner are If the second threshold corresponding to the coordinate offset is set to δ and the frame number threshold is set to α, then if formula (9) holds, that is, if the coordinate difference between any two frames in the i-th frame video stream image to the i+α-th frame video stream image is less than the second threshold δ, then it can be determined that the excavation equipment is in a stationary state.

[0294] Formula (9) calculates and determines the coordinate difference between the top-left pixel coordinates of the i-th and (i+j)-th video stream frames. Therefore, This represents the x-coordinate of the top-left pixel of the (i+j)th frame of the video stream. This represents the ordinate of the top-left pixel of the (i+j)th frame of the video stream.

[0295]

[0296] It is understood that formula (9) calculates and judges the coordinate difference between the upper left corner pixel coordinates of the i-th frame video stream image and the (i+j)-th frame video stream image, but this is only an optional embodiment. In fact, the coordinate difference can be calculated and judged between the upper right corner pixel coordinates, lower left corner pixel coordinates, or lower right corner pixel coordinates of the i-th frame video stream image and the (i+j)-th frame video stream image.

[0297] Since step S3 obtains the target area number, the position information of the corresponding rectangle, and whether it contains a second bounding box after determining the target area, step E3 can determine whether the current target area contains a self-unloading device based on whether the current target area contains the information corresponding to the second bounding box. Alternatively, target detection technology can be used to perform target detection on the image corresponding to the current target area to determine whether the current target area contains a self-unloading device.

[0298] Step E5 determines whether there is overlap between the dump truck and the excavator in the current target area, specifically including:

[0299] Step F1: Determine the intersection-union ratio (IUR) index between the second bounding box corresponding to the dumping equipment in the current target area and the first bounding box corresponding to the excavating equipment in the current target area;

[0300] Step F2: If the intersection-to-union ratio index value is greater than the first threshold, it is determined that there is an overlap between the dumping equipment and the excavating equipment in the current target area.

[0301] Formula (10) shows the calculation process of the intersection-union ratio index between the second bounding box and the first bounding box.

[0302]

[0303] Where A represents the second bounding box, B represents the first bounding box, A∩B represents the intersection area between the second bounding box and the first bounding box, and A∪B represents the union area between the second bounding box and the first bounding box.

[0304] In practical applications, when the intersection-union ratio (IUGR) is greater than 0, it can be considered that there is overlap between dump trucks and excavators in the current target area. Therefore, examples of the first threshold can include 0 or a real number greater than 0.

[0305] In step S5, the excavation status of the current target area can be monitored. If the excavation status changes from loading to not excavated or excavating, the current target area can be considered to have completed the current loading. Based on the rated loading capacity of the dump truck in the current target area, the excavation volume corresponding to the current loading is determined, and the cumulative excavation volume of the current target area is updated. The cumulative excavation volume of the current target area can be the sum of the excavation volumes corresponding to multiple loadings in the current target area.

[0306] The process of determining the rated loading capacity of the self-unloading equipment in the current target area specifically includes:

[0307] Step G1: Use the self-unloading equipment model classification model to determine the target model of the self-unloading equipment in the current target area;

[0308] Step G2: Based on the target model, search for the mapping relationship between the model and the rated loading capacity to determine the rated loading capacity corresponding to the self-unloading equipment in the current target area.

[0309] The self-unloading equipment model classification model in this application embodiment can have the ability to classify self-unloading equipment models. In other words, it can determine the target model of the self-unloading equipment in the regional image based on the regional image corresponding to the current target area.

[0310] In a specific implementation, the embodiments of this application can label the model categories of the self-unloading equipment image dataset, and train a self-unloading equipment model classification model based on the self-unloading equipment image dataset.

[0311] This document describes a process for acquiring a dataset of images of self-unloading equipment. Specifically, it involves collecting images of self-unloading equipment under various construction backgrounds, lighting conditions, and operating postures, based on a video surveillance system. Furthermore, it is necessary to acquire combined images of self-unloading equipment and excavating equipment working together in excavation operations to demonstrate the actual situation of mutual occlusion between the two types of construction machinery. Therefore, the self-unloading equipment image dataset can include both self-unloading equipment images and combined images.

[0312] This application does not limit the specific structure of the self-unloading equipment model classification model. For example, the self-unloading equipment model classification model may include a convolutional neural network structure such as VGG (Visual Geometry Group).

[0313] This application embodiment can pre-save the mapping relationship between model and rated loading capacity. In this way, the rated loading capacity of the self-unloading equipment in the current target area can be obtained by searching in the above mapping relationship according to the target model of the self-unloading equipment in the current target area.

[0314] In this embodiment of the application, when the excavation status of the current target area is updated from loading to not excavated or excavating, it can be considered that the current loading of the current target area has been completed. Based on the rated loading capacity of the dump truck in the current target area, the excavation volume corresponding to the current loading of the current target area is determined. The excavation volume corresponding to the current loading can be equal to the rated loading capacity of the dump truck in the current target area.

[0315] In step S6, the cumulative excavation volume of all current target areas in N target areas can be summed to obtain the total real-time excavation volume corresponding to the construction area to be counted.

[0316] In one example of this application, the total real-time excavation volume corresponding to the construction area to be counted can be set to 0 at the beginning of a natural day. Subsequently, the total real-time excavation volume can be updated based on the completion of the current loading of the current target area to obtain the continuously accumulated total real-time excavation volume. Therefore, the real-time monitoring process of excavation volume can be a process of continuously updating the total real-time excavation volume based on the completion of the current loading of N current target areas.

[0317] In practical applications, if there are multiple construction areas to be statistically analyzed in a hydropower project, the total real-time excavation volume corresponding to each construction area can be monitored separately on a daily basis. After a daily period ends, the total real-time excavation volumes corresponding to multiple construction areas can be merged. It is understood that the embodiments of this application do not limit the specific merging method.

[0318] In summary, the real-time excavation volume monitoring method of this application embodiment determines N target areas in the video stream image based on target information corresponding to targets such as excavating equipment and dumping equipment. Furthermore, it determines the excavation volume corresponding to the current loading of the current target area based on the objective laws governing the excavation operation status of the target area, the operation status of the excavating equipment, the presence of dumping equipment, and the positions of the excavating equipment and dumping equipment. Thus, this application embodiment can continuously determine the target areas present in the video stream image and the corresponding operation status of the target areas based on the constantly updated video stream image. On this basis, this application embodiment can continuously determine the excavation volume corresponding to the current loading of the current target area in the video stream image, as well as the cumulative excavation volume of all current target areas in the video stream image. Therefore, this application embodiment can achieve real-time monitoring of earthwork excavation volume with lower cost and faster speed based on the analysis of video stream images.

[0319] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0320] Based on the above embodiments, this embodiment also provides a construction scene object recognition device, referring to... Figure 12 The device may specifically include: a data acquisition module 1201 and an object recognition module 1202.

[0321] Acquisition module 1201 is used to acquire construction images;

[0322] The object recognition module 1202 is used to recognize the above construction image according to the target detection model and obtain the recognition result; the recognition result includes: the category of the target object and the location information of the target object in the above construction image;

[0323] The aforementioned target detection model specifically includes: a backbone network, a neck network, and a detection network connected in sequence; the backbone network specifically includes: a convolutional processing module, a multi-level hourglass module, a multi-level deep convolutional module, and a spatial pyramid pooling module; the neck network specifically includes: a multi-level feature fusion module and a multi-level attention module.

[0324] The object recognition module 1202 specifically includes:

[0325] Feature extraction module 1221 is used to determine multiple first image features corresponding to the above construction images using the backbone network;

[0326] The feature fusion module 1222 is used to perform feature fusion processing based on the above-mentioned multiple first image features using the multi-level feature fusion module in the neck network to obtain multiple fused image features;

[0327] Attention processing module 1223 is used to perform multi-head self-attention processing on the above-mentioned multiple fused image features using the multi-level attention module in the neck network, so as to obtain multiple multi-head self-attention image features;

[0328] The recognition result determination module 1224 is used to determine the recognition result based on the above-mentioned multiple multi-head self-attention image features using the detection network.

[0329] Optionally, the multi-level attention modules in the aforementioned neck network perform multi-head self-attention processing on the various fused image features, including:

[0330] The fused image features are weighted to obtain the query matrix, key matrix, and value matrix;

[0331] The horizontal and vertical position weight matrices corresponding to the above fused image features are added together to obtain the position weight matrix;

[0332] Multiply the query matrix and the key matrix to obtain the first product result;

[0333] The query matrix and the position weight matrix are multiplied to obtain the second product result;

[0334] Add the first product result to the second product result to obtain the sum;

[0335] The addition result is activated to obtain the activated result;

[0336] The activation result is multiplied by the value matrix to obtain the multi-head self-attention image features.

[0337] Optionally, the backbone network includes: an n+1 level hourglass module and an n-level depthwise convolutional module; the first-level hourglass module includes: at least one hourglass module; the first-level depthwise convolutional module includes: at least one depthwise convolutional module.

[0338] Here, the i-th level hourglass module is connected in front of the i-th level depthwise convolutional module; n is a positive integer greater than 1, and i is a positive integer not greater than n;

[0339] The (n+1)th level hourglass module is connected after the nth level depthwise convolutional module and before the spatial pyramid pooling module.

[0340] Optionally, the second-level hourglass module is connected to the aforementioned neck network and outputs the first image feature A to the aforementioned neck network; the third-level hourglass module is connected to the aforementioned neck network and outputs the first image feature B to the aforementioned neck network; the spatial pyramid pooling module is connected to the aforementioned neck network and outputs the first image feature C to the aforementioned neck network.

[0341] Optionally, the hourglass module mentioned above includes: a first lightweight convolution module, a second lightweight convolution module, a first splicing module, a third convolution module, and a fourth convolution module;

[0342] The first lightweight convolutional module and the second lightweight convolutional model mentioned above correspond to the same network structure; the first lightweight convolutional module includes: a first convolutional module and a fourth depthwise separable convolutional module; the second lightweight convolutional module includes: a second convolutional module and a fifth depthwise separable convolutional module.

[0343] The output of the first lightweight convolution module is input to the second lightweight convolution module and the first stitching module, respectively; the output of the second lightweight convolution module is input to the first stitching module; the output of the first stitching module is input to the third convolution module; and the output of the third convolution module is input to the fourth convolution module.

[0344] Optionally, the aforementioned spatial pyramid pooling module includes: convolution module A, p first max pooling modules, a second concatenation module, and convolution module B; where p is a positive integer greater than 1.

[0345] Optionally, the backbone network mentioned above also includes a convolution processing module connected before the first-level hourglass module;

[0346] The above convolution processing modules include: the fifth convolution module, the sixth convolution module, the seventh convolution module, the second max pooling module, the third concatenation module, the eighth convolution module, and the ninth convolution module;

[0347] The fifth convolutional module receives input features and provides outputs to the sixth convolutional module and the second max pooling module, respectively; the output of the sixth convolutional module is input to the seventh convolutional module; the third concatenation module receives the outputs of the second max pooling module and the seventh convolutional module, respectively, and provides outputs to the eighth convolutional module; the ninth convolutional module is connected to the eighth convolutional module.

[0348] Optionally, the multi-level feature fusion module includes: a first-level feature fusion module, a second-level feature fusion module, and a third-level feature fusion module; the multi-level attention module includes: a first attention module, a second attention module, and a third attention module;

[0349] The first-level feature fusion module includes: a first upsampling module, a fourth stitching module, a first channel-to-pixel module, a second upsampling module, a fifth stitching module, and a second channel-to-pixel module connected in series; the output of the second channel-to-pixel module is input to the first attention module.

[0350] The second-level feature fusion module includes: a tenth convolutional module, a sixth concatenation module, and a third channel-to-pixel module connected in series; the input of the third channel-to-pixel module is input to the second attention module; the output of the first attention module is input to the tenth convolutional module.

[0351] The third-level feature fusion module includes: an eleventh convolutional module, a seventh concatenation module, and a fourth channel-to-pixel module connected in series; the output of the fourth channel-to-pixel module is input to the third attention module; the output of the second attention module is input to the eleventh convolutional module.

[0352] The fifth stitching module receives the first image feature A output by the backbone network, the fourth stitching module receives the first image feature B output by the backbone network, and the first upsampling module and the seventh stitching module respectively receive the first image feature C output by the backbone network; the first channel to pixel module is also connected to the sixth stitching module; the first attention module outputs multi-head self-attention image feature A, the second attention module outputs multi-head self-attention image feature B, and the third attention module outputs multi-head self-attention image feature C.

[0353] Optionally, the training process of the above object detection model includes:

[0354] The construction image sample is input into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: the predicted bounding box information corresponding to the target object category; and the ground truth bounding box information corresponding to the construction image sample.

[0355] Determine the loss information corresponding to the above ground truth bounding box information and the above predicted bounding box information; the above loss information includes: cross-entropy loss corresponding to the predicted bounding box category, cross-union ratio loss corresponding to the predicted bounding box position, and focus loss corresponding to the predicted bounding box position;

[0356] Based on the aforementioned loss information, the parameters of the target detection model are updated.

[0357] Optionally, the process of acquiring the above-mentioned construction image samples includes:

[0358] Four original construction images are randomly selected from the collection of construction images;

[0359] Random enhancement operations were performed on the four original construction images to obtain four enhanced construction images;

[0360] Four enhanced construction images were merged into one merged image;

[0361] Annotate the ground truth bounding boxes on a fused image to obtain a training sample.

[0362] In summary, the construction scene object recognition device of this application embodiment incorporates an attention module in the neck network of the target detection model. This attention module performs multi-head self-attention processing on the aforementioned multiple fused image features to obtain various multi-head self-attention image features. Specifically, the multi-head self-attention processing can capture complex dependencies between elements within a sequence. In this application embodiment, the sequence specifically refers to fused image features, and the elements within the sequence can represent image features such as shape, texture, and contextual information of multiple object categories. Thus, the multi-head self-attention processing of this application embodiment can effectively capture the nonlinear relationships between image features of multiple object categories in the fused image features and perform cross-channel information interaction. The information interaction corresponding to the aforementioned nonlinear relationships can improve the feature fusion and feature processing capabilities of the neck network. Based on improving the feature fusion and feature processing capabilities of the neck network, the three layers of fused image features with different resolutions output by the neck network can simultaneously improve the detection accuracy of large, medium, and small-sized targets in construction scene images. Therefore, this application embodiment can improve the recognition accuracy of construction scene objects.

[0363] Furthermore, the backbone network of this embodiment specifically includes: a convolution processing module, a multi-level hourglass module, a multi-level depthwise convolutional module, and a spatial pyramid pooling module. The aforementioned multi-level hourglass module and multi-level depthwise convolutional module, through repeated computation, can extract depth feature information from different levels of the construction scene image, capturing multi-scale object features and enhancing the backbone network's adaptability to construction scene objects of different sizes. Moreover, the backbone network of this embodiment combines upsampling and downsampling operations into a single module, preserving high-level semantic information and low-level detail information in the deep network. This improves the recognition accuracy of construction scene objects of different sizes, particularly small-sized construction scene objects.

[0364] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0365] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0366] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices and servers (clusters). Figure 13 An exemplary apparatus 1300 is schematically shown that can be used to implement the various embodiments described in this application.

[0367] In one embodiment, Figure 13 An exemplary device 1300 is shown, which includes one or more processors 1302, a control module (chipset) 1304 coupled to at least one of the processors 1302, a memory 1306 coupled to the control module 1304, an NVM (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0368] Processor 1302 may include one or more single-core or multi-core processors, and processor 1302 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1300 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0369] In some embodiments, apparatus 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314 and one or more processors 1302 that are combined with the one or more computer-readable media and configured to execute instructions 1314 to implement modules and thereby perform the actions described in this disclosure.

[0370] In one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1302 and / or any suitable device or component communicating with the control module 1304.

[0371] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0372] Memory 1306 may be used, for example, to load and store data and / or instructions 1314 for device 1300. In one embodiment, memory 1306 may include any suitable volatile memory, such as suitable DRAM (Dynamic Random Access Memory). In some embodiments, memory 1306 may include double data rate type quad synchronous dynamic random access memory.

[0373] In one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1308 and (one or more) input / output devices 1310.

[0374] For example, non-volatile memory / storage device 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage device 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital universal optical disk drives).

[0375] The non-volatile memory / storage device 1308 may include storage resources that are physically part of a device on which the device 1300 is mounted, or that can be accessed by the device without being part of the device. For example, the non-volatile memory / storage device 1308 may be accessed via a network via one or more input / output devices 1310.

[0376] One or more input / output devices 1310 may provide an interface for device 1300 to communicate with any other suitable device. Input / output devices 1310 may include communication components, audio components, sensor components, etc. Network interface 1312 may provide an interface for device 1300 to communicate via one or more networks. Device 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi (Wireless Fidelity), 2G (2-Generation wireless telephone technology), 3G (3-Generation wireless telephone technology), 4G (4-Generation wireless telephone technology), 5G (5-Generation wireless telephone technology), etc., or combinations thereof.

[0377] In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1304. In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers of the control module 1304 to form a system-in-package. In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die. In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die to form a system-on-a-chip.

[0378] In various embodiments, device 1300 may be, but is not limited to, a terminal device such as a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, touchscreen device, netbook, etc.). In various embodiments, device 1300 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0379] The detection device may use a main control chip as a processor or control module, and sensor data, position information, etc. may be stored in a memory or non-volatile memory / storage device. The sensor group may be used as an input / output device, and the communication interface may include a network interface.

[0380] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0381] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0382] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0383] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0384] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0385] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0386] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0387] The above provides a detailed description of the construction scene object recognition method and apparatus, an electronic device, and a machine-readable medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application sets based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for object recognition in a construction scene, characterized in that, The method includes: Acquire construction images; The construction image is identified based on the target detection model to obtain the identification result; the identification result includes: the category of the target object and the location information of the target object in the construction image; The target detection model comprises a backbone network, a neck network, and a detection network connected in sequence; the backbone network comprises a convolutional processing module, a multi-level hourglass module, a multi-level deep convolutional module, and a spatial pyramid pooling module; the neck network comprises a multi-level feature fusion module and a multi-level attention module. The step of identifying the construction image based on the target detection model includes: The backbone network determines multiple first image features corresponding to the construction images; The multi-level feature fusion module in the neck network performs feature fusion processing based on the multiple first image features to obtain multiple fused image features; The multi-level attention modules in the neck network perform multi-head self-attention processing on the various fused image features to obtain various multi-head self-attention image features; The detection network determines the recognition result based on the various multi-head self-attention image features; The multi-level attention module in the neck network performs multi-head self-attention processing on the various fused image features, including: weighting the fused image features to obtain a query matrix, a key matrix, and a value matrix; adding the horizontal and vertical position weight matrices corresponding to the fused image features to obtain a position weight matrix; multiplying the query matrix and the key matrix to obtain a first product result; multiplying the query matrix and the position weight matrix to obtain a second product result; adding the first and second product results to obtain a sum result; performing activation processing on the sum result to obtain an activation result; and multiplying the activation result with the value matrix to obtain the multi-head self-attention image features. The multi-level feature fusion module includes: a first-level feature fusion module, a second-level feature fusion module, and a third-level feature fusion module; the multi-level attention module includes: a first attention module, a second attention module, and a third attention module; The first-level feature fusion module includes: a first upsampling module, a fourth stitching module, a first channel-to-pixel module, a second upsampling module, a fifth stitching module, and a second channel-to-pixel module connected in series; the output of the second channel-to-pixel module is input to the first attention module. The second-level feature fusion module includes: a tenth convolutional module, a sixth concatenation module, and a third channel-to-pixel module connected in series; the output of the third channel-to-pixel module is input to the second attention module; the output of the first attention module is input to the tenth convolutional module. The third-level feature fusion module includes: an eleventh convolutional module, a seventh concatenation module, and a fourth channel-to-pixel module connected in series; the output of the fourth channel-to-pixel module is input to the third attention module; the output of the second attention module is input to the eleventh convolutional module. The fifth stitching module receives the first image feature A output by the backbone network, the fourth stitching module receives the first image feature B output by the backbone network, and the first upsampling module and the seventh stitching module respectively receive the first image feature C output by the backbone network; the first channel to pixel module is also connected to the sixth stitching module; the first attention module outputs multi-head self-attention image feature A, the second attention module outputs multi-head self-attention image feature B, and the third attention module outputs multi-head self-attention image feature C.

2. The method according to claim 1, characterized in that, The backbone network includes: an n+1 level hourglass module and an n-level depthwise convolutional module; the first-level hourglass module includes: at least one hourglass module; the first-level depthwise convolutional module includes: at least one depthwise convolutional module; Here, the i-th level hourglass module is connected in front of the i-th level depthwise convolutional module; n is a positive integer greater than 1, and i is a positive integer not greater than n; The (n+1)th level hourglass module is connected after the nth level depthwise convolutional module and before the spatial pyramid pooling module.

3. The method according to claim 2, characterized in that, The second-level hourglass module is connected to the neck network and outputs the first image feature A to the neck network; the third-level hourglass module is connected to the neck network and outputs the first image feature B to the neck network; the spatial pyramid pooling module is connected to the neck network and outputs the first image feature C to the neck network.

4. The method according to any one of claims 1 to 3, characterized in that, The hourglass module includes: a first lightweight convolution module, a second lightweight convolution module, a first splicing module, a third convolution module, and a fourth convolution module; The first lightweight convolutional module and the second lightweight convolutional module correspond to the same network structure; the first lightweight convolutional module includes: a first convolutional module and a fourth depthwise separable convolutional module; the second lightweight convolutional module includes: a second convolutional module and a fifth depthwise separable convolutional module. The output of the first lightweight convolution module is input to the second lightweight convolution module and the first stitching module, respectively; the output of the second lightweight convolution module is input to the first stitching module; the output of the first stitching module is input to the third convolution module; and the output of the third convolution module is input to the fourth convolution module.

5. The method according to any one of claims 1 to 3, characterized in that, The spatial pyramid pooling module includes: convolution module A, p first max pooling modules, a second concatenation module, and convolution module B; p is a positive integer greater than 1.

6. The method according to any one of claims 1 to 3, characterized in that, The convolution processing module includes: a fifth convolution module, a sixth convolution module, a seventh convolution module, a second max pooling module, a third concatenation module, an eighth convolution module, and a ninth convolution module; The fifth convolutional module receives input features and provides outputs to the sixth convolutional module and the second max pooling module, respectively; the output of the sixth convolutional module is input to the seventh convolutional module; the third concatenation module receives the outputs of the second max pooling module and the seventh convolutional module, respectively, and provides outputs to the eighth convolutional module; the ninth convolutional module is connected to the eighth convolutional module.

7. The method according to any one of claims 1 to 3, characterized in that, The training process of the target detection model includes: A construction image sample is input into a target detection model, which outputs a prediction result corresponding to the construction image sample. The prediction result includes: predicted bounding box information corresponding to the target object category; and ground truth bounding box information corresponding to the construction image sample. Determine the loss information corresponding to the ground truth bounding box information and the predicted bounding box information; the loss information includes: cross-entropy loss corresponding to the predicted bounding box category, cross-union ratio loss corresponding to the predicted bounding box position, and focus loss corresponding to the predicted bounding box position; The parameters of the target detection model are updated based on the loss information.

8. The method according to claim 7, characterized in that, The process of acquiring the construction image samples includes: Four original construction images are randomly selected from the collection of construction images; Random enhancement operations were performed on the four original construction images to obtain four enhanced construction images; Four enhanced construction images were merged into one merged image; Annotate the ground truth bounding boxes on a fused image to obtain a training sample.

9. A construction scene object recognition device, characterized in that, The device includes: The acquisition module is used to acquire construction images; The object recognition module is used to recognize the construction image according to the target detection model and obtain the recognition result; the recognition result includes: the category of the target object and the location information of the target object in the construction image; The target detection model comprises a backbone network, a neck network, and a detection network connected in sequence; the backbone network comprises a convolutional processing module, a multi-level hourglass module, a multi-level deep convolutional module, and a spatial pyramid pooling module; the neck network comprises a multi-level feature fusion module and a multi-level attention module. The object recognition module includes: The feature extraction module is used to determine multiple first image features corresponding to the construction image using the backbone network; The feature fusion module is used to perform feature fusion processing based on the multiple first image features using the multi-level feature fusion module in the neck network to obtain multiple fused image features; An attention processing module is used to perform multi-head self-attention processing on the various fused image features using multi-level attention modules in the neck network, so as to obtain various multi-head self-attention image features. The recognition result determination module is used to determine the recognition result based on the various multi-head self-attention image features using the detection network; The multi-level attention module in the neck network performs multi-head self-attention processing on the various fused image features, including: weighting the fused image features to obtain a query matrix, a key matrix, and a value matrix; adding the horizontal and vertical position weight matrices corresponding to the fused image features to obtain a position weight matrix; multiplying the query matrix and the key matrix to obtain a first product result; multiplying the query matrix and the position weight matrix to obtain a second product result; adding the first and second product results to obtain a sum result; performing activation processing on the sum result to obtain an activation result; and multiplying the activation result with the value matrix to obtain the multi-head self-attention image features. The multi-level feature fusion module includes: a first-level feature fusion module, a second-level feature fusion module, and a third-level feature fusion module; the multi-level attention module includes: a first attention module, a second attention module, and a third attention module; The first-level feature fusion module includes: a first upsampling module, a fourth stitching module, a first channel-to-pixel module, a second upsampling module, a fifth stitching module, and a second channel-to-pixel module connected in series; the output of the second channel-to-pixel module is input to the first attention module. The second-level feature fusion module includes: a tenth convolutional module, a sixth concatenation module, and a third channel-to-pixel module connected in series; the output of the third channel-to-pixel module is input to the second attention module; the output of the first attention module is input to the tenth convolutional module. The third-level feature fusion module includes: an eleventh convolutional module, a seventh concatenation module, and a fourth channel-to-pixel module connected in series; the output of the fourth channel-to-pixel module is input to the third attention module; the output of the second attention module is input to the eleventh convolutional module. The fifth stitching module receives the first image feature A output by the backbone network, the fourth stitching module receives the first image feature B output by the backbone network, and the first upsampling module and the seventh stitching module respectively receive the first image feature C output by the backbone network; the first channel to pixel module is also connected to the sixth stitching module; the first attention module outputs multi-head self-attention image feature A, the second attention module outputs multi-head self-attention image feature B, and the third attention module outputs multi-head self-attention image feature C.

10. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as described in any one of claims 1-8.

11. A machine-readable medium having executable code stored thereon, which, when executed, causes a processor to perform the method as claimed in any one of claims 1-8.

Images