Method, apparatus and product for identifying object in construction site based on dual-backbone fusion

Abstract

To provide an object identification method capable of monitoring an abnormal operation of a construction image in real time, saving labor costs required for safety monitoring at a construction site, improving an accuracy rate of the safety monitoring at the construction site, and reducing an occurrence rate of safety accidents.SOLUTION: The method includes: collecting a construction image; recognizing the construction image based on a target detection model to obtain a recognition result, the recognition result including a category of a construction object and location information of the construction object; and determining whether the construction image is associated with an abnormal action based on the recognition result. The ith feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the ith type of first image feature, and outputs the jth type of sub-image feature to the jth feature fusion module connected to the ith feature segmentation module.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The embodiments of the present application relate to the technical field of image identification, and in particular to a method, apparatus and product for identifying objects in construction sites based on dual backbone fusion. [Background technology]

[0002] With the continuous development of society, the engineering industry, including hydropower generation and civil engineering and construction, is developing rapidly. However, in recent years, there is a risk of safety accidents occurring during construction due to factors such as people and vehicles getting too close.

[0003] In order to reduce safety accidents, conventional monitoring methods usually use a personnel patrol method, specifically, by assigning specialized patrol inspectors to the construction site, who periodically patrol and inspect the construction site to monitor the distance between the construction workers and the construction machinery, and issue an alarm if the distance is shorter than the safe distance.

[0004] The traditional monitoring method requires a certain amount of labor costs, and the patrol inspection personnel cannot inspect the construction site in real time, so the traditional monitoring method still has certain safety risks and the accuracy rate of safety monitoring is relatively low. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application provide a method for identifying objects at construction sites based on dual backbone fusion, which can reduce the labor costs required for safety monitoring at construction sites. The real-time monitoring of the embodiments of the present application can improve the accuracy of safety monitoring at construction sites and further reduce the occurrence rate of safety accidents.

[0006] Correspondingly, the embodiments of the present application further provide an object identification device, an electronic device, and a machine-readable medium in a construction site based on dual backbone fusion, thereby ensuring the realization and application of the above method. [Means for solving the problem]

[0007] In order to solve the above problem, an embodiment of the present application is a method for identifying objects in a construction site based on dual backbone fusion, the method comprising: Collecting construction images; A step of identifying the construction image based on a target detection model to obtain an identification result, wherein the identification result includes a construction object category and position information of the construction object category in the construction image, and the construction object category includes at least one of a human category, a construction machine category, a safety protector wearing category, and a safety sign category; and determining whether the construction image is related to an abnormal operation based on the identification result; The target detection model includes a first backbone network, a feature division network, a second backbone network, a neck network, and a detection head network, wherein the feature division network includes n feature division modules, and the second backbone network includes n feature fusion modules, and a channel-to-pixel module is connected after at least some of the feature fusion modules, where n is a positive integer greater than 1; The step of identifying the construction image based on a target detection model includes: The first backbone network determines n first image features corresponding to the construction image; the i-th feature division module in the feature division network performs convolution processing and division processing on the i-th first image feature, the obtained division processing result includes i-th sub-image feature, and outputs the j-th sub-image feature to the connected j-th feature fusion module, where i and j are positive integers, i is less than or equal to n, j is less than or equal to i, and when i is greater than 1, the i-th sub-image feature output by the same i-th feature division module is a sub-image feature with a different number of channels, and one feature fusion module corresponds to one number of channels, and different feature fusion modules correspond to different numbers of channels; the i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result; the channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result, and outputs the second image feature to the backbone network; the neck network performs a second fusion process on the second image features to obtain a second fusion process result; The detection head network discloses a method for identifying objects in a construction site based on dual backbone fusion, including a step of determining the identification result based on the second fusion processing result.

[0008] An embodiment of the present application is an object identification device, the device comprising: a collection module for collecting construction images; an object identification module for identifying the construction image based on a target detection model to obtain an identification result, the identification result including a construction object category and position information of the construction object category in the construction image, the construction object category including at least one of a human category, a construction machine category, a safety protector wearing category, and a safety sign category; an abnormality determination module for determining whether the construction image is related to an abnormal operation based on the identification result; The target detection model includes a first backbone network, a feature division network, a second backbone network, a neck network, and a detection head network, wherein the feature division network includes n feature division modules, and the second backbone network includes n feature fusion modules, and a channel-to-pixel module is connected after at least some of the feature fusion modules, where n is a positive integer greater than 1; The object identification module includes: a first image feature determination module for determining n kinds of first image features corresponding to the construction image by using a first backbone network; a convolution and decomposition module for using an i-th feature decomposition module in the feature decomposition network to perform convolution and decomposition on an i-th first image feature, the resulting decomposition process including i-th sub-image features, and outputting the j-th sub-image feature to a connected j-th feature fusion module, where i and j are positive integers, i is less than or equal to n, j is less than or equal to i, and when i is greater than 1, the i-th sub-image features output by the same i-th feature decomposition module are sub-image features with different channel numbers, one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers; a first fusion module for using an ith feature fusion module in a second backbone network to perform a first fusion process on at least one sub-image feature to obtain a first fusion result, and using a channel-to-pixel module in the second backbone network to determine a second image feature based on the first fusion result, and outputting the second image feature to a neck network; a second fusion module for performing a second fusion process on the second image features by using a neck network to obtain a second fusion process result; An object identification apparatus is further disclosed, including: an identification result determination module for determining the identification result based on the second fusion processing result by utilizing a detection head network.

[0009] Optionally, the safety protector wearing category includes a safety protector not wearing category; The abnormality determination module a first anomaly determination module for determining that the construction image is related to an abnormal operation when the identification result includes a category of safety protectors not being worn; or The system includes a second abnormality determination module for determining distance information between the human category and the construction machinery category based on position information corresponding to the human category and the construction machinery category included in the identification result, and determining whether the construction image is related to abnormal operation based on the distance information.

[0010] Optionally, the convolutional decomposition module: a convolution module for performing convolution processing on the i-th first image feature using the i-th feature division module in the feature division network to obtain image features with c channels; A segmentation module for segmenting image features with c channels into i sub-image features with different channel numbers according to a channel dimension, wherein the jth sub-image feature in the i sub-image features has a channel number of t*2 (j-1) and a division module,

[0011] Optionally, the second backbone network further includes a first convolution module connected before the feature fusion module; The first fusion module comprises: a receiving module for receiving the n-i+1 kinds of sub-image features and the convolved image features output by the first convolution module, wherein the number of channels corresponding to the n-i+1 kinds of sub-image features is a first number of channels, and the number of channels corresponding to the convolved image features is a second number of channels; an interpolation module for performing an interpolation process on n-i+1 kinds of sub-image features using the i-th feature fusion module, wherein the number of channels corresponding to the n-i+1 kinds of sub-image features after the interpolation process is a second number of channels; and a fusion processing module for performing fusion processing on the n-i+1 kinds of sub-image features and the convolved image features after the interpolation processing using the i-th feature fusion module.

[0012] Optionally, a first convolution module connected before the first feature fusion module is used to perform convolution processing on the construction image and output corresponding convolved image features to the first feature fusion module.

[0013] Optionally, the first backbone network includes a second convolution module and n-1 processing units connected in series, and the processing units include a third convolution module and a channel-to-pixel module; The second convolution module is connected to the first feature segmentation module and is used to output a first type of first image feature to the first feature segmentation module, and the channel-to-pixel modules included in the n-1 processing units are respectively connected to corresponding n-1 feature segmentation modules and are used to output first image features to the corresponding feature segmentation modules.

[0014] Optionally, the process of training the target detection model comprises: inputting a construction image sample into a target detection model, and the target detection model outputting a prediction result corresponding to the construction image sample, the prediction result including predicted box information corresponding to a category of a construction object, and the construction image sample corresponding to actual measurement box information; determining loss information corresponding to the measured box information and the predicted box information based on a minimum point distance function based on a horizontal rectangle; and updating parameters of the target detection model based on the loss information.

[0015] An embodiment of the present application further discloses an electronic device including a processor and a memory in which executable code is stored, the executable code, when executed, causing the processor to perform a method described in an embodiment of the present application.

[0016] An embodiment of the present application further discloses a machine-readable medium having executable code stored thereon, the executable code, when executed, causing a processor to perform a method according to an embodiment of the present application. [Effects of the Invention]

[0017] The embodiments of the present application include the following advantages:

[0018] The technical solution of the embodiment of this application uses image recognition to monitor abnormal behavior in construction images in real time, which can save labor costs for safety monitoring at construction sites, and the real-time monitoring can improve the accuracy of safety monitoring at construction sites and further reduce the occurrence rate of safety accidents.

[0019] First, the n feature splitting modules of the feature splitting network in the embodiment of the present application can play the role of aggregating multiple levels of sub-image features. Based on this, the n feature fusion modules of the second backbone network can effectively fuse multiple levels of sub-image features, which can provide richer image features in the forward propagation stage of the target detection model and improve the expressive ability of the second image features output by the second backbone network.

[0020] Furthermore, in the backpropagation calculation process of the target detection model, the deviation calculation between the aggregated multi-level sub-image feature information and the true value can transmit more reliable gradient information to guide the parameter learning of the target detection model, thereby helping the first backbone network extract more important and accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the construction image recognition results, and based on this, the accuracy rate of safety monitoring at construction sites can be improved. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a schematic diagram illustrating an application environment of a method for identifying objects in a construction site based on dual backbone fusion according to an embodiment of the present application; [Figure 2] 1 is a schematic diagram of a step flow of a method for identifying objects in a construction site based on dual backbone fusion according to an embodiment of the present application; [Figure 3] FIG. 2 is a structural schematic diagram of a target detection model according to an embodiment of the present application; [Figure 4] 1 is a structural schematic diagram of a first backbone network 301, a feature division network 302, and a second backbone network 303 in an embodiment of the present application. [Figure 5] 1 is a structural schematic diagram of a channel-to-pixel module according to an embodiment of the present application; [Figure 6] 1 is a structural schematic diagram of a bottleneck module according to an embodiment of the present application; [Figure 7] FIG. 2 is a structural schematic diagram of a second backbone network according to an embodiment of the present application; [Figure 8] FIG. 1 is a structural schematic diagram of a first backbone network according to an embodiment of the present application; [Figure 9] FIG. 1 is a structural schematic diagram of a neck network according to an embodiment of the present application. [Figure 10] FIG. 2 is a structural schematic diagram of a spatial pyramid pooling module according to an embodiment of the present application; [Figure 11]1 is a structural schematic diagram of an object identification device for a construction site based on dual backbone fusion according to an embodiment of the present application; [Figure 12] 1 is a structural schematic diagram of a device provided in an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0022] In order to make the above objects, features, and advantages of the present application clearer and easier to understand, the present application will be described in more detail below in combination with drawings and specific embodiments.

[0023] The embodiments of the present application can be applied to engineering industries such as hydroelectric power generation and civil engineering and construction, and can be used to realize safety monitoring of construction processes while saving labor costs, reducing the incidence of safety accidents during construction, and improving the accuracy of safety monitoring at construction sites.

[0024] In the prior art, professional patrol inspectors are usually deployed at construction sites. The patrol inspectors regularly inspect the construction site, monitor the distance between the construction workers and the construction machinery, and issue an alarm when the distance is shorter than the safe distance. The prior art requires a certain amount of labor costs. In addition, since the patrol inspectors cannot inspect the construction site in real time, the traditional monitoring method still poses certain safety risks and has a relatively low accuracy rate for safety monitoring.

[0025] In response to the technical problems in the prior art, such as the high labor costs and the relatively low accuracy rate of safety monitoring, an embodiment of the present application provides an object identification method, which specifically includes: collecting construction images; identifying the construction images based on a target detection model to obtain an identification result, where the identification result specifically includes a construction object category and location information of the construction object category in the construction image, and the construction object category specifically includes at least one of a human category, a construction machine category, a safety protector wearing category, and a safety sign category; and determining whether the construction image is related to abnormal behavior based on the identification result. The target detection model specifically includes a first backbone network, a feature division network, a second backbone network, a neck network, and a detection head network, the feature division network specifically includes n feature division modules, the second backbone network specifically includes n feature fusion modules, and a channel-to-pixel module is connected after at least some of the feature fusion modules, where n is a positive integer greater than 1; The process of identifying the construction image based on the target detection model specifically includes: The first backbone network determines n kinds of first image features corresponding to the construction image; the i-th feature division module in the feature division network performs convolution processing and division processing on the i-th first image feature, the obtained division processing result includes i-th sub-image feature, and outputs the j-th sub-image feature to the connected j-th feature fusion module, where i and j are positive integers, i is less than or equal to n, j is less than or equal to i, and when i is greater than 1, the i-th sub-image feature output by the same i-th feature division module is a sub-image feature with a different number of channels, and one feature fusion module corresponds to one number of channels, and different feature fusion modules correspond to different numbers of channels; the i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result; the channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result, and outputs the second image feature to the backbone network; The neck network performs a second fusion process on the second image feature to obtain a second fusion process result; The detection head network includes determining the identification result based on the second fusion processing result.

[0026] The image recognition method of the present invention can monitor abnormal behavior in construction images in real time, thereby reducing the labor costs required for safety monitoring at construction sites. Moreover, the real-time monitoring can improve the accuracy of safety monitoring and further reduce the occurrence of safety accidents.

[0027] In the embodiment of the present application, the i-th feature division module divides the first image feature to obtain i sub-image features with different channel numbers, and outputs the j-th sub-image feature to the connected j-th feature fusion module. Since different i-th feature fusion modules can process different levels of the first image feature, the n feature division modules in the embodiment of the present application can serve to aggregate sub-image features at multiple levels. For example, the n feature division modules can provide n-level sub-image features to the first feature fusion module, and the n feature division modules can provide (n-1)-level sub-image features to the second feature fusion module.

[0028] First, the n feature splitting modules of the feature splitting network in the embodiment of the present application can play the role of aggregating multiple levels of sub-image features. Based on this, the n feature fusion modules of the second backbone network can effectively fuse multiple levels of sub-image features, which can provide richer image features in the forward propagation stage of the target detection model and improve the expressive ability of the second image features output by the second backbone network.

[0029] Furthermore, in the backpropagation calculation process of the target detection model, the deviation calculation between the aggregated multi-level sub-image feature information and the true value can transmit more reliable gradient information to guide the parameter learning of the target detection model, thereby helping the first backbone network extract more important and accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the construction image recognition results, and based on this, the accuracy rate of safety monitoring at construction sites can be improved.

[0030] Referring to Figure 1, it shows a schematic diagram of the application environment of the object identification method in construction sites based on dual backbone fusion in an embodiment of the present invention, where the image collection terminal 101 and the server 102 can exchange data based on a wireless network or a wired network.

[0031] In practical application, the image collection terminal 101 may be equipped with an image collection device having an image collection function such as an image sensor. The image collection device can collect construction images, and the image collection terminal 101 can transmit the construction images to the server 102 according to a preset time period. Of course, the image collection device can collect construction videos, and the image collection terminal 101 can transmit the construction videos to the server 102 according to a preset time period, and in this case, the server 102 can analyze the construction images from the construction videos.

[0032] After receiving the construction images sent by the image collection terminal 101, the server 102 can process the construction images using the method of the embodiment of the present application to determine whether the construction images are related to abnormal activity, and if so, output notification information. For example, the notification information can be sent to a terminal of a preset user. The notification information notifies the occurrence of abnormal activity and prompts the preset user to take appropriate action.

[0033] Examples of abnormal behavior include the distance between a human (construction worker) and construction machinery being shorter than the safe distance, the human not wearing safety gear, or the construction environment corresponding to the construction image not having safety signs installed.

[0034] When an abnormal operation occurs, the server 102 can send notification information to a predetermined user, prompting the predetermined user to take a corresponding processing action. The predetermined user can be a construction environment manager. For example, if the distance between a person and a construction machine is shorter than the safe distance, an example of a processing action can be to play a first warning sound through a speaker closest to the construction site corresponding to the construction image, prompting the construction worker in the construction image to move away from the construction machine. If a person is not wearing safety protection gear, an example of a processing action can be to play a second warning sound through a speaker closest to the construction site corresponding to the construction image, prompting the construction worker in the construction image to put on a safety cap. Furthermore, if no safety signs have been installed in the construction environment corresponding to the construction image, the construction environment manager can install safety signs, such as safety slogans, in the construction environment. Those skilled in the art can understand that various processing actions can be used depending on the requirements of actual applications, and the embodiments of this application do not limit the specific processing actions.

[0035] Example of the method Please refer to FIG. 2, which shows a step flow diagram of a method for identifying objects in a construction site based on dual backbone fusion in one embodiment of the present application, which specifically includes: Step 201 of collecting construction images; Step 202: based on a target detection model, classifying the construction image to obtain a classification result, the classification result specifically includes a construction object category and position information of the construction object category in the construction image, the construction object category specifically includes at least one of a human category, a construction machine category, a safety protector wearing category, and a safety sign category; and step 203, based on the identification result, determining whether the construction image is related to an abnormal operation; The target detection model specifically includes a first backbone network, a feature division network, a second backbone network, a neck network, and a detection head network, the feature division network specifically includes n feature division modules, the second backbone network specifically includes n feature fusion modules, and a channel-to-pixel module is connected after at least some of the feature fusion modules, where n can be a positive integer greater than 1; The process of identifying the construction image based on the target detection model in step 202 specifically includes: The first backbone network includes a step 221 of determining n kinds of first image features corresponding to the construction image; The i-th feature division module in the feature division network performs convolution processing and division processing on the i-th first image feature, and the obtained division processing result includes i-th sub-image features, and outputs the j-th sub-image feature to the connected j-th feature fusion module in step 222, where i and j are positive integers, i is less than or equal to n, j is less than or equal to i, and when i is greater than 1, the i-th sub-image features output by the same i-th feature division module have different channel numbers, and one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers; Step 223: the i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result; the channel-to-pixel module in the second backbone network determines a second image feature based on the first fusion result, and outputs the second image feature to the backbone network; Step 224: the neck network performs a second fusion process on the second image features to obtain a second fusion process result; The detection head network includes a step 225 of determining an identification result based on the second fusion processing result.

[0036] The steps included in the embodiment of the method shown in Figure 2 can be performed by a server, and the server can take advantage of its abundant computing resources to quickly process the construction images and monitor abnormal behavior of the construction images in real time based on image recognition. The embodiment of the present application does not limit the specific performing entity of the embodiment of the method shown in Figure 2.

[0037] In the embodiment of the present application, the construction site can be monitored in real time based on construction images, eliminating the need for manual real-time monitoring and saving on labor costs for construction monitoring.

[0038] In step 201, the server can receive the construction images sent by the image collection terminal according to a preset time period. The image collection terminal is installed at the construction site, and can be used to collect construction images at the construction site in real time and send the construction images to the server according to a preset time period.

[0039] In step 202, the construction image is identified based on a target detection model, and the obtained identification result specifically includes the category of the construction object and the location information of the category of the construction object in the construction image, and the category of the construction object specifically includes at least one of the following categories: human category, construction machinery category, safety protective equipment wearing category, and safety sign category.

[0040] An example of the people category is a construction worker. Examples of the construction machinery category are a backhoe excavator, a tower crane, a dump truck, a truck crane, a loader, a hook, a pump truck, a smooth roller, a concrete mixer truck, a pile driver, etc. The safety equipment wearing category specifically includes a safety equipment wearing category and a safety equipment not wearing category. The safety sign category specifically includes a safety slogan, etc.

[0041] Referring to Figure 3, it shows a structural schematic diagram of a target detection model in one embodiment of the present application, which specifically includes a first backbone network 301, a feature division network 302, a second backbone network 303, a neck network 304 and a detection head network 305 connected in sequence.

[0042] The first backbone network 301 is used to perform feature extraction on the input image to obtain n first image features. In the training phase, the input image may be a construction image sample. In the image classification phase, the input image may be a real-time construction image.

[0043] The feature decomposition network 302 is used to perform convolution processing and decomposition processing on the n types of first image features, and output the multiple types of sub-image features obtained by the decomposition to the second backbone network 303.

[0044] The second backbone network 303 is used to perform a first fusion process and a channel-to-pixel process on the multiple sub-image features to obtain a second image feature, and output the second image feature to the neck network 304 .

[0045] The neck network 304 is used to perform a second fusion process on the second image features and obtain a second fusion process result.

[0046] The detection head network 305 is used to determine the classification result based on the second fusion processing result.

[0047] Referring to FIG. 4, it shows a structural schematic diagram of a first backbone network 301, a feature partitioning network 302, and a second backbone network 303 in one embodiment of the present application, where the first backbone network 301 outputs n kinds of first image features to n feature partitioning modules of the feature partitioning network 302.

[0048] The feature division network 302 specifically includes n feature division modules, which are respectively denoted as a first feature division module 321, a second feature division module 322, . . . and an n-th feature division module 32n.

[0049] The i-th feature division module in the feature division network performs convolution processing and division processing on the i-th first image feature, the obtained division processing result includes i-th sub-image feature, and outputs the j-th sub-image feature to the connected j-th feature fusion module, where i and j are positive integers, i is less than or equal to n, j is less than or equal to i, and when i is greater than 1, the i-th sub-image feature output by the same i-th feature division module is a sub-image feature with a different number of channels, and one feature fusion module corresponds to one number of channels, and different feature fusion modules correspond to different numbers of channels.

[0050] The second backbone network 303 specifically includes n feature fusion modules, which are respectively denoted as a first feature fusion module 331, a second feature fusion module 332, . . . and an n-th feature fusion module 33n.

[0051] In a specific implementation, the process in which the i-th feature division module in the feature division network performs convolution processing and division processing on the i-th first image feature is specifically as follows: A step A1 in which an i-th feature division module in the feature division network performs a convolution process on an i-th type first image feature to obtain image features with c channels; A step A2 of dividing the image features with c channels into i sub-image features with different channel numbers according to a channel dimension, wherein the jth sub-image feature in the i sub-image features has a channel number of t*2 (j-1) and step A2, where t is the number of times ...

[0052] The number of channels of i sub-image features is t*2 0 , t*2 1 , t*2 2 …, t*2 (i-1) The total number of channels corresponding to i sub-image features can be c, i.e., t*2 0 +t*2 1 +t*2 2 …+t*2 (i-1) =c.

[0053] When n is 5, the i-th feature division module can output the j-th sub-image feature to the j-th feature fusion module connected thereto, and specific examples include:

[0054] For example, the segmentation processing result of the first feature segmentation module 321 includes one kind of m-channel first sub-image feature, and the one kind of m-channel first sub-image feature is output to the first feature fusion module 331 .

[0055] For example, the segmentation result of the second feature segmentation module 322 includes one m-channel second sub-image feature and one 2m-channel third sub-image feature. The one m-channel second sub-image feature is output to the first feature fusion module 331. The one 2m-channel third sub-image feature is output to the second feature fusion module 332.

[0056] Furthermore, for example, the segmentation processing result of the third feature segmentation module 323 (not shown) includes one m-channel fourth sub-image feature, one 2m-channel fifth sub-image feature, and one 4m-channel sixth sub-image feature. The one m-channel fourth sub-image feature is output to the first feature fusion module 331. The one 2m-channel fifth sub-image feature is output to the second feature fusion module 332. The one 4m-channel sixth sub-image feature is output to the third feature fusion module 333.

[0057] Alternatively, the segmentation processing result of the fourth feature segmentation module 324 (not shown) includes one m-channel seventh sub-image feature, one 2m-channel eighth sub-image feature, one 4m-channel ninth sub-image feature, and one 8m-channel tenth sub-image feature. The one m-channel seventh sub-image feature is output to the first feature fusion module 331. The one 2m-channel eighth sub-image feature is output to the second feature fusion module 332. The one 4m-channel ninth sub-image feature is output to the third feature fusion module 333. The one 8m-channel tenth sub-image feature is output to the fourth feature fusion module 334.

[0058] Alternatively, the segmentation processing result of the fifth feature segmentation module 325 (not shown) includes one m-channel eleventh sub-image feature, one 2m-channel twelfth sub-image feature, one 4m-channel thirteenth sub-image feature, one 8m-channel fourteenth sub-image feature, and one 16m-channel fifteenth sub-image feature. The one m-channel eleventh sub-image feature is output to the first feature fusion module 331. The one 2m-channel twelfth sub-image feature is output to the second feature fusion module 332. The one 4m-channel thirteenth sub-image feature is output to the third feature fusion module 333. The one 8m-channel 14th sub-image feature is output to the fourth feature fusion module 334. The one 16m-channel fifteenth sub-image feature is output to the fifth feature fusion module 335.

[0059] The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result, and the channel-to-pixel module in the second backbone network determines a second image feature based on the first fusion result and outputs the second image feature to the neck network.

[0060] In an alternative implementation form of the present application, the second backbone network may further include a first convolution module connected before the feature fusion module; In this case, the process in which the i-th feature fusion module in the second backbone network performs the first fusion process on at least one sub-image feature is specifically as follows: Step B1: the i-th feature fusion module receives n-i+1 kinds of sub-image features and convolved image features output by the first convolution module, where the number of channels corresponding to the n-i+1 kinds of sub-image features is a first number of channels, and the number of channels corresponding to the convolved image features is a second number of channels; Step B2: the i-th feature fusion module performs interpolation processing on the n-i+1 kinds of sub-image features, and the number of channels corresponding to the n-i+1 kinds of sub-image features after the interpolation processing is a second number of channels; The i-th feature fusion module includes step B3 of performing fusion processing on the n-i+1 kinds of sub-image features after the interpolation processing and the convolution image features.

[0061] In the embodiments of the present application, the first convolution module, the second convolution module, or the first convolution processing module all fall within the scope of a convolution structure. In one example, the convolution structure specifically includes at least one convolution layer, at least one batch normalization layer, and at least one activation function. Those skilled in the art can use any desired convolution structure according to the needs of the actual application, and it is understood that the embodiments of the present application do not limit the specific convolution structure.

[0062] For example, the number of first channels corresponding to the n sub-image features received by the first feature fusion module is t*2 0 Those skilled in the art can use corresponding interpolation techniques according to the needs of actual applications, and the embodiments of this application do not limit the specific interpolation techniques.

[0063] The process in which the i-th feature fusion module performs fusion processing on the n-i+1 kinds of sub-image features and the convolved image features after interpolation specifically includes the i-th feature fusion module adding values ​​corresponding to the n-i+1 kinds of sub-image features and the convolved image features after interpolation, etc. The sub-image features and the convolved image features may correspond to multidimensional matrices, and the above addition processing may be adding element values ​​of the multidimensional matrices.

[0064] In another optional implementation form of the present application, a first convolution module connected before the first feature fusion module is used to perform convolution processing on the construction image and output corresponding convolved image features to the first feature fusion module.

[0065] At least some of the feature fusion modules are followed by a channel-to-pixel module. Referring to Figure 5, it shows a structural schematic diagram of the channel-to-pixel module in one embodiment of the present application, which specifically includes a first convolution processing module 501, a division processing module 502, M bottleneck modules 503, a connection module 504, and a second convolution processing module 505. M can be a positive integer greater than 1.

[0066] The first convolution processing module 501 is used to perform a first convolution processing on the first fusion result output by the feature fusion module to obtain a first convolution processing result.

[0067] The splitting module 502 is used to split the first convolution result into two parts with the same number of channels. The two parts may include features of the first part and features of the second part. Assuming that the number of channels before splitting is the third number of channels, the number of channels after splitting may be the fourth number of channels, and the third number of channels may be twice the fourth number of channels.

[0068] After the first part of the features are processed by the M bottleneck modules 503, the obtained bottleneck processing features enter the connection module 504. The first convolution processing result and the second part of the features also enter the connection module 504. The connection module 504 is used to perform a connection operation on the first convolution processing result, the second part of the features, and the bottleneck processing features to obtain connection features.

[0069] The connection features enter the second convolution processing module 505, which restores the channel number for the connection features, for example, from the fourth channel number to the third channel number.

[0070] Referring to FIG. 6, it shows a structural schematic diagram of a bottleneck module in one embodiment of the present application, which specifically includes a third convolution processing module 601 and a fourth convolution processing module 602.

[0071] The third convolution processing module 601 is used to reduce the number of channels of the input features to half of the original number to obtain the second convolution processing result.

[0072] The fourth convolution module 603 is used to double the number of channels of the second convolution result to obtain a third convolution result, which has the same number of channels as the input features. The third convolution result is then combined with the input features to obtain output features.

[0073] Referring to FIG. 7, it shows a structural schematic diagram of a second backbone network in one embodiment of the present application, the second backbone network specifically includes: a first convolution module A 701, a first feature fusion module 702, a first convolution module B 703, a second feature fusion module 704, a first channel-to-pixel module 705, a first convolution module C 706, a third feature fusion module 707, a second channel-to-pixel module 708, a first convolution module D 709, a fourth feature fusion module 710, a third channel-to-pixel module 711, a first convolution module E 712, a fifth feature fusion module 713, and a fourth channel-to-pixel module 714.

[0074] The first convolution module A701 is used to perform convolution processing on the construction image and output the corresponding convolved image feature A to the first feature fusion module.

[0075] The first feature fusion module 702 is used to receive five types of sub-image features A from the first feature division module, receive convolved image features A from the first convolution module A701, perform interpolation processing on the five types of sub-image features A, assume that the number of channels corresponding to the five types of sub-image features A after the interpolation processing is the second channel number, and perform fusion processing on the five types of sub-image features A after the interpolation processing and the convolved image features A to obtain a first fusion result A.

[0076] The first convolution module B703 is used to perform a convolution process on the first fusion result A to obtain a convolved image feature B.

[0077] The second feature fusion module 704 is used to receive four types of sub-image features B from the second feature division module, receive convolved image features B from the first convolution module B 703, perform interpolation processing on the four types of sub-image features B, assume that the number of channels corresponding to the four types of sub-image features B after the interpolation processing is the second channel number, and perform fusion processing on the four types of sub-image features B after the interpolation processing and the convolved image features B to obtain a first fusion result B.

[0078] The first channel-to-pixel module 705 is used to determine intermediate image features based on the first fusion result B.

[0079] The first convolution module C706 is used to perform a convolution process on the intermediate image features to obtain convolved image features C.

[0080] The third feature fusion module 707 is used to receive three types of sub-image features C from the third feature division module, receive convolved image features C from the first convolution module C 706, perform interpolation processing on the three types of sub-image features C, assume that the number of channels corresponding to the three types of sub-image features C after the interpolation processing is the second channel number, and perform fusion processing on the three types of sub-image features C after the interpolation processing and the convolved image features C to obtain a first fusion result C.

[0081] The second channel-to-pixel module 708 is used to determine a second image feature A according to the first fusion result C, and output the second image feature A to the neck network.

[0082] The first convolution module D709 is used to perform a convolution process on the second image feature A to obtain a convolved image feature D.

[0083] The fourth feature fusion module 710 is used to receive two types of sub-image features D from the fourth feature division module, receive convolved image features D from the first convolution module D709, perform interpolation processing on the three types of sub-image features D, assume that the number of channels corresponding to the two types of sub-image features D after the interpolation processing is the second channel number, and perform fusion processing on the two types of sub-image features D after the interpolation processing and the convolved image features D to obtain a first fusion result D.

[0084] The third channel-to-pixel module 711 is used to determine a second image feature B according to the first fusion result D, and output the second image feature B to the neck network.

[0085] The first convolution module E712 is used to perform a convolution process on the second image feature B to obtain a convolved image feature E.

[0086] The fifth feature fusion module 713 is used to receive one kind of sub-image feature E from the fifth feature division module, receive convolved image feature E from the first convolution module E706, perform interpolation processing on the one kind of sub-image feature E, assume that the number of channels corresponding to the one kind of sub-image feature E after the interpolation processing is the second channel number, and perform fusion processing on the one kind of sub-image feature E after the interpolation processing and the convolved image feature E to obtain a first fusion result E.

[0087] The fourth channel-to-pixel module 714 is used to determine a second image feature C according to the first fusion result E, and output the second image feature C to the neck network.

[0088] In a specific implementation, the first backbone network may specifically include a second convolution module and n-1 processing units connected in sequence, and the processing unit specifically includes a third convolution module and a channel-to-pixel module; The second convolution module is connected to the first feature segmentation module and is used to output a first type of first image feature to the first feature segmentation module, and the channel-to-pixel modules included in the n-1 processing units are respectively connected to corresponding n-1 feature segmentation modules and are used to output first image features to the corresponding feature segmentation modules.

[0089] 8, which shows a structural schematic diagram of a first backbone network in one embodiment of the present application, the first backbone network specifically includes, connected in sequence, a second convolution module 801, a third convolution module A 802, a fifth channel-to-pixel module 803, a third convolution module B 804, a sixth channel-to-pixel module 805, a third convolution module C 806, a seventh channel-to-pixel module 807, a third convolution module D 808, and an eighth channel-to-pixel module 809. The structure of the fifth channel-to-pixel module 803 is shown in FIG. 5 and will not be described in detail here.

[0090] The second convolution module 801 is used to perform convolution processing on the construction image to obtain a first image feature A, and send the first image feature A to the first feature division module.

[0091] The third convolution module A802 is used to perform a convolution process on the first image feature A to obtain a first convolution result.

[0092] The fifth channel-to-pixel module 803 is used to determine a first image feature B according to the first convolution result, and send the first image feature B to the second feature segmentation module.

[0093] The third convolution module B 804 is used to perform a convolution process on the first image feature B to obtain a second convolution result.

[0094] The sixth channel-to-pixel module 805 is used to determine a first image feature C based on the second convolution result, and send the first image feature C to the third feature segmentation module.

[0095] The third convolution module C806 is used to perform a convolution process on the first image feature C to obtain a third convolution result.

[0096] The seventh channel-to-pixel module 807 is used to determine a first image feature D according to the third convolution result, and send the first image feature D to the fourth feature segmentation module.

[0097] The third convolution module D808 is used to perform a convolution process on the first image feature D to obtain a fourth convolution result.

[0098] The seventh channel-to-pixel module 807 is used to determine a first image feature E according to the fourth convolution result, and send the first image feature E to the fifth feature division module.

[0099] In step 224, the neck network is used to perform a second fusion process on the second image features to obtain a second fusion process result.

[0100] Referring to Figure 9, it shows a structural schematic diagram of a neck network in one embodiment of the present application, which specifically includes: a spatial pyramid pooling module 901, a first upsampling module 902, a first connection module 903, a ninth channel-to-pixel module 904, a second upsampling module 905, a second connection module 906, a tenth channel-to-pixel module 907, a fourth convolution module 908, a third connection module 909, an eleventh channel-to-pixel module 910, a fifth convolution module 911, a fourth connection module 912, and a twelfth channel-to-pixel module 913.

[0101] The spatial pyramid pooling module 901 is used to receive the second image feature C and perform spatial pyramid pooling on the second image feature C. The spatial pyramid pooling may include a convolution operation and a max pooling operation, and can realize deep fusion of the second image feature C.

[0102] 10, which shows a structural schematic diagram of a spatial pyramid pooling module in one embodiment of the present application, the spatial pyramid pooling module specifically includes a sixth convolution module 1001, p max pooling modules 1002, a fifth connection module 1003, and a seventh convolution module 1004. p can be a positive integer greater than 1, and the value of p in the figure is 3.

[0103] The sixth convolution module 1001 is used to perform a convolution process on the input second image feature C to obtain a fifth convolution result.

[0104] The p max pooling modules 1002 are used to perform max pooling on the input image features, respectively, to obtain p corresponding max pooling results.

[0105] The fifth convolution result and the p max pooling processing results are respectively input to the fifth connection module 1003, and the fifth connection module 1003 performs deep fusion on the fifth convolution result and the p max pooling processing results to obtain a corresponding deep fusion result.

[0106] The seventh convolution module 1004 is used to perform a convolution process on the deep fusion result to obtain a sixth convolution result, which is provided to the first up-sampling module 902.

[0107] The first upsampling module 902 is used to perform a first upsampling process on the sixth convolution result to obtain a first upsampling process result.

[0108] The first connection module 903 is used to perform a connection process on the second image feature B and the first upsampling process result to obtain a first connection result.

[0109] The ninth channel-to-pixel module 904 is used to determine a first processing result according to the first connection result.

[0110] The second upsampling module 905 is used to perform a second upsampling process on the first processing result to obtain a second upsampling process result.

[0111] The second connection module 906 is used to perform a connection process on the second upsampling process result and the second image feature A to obtain a second connection result. The connection process may be a joining process of two kinds of image features.

[0112] The tenth channel-to-pixel module 907 is used to determine a second fusion processing result A according to the second connection result, and the second fusion processing result A is output to the detection head network.

[0113] The fourth convolution module 908 is used to perform convolution processing on the second fusion processing result A to obtain a sixth convolution result.

[0114] The third connection module 909 is used to perform connection processing on the sixth convolution result and the first processing result to obtain a third connection result.

[0115] The eleventh channel-to-pixel module 910 is used to determine a second fusion processing result B according to the third connection result, and the second fusion processing result B is output to the detection head network.

[0116] The fifth convolution module 911 is used to perform a convolution process on the second fusion result B to obtain a seventh convolution result.

[0117] The fourth connection module 912 is used to perform connection processing on the sixth convolution result and the seventh convolution result to obtain a fourth connection result.

[0118] The twelfth channel-to-pixel module 913 is used to determine a second fusion processing result C according to the fourth connection result, and the second fusion processing result C is output to the detection head network.

[0119] In step 225, the detection head network can determine an identification result based on the second fusion process result.

[0120] The detection head network may include at least one detection module, which can be used to perform classification and regression calculations using a convolution module and a convolution layer based on the second fusion processing result to obtain the category of the construction object in the construction image and the location information of the category of the construction object in the construction image.

[0121] In summary, in the embodiment of the present application, the n feature splitting modules of the feature splitting network can play the role of aggregating multiple levels of sub-image features, and based on this, the n feature fusion modules of the second backbone network can effectively fuse multiple levels of sub-image features, thus providing richer image features in the forward propagation stage of the target detection model, and improving the expressive ability of the second image features output by the second backbone network.

[0122] Furthermore, in the backpropagation calculation process of the target detection model, the deviation calculation between the aggregated multi-level sub-image feature information and the true value can transmit more reliable gradient information to guide the parameter learning of the target detection model, thereby helping the first backbone network extract more important and accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the construction image recognition results, and based on this, the accuracy rate of safety monitoring at construction sites can be improved.

[0123] The target detection model of the present embodiment is experimentally analyzed using a construction site target detection reference dataset. 19,404 marked training set images in the construction site target detection reference dataset are used as the training dataset, and 4,000 validation set images are used as the test dataset. AP (Average Precision)50 and AP75 are defined as evaluation metrics. AP50 represents the average precision when the intersection over union threshold is 0.5. AP75 represents the average precision when the intersection over union threshold is 0.75.

[0124] Experimental results show that the detection accuracy of the present embodiment on a construction site target detection reference dataset is superior to that of conventional techniques. Taking the AP50 index as an example, the detection accuracy of the first, second, and third versions of the target detection model was improved by 13.6%, 8.7%, and 4.3%, respectively, compared to conventional techniques, verifying the effectiveness of the present method. The first, second, and third versions of the target detection model correspond to different parameter amounts.

[0125] In an embodiment of the present application, the training process of the target detection model may include forward propagation and backward propagation.

[0126] Forward propagation is performed based on the parameters of the target detection model, from the embedding layer to the processing layer, and finally predicts the classification result. The predicted information is used to determine the loss information.

[0127] Backward propagation can update the parameters of the target detection model by sequentially calculating from the output layer to the input layer based on the loss information. The target detection model usually uses a neural network structure, and the parameters of the target detection model may include parameters such as neural network weights. In the backward propagation process, gradient information of the parameters of the target detection model is determined, and the parameters of the target detection model can be updated using the gradient information. For example, backward propagation can sequentially calculate from the processing layer to the embedding layer based on the chain rule of calculus, and store the gradient information of the parameters of the target detection model.

[0128] In an embodiment, the process of training the target detection model comprises: Step C1: inputting a construction image sample into a target detection model, and the target detection model outputs a prediction result corresponding to the construction image sample, wherein the prediction result includes predicted box information corresponding to a category of a construction object, and the construction image sample corresponds to actual measurement box information; a step C2 of determining loss information corresponding to the measured box information and the predicted box information based on a minimum point distance function based on a horizontal rectangle; and step C3 of updating parameters of the target detection model based on the loss information.

[0129] The process for obtaining the above construction image samples is as follows: Step D1: randomly reading four original construction images from the construction image set; Step D2: performing a random reinforcement operation on each of the four original construction images to obtain four reinforced construction images; Step D3 of fusing the four enhanced construction images into one fused image; and step D4 of marking the measured box information on one fused image to obtain one training sample.

[0130] The random enhancement operation specifically includes at least one of the following operations: a reversal operation, a random scaling operation, a random color tone transformation operation, and a random perspective transformation operation.

[0131] The process of fusing the four enhanced construction images into one fused image in step D3 specifically includes the steps of arranging the four enhanced construction images into one intermediate image according to offsets of [0,0], [0,243], [320,320], and [320,0], and then cropping the parts of the intermediate images that exceed the size range and reducing the range of the actual measurement box so that it does not exceed the boundary, thereby finally obtaining one fused image. The size range specifically includes the coordinate range corresponding to [0,0] to [320,320].

[0132] Referring to equation (1), it shows the process of determining the loss information corresponding to the measured box information and the predicted box information based on a minimum point distance function based on a horizontal rectangle. JPEG2026017510000002.jpg140170

[0133] The horizontal rectangle-based minimum point distance function helps ensure that the predicted box is geometrically close to the measured box, and performs metric optimization by calculating the distance between the upper left and lower right corners of the predicted box and the measured box, especially when the predicted box and the measured box have the same aspect ratio but different width and height values. The minimum point distance function is more sensitive to the position error of the predicted box, and therefore may be useful in improving the identification accuracy of the target detection model.

[0134] In step 203, based on the identification result, it can be determined whether the construction image is related to an abnormal operation.

[0135] Examples of abnormal behavior include the distance between a human (construction worker) and construction machinery being shorter than the safe distance, the human not wearing safety gear, or the construction environment corresponding to the construction image not having safety signs installed.

[0136] In an implementation form of the present application, the safety protector wearing category includes a safety protector not wearing category, The process of determining whether the construction image is related to an abnormal operation based on the identification result specifically includes: Step C1 of determining that the construction image is related to an abnormal operation when the identification result includes a category of safety protectors not being worn; or The method includes a step C2 of determining distance information between the human category and the construction machinery category based on position information corresponding to the human category and the construction machinery category included in the identification result, and determining whether the construction image is related to abnormal operation based on the distance information.

[0137] In one example, the process of determining the distance information between the human category and the construction machinery category specifically includes: first, based on the position information corresponding to the human category and the construction machinery category respectively included in the identification result, determining the image distance between the human category and the construction machinery category; and then, based on the scale factor of the construction image, converting the image distance into an actual distance, and the actual distance can be used as the distance information between the human category and the construction machinery category.

[0138] If the distance value corresponding to the distance information is shorter than the safe distance, it can be determined that the construction image is related to an abnormal operation. If the construction image is related to an abnormal operation, the embodiment of the present application can send corresponding prompting information to a predetermined user, so that the predetermined user can handle the abnormal operation.

[0139] In summary, the object identification method of the present application uses image identification to monitor abnormal behavior in construction images in real time, thereby reducing the labor costs required for safety monitoring at construction sites. Moreover, the real-time monitoring can improve the accuracy of safety monitoring at construction sites and further reduce the incidence of safety accidents.

[0140] First, the n feature splitting modules of the feature splitting network in the embodiment of the present application can play the role of aggregating multiple levels of sub-image features. Based on this, the n feature fusion modules of the second backbone network can effectively fuse multiple levels of sub-image features. In this way, richer image features can be provided in the forward propagation stage of the target detection model, and the expressive ability of the second image features output by the second backbone network can be improved.

[0141] Furthermore, in the backpropagation calculation process of the target detection model, the deviation calculation between the aggregated multi-level sub-image feature information and the true value can transmit more reliable gradient information to guide the parameter learning of the target detection model, thereby helping the first backbone network extract more important and accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the construction image recognition results, and based on this, the accuracy rate of safety monitoring at construction sites can be improved.

[0142] Although the embodiments of the method are all expressed as a combination of a series of operations for ease of explanation, those skilled in the art will recognize that the embodiments of the present application are not limited to the order of operations described, as certain steps may be performed in other orders or simultaneously. Those skilled in the art will also recognize that the embodiments described in the specification are all preferred embodiments, and that the related operations are not necessarily required for the embodiments of the present application.

[0143] Based on the above embodiment, this embodiment further provides an object identification device, and referring to FIG. 11, the device may specifically include: a collection module 1101, an object identification module 1102, and an anomaly determination module 1103.

[0144] The collection module 1101 is used to collect construction images; The object identification module 1102 is used to identify the construction image based on a target detection model to obtain an identification result, and the identification result includes a construction object category and position information of the construction object category in the construction image, and the construction object category includes at least one of a human category, a construction machine category, a safety protector wearing category, and a safety sign category; The abnormality determination module 1103 is used to determine whether the construction image is related to an abnormal operation based on the identification result; The target detection model includes a first backbone network, a feature division network, a second backbone network, a neck network, and a detection head network, wherein the feature division network includes n feature division modules, and the second backbone network includes n feature fusion modules, and a channel-to-pixel module is connected after at least some of the feature fusion modules, where n is a positive integer greater than 1; The object identification module 1102 specifically includes: a first image feature determining module 1121 for determining n kinds of first image features corresponding to the construction image by using a first backbone network; a convolution and division module 1122 for using an i-th feature division module in the feature division network to perform convolution and division on an i-th first image feature, the division result of which includes i-th sub-image features, and outputting the j-th sub-image feature to a connected j-th feature fusion module, where i and j are positive integers, i is less than or equal to n, j is less than or equal to i, and when i is greater than 1, the i-th sub-image features output by the same i-th feature division module are sub-image features with different channel numbers, and one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers; a first fusion module 1123 for using an ith feature fusion module in a second backbone network to perform a first fusion process on at least one sub-image feature to obtain a first fusion result, and using a channel-to-pixel module in the second backbone network to determine a second image feature based on the first fusion result, and outputting the second image feature to a backbone network; a second fusion module 1124 for performing a second fusion process on the second image features by using a neck network to obtain a second fusion process result; and an identification result determination module 1125 for determining the identification result based on the second fusion processing result using a detection head network.

[0145] Optionally, the safety protector wearing category includes a safety protector not wearing category; The abnormality determination module a first anomaly determination module for determining that the construction image is related to an abnormal operation when the identification result includes a category of safety protectors not being worn; or The system includes a second abnormality determination module for determining distance information between the human category and the construction machinery category based on position information corresponding to the human category and the construction machinery category included in the identification result, and determining whether the construction image is related to abnormal operation based on the distance information.

[0146] Optionally, the convolutional decomposition module: a convolution module for performing convolution processing on the i-th first image feature using the i-th feature division module in the feature division network to obtain image features with c channels; A segmentation module for segmenting image features with c channels into i sub-image features with different channel numbers according to a channel dimension, wherein the jth sub-image feature in the i sub-image features has a channel number of t*2 (j-1 ), and a division module.

[0147] Optionally, the second backbone network further includes a first convolution module connected before the feature fusion module; The first fusion module comprises: a receiving module for receiving the n-i+1 kinds of sub-image features and the convolved image features output by the first convolution module, wherein the number of channels corresponding to the n-i+1 kinds of sub-image features is a first number of channels, and the number of channels corresponding to the convolved image features is a second number of channels; an interpolation module for performing an interpolation process on n-i+1 kinds of sub-image features using the i-th feature fusion module, wherein the number of channels corresponding to the n-i+1 kinds of sub-image features after the interpolation process is a second number of channels; and a fusion processing module for performing fusion processing on the n-i+1 kinds of sub-image features and the convolved image features after the interpolation processing using the i-th feature fusion module.

[0148] Optionally, a first convolution module connected before the first feature fusion module is used to perform convolution processing on the construction image and output corresponding convolved image features to the first feature fusion module.

[0149] Optionally, the first backbone network includes a second convolution module and n-1 processing units connected in series, and the processing units include a third convolution module and a channel-to-pixel module; The second convolution module is connected to the first feature segmentation module and is used to output a first type of first image feature to the first feature segmentation module, and the channel-to-pixel modules included in the n-1 processing units are respectively connected to corresponding n-1 feature segmentation modules and are used to output first image features to the corresponding feature segmentation modules.

[0150] Optionally, the process of training the target detection model comprises: inputting a construction image sample into a target detection model, and the target detection model outputting a prediction result corresponding to the construction image sample, the prediction result including predicted box information corresponding to a category of a construction object, and the construction image sample corresponding to actual measurement box information; determining loss information corresponding to the measured box information and the predicted box information based on a minimum point distance function based on a horizontal rectangle; and updating parameters of the target detection model based on the loss information.

[0151] In summary, the object identification device of the present embodiment uses image identification to monitor abnormal behavior in construction images in real time, thereby reducing the labor costs required for safety monitoring at construction sites. Moreover, the real-time monitoring can improve the accuracy of safety monitoring at construction sites and further reduce the incidence of safety accidents.

[0152] First, the n feature splitting modules of the feature splitting network in the embodiment of the present application can play the role of aggregating multiple levels of sub-image features. Based on this, the n feature fusion modules of the second backbone network can effectively fuse multiple levels of sub-image features. In this way, richer image features can be provided in the forward propagation stage of the target detection model, and the expressive ability of the second image features output by the second backbone network can be improved.

[0153] Furthermore, in the backpropagation calculation process of the target detection model, the deviation calculation between the aggregated multi-level sub-image feature information and the true value can transmit more reliable gradient information to guide the parameter learning of the target detection model, thereby helping the first backbone network extract more important and accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the construction image recognition results, and based on this, the accuracy rate of safety monitoring at construction sites can be improved.

[0154] An embodiment of the present application further provides a non-volatile readable storage medium, which stores one or more modules (programs), and when the one or more modules are applied to an apparatus, causes the apparatus to execute instructions for each step of the method in the embodiment of the present application.

[0155] Embodiments of the present application provide one or more machine-readable media having instructions stored thereon that, when executed by one or more processors, cause an electronic device to perform the method according to one or more of the above embodiments. In embodiments of the present application, the electronic device includes various devices such as a terminal device, a server (cluster), etc.

[0156] The embodiments of the present disclosure may be implemented as a desired configuration device using any suitable hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices, servers (clusters), etc. Figure 12 illustrates an exemplary device 1300 that may be used to implement the embodiments described herein.

[0157] For one embodiment, FIG. 12 illustrates an exemplary device 1300 having one or more processors 1302, a control module (chipset) 1304 coupled to at least one of the processor(s) 1302, memory 1306 coupled to the control module 1304, an NVM (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0158] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the apparatus 1300 may be used as a terminal device, a server (cluster), or other device described in embodiments of the present application.

[0159] In some examples, apparatus 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314, and one or more processors 1302 configured to execute the instructions 1314 and implement modules in combination with the one or more computer-readable media to perform operations described in this disclosure.

[0160] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304.

[0161] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0162] Memory 1306 may be used, for example, to load and store data and / or instructions 1314 for device 1300. For one embodiment, memory 1306 may include any suitable volatile memory, such as suitable DRAM (Dynamic Random Access Memory). In some embodiments, memory 1306 may include DDR4 synchronous dynamic random access memory.

[0163] For one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to a non-volatile memory / storage device 1308 and to one or more input / output device(s) 1310 .

[0164] For example, non-volatile memory / storage device 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage device 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile optical disk drives).

[0165] The non-volatile memory / storage device 1308 may include storage resources that are physically part of the device in which the apparatus 1300 is installed, or may be accessible by but not necessarily part of the device. For example, the non-volatile memory / storage device 1308 may be accessed over a network via the input / output device(s) 1310.

[0166] Input / output device(s) 1310 may provide an interface for device 1300 to communicate with any other suitable devices, and input / output device(s) 1310 may include communication components, audio components, sensor components, etc. Network interface 1312 may provide an interface for device 1300 to communicate over one or more networks, and device 1300 may wirelessly communicate with one or more components of a wireless network based on any of one or more wireless network standards and / or protocols, for example, to access and wirelessly communicate with a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2-Generation wireless telephone technology), 3G (3-Generation wireless telephone technology), 4G (4-Generation wireless telephone technology), 5G (5-Generation wireless telephone technology), etc., or a combination thereof.

[0167] For one embodiment, at least one of the processor(s) 1302 may be co-packaged with logic of one or more controllers (e.g., memory controller modules) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be co-packaged with logic of one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die as logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die as logic of one or more controllers of the control module 1304 to form a system-on-chip.

[0168] In various embodiments, device 1300 may include, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touchscreen device, a netbook, etc.). In various embodiments, device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, device 1300 includes one or more video cameras, a keyboard, a liquid crystal display screen (e.g., a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit, and a speaker.

[0169] In the detection device, a master chip can be used as a processor or control module, sensor data, location information, etc. are stored in memory or non-volatile memory / storage device, sensors can be used as input / output devices, and the communication interface may include a network interface.

[0170] In the case of the device embodiment, since it is basically the same as the method embodiment, the explanation is relatively simple, and relevant parts can be referred to the partial explanation of the method embodiment.

[0171] Each embodiment in this specification will be described step by step, with emphasis on the differences between each embodiment and other embodiments, and identical and similar parts between the embodiments may be mutually referenced.

[0172] The embodiments of the present application will be described with reference to flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device, and the instructions executed by the processor of the computer or other programmable data processing terminal device can generate a machine to implement the function(s) specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0173] These computer program instructions may also be stored in a computer-readable memory that can cause a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0174] These computer program instructions may be loaded into a computer or other programmable data processing terminal device, whereby a series of operational steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, the instructions executing on the computer or other programmable terminal device providing steps for realizing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0175] Although preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have acquired the basic creative concept. Therefore, it is intended that the appended claims be interpreted to include all changes and modifications that fall within the scope of the preferred embodiments and the present application.

[0176] Finally, it should be explained that relational terms such as "first" and "second" are used herein merely to distinguish one entity or operation from another, and do not necessarily require or imply the existence of any actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "include," and any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal device that includes a set of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such a process, method, article, or terminal device. Unless further limited, an element defined by the phrase "comprises" does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes that element.

[0177] The object identification method, device, electronic device, and machine-readable medium according to the present invention have been described in detail above, and the present specification uses specific examples to explain the principles and embodiments of the present application. However, the description of the above examples is only intended to facilitate understanding of the method and its core idea of ​​the present application. Furthermore, those skilled in the art will recognize that there may be changes in the specific embodiments and application scope based on the idea of ​​the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

1. 1. A method for identifying objects in a construction site based on dual backbone fusion, the method comprising: Collecting construction images; A step of identifying the construction image based on a target detection model to obtain an identification result, wherein the identification result includes a construction object category and position information of the construction object category in the construction image, and the construction object category includes at least one of a human category, a construction machine category, a safety protector wearing category, and a safety sign category; and determining whether the construction image is related to an abnormal operation based on the identification result; the target detection model includes a first backbone network, a feature division network, a second backbone network, a neck network, and a detection head network, the feature division network includes n feature division modules, the second backbone network includes n feature fusion modules, and a channel-to-pixel module is connected after at least some of the feature fusion modules, where n is a positive integer greater than 1; The step of identifying the construction image based on a target detection model includes: The first backbone network determines n first image features corresponding to the construction image; the i-th feature division module in the feature division network performs convolution processing and division processing on the i-th first image feature, the obtained division processing result includes i-th sub-image features, and outputs the j-th sub-image feature to the connected j-th feature fusion module, where i and j are positive integers, i is less than or equal to n, j is less than or equal to i, and when i is greater than 1, the i-th sub-image features output by the same i-th feature division module have different channel numbers, and one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers; the i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result; the channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result, and outputs the second image feature to the neck network; The neck network performs a second fusion process on the second image features to obtain a second fusion process result; The detection head network determines the identification result based on the second fusion processing result,

2. The safety protector wearing category includes a safety protector not wearing category, The step of determining whether the construction image is related to an abnormal operation based on the identification result includes: If the identification result includes a category of not wearing safety protectors, determining that the construction image is related to an abnormal operation; or 2. The method of claim 1, further comprising: determining distance information between the human category and the construction machine category based on position information corresponding to the human category and the construction machine category included in the identification result; and determining whether the construction image is related to abnormal operation based on the distance information.

3. The step of the i-th feature division module in the feature division network performing convolution processing and division processing on the i-th type of first image feature includes: The i-th feature division module in the feature division network performs a convolution process on the i-th first image feature to obtain image features with c channels; A step of dividing the image features with c channels into i sub-image features with different channel numbers according to a channel dimension, wherein the number of channels of the j-th sub-image feature in the i sub-image features is t*2 （ｊ－１） 2. The method of claim 1, further comprising the steps of:

4. the second backbone network further includes a first convolution module connected before the feature fusion module; The step of the i-th feature fusion module in the second backbone network performing a first fusion process on at least one sub-image feature includes: the i-th feature fusion module receives n-i+1 kinds of sub-image features and the convolved image features output by the first convolution module, where the number of channels corresponding to the n-i+1 kinds of sub-image features is a first number of channels, and the number of channels corresponding to the convolved image features is a second number of channels; the i-th feature fusion module performs an interpolation process on the n-i+1 kinds of sub-image features, and the number of channels corresponding to the n-i+1 kinds of sub-image features after the interpolation process is a second number of channels; The method of claim 1, wherein the i-th feature fusion module performs fusion processing on the n-i+1 kinds of sub-image features after interpolation processing and the convolved image features.

5. The method of claim 4, wherein a first convolution module connected before the first feature fusion module is used to perform convolution processing on the construction image and output corresponding convolved image features to the first feature fusion module.

6. The n feature division modules include a first feature division module, a second feature division module, a third feature division module, a fourth feature division module, and a fifth feature division module; and the n feature fusion modules include a first feature fusion module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module, and a fifth feature fusion module; The segmentation processing result of the first feature segmentation module includes one kind of m-channel first sub-image feature, and the one kind of m-channel first sub-image feature is output to the first feature fusion module; The segmentation processing result of the second feature segmentation module includes one kind of m-channel second sub-image feature and one kind of 2m-channel third sub-image feature, and the one kind of m-channel second sub-image feature is output to the first feature fusion module, and the one kind of 2m-channel third sub-image feature is output to the second feature fusion module; The segmentation processing result of the third feature segmentation module includes one kind of m-channel fourth sub-image feature, one kind of 2m-channel fifth sub-image feature, and one kind of 4m-channel sixth sub-image feature, wherein the one kind of m-channel fourth sub-image feature is output to the first feature fusion module, the one kind of 2m-channel fifth sub-image feature is output to the second feature fusion module, and the one kind of 4m-channel sixth sub-image feature is output to the third feature fusion module; The segmentation processing result of the fourth feature segmentation module includes one m-channel seventh sub-image feature, one 2m-channel eighth sub-image feature, one 4m-channel ninth sub-image feature, and one 8m-channel tenth sub-image feature, wherein the one m-channel seventh sub-image feature is output to the first feature fusion module 331, the one 2m-channel eighth sub-image feature is output to the second feature fusion module, the one 4m-channel ninth sub-image feature is output to the third feature fusion module 333, and the one 8m-channel tenth sub-image feature is output to the fourth feature fusion module 334; 2. The method of claim 1, wherein the segmentation processing result of the fifth feature segmentation module includes one eleventh sub-image feature with m channels, one twelfth sub-image feature with 2m channels, one thirteenth sub-image feature with 4m channels, one fourteenth sub-image feature with 8m channels, and one fifteenth sub-image feature with 16m channels, wherein the one eleventh sub-image feature with m channels is output to the first feature fusion module, the one twelfth sub-image feature with 2m channels is output to the second feature fusion module, the one thirteenth sub-image feature with 4m channels is output to the third feature fusion module, the one fourteenth sub-image feature with 8m channels is output to the fourth feature fusion module, and the one fifteenth sub-image feature with 16m channels is output to the fifth feature fusion module.

7. 2. The method of claim 1 , wherein the second backbone network includes, connected in sequence, a first convolution module A, a first feature fusion module, a first convolution module B, a second feature fusion module, a first channel-to-pixel module, a first convolution module C, a third feature fusion module, a second channel-to-pixel module, a first convolution module D, a fourth feature fusion module, a third channel-to-pixel module, a first convolution module E, a fifth feature fusion module, and a fourth channel-to-pixel module.

8. The first convolution module A is used to perform convolution processing on the construction image and output the corresponding convolved image feature A to the first feature fusion module; the first feature fusion module is used to receive five kinds of sub-image features A from the first feature division module, receive convolved image features A from the first convolution module A, perform interpolation processing on the five kinds of sub-image features A, and assume that the number of channels corresponding to the five kinds of sub-image features A after the interpolation processing is a second channel number; and perform fusion processing on the five kinds of sub-image features A after the interpolation processing and the convolved image features A to obtain a first fusion result A; The first convolution module B is used to perform a convolution process on the first fusion result A to obtain a convolved image feature B; the second feature fusion module is used to receive four sub-image features B from the second feature division module, receive convolved image features B from the first convolution module B, perform interpolation on the four sub-image features B, and assume that the number of channels corresponding to the four sub-image features B after the interpolation is a second channel number; and perform fusion on the four sub-image features B after the interpolation and the convolved image features B to obtain a first fusion result B; a first channel-to-pixel module for determining intermediate image features based on the first fusion result B; The first convolution module C is used to perform a convolution process on the intermediate image features to obtain convolved image features C; the third feature fusion module is used to receive three kinds of sub-image features C from the third feature division module, receive convolved image features C from the first convolution module C, perform interpolation processing on the three kinds of sub-image features C, and assume that the number of channels corresponding to the three kinds of sub-image features C after the interpolation processing is the second number of channels; and perform fusion processing on the three kinds of sub-image features C after the interpolation processing and the convolved image features C to obtain a first fusion result C; a second channel-to-pixel module for determining a second image feature A according to the first fusion result C, and outputting the second image feature A to the neck network; The first convolution module D is used to perform a convolution process on the second image feature A to obtain a convolved image feature D; the fourth feature fusion module receives two kinds of sub-image features D from the fourth feature division module, receives convolved image features D from the first convolution module D, performs interpolation processing on the three kinds of sub-image features D, and assumes that the number of channels corresponding to the two kinds of sub-image features D after the interpolation processing is the second number of channels; and performs fusion processing on the two kinds of sub-image features D after the interpolation processing and the convolved image features D to obtain a first fusion result D; a third channel-to-pixel module for determining a second image feature B according to the first fusion result D, and outputting the second image feature B to the neck network; The first convolution module E is used to perform a convolution process on the second image feature B to obtain a convolved image feature E; the fifth feature fusion module is used to receive one kind of sub-image feature E from the fifth feature division module, receive convolved image feature E from the first convolution module E, perform an interpolation process on the one kind of sub-image feature E, and assume that the number of channels corresponding to the one kind of sub-image feature E after the interpolation process is a second number of channels; and perform a fusion process on the one kind of sub-image feature E after the interpolation process and the convolved image feature E to obtain a first fusion result E; The method of claim 7, wherein a fourth channel-to-pixel module is used to determine a second image feature C based on the first fusion result E and output the second image feature C to the neck network.

9. The first backbone network includes a second convolution module and n-1 processing units connected in series, and the processing units include a third convolution module and a channel-to-pixel module; 2. The method of claim 1, wherein the second convolution module is connected to a first feature segmentation module and is used to output a first type of first image feature to the first feature segmentation module; and the channel-to-pixel modules included in the n-1 processing units are respectively connected to corresponding n-1 feature segmentation modules and are used to output the first image feature to the corresponding feature segmentation module.

10. In the process of training the target detection model, Input a construction image sample into a target detection model, and the target detection model outputs a prediction result corresponding to the construction image sample, the prediction result including predicted box information corresponding to a category of a construction object, and the construction image sample corresponds to actual measurement box information; determining loss information corresponding to the measured box information and the predicted box information based on a minimum point distance function based on a horizontal rectangle; The method according to any one of claims 1 to 9, further comprising updating parameters of the target detection model based on the loss information.

11. 1. An object identification device, comprising: a collection module for collecting construction images; an object identification module for identifying the construction image based on a target detection model to obtain an identification result, the identification result including a construction object category and position information of the construction object category in the construction image, the construction object category including at least one of a human category, a construction machine category, a safety protector wearing category, and a safety sign category; an abnormality determination module for determining whether the construction image is related to an abnormal operation based on the identification result; the target detection model includes a first backbone network, a feature division network, a second backbone network, a neck network, and a detection head network, the feature division network includes n feature division modules, the second backbone network includes n feature fusion modules, and a channel-to-pixel module is connected after at least some of the feature fusion modules, where n is a positive integer greater than 1; The object identification module includes: a first image feature determining module for determining n kinds of first image features corresponding to the construction image by using a first backbone network; a convolution and decomposition module for using an i-th feature decomposition module in the feature decomposition network to perform convolution and decomposition on an i-th first image feature, the resulting decomposition processing result including i-th sub-image features, and outputting the j-th sub-image feature to a connected j-th feature fusion module, where i and j are positive integers, i is less than or equal to n, j is less than or equal to i, and when i is greater than 1, the i-th sub-image features output by the same i-th feature decomposition module are sub-image features with different channel numbers, one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers; a first fusion module for using an ith feature fusion module in a second backbone network to perform a first fusion process on at least one sub-image feature to obtain a first fusion result, and using a channel-to-pixel module in the second backbone network to determine a second image feature based on the first fusion result, and outputting the second image feature to a neck network; a second fusion module for performing a second fusion process on the second image features by using a neck network to obtain a second fusion process result; and an identification result determination module for determining the identification result based on the second fusion processing result by utilizing a detection head network.

12. An electronic device, a processor; An electronic device comprising: a memory in which executable code is stored; and the executable code, when executed, causes the processor to perform the method according to any one of claims 1 to 10.

13. A machine-readable medium having stored thereon executable code that, when executed, causes a processor to perform the method of any of claims 1 to 10.

Images