Construction scene object recognition method, device and product based on dual-trunk fusion

Abstract

The embodiment of the present application provides a construction scene object recognition method, device and product based on dual-trunk fusion, wherein the method specifically includes: collecting construction images; according to the target detection model, recognizing the construction images to obtain recognition results; the recognition results include: the construction object category and the location information of the construction object category in the construction image; according to the recognition results, determining whether the construction image involves abnormal behavior; the i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and outputs the j-th seed image feature to the connected j-th feature fusion module. The embodiment of the present application can save the manpower cost spent on safety monitoring in construction scenes; and the real-time monitoring of the embodiment of the present application can improve the accuracy of safety monitoring in construction scenes, thereby reducing the incidence of safety responsibility accidents.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image recognition technology, and in particular to a method, device and product for construction scene object recognition based on dual-trunk fusion. Background Art

[0002] With the continuous development of society, engineering industries such as hydropower and civil engineering have achieved rapid development. However, in recent years, safety accidents may occur during construction due to the close distance between people and vehicles.

[0003] In order to reduce safety accidents, traditional monitoring methods usually use personnel inspections for management; specifically, special inspection personnel are deployed at the construction site to inspect the construction site regularly and monitor the distance between construction personnel and construction machinery. If the distance is less than the safe distance, an alarm message is issued.

[0004] Traditional monitoring methods require a certain amount of manpower costs. In addition, since inspection personnel cannot inspect the construction site in real time, traditional monitoring methods still have certain safety hazards, resulting in a low accuracy rate of safety monitoring. Summary of the invention

[0005] The embodiment of the present application provides a construction scene object recognition method based on dual-trunk fusion, which can save the manpower cost spent on safety monitoring in construction scenes; and the real-time monitoring of the embodiment of the present application can improve the accuracy of safety monitoring in construction scenes, thereby reducing the occurrence rate of safety liability accidents.

[0006] Correspondingly, an embodiment of the present application also provides a construction scene object recognition device based on dual-trunk fusion, an electronic device and a machine-readable medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the embodiment of the present application discloses a method for construction scene object recognition based on dual-trunk fusion, including:

[0008] Collect construction images;

[0009] According to the target detection model, the construction image is recognized to obtain a recognition result; the recognition result includes: a construction object category and position information of the construction object category in the construction image, and the construction object category includes at least one of the following categories: a human category, a construction machinery category, a safety protection facility wearing category, and a safety sign category;

[0010] Determining whether the construction image involves abnormal behavior according to the recognition result;

[0011] The target detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1;

[0012] The identifying the construction image according to the target detection model includes:

[0013] The first backbone network determines n first image features corresponding to the construction image;

[0014] The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers;

[0015] The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result, and the channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result, and outputs the second image feature to the neck network;

[0016] The neck network performs a second fusion process on the second image feature to obtain a second fusion process result;

[0017] The detection head network determines the recognition result according to the second fusion processing result.

[0018] The present application also discloses a device for identifying an object, the device comprising:

[0019] A collection module, used for collecting construction images;

[0020] An object recognition module is used to recognize the construction image according to the target detection model to obtain a recognition result; the recognition result includes: a construction object category and position information of the construction object category in the construction image, and the construction object category includes at least one of the following categories: a human category, a construction machinery category, a safety protection facility wearing category, and a safety sign category;

[0021] an abnormality determination module, used to determine whether the construction image involves abnormal behavior according to the recognition result;

[0022] The target detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1;

[0023] The object recognition module comprises:

[0024] A first image feature determination module, used to determine n first image features corresponding to the construction image using a first backbone network;

[0025] The convolution segmentation module is used to perform convolution processing and segmentation processing on the i-th first image feature using the i-th feature segmentation module in the feature segmentation network, and the obtained segmentation processing result includes: i-seed image features, and outputs the j-seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i-seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers;

[0026] a first fusion module, configured to perform a first fusion process on at least one sub-image feature using the i-th feature fusion module in the second backbone network to obtain a first fusion result, and to determine a second image feature based on the first fusion result using the channel-to-pixel module in the second backbone network, and output the second image feature to the neck network;

[0027] a second fusion module, configured to perform a second fusion process on the second image feature by using a neck network to obtain a second fusion process result;

[0028] The recognition result determination module is used to determine the recognition result according to the second fusion processing result by using the detection head network.

[0029] Optionally, the safety protection facility wearing category includes: safety protection facility not wearing category;

[0030] The abnormality determination module comprises:

[0031] A first abnormality determination module is used to determine that the construction image involves abnormal behavior when the recognition result includes the category of not wearing safety protection equipment; or

[0032] The second abnormality determination module is used to determine the distance information between the human category and the construction machinery category according to the position information corresponding to the human category and the construction machinery category respectively contained in the recognition result; and determine whether the construction image involves abnormal behavior according to the distance information.

[0033] Optionally, the convolution segmentation module includes:

[0034] A convolution module is used to perform convolution processing on the i-th first image feature using the i-th feature segmentation module in the feature segmentation network to obtain image features with c channels;

[0035] The segmentation module is used to segment the image features with c channels into i seed image features with different numbers of channels according to the channel dimension; the number of channels of the jth seed image feature in the i seed image features is: t*2 (j-1) .

[0036] Optionally, the second backbone network further includes: a first convolution module connected in front of the feature fusion module;

[0037] The first fusion module includes:

[0038] A receiving module, used to receive n-i+1 seed image features and convolution image features output by the first convolution module; the number of channels corresponding to the n-i+1 seed image features is the first number of channels, and the number of channels corresponding to the convolution image features is the second number of channels;

[0039] An interpolation module is used to perform interpolation processing on n-i+1 seed image features using the i-th feature fusion module, and the number of channels corresponding to the n-i+1 seed image features after the interpolation processing is the second number of channels;

[0040] The fusion processing module is used to use the i-th feature fusion module to fuse the n-i+1 seed image features and the convolution image features after interpolation processing.

[0041] Optionally, the first convolution module connected in front of the first feature fusion module is used to perform convolution processing on the construction image and output corresponding convolution image features to the first feature fusion module.

[0042] Optionally, the first backbone network comprises: a second convolution module and n-1 processing units connected in sequence; the processing unit comprises: a third convolution module and a channel to pixel module;

[0043] Among them, the second convolution module is connected to the first feature segmentation module, and is used to output the first first image feature to the first feature segmentation module; the channel-to-pixel modules contained in the n-1 processing units are respectively connected to the corresponding n-1 feature segmentation modules, and are used to output the first image feature to the corresponding feature segmentation module.

[0044] Optionally, the training process of the target detection model includes:

[0045] Input the construction image sample into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: prediction frame information corresponding to the construction object category; and real frame information corresponding to the construction image sample;

[0046] Determining loss information corresponding to the real frame information and the predicted frame information according to a minimum point distance function based on a horizontal rectangle;

[0047] According to the loss information, the parameters of the target detection model are updated.

[0048] The embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the method described in the embodiment of the present application.

[0049] The embodiment of the present application further discloses a machine-readable medium on which executable code is stored. When the executable code is executed, a processor executes the method described in the embodiment of the present application.

[0050] The embodiments of the present application include the following advantages:

[0051] In the technical solution of the embodiment of the present application, abnormal behavior of construction images is monitored in real time based on image recognition, which can save the manpower cost of safety monitoring in construction scenes. In addition, the above real-time monitoring can improve the accuracy of safety monitoring in construction scenes, thereby reducing the incidence of safety liability accidents.

[0052] First, the n feature segmentation modules of the feature segmentation network of the embodiment of the present application can play the role of aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In the forward propagation stage of the target detection model, richer image features can be provided, which can enhance the expression ability of the second image features output by the second backbone network.

[0053] Secondly, in the back propagation calculation process of the target detection model, by calculating the deviation between the converged multi-level sub-image feature information and the true value, more reliable gradient information can be transmitted to guide the parameter learning of the target detection model, thereby helping the first backbone network to extract more critical and more accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the recognition results of the construction image, and on this basis can improve the accuracy of safety monitoring in the construction scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of an application environment of a construction scene object recognition method based on dual-trunk fusion according to an embodiment of the present application;

[0055] Figure 2 It is a schematic diagram of the steps of a construction scene object recognition method based on dual-trunk fusion according to an embodiment of the present application;

[0056] Figure 3 is a schematic diagram of the structure of a target detection model according to an embodiment of the present application;

[0057] Figure 4 It is a schematic diagram of the structure of a first backbone network 301, a feature segmentation network 302, and a second backbone network 303 in one embodiment of the present application;

[0058] Figure 5 is a schematic diagram of the structure of a channel-to-pixel module according to an embodiment of the present application;

[0059] Figure 6 is a schematic diagram of the structure of a bottleneck module of an embodiment of the present application;

[0060] Figure 7 is a schematic diagram of the structure of a second backbone network according to an embodiment of the present application;

[0061] Figure 8 is a schematic diagram of the structure of a first backbone network according to an embodiment of the present application;

[0062] Fig. 9 is a schematic diagram of the structure of a neck network according to an embodiment of the present application;

[0063] Fig.10 It is a structural diagram of a spatial pyramid pooling module according to an embodiment of the present application;

[0064] Fig.11 It is a structural schematic diagram of a construction scene object recognition device based on dual-trunk fusion according to an embodiment of the present application;

[0065] Fig.12 It is a schematic diagram of the structure of a device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0067] The embodiments of the present application can be applied to engineering industries such as hydropower, civil engineering, etc., to achieve safety monitoring of the construction process while saving labor costs, and can reduce the occurrence rate of safety responsibility accidents during the construction process, and can improve the accuracy of safety monitoring in construction scenarios.

[0068] Traditional technology usually deploys special inspection personnel at the construction site. The inspection personnel regularly inspect the construction site to monitor the distance between the construction personnel and the construction machinery. If the distance is less than the safe distance, an alarm message will be issued. Traditional technology requires a certain amount of manpower cost. In addition, since the inspection personnel cannot inspect the construction site in real time, the traditional monitoring method still has certain safety hazards, resulting in a low accuracy rate of safety monitoring.

[0069] In view of the technical problems that traditional technologies consume manpower costs and have low accuracy of safety monitoring, an embodiment of the present application provides a method for identifying objects, which specifically includes the following steps: collecting construction images; identifying the above construction images according to the target detection model to obtain identification results; the above identification results specifically include: the category of construction objects and the location information of the above construction object categories in the above construction images, and the above construction object categories specifically include at least one of the following categories: human category, construction machinery category, safety protection facility wearing category and safety sign category; according to the above identification results, determining whether the above construction image involves abnormal behavior;

[0070] The target detection model specifically includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network and a detection head network; the feature segmentation network specifically includes: n feature segmentation modules; the second backbone network specifically includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1;

[0071] The process of identifying the construction image according to the target detection model specifically includes:

[0072] The first backbone network determines n first image features corresponding to the construction image;

[0073] The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers;

[0074] The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result, and the channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result, and outputs the second image feature to the neck network;

[0075] The neck network performs a second fusion process on the second image feature to obtain a second fusion process result;

[0076] The detection head network determines the recognition result based on the second fusion processing result.

[0077] The embodiment of the present application monitors abnormal behaviors of construction images in real time based on image recognition, which can save the manpower cost of safety monitoring in construction scenes. In addition, the above real-time monitoring can improve the accuracy of safety monitoring, thereby reducing the incidence of safety liability accidents.

[0078] The i-th feature segmentation module of the embodiment of the present application splits the first image feature to obtain i seed image features with different numbers of channels, and outputs the j-th seed image feature to the connected j-th feature fusion module. Since different i-th feature fusion modules can process first image features of different levels, the n feature segmentation modules of the embodiment of the present application can play a role in aggregating sub-image features of multiple levels. For example, the n feature segmentation modules can provide n-level sub-image features to the first feature fusion module, the n feature segmentation modules can provide (n-1)-level sub-image features to the second feature fusion module, and so on.

[0079] First, the n feature segmentation modules of the feature segmentation network of the embodiment of the present application can play the role of aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In the forward propagation stage of the target detection model, richer image features can be provided, which can enhance the expression ability of the second image features output by the second backbone network.

[0080] Secondly, in the back propagation calculation process of the target detection model, by calculating the deviation between the converged multi-level sub-image feature information and the true value, more reliable gradient information can be transmitted to guide the parameter learning of the target detection model, thereby helping the first backbone network to extract more critical and more accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the recognition results of the construction image, and on this basis can improve the accuracy of safety monitoring in the construction scene.

[0081] Reference Figure 1 , showing a schematic diagram of an application environment of a method for construction scene object recognition based on dual-trunk fusion according to an embodiment of the present invention, wherein the image acquisition end 101 and the service end 102 can perform data exchange based on a wireless network or a wired network.

[0082] In practical applications, the image acquisition terminal 101 may be equipped with an image acquisition device having an image acquisition function, such as an image sensor. The image acquisition device can acquire construction images, and the image acquisition terminal 101 can send the construction images to the server 102 according to a preset time period. Of course, the image acquisition device can acquire construction videos, and the image acquisition terminal 101 can send the construction videos to the server 102 according to a preset time period; in this case, the server 102 can parse the construction images from the construction videos.

[0083] After receiving the construction image sent by the image acquisition terminal 101, the server 102 can use the method of the embodiment of the present application to process the construction image to determine whether the construction image involves abnormal behavior. If so, a prompt message can be output. For example, a prompt message can be sent to a terminal of a preset user. The above prompt message can indicate the occurrence of abnormal behavior, thereby enabling the preset user to take corresponding processing measures.

[0084] Examples of abnormal behavior may include: the distance between a person (construction worker) and a construction machine is less than a safe distance, or the person is not wearing safety protection equipment, or there is no safety sign set in the construction environment corresponding to the construction image.

[0085] In the case of abnormal behavior, the server 102 sends a prompt message to the preset user, which can make the preset execute the corresponding processing measures. The preset user can be the management user of the construction environment. For example, when the distance between a person and a construction machine is less than the safe distance, examples of processing measures may include: using the speaker closest to the construction site corresponding to the construction image to play a first alarm voice to prompt the construction personnel in the construction image to stay away from the construction machine. For another example, when a person does not wear safety protection equipment, examples of processing measures may include: using the speaker closest to the construction site corresponding to the construction image to play a second alarm voice to prompt the construction personnel in the construction image to wear a safety helmet. For another example, when no safety signs are set in the construction environment corresponding to the construction image, the management user of the construction environment can set safety signs such as safety slogans in the construction environment. It can be understood that those skilled in the art can adopt various processing measures according to actual application requirements, and the embodiments of the present application do not limit the specific processing measures.

[0086] Method Embodiment

[0087] refer to Figure 2 , shows a schematic flow chart of the steps of a construction scene object recognition method based on dual-trunk fusion according to an embodiment of the present application, the method specifically comprises the following steps:

[0088] Step 201: Collect construction images;

[0089] Step 202: According to the target detection model, the construction image is recognized to obtain a recognition result; the recognition result specifically includes: a construction object category and location information of the construction object category in the construction image; the construction object category specifically includes at least one of the following categories: a human category, a construction machinery category, a safety protection facility wearing category, and a safety sign category;

[0090] Step 203: Determine whether the construction image involves abnormal behavior based on the recognition result;

[0091] The target detection model specifically includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network and a detection head network; the feature segmentation network specifically includes: n feature segmentation modules; the second backbone network specifically includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n can be a positive integer greater than 1;

[0092] The process of identifying the construction image according to the target detection model in 202 specifically includes:

[0093] Step 221: The first backbone network determines n first image features corresponding to the construction image;

[0094] Step 222, the i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers;

[0095] Step 223: the i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result; the channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result, and outputs the second image feature to the neck network;

[0096] Step 224: The neck network performs a second fusion process on the second image feature to obtain a second fusion process result.

[0097] Step 225: The detection head network determines the recognition result according to the second fusion processing result.

[0098] Figure 2 The steps included in the method embodiment shown can be executed by the server, which can take advantage of its rich computing resources to quickly process the construction image, so as to monitor the abnormal behavior of the construction image in real time based on image recognition. Figure 2 The specific execution subject of the illustrated method embodiment is not limited.

[0099] Since the embodiments of the present application can monitor the construction site in real time based on the construction images, manual real-time monitoring is no longer required, thus saving the manpower cost spent on construction monitoring.

[0100] In step 201, the server may receive the construction images sent by the image acquisition terminal according to a preset time period. The image acquisition terminal may be set at the construction site to collect the construction images of the construction site in real time and send the construction images to the server according to the preset time period.

[0101] In step 202, the construction image is identified according to the target detection model, and the obtained identification result specifically includes: the construction object category and the position information of the construction object category in the construction image, and the construction object category specifically includes at least one of the following categories: human category, construction machinery category, safety protection facility wearing category and safety sign category.

[0102] Examples of the human category may include: construction workers. Examples of the construction machinery category may include: backhoe excavators, tower cranes, dump trucks, truck cranes, loaders, hooks, pump trucks, smooth rollers, concrete mixers, pile drivers, etc. The safety protection facility wearing category specifically includes: safety protection facility wearing category, or safety protection facility not wearing category. The safety sign category specifically includes: safety slogans, etc.

[0103] Reference Figure 3 , shows a schematic diagram of the structure of a target detection model of an embodiment of the present application, which specifically includes: a first backbone network 301, a feature segmentation network 302, a second backbone network 303, a neck network 304 and a detection head network 305 connected in sequence.

[0104] The first backbone network 301 is used to extract features from the input image to obtain n first image features. In the training phase, the input image may be a construction image sample. In the image recognition phase, the input image may be a real-time construction image.

[0105] The feature segmentation network 302 is used to perform convolution processing and segmentation processing on the n types of first image features, and output the multiple sub-image features obtained by segmentation to the second backbone network 303.

[0106] The second backbone network 303 is used to perform a first fusion process and a channel-to-pixel process on the multiple sub-image features to obtain a second image feature, and output the second image feature to the neck network 304 .

[0107] The neck network 304 is used to perform a second fusion process on the second image feature to obtain a second fusion process result.

[0108] The detection head network 305 is used to determine the recognition result according to the above second fusion processing result.

[0109] Reference Figure 4 , shows a schematic diagram of the structure of a first backbone network 301, a feature segmentation network 302 and a second backbone network 303 of an embodiment of the present application, wherein the first backbone network 301 outputs n first image features to n feature segmentation modules of the feature segmentation network 302.

[0110] The feature segmentation network 302 specifically includes: n feature segmentation modules, which are respectively recorded as: a first feature segmentation module 321, a second feature segmentation module 322... and an nth feature segmentation module 32n.

[0111] The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers.

[0112] The second backbone network 303 specifically includes: n feature fusion modules, which are respectively recorded as: a first feature fusion module 331, a second feature fusion module 332... and an nth feature fusion module 33n.

[0113] In a specific implementation, the process of performing convolution processing and segmentation processing on the i-th first image feature by the i-th feature segmentation module in the feature segmentation network specifically includes:

[0114] Step A1: the i-th feature segmentation module in the feature segmentation network performs convolution processing on the i-th first image feature to obtain image features with c channels;

[0115] Step A2: according to the channel dimension, the image features with c channels are divided into i seed image features with different numbers of channels; the number of channels of the jth seed image feature in the i seed image features is: t*2 (j-1) The value of t may be determined by those skilled in the art according to actual application requirements, and examples of the value of t may include: 32, 64, and the like.

[0116] The number of channels of the i seed image features is t*2 0 , t*2 1 , t*2 2 ..., t*2 (i-1) Among them, the sum of the number of channels corresponding to the i seed image features can be c, that is: t*2 0 +t*2 1 +t*2 2 ……+t*2 (i-1) =c.

[0117] When n is 5, the i-th feature segmentation module can output the j-th seed image feature to the connected j-th feature fusion module. Specific examples include:

[0118] For example, the segmentation processing result of the first feature segmentation module 321 includes: 1 m-channel first sub-image feature, and the 1 m-channel first sub-image feature is output to the first feature fusion module 331 .

[0119] For another example, the segmentation processing result of the second feature segmentation module 322 includes: one m-channel second sub-image feature and one 2m-channel third sub-image feature. One m-channel second sub-image feature is output to the first feature fusion module 331. One 2m-channel third sub-image feature is output to the second feature fusion module 332.

[0120] For another example, the segmentation processing result of the third feature segmentation module 323 (not shown in the figure) includes: one fourth sub-image feature of the m channel, one fifth sub-image feature of the 2m channel, and one sixth sub-image feature of the 4m channel. One fourth sub-image feature of the m channel is output to the first feature fusion module 331. One fifth sub-image feature of the 2m channel is output to the second feature fusion module 332. One sixth sub-image feature of the 4m channel is output to the third feature fusion module 333.

[0121] Alternatively, the segmentation processing result of the fourth feature segmentation module 324 (not shown in the figure) includes: one seventh sub-image feature of the m channel, one eighth sub-image feature of the 2m channel, one ninth sub-image feature of the 4m channel, and one tenth sub-image feature of the 8m channel. One seventh sub-image feature of the m channel is output to the first feature fusion module 331. One eighth sub-image feature of the 2m channel is output to the second feature fusion module 332. One ninth sub-image feature of the 4m channel is output to the third feature fusion module 333. One tenth sub-image feature of the 8m channel is output to the fourth feature fusion module 334.

[0122] Alternatively, the segmentation processing result of the fifth feature segmentation module 325 (not shown in the figure) includes: one eleventh sub-image feature of the m channel, one twelfth sub-image feature of the 2m channel, one thirteenth sub-image feature of the 4m channel, one fourteenth sub-image feature of the 8m channel, and one fifteenth sub-image feature of the 16m channel. One eleventh sub-image feature of the m channel is output to the first feature fusion module 331. One twelfth sub-image feature of the 2m channel is output to the second feature fusion module 332. One thirteenth sub-image feature of the 4m channel is output to the third feature fusion module 333. One fourteenth sub-image feature of the 8m channel is output to the fourth feature fusion module 334. One fifteenth sub-image feature of the 16m channel is output to the fifth feature fusion module 335.

[0123] The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result. The channel-to-pixel module in the second backbone network determines the second image feature based on the first fusion result and outputs the second image feature to the neck network.

[0124] In an optional implementation of the present application, the second backbone network may further include: a first convolution module connected in front of the feature fusion module;

[0125] Then the process of the i-th feature fusion module in the second backbone network performing the first fusion processing on at least one sub-image feature specifically includes:

[0126] Step B1, the i-th feature fusion module receives the n-i+1 seed image features and the convolution image features output by the first convolution module; the number of channels corresponding to the n-i+1 seed image features is the first number of channels, and the number of channels corresponding to the convolution image features is the second number of channels;

[0127] Step B2: the i-th feature fusion module performs interpolation processing on the n-i+1 seed image features, and the number of channels corresponding to the n-i+1 seed image features after the interpolation processing is the second number of channels;

[0128] Step B3: The i-th feature fusion module fuses the n-i+1 seed image features and the convolution image features after interpolation processing.

[0129] The first convolution module, the second convolution module, or the first convolution processing module of the embodiment of the present application all belong to the category of convolution structure. In one example, the above-mentioned convolution structure specifically includes: at least one convolution layer, at least one batch normalization layer, and at least one activation function. It can be understood that those skilled in the art can adopt the required convolution structure according to the actual application requirements, and the embodiment of the present application does not limit the specific convolution structure.

[0130] For example, the number of first channels corresponding to the n seed image features received by the first feature fusion module is t*2 0 Those skilled in the art may adopt corresponding interpolation processing technology according to actual application requirements, and the embodiments of the present application do not limit the specific interpolation processing technology.

[0131] The process of fusing the n-i+1 seed image features and the convolution image features after the interpolation processing by the i-th feature fusion module specifically includes: the i-th feature fusion module performs addition processing on the values ​​corresponding to the n-i+1 seed image features and the convolution image features after the interpolation processing. The sub-image features and the convolution image features may correspond to a multi-dimensional matrix, and the above-mentioned addition processing may be to add the element values ​​of the multi-dimensional matrix.

[0132] In another optional implementation of the present application, a first convolution module connected in front of the first feature fusion module is used to perform convolution processing on the construction image and output corresponding convolution image features to the first feature fusion module.

[0133] At least some of the feature fusion modules are connected to the channel-to-pixel module. Figure 5 , shows a schematic diagram of the structure of a channel-to-pixel module of an embodiment of the present application, the channel-to-pixel module specifically includes: a first convolution processing module 501, a segmentation processing module 502, M bottleneck modules 503, a connection module 504, and a second convolution processing module 505. M can be a positive integer greater than 1.

[0134] Among them, the first convolution processing module 501 is used to perform a first convolution processing on the first fusion result output by the feature fusion module to obtain a first convolution processing result.

[0135] The segmentation processing module 502 is used to segment the first convolution processing result into two parts with the same number of channels. The two parts may include: a first part of features and a second part of features. Assuming that the number of channels before segmentation is the third number of channels, the number of channels after segmentation may be the fourth number of channels, and the third number of channels may be twice the fourth number of channels.

[0136] After the first part of the features are processed by the M bottleneck modules 503, the obtained bottleneck processing features enter the connection module 504. The first convolution processing result and the second part of the features also enter the connection module 504. The connection module 504 is used to connect the first convolution processing result, the second part of the features and the bottleneck processing features to obtain the connection features.

[0137] The connection feature enters the second convolution processing module 505, and the second convolution processing module 505 restores the number of channels for the connection feature, for example, restores the number of channels from the fourth number of channels to the third number of channels.

[0138] Reference Figure 6 , shows a schematic structural diagram of a bottleneck module of an embodiment of the present application, and the bottleneck module specifically includes: a third convolution processing module 601 and a fourth convolution processing module 602.

[0139] The third convolution processing module 601 is used to reduce the number of channels of the input feature to half of the original number to obtain a second convolution processing result.

[0140] The fourth convolution processing module 603 is used to increase the number of channels of the second convolution processing result to twice the original number to obtain a third convolution processing result. The third convolution processing result has the same number of channels as the input feature. The third convolution processing result is fused with the input feature to obtain an output feature.

[0141] Reference Figure 7, shows a schematic diagram of the structure of the second backbone network of an embodiment of the present application, and the second backbone network specifically includes: a first convolution module A701, a first feature fusion module 702, a first convolution module B703, a second feature fusion module 704, a first channel to pixel module 705, a first convolution module C706, a third feature fusion module 707, a second channel to pixel module 708, a first convolution module D709, a fourth feature fusion module 710, a third channel to pixel module 711, a first convolution module E712, a fifth feature fusion module 713 and a fourth channel to pixel module 714.

[0142] Among them, the first convolution module A701 is used to perform convolution processing on the construction image and output the corresponding convolution image feature A to the first feature fusion module.

[0143] The first feature fusion module 702 is used to receive the five seed image features A from the first feature segmentation module and the convolution image feature A from the first convolution module A701, perform interpolation processing on the five seed image features A, and the number of channels corresponding to the five seed image features A after the interpolation processing is the second number of channels; and, perform fusion processing on the five seed image features A after the interpolation processing and the convolution image feature A, and assume that the first fusion result A is obtained.

[0144] The first convolution module B703 is used to perform convolution processing on the first fusion result A to obtain a convolution image feature B.

[0145] The second feature fusion module 704 is used to receive the four seed image features B from the second feature segmentation module and the convolution image feature B from the first convolution module B703, perform interpolation processing on the four seed image features B, and the number of channels corresponding to the four seed image features B after the interpolation processing is the second number of channels; and, perform fusion processing on the four seed image features B after the interpolation processing and the convolution image feature B, and assume that the first fusion result B is obtained.

[0146] The first channel to pixel module 705 is used to determine the intermediate image features according to the first fusion result B mentioned above.

[0147] The first convolution module C706 is used to perform convolution processing on the intermediate image features to obtain convolution image features C.

[0148] The third feature fusion module 707 is used to receive the three seed image features C from the third feature segmentation module and the convolution image feature C from the first convolution module C706, perform interpolation processing on the three seed image features C, and the number of channels corresponding to the three seed image features C after the interpolation processing is the second number of channels; and, perform fusion processing on the three seed image features C after the interpolation processing and the convolution image feature C, and assume that the first fusion result C is obtained.

[0149] The second channel to pixel module 708 is used to determine the second image feature A according to the first fusion result C, and output the second image feature A to the neck network.

[0150] The first convolution module D709 is used to perform convolution processing on the second image feature A to obtain a convolution image feature D.

[0151] The fourth feature fusion module 710 is used to receive the 2-seed image features D from the fourth feature segmentation module and the convolution image features D from the first convolution module D709, perform interpolation processing on the 3-seed image features D, and the number of channels corresponding to the 2-seed image features D after the interpolation processing is the second number of channels; and, perform fusion processing on the 2-seed image features D after the interpolation processing and the convolution image features D, and assume that the first fusion result D is obtained.

[0152] The third channel-to-pixel module 711 is used to determine the second image feature B according to the first fusion result D, and output the second image feature B to the neck network.

[0153] The first convolution module E712 is used to perform convolution processing on the second image feature B to obtain a convolution image feature E.

[0154] The fifth feature fusion module 713 is used to receive the 1 seed image feature E from the fifth feature segmentation module and the convolution image feature E from the first convolution module E706, perform interpolation processing on the 1 seed image feature E, and the number of channels corresponding to the 1 seed image feature E after the interpolation processing is the second number of channels; and, perform fusion processing on the 1 seed image feature E after the interpolation processing and the convolution image feature E, and assume that the first fusion result E is obtained.

[0155] The fourth channel-to-pixel module 714 is used to determine the second image feature C according to the first fusion result E, and output the second image feature C to the neck network.

[0156] In a specific implementation, the first backbone network may specifically include: a second convolution module and n-1 processing units connected in sequence; the processing unit specifically includes: a third convolution module and a channel to pixel module;

[0157] Among them, the second convolution module is connected to the first feature segmentation module, and is used to output the first first image feature to the first feature segmentation module; the channel-to-pixel modules contained in the n-1 processing units are respectively connected to the corresponding n-1 feature segmentation modules, and are used to output the first image feature to the corresponding feature segmentation module.

[0158] Reference Figure 8, shows a schematic diagram of the structure of the first backbone network of an embodiment of the present application, wherein the first backbone network specifically includes: a second convolution module 801, a third convolution module A802, a fifth channel to pixel module 803, a third convolution module B804, a sixth channel to pixel module 805, a third convolution module C806, a seventh channel to pixel module 807, a third convolution module D808 and an eighth channel to pixel module 809 connected in sequence. The structure of the fifth channel to pixel module 803 can be found in Figure 5 , I will not go into details here.

[0159] The second convolution module 801 is used to perform convolution processing on the construction image to obtain the first image feature A, and send the first image feature A to the first feature segmentation module.

[0160] The third convolution module A802 is used to perform convolution processing on the first image feature A to obtain a first convolution result.

[0161] The fifth channel to pixel module 803 is used to determine the first image feature B according to the first convolution result, and send the first image feature B to the second feature segmentation module.

[0162] The third convolution module B804 is used to perform convolution processing on the first image feature B to obtain a second convolution result.

[0163] The sixth channel to pixel module 805 is used to determine the first image feature C according to the second convolution result, and send the first image feature C to the third feature segmentation module.

[0164] The third convolution module C806 is used to perform convolution processing on the first image feature C to obtain a third convolution result.

[0165] The seventh channel-to-pixel module 807 is used to determine the first image feature D according to the third convolution result, and send the first image feature D to the fourth feature segmentation module.

[0166] The third convolution module D808 is used to perform convolution processing on the first image feature D to obtain a fourth convolution result.

[0167] The seventh channel to pixel module 807 is used to determine the first image feature E according to the fourth convolution result, and send the first image feature E to the fifth feature segmentation module.

[0168] In step 224, the neck network is used to perform a second fusion process on the second image feature to obtain a second fusion process result.

[0169] Reference Fig. 9, shows a schematic diagram of the structure of a neck network of an embodiment of the present application, which specifically includes: a spatial pyramid pooling module 901, a first upsampling module 902, a first connection module 903, a ninth channel to pixel module 904, a second upsampling module 905, a second connection module 906, a tenth channel to pixel module 907, a fourth convolution module 908, a third connection module 909, an eleventh channel to pixel module 910, a fifth convolution module 911, a fourth connection module 912 and a twelfth channel to pixel module 913.

[0170] The spatial pyramid pooling module 901 is used to receive the second image feature C and perform spatial pyramid pooling processing on the second image feature C. The spatial pyramid pooling processing may include: a convolution operation and a maximum pooling operation, which can achieve deep fusion of the second image feature C.

[0171] Reference Fig.10 , shows a schematic diagram of the structure of a spatial pyramid pooling module of an embodiment of the present application, and the spatial pyramid pooling module specifically includes: a sixth convolution module 1001, p maximum pooling modules 1002, a fifth connection module 1003 and a seventh convolution module 1004. p can be a positive integer greater than 1, and the value of p in the figure is 3.

[0172] Among them, the sixth convolution module 1001 is used to perform convolution processing on the input second image feature C to obtain a fifth convolution result.

[0173] The p maximum pooling modules 1002 are respectively used to perform maximum pooling processing on the input image features to obtain corresponding p maximum pooling processing results.

[0174] The fifth convolution result and the p maximum pooling processing results are respectively input into the fifth connection module 1003, and the fifth connection module 1003 performs deep fusion on the fifth convolution result and the p maximum pooling processing results to obtain a corresponding deep fusion result.

[0175] The seventh convolution module 1004 is used to perform convolution processing on the depth fusion result to obtain a sixth convolution result. The sixth convolution result is provided to the first upsampling module 902.

[0176] The first up-sampling module 902 is used to perform a first up-sampling process on the sixth convolution result to obtain a first up-sampling process result.

[0177] The first connection module 903 is used to perform connection processing on the second image feature B and the first upsampling processing result to obtain a first connection result.

[0178] The ninth channel-to-pixel module 904 is configured to determine a first processing result according to the first connection result.

[0179] The second up-sampling module 905 is used to perform a second up-sampling process on the first processing result to obtain a second up-sampling processing result.

[0180] The second connection module 906 is used to connect the second upsampling processing result with the second image feature A to obtain a second connection result. The connection processing may be a splicing processing of two image features.

[0181] The tenth channel-to-pixel module 907 is used to determine a second fusion processing result A according to the second connection result. The second fusion processing result A is output to the detection head network.

[0182] The fourth convolution module 908 is used to perform convolution processing on the second fusion processing result A to obtain a sixth convolution result.

[0183] The third connection module 909 is used to connect the sixth convolution result and the first processing result to obtain a third connection result.

[0184] The eleventh channel-to-pixel module 910 is used to determine a second fusion processing result B according to the third connection result. The second fusion processing result B is output to the detection head network.

[0185] The fifth convolution module 911 is used to perform convolution processing on the second fusion result B to obtain a seventh convolution result.

[0186] The fourth connection module 912 is used to connect the sixth convolution result and the seventh convolution result to obtain a fourth connection result.

[0187] The twelfth channel-to-pixel module 913 is used to determine a second fusion processing result C according to the fourth connection result. The second fusion processing result C is output to the detection head network.

[0188] In step 225, the detection head network may determine a recognition result based on the second fusion processing result.

[0189] The detection head network may include: at least one detection module. The detection module may be used to perform classification and regression calculations respectively based on the second fusion processing result using a convolution module and a convolution layer to obtain the construction object category in the construction image and the position information of the construction object category in the construction image.

[0190] In summary, the n feature segmentation modules of the feature segmentation network of the embodiment of the present application can play the role of aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In this way, in the forward propagation stage of the target detection model, richer image features can be provided, which can enhance the expression ability of the second image features output by the second backbone network.

[0191] Secondly, in the back propagation calculation process of the target detection model, by calculating the deviation between the converged multi-level sub-image feature information and the true value, more reliable gradient information can be transmitted to guide the parameter learning of the target detection model, thereby helping the first backbone network to extract more critical and more accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the recognition results of the construction image, and on this basis can improve the accuracy of safety monitoring in the construction scene.

[0192] The target detection model of the embodiment of the present application was experimentally analyzed using the construction scene target detection benchmark data set. Among them, the 19404 training set images that have been annotated in the construction scene target detection benchmark data set are used as the training data set, and the 4000 validation set images are used as the test data set, and AP (average precision) 50 and AP75 are defined as evaluation indicators. Among them, AP50 represents the average precision when the intersection-over-union ratio threshold is 0.5. AP75 represents the average precision when the intersection-over-union ratio threshold is 0.75.

[0193] The experimental results show that the detection accuracy of the embodiment of the present application on the construction scene target detection benchmark dataset is better than the existing technology. Taking the AP50 index as an example, compared with the existing technology, the detection accuracy of the first version, second version and third version of the target detection model increased by 13.6%, 8.7% and 4.3% respectively, verifying the effectiveness of the method proposed in this application. Among them, the first version, second version and third version of the target detection model correspond to different parameter quantities.

[0194] In an embodiment of the present application, the training process of the target detection model may include: forward propagation and back propagation.

[0195] Among them, forward propagation can calculate the prediction information of the recognition result in sequence from the embedding layer to the processing layer according to the parameters of the target detection model. The prediction information is used to determine the loss information

[0196] Backward Propagation can calculate and update the parameters of the target detection model in sequence from the output layer to the input layer according to the loss information. The target detection model usually adopts the structure of a neural network, and the parameters of the target detection model may include: parameters such as the weights of the neural network. Among them, during the back propagation process, the gradient information of the parameters of the target detection model can be determined, and the parameters of the target detection model can be updated using the gradient information. For example, back propagation can calculate and store the gradient information of the parameters of the target detection model in sequence from the processing layer to the embedding layer according to the chain rule in calculus.

[0197] In one embodiment, the training process of the target detection model includes:

[0198] Step C1, inputting the construction image sample into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: the prediction frame information corresponding to the construction object category; the real frame information corresponding to the construction image sample;

[0199] Step C2: determining loss information corresponding to the real frame information and the predicted frame information according to a minimum point distance function based on a horizontal rectangle;

[0200] Step C3 updates the parameters of the target detection model according to the loss information.

[0201] The process of obtaining the above construction image samples specifically includes:

[0202] Step D1, randomly reading four original construction images from the construction image set;

[0203] Step D2, performing random enhancement operations on the four original construction images respectively to obtain four enhanced construction images;

[0204] Step D3, fusing the four enhanced construction images into a fused image;

[0205] Step D4: annotate a pair of fused images with real frame information to obtain a training sample.

[0206] The random enhancement operation specifically includes at least one of the following operations: a flip operation, a random scaling operation, a random tone transformation operation, and a random perspective transformation operation.

[0207] Step D3 is a process of fusing the four enhanced construction images into a fused image, specifically including: placing the four enhanced construction images on an intermediate image according to the offsets of [0,0], [0,243], [320,320], [320,0]; cropping the part of the intermediate image that exceeds the size range, and shrinking the range of the real frame to prevent crossing the boundary, and finally obtaining a fused image. The size range specifically includes: the coordinate range corresponding to [0,0] to [320,320].

[0208] Referring to formula (1), a process of determining the loss information corresponding to the real box information and the predicted box information according to the minimum point distance function based on the horizontal rectangle is shown.

[0209]

[0210] Among them, L MPDIoU Represents loss information; A represents the real box; B represents the predicted box; Indicates the pixel coordinates of the upper left corner of the real box; Indicates the pixel coordinates of the lower right corner of the real box; Indicates the pixel coordinate of the upper left corner of the prediction box, represents the pixel coordinate of the lower right corner of the prediction box; w and h represent the width and height of the construction image sample respectively; d 1 Represents the first distance between the upper left corner pixel coordinates of the real box and the predicted box; d 2 Represents the second distance between the lower right corner pixel coordinates of the true box and the predicted box.

[0211] The minimum point distance function based on the horizontal rectangle helps ensure that the predicted box is geometrically closer to the true box, especially when the predicted box has the same aspect ratio as the true box but different width and height values. The metric optimization is performed by calculating the distance between the upper left corner and the lower right corner of the predicted box and the true box. Since the minimum point distance function is more sensitive to the position error of the predicted box, it may help improve the recognition accuracy of the target detection model.

[0212] In step 203, it may be determined whether the construction image involves abnormal behavior based on the recognition result.

[0213] Examples of abnormal behavior may include: the distance between a person (construction worker) and a construction machine is less than a safe distance, or the person is not wearing safety protection equipment, or there is no safety sign set in the construction environment corresponding to the construction image.

[0214] In one implementation of the present application, the safety protection facility wearing category includes: a safety protection facility not wearing category;

[0215] The process of determining whether the construction image involves abnormal behavior based on the recognition result specifically includes:

[0216] Step C1: when the recognition result includes the category of "safety protection equipment not being worn", determining that the construction image involves abnormal behavior; or

[0217] Step C2: Determine the distance information between the human category and the construction machinery category according to the position information corresponding to the human category and the construction machinery category respectively contained in the recognition result; and determine whether the construction image involves abnormal behavior according to the distance information.

[0218] In one example, the process of determining the distance information between the human category and the construction machinery category specifically includes: first, determining the image distance between the human category and the construction machinery category based on the position information corresponding to the human category and the construction machinery category respectively contained in the recognition result; then, converting the image distance into the actual distance based on the scale factor of the construction image; the actual distance can be used as the distance information between the human category and the construction machinery category.

[0219] When the distance value corresponding to the above distance information is less than the safe distance, it can be determined whether the construction image involves abnormal behavior. When the above construction image involves abnormal behavior, the embodiment of the present application can send corresponding prompt information to the preset user so that the preset user can handle the abnormal behavior.

[0220] In summary, the object recognition method of the embodiment of the present application monitors abnormal behaviors of construction images in real time based on image recognition, which can save the manpower cost of safety monitoring in construction scenes. In addition, the above-mentioned real-time monitoring can improve the accuracy of safety monitoring in construction scenes, thereby reducing the incidence of safety liability accidents.

[0221] First, the n feature segmentation modules of the feature segmentation network of the embodiment of the present application can play the role of aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In this way, in the forward propagation stage of the target detection model, richer image features can be provided, which can enhance the expression ability of the second image features output by the second backbone network.

[0222] Secondly, in the back propagation calculation process of the target detection model, by calculating the deviation between the converged multi-level sub-image feature information and the true value, more reliable gradient information can be transmitted to guide the parameter learning of the target detection model, thereby helping the first backbone network to extract more critical and more accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the recognition results of the construction image, and on this basis can improve the accuracy of safety monitoring in the construction scene.

[0223] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0224] Based on the above embodiment, this embodiment further provides a device for identifying an object. Fig.11 The device may specifically include: a collection module 1101, an object recognition module 1102 and an abnormality determination module 1103.

[0225] Wherein, the acquisition module 1101 is used to acquire construction images;

[0226] The object recognition module 1102 is used to recognize the construction image according to the target detection model to obtain a recognition result; the recognition result includes: the construction object category and the location information of the construction object category in the construction image, and the construction object category includes at least one of the following categories: human category, construction machinery category, safety protection facility wearing category and safety sign category;

[0227] An abnormality determination module 1103, used to determine whether the construction image involves abnormal behavior according to the recognition result;

[0228] The target detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1;

[0229] The object recognition module 1102 specifically includes:

[0230] A first image feature determination module 1121, configured to determine n first image features corresponding to the construction image using a first backbone network;

[0231] The convolution segmentation module 1122 is used to use the i-th feature segmentation module in the feature segmentation network to perform convolution processing and segmentation processing on the i-th first image feature, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers;

[0232] A first fusion module 1123 is used to perform a first fusion process on at least one sub-image feature using the i-th feature fusion module in the second backbone network to obtain a first fusion result, and to determine a second image feature according to the first fusion result using the channel-to-pixel module in the second backbone network, and output the second image feature to the neck network;

[0233] A second fusion module 1124 is used to perform a second fusion process on the second image feature by using a neck network to obtain a second fusion process result;

[0234] The recognition result determination module 1125 is used to determine the recognition result according to the second fusion processing result by using the detection head network.

[0235] Optionally, the safety protection facility wearing category includes: safety protection facility not wearing category;

[0236] The abnormality determination module comprises:

[0237] A first abnormality determination module is used to determine that the construction image involves abnormal behavior when the recognition result includes the category of not wearing safety protection equipment; or

[0238] The second abnormality determination module is used to determine the distance information between the human category and the construction machinery category according to the position information corresponding to the human category and the construction machinery category respectively contained in the recognition result; and determine whether the construction image involves abnormal behavior according to the distance information.

[0239] Optionally, the convolution segmentation module includes:

[0240] A convolution module is used to perform convolution processing on the i-th first image feature using the i-th feature segmentation module in the feature segmentation network to obtain image features with c channels;

[0241] The segmentation module is used to segment the image features with c channels into i seed image features with different numbers of channels according to the channel dimension; the number of channels of the jth seed image feature in the i seed image features is: t*2 (j-1) .

[0242] Optionally, the second backbone network further includes: a first convolution module connected in front of the feature fusion module;

[0243] The first fusion module includes:

[0244] A receiving module, used to receive n-i+1 seed image features and convolution image features output by the first convolution module; the number of channels corresponding to the n-i+1 seed image features is the first number of channels, and the number of channels corresponding to the convolution image features is the second number of channels;

[0245] An interpolation module is used to perform interpolation processing on n-i+1 seed image features using the i-th feature fusion module, and the number of channels corresponding to the n-i+1 seed image features after the interpolation processing is the second number of channels;

[0246] The fusion processing module is used to use the i-th feature fusion module to fuse the n-i+1 seed image features and the convolution image features after interpolation processing.

[0247] Optionally, the first convolution module connected in front of the first feature fusion module is used to perform convolution processing on the construction image and output corresponding convolution image features to the first feature fusion module.

[0248] Optionally, the first backbone network comprises: a second convolution module and n-1 processing units connected in sequence; the processing unit comprises: a third convolution module and a channel to pixel module;

[0249] Among them, the second convolution module is connected to the first feature segmentation module, and is used to output the first first image feature to the first feature segmentation module; the channel-to-pixel modules contained in the n-1 processing units are respectively connected to the corresponding n-1 feature segmentation modules, and are used to output the first image feature to the corresponding feature segmentation module.

[0250] Optionally, the training process of the target detection model includes:

[0251] Input the construction image sample into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: prediction frame information corresponding to the construction object category; and real frame information corresponding to the construction image sample;

[0252] Determining loss information corresponding to the real frame information and the predicted frame information according to a minimum point distance function based on a horizontal rectangle;

[0253] According to the loss information, the parameters of the target detection model are updated.

[0254] In summary, the device for identifying objects in the embodiment of the present application monitors abnormal behaviors of construction images in real time based on image recognition, which can save the manpower cost of safety monitoring in construction scenes. In addition, the above-mentioned real-time monitoring can improve the accuracy of safety monitoring in construction scenes, thereby reducing the incidence of safety liability accidents.

[0255] First, the n feature segmentation modules of the feature segmentation network of the embodiment of the present application can play the role of aggregating sub-image features of multiple levels. On this basis, the n feature fusion modules of the second backbone network can effectively fuse sub-image features of multiple levels. In this way, in the forward propagation stage of the target detection model, richer image features can be provided, which can enhance the expression ability of the second image features output by the second backbone network.

[0256] Secondly, in the back propagation calculation process of the target detection model, by calculating the deviation between the converged multi-level sub-image feature information and the true value, more reliable gradient information can be transmitted to guide the parameter learning of the target detection model, thereby helping the first backbone network to extract more critical and more accurate image features. Therefore, the embodiment of the present application can further improve the accuracy of the recognition results of the construction image, and on this basis can improve the accuracy of safety monitoring in the construction scene.

[0257] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0258] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0259] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform a desired configuration, and the device may include: electronic devices such as terminal devices and servers (clusters). Fig.12 An exemplary apparatus 1300 that can be used to implement various embodiments described in this application is schematically shown.

[0260] For one embodiment, Fig.12 An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, an NVM (non-volatile memory) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0261] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a terminal device, server (cluster), and other devices described in the embodiments of the present application.

[0262] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or non-volatile memory / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0263] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0264] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0265] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, such as a suitable DRAM (Dynamic Random Access Memory). In some embodiments, the memory 1306 may include a double data rate type four synchronous dynamic random access memory.

[0266] For one embodiment, the control module 1304 may include one or more input / output controllers to provide an interface to the non-volatile memory / storage device 1308 and the input / output device(s) 1310 .

[0267] For example, non-volatile memory / storage 1308 may be used to store data and / or instructions 1314. Non-volatile memory / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives, one or more optical disk drives, and / or one or more digital versatile optical disk drives).

[0268] The non-volatile memory / storage device 1308 may include storage resources that are physically part of the device on which the apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, the non-volatile memory / storage device 1308 may be accessed via (one or more) input / output devices 1310 over a network.

[0269] (One or more) input / output devices 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, an audio component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2nd Generation Wireless Telephone Technology), 3G (3rd Generation Wireless Telephone Technology), 4G (4th Generation Wireless Telephone Technology), 5G (5th Generation Wireless Telephone Technology), etc., or a combination thereof for wireless communication.

[0270] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip.

[0271] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a touch screen device, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, a dedicated integrated circuit, and a speaker.

[0272] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or non-volatile memory / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0273] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0274] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0275] The present application embodiment is described with reference to the flowchart and / or block diagram of the method, terminal device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0276] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0277] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0278] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0279] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0280] The above is a detailed introduction to a method and device for identifying an object, an electronic device and a machine-readable medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application sets. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A construction scene object recognition method based on dual-trunk fusion, characterized in that: The method comprises: Collect construction images; According to the target detection model, the construction image is recognized to obtain a recognition result; the recognition result includes: a construction object category and position information of the construction object category in the construction image, and the construction object category includes at least one of the following categories: a human category, a construction machinery category, a safety protection facility wearing category, and a safety sign category; Determining whether the construction image involves abnormal behavior according to the recognition result; The target detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1; The identifying the construction image according to the target detection model includes: The first backbone network determines n first image features corresponding to the construction image; The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers; The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature to obtain a first fusion result, and the channel-to-pixel module in the second backbone network determines a second image feature according to the first fusion result, and outputs the second image feature to the neck network; The neck network performs a second fusion process on the second image feature to obtain a second fusion process result; The detection head network determines the recognition result according to the second fusion processing result; The first backbone network comprises: a second convolution module and n-1 processing units connected in sequence; the processing unit comprises: a third convolution module and a channel-to-pixel module; wherein the second convolution module is connected to the first feature segmentation module, and is used to output the first first image feature to the first feature segmentation module; the channel-to-pixel modules included in the n-1 processing units are respectively connected to the corresponding n-1 feature segmentation modules, and are used to output the first image feature to the corresponding feature segmentation module; The second backbone network includes: a first convolution module A, a first feature fusion module, a first convolution module B, a second feature fusion module, a first channel to pixel module, a first convolution module C, a third feature fusion module, a second channel to pixel module, a first convolution module D, a fourth feature fusion module, a third channel to pixel module, a first convolution module E, a fifth feature fusion module and a fourth channel to pixel module, which are connected in sequence.

2. The method according to claim 1, characterized in that The categories of wearing safety protection facilities include: categories of not wearing safety protection facilities; Determining whether the construction image involves abnormal behavior according to the recognition result includes: In the case where the recognition result includes the category of not wearing safety protection equipment, determining that the construction image involves abnormal behavior; or According to the position information respectively corresponding to the human category and the construction machinery category included in the recognition result, the distance information between the human category and the construction machinery category is determined; according to the distance information, it is determined whether the construction image involves abnormal behavior.

3. The method according to claim 1, characterized in that The i-th feature segmentation module in the feature segmentation network performs convolution processing and segmentation processing on the i-th first image feature, including: The i-th feature segmentation module in the feature segmentation network performs convolution processing on the i-th first image feature to obtain image features with c channels; According to the channel dimension, the image features with c channels are divided into i seed image features with different numbers of channels; the number of channels of the jth seed image feature in the i seed image features is: t*2 (j-1) .

4. The method according to claim 1, characterized in that The second backbone network further includes: a first convolution module connected in front of the feature fusion module; The i-th feature fusion module in the second backbone network performs a first fusion process on at least one sub-image feature, including: The i-th feature fusion module receives the n-i+1 seed image features and the convolution image features output by the first convolution module; the number of channels corresponding to the n-i+1 seed image features is the first number of channels, and the number of channels corresponding to the convolution image features is the second number of channels; The i-th feature fusion module performs interpolation processing on the n-i+1 seed image features, and the number of channels corresponding to the n-i+1 seed image features after the interpolation processing is the second channel number; The i-th feature fusion module fuses the n-i+1 seed image features and convolution image features after interpolation.

5. The method according to claim 4, characterized in that The first convolution module connected in front of the first feature fusion module is used to perform convolution processing on the construction image and output the corresponding convolution image features to the first feature fusion module.

6. The method according to claim 1, characterized in that The n feature segmentation modules include: a first feature segmentation module, a second feature segmentation module, a third feature segmentation module, a fourth feature segmentation module and a fifth feature segmentation module; the n feature fusion modules include: a first feature fusion module, a second feature fusion module, a third feature fusion module, a fourth feature fusion module and a fifth feature fusion module; The segmentation processing result of the first feature segmentation module includes: 1 m-channel first sub-image feature, and 1 m-channel first sub-image feature is output to the first feature fusion module; The segmentation processing result of the second feature segmentation module includes: one m-channel second sub-image feature and one 2m-channel third sub-image feature; one m-channel second sub-image feature is output to the first feature fusion module; one 2m-channel third sub-image feature is output to the second feature fusion module; The segmentation processing result of the third feature segmentation module includes: one fourth sub-image feature of the m channel, one fifth sub-image feature of the 2m channel and one sixth sub-image feature of the 4m channel; one fourth sub-image feature of the m channel is output to the first feature fusion module; one fifth sub-image feature of the 2m channel is output to the second feature fusion module; one sixth sub-image feature of the 4m channel is output to the third feature fusion module; The segmentation processing result of the fourth feature segmentation module includes: one seventh sub-image feature of the m channel, one eighth sub-image feature of the 2m channel, one ninth sub-image feature of the 4m channel, and one tenth sub-image feature of the 8m channel; one seventh sub-image feature of the m channel is output to the first feature fusion module 331; one eighth sub-image feature of the 2m channel is output to the second feature fusion module; one ninth sub-image feature of the 4m channel is output to the third feature fusion module 333; and one tenth sub-image feature of the 8m channel is output to the fourth feature fusion module 334; The segmentation processing results of the fifth feature segmentation module include: one eleventh sub-image feature of the m channel, one twelfth sub-image feature of the 2m channel, one thirteenth sub-image feature of the 4m channel, one fourteenth sub-image feature of the 8m channel and one fifteenth sub-image feature of the 16m channel; one eleventh sub-image feature of the m channel is output to the first feature fusion module; one twelfth sub-image feature of the 2m channel is output to the second feature fusion module; one thirteenth sub-image feature of the 4m channel is output to the third feature fusion module; one fourteenth sub-image feature of the 8m channel is output to the fourth feature fusion module; and one fifteenth sub-image feature of the 16m channel is output to the fifth feature fusion module.

7. The method according to claim 1, characterized in that The first convolution module A is used to perform convolution processing on the construction image and output the corresponding convolution image feature A to the first feature fusion module; The first feature fusion module is used to receive the five seed image features A from the first feature segmentation module and the convolution image feature A from the first convolution module A, perform interpolation processing on the five seed image features A, and the number of channels corresponding to the five seed image features A after the interpolation processing is the second number of channels; and perform fusion processing on the five seed image features A after the interpolation processing and the convolution image feature A, and assume that a first fusion result A is obtained; The first convolution module B is used to perform convolution processing on the first fusion result A to obtain a convolution image feature B; The second feature fusion module is used to receive the four sub-image features B from the second feature segmentation module and the convolution image feature B from the first convolution module B, perform interpolation processing on the four sub-image features B, and the number of channels corresponding to the four sub-image features B after the interpolation processing is the second number of channels; and, perform fusion processing on the four sub-image features B after the interpolation processing and the convolution image feature B, and assume that a first fusion result B is obtained; The first channel to pixel module is used to determine the intermediate image features according to the first fusion result B; The first convolution module C is used to perform convolution processing on the intermediate image features to obtain convolution image features C; The third feature fusion module is used to receive the three sub-image features C from the third feature segmentation module and the convolution image features C from the first convolution module C, perform interpolation processing on the three sub-image features C, and the number of channels corresponding to the three sub-image features C after the interpolation processing is the second number of channels; and perform fusion processing on the three sub-image features C after the interpolation processing and the convolution image features C, and assume that a first fusion result C is obtained; The second channel-to-pixel module is used to determine a second image feature A according to the first fusion result C, and output the second image feature A to the neck network; The first convolution module D is used to perform convolution processing on the second image feature A to obtain a convolution image feature D; The fourth feature fusion module is used to receive the two sub-image features D from the fourth feature segmentation module and the convolution image feature D from the first convolution module D, perform interpolation processing on the three sub-image features D, and the number of channels corresponding to the two sub-image features D after the interpolation processing is the second number of channels; and, perform fusion processing on the two sub-image features D after the interpolation processing and the convolution image feature D, and assume that a first fusion result D is obtained; The third channel-to-pixel module is used to determine the second image feature B according to the first fusion result D, and output the second image feature B to the neck network; The first convolution module E is used to perform convolution processing on the second image feature B to obtain a convolution image feature E; The fifth feature fusion module is used to receive the 1 seed image feature E from the fifth feature segmentation module and the convolution image feature E from the first convolution module E, perform interpolation processing on the 1 seed image feature E, and the number of channels corresponding to the 1 seed image feature E after the interpolation processing is the second number of channels; and, perform fusion processing on the 1 seed image feature E after the interpolation processing and the convolution image feature E, and assume that a first fusion result E is obtained; The fourth channel-to-pixel module is used to determine the second image feature C according to the first fusion result E, and output the second image feature C to the neck network.

8. The method according to any one of claims 1 to 7, characterized in that: The training process of the target detection model includes: Input the construction image sample into the target detection model, and the target detection model outputs the prediction result corresponding to the construction image sample; the prediction result includes: prediction frame information corresponding to the construction object category; and real frame information corresponding to the construction image sample; Determining loss information corresponding to the real frame information and the predicted frame information according to a minimum point distance function based on a horizontal rectangle; According to the loss information, the parameters of the target detection model are updated.

9. A device for identifying an object, characterized in that: The device comprises: A collection module, used for collecting construction images; An object recognition module is used to recognize the construction image according to the target detection model to obtain a recognition result; the recognition result includes: a construction object category and position information of the construction object category in the construction image, and the construction object category includes at least one of the following categories: a human category, a construction machinery category, a safety protection facility wearing category, and a safety sign category; an abnormality determination module, used to determine whether the construction image involves abnormal behavior based on the recognition result; The target detection model includes: a first backbone network, a feature segmentation network, a second backbone network, a neck network and a detection head network; the feature segmentation network includes: n feature segmentation modules; the second backbone network includes: n feature fusion modules; at least some of the feature fusion modules are connected to a channel to pixel module; n is a positive integer greater than 1; The object recognition module comprises: A first image feature determination module, used to determine n first image features corresponding to the construction image using a first backbone network; The convolution segmentation module is used to perform convolution processing and segmentation processing on the i-th first image feature using the i-th feature segmentation module in the feature segmentation network, and the obtained segmentation processing result includes: i seed image features, and outputs the j-th seed image features to the connected j-th feature fusion module; i and j are positive integers, i is not greater than n, and j is not greater than i; when i is greater than 1, the i seed image features output by the same i-th feature segmentation module are sub-image features with different channel numbers; one feature fusion module corresponds to one channel number, and different feature fusion modules correspond to different channel numbers; a first fusion module, configured to perform a first fusion process on at least one sub-image feature using the i-th feature fusion module in the second backbone network to obtain a first fusion result, and to determine a second image feature based on the first fusion result using the channel-to-pixel module in the second backbone network, and output the second image feature to the neck network; a second fusion module, configured to perform a second fusion process on the second image feature by using a neck network to obtain a second fusion process result; A recognition result determination module, used to determine the recognition result according to the second fusion processing result by using the detection head network; The first backbone network comprises: a second convolution module and n-1 processing units connected in sequence; the processing unit comprises: a third convolution module and a channel-to-pixel module; wherein the second convolution module is connected to the first feature segmentation module, and is used to output the first first image feature to the first feature segmentation module; the channel-to-pixel modules included in the n-1 processing units are respectively connected to the corresponding n-1 feature segmentation modules, and are used to output the first image feature to the corresponding feature segmentation module; The second backbone network includes: a first convolution module A, a first feature fusion module, a first convolution module B, a second feature fusion module, a first channel to pixel module, a first convolution module C, a third feature fusion module, a second channel to pixel module, a first convolution module D, a fourth feature fusion module, a third channel to pixel module, a first convolution module E, a fifth feature fusion module and a fourth channel to pixel module, which are connected in sequence.

10. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the method according to any one of claims 1 to 8.

11. A machine-readable medium having executable codes stored thereon, which, when executed, causes a processor to perform the method according to any one of claims 1 to 8.

Images