Neural network image recognition method and multistage feedback construction monitoring system

Abstract

The invention discloses a neural network image recognition method and a multistage feedback construction monitoring system in the technical field of image monitoring, and the method comprises the steps: fusing a first algorithm and a second algorithm to construct a small target detection neural network, the first algorithm being a target detection algorithm, and the second algorithm being a small target detection algorithm; video images are collected and input into the small target detection neural network, the construction state in the video images is detected, and the video images are video images of the construction area; judging whether the construction state in the video image is qualified or not to obtain a judgment result; and synthesizing the video image and the judgment result into communication data, and sending the communication data to a receiving terminal. According to the invention, a neural network image recognition method, a multistage feedback construction monitoring system and a built-in wide-angle camera are adopted to collect on-site videos, convolution feature extraction is carried out on the images, whether the constructors have potential safety hazards or not is recognized, the on-site management efficiency and the accident prevention capability can be improved, and the construction behaviors can be standardized; and the safety risk of a construction area is effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image monitoring, in particular to a neural network image recognition method and a multi-level feedback construction monitoring system. BACKGROUND

[0002] The construction industry has always been a high-incidence area of safety accidents due to its labor-intensive industry characteristics and the complexity of the construction environment. It is particularly important to strengthen the research on construction safety monitoring and real-time intervention.

[0003] The causes of construction accidents can be summarized into two aspects: one is the lack of standardization of construction personnel, which is prone to accidents due to negligence of safety regulations, fatigue and carelessness; the other is the negligence of safety management in the construction area, which makes it difficult to implement safety warning work and effectively manage the safety risks of the construction area, which is not conducive to reducing the accident rate in the construction area. In order to reduce the accident rate, video image monitoring facilities are currently used to monitor the construction area.

[0004] However, the existing video image monitoring facilities often have the defects of response lag and single feedback mechanism when processing video images due to their dependence on centralized systems, and it is difficult to effectively identify the construction risks in the construction area. SUMMARY

[0005] The purpose of the present application is to provide a neural network image recognition method and a multi-level feedback construction monitoring system, which collects on-site videos through a built-in wide-angle camera, uses a neural network program based on the secondary development of the YOLOv8 algorithm to extract convolutional features from the images and identify whether there are safety hazards in the construction personnel, and transmits feedback information to local and remote terminals through a sound card module, a Wi-Fi bridge and a 4G-DTU module. Through real-time identification, hierarchical feedback and remote control of the safety state of the construction area, the efficiency of on-site management and the accident prevention ability can be improved, and the construction behavior can be standardized, which can effectively reduce the safety risks in the construction area.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] In a first aspect, a neural network image recognition method is provided, comprising:

[0008] Fusing a first algorithm and a second algorithm to construct a small target detection neural network, wherein the first algorithm is a target detection algorithm, and the second algorithm is a small target detection algorithm;

[0009] Collecting video images and inputting the video images into the small target detection neural network to detect the construction state in the video images, wherein the video images are video images of the construction area;

[0010] Judging whether the construction state in the video images is qualified to obtain a judgment result;

[0011] The video image and the judgment result thereof are synthesized into communication data, and the communication data is sent to a receiving terminal.

[0012] As a further scheme of the present application, the construction state in the video image is detected, and the detection comprises:

[0013] a. feature extraction is performed on the video image to generate N multi-scale feature maps, wherein the size of the i-th feature map is 1 / 2 of the size of the video image (i+1) , i≤N; wherein the feature map set is a plurality of feature maps with sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the video image, denoted as {P2, P3, P4, P5};

[0014] b. the N-th feature map in the feature map set is up-sampled to generate a set of M up-sampled feature maps, M≤N, and the sampling formula is:

[0015] U i = Upsample(Pi, s = 2, mode = nearest), wherein Pi represents the feature map, s represents the up-sampling multiple, and mode represents the interpolation mode;

[0016] c. the M-th up-sampled feature map is fused with the M-1-th feature map to generate a fused feature map;

[0017] d. a feature enhancement algorithm is used to perform feature enhancement on the fused feature map to generate an enhanced feature map.

[0018] e. steps b to d are repeated to sequentially perform up-sampling and feature fusion on the feature maps to generate corresponding enhanced feature maps;

[0019] f. convolution and down-sampling are respectively performed on the enhanced feature maps to generate multi-scale detection feature maps;

[0020] g. a detection head algorithm is used to make the detection feature maps generate detection algorithm configuration parameters, wherein the optimization function of the detection algorithm configuration parameters is:

[0021] L = λ1L cls + λ2L box + λ3L obj , wherein L cls , L box , L obj respectively represent a classification loss, a bounding box regression loss and a target confidence loss, and λ1, λ2, λ3 are weight coefficients.

[0022] As a further scheme of the present application, the feature extraction on the video image comprises:

[0023] The feature of the video image is extracted by using a backbone network, wherein the backbone network comprises a plurality of convolution layers and pooling layers;

[0024] The Nth feature map in the feature map set is up-sampled, comprising:

[0025] The feature map size is expanded to four times of the original feature map size by using the nearest neighbor interpolation method, that is, the feature map size is expanded from HxW to 2Hx2W;

[0026] The feature enhancement algorithm in step d includes N=Favored by default (3.0), which improves the key information extraction capability in the image by adaptively enhancing the feature map. Specifically, the feature enhancement algorithm automatically adjusts the enhancement degree according to the importance of the feature map, so that important features are more significantly represented, and secondary features are relatively weakened. This adaptive enhancement mechanism helps to improve the accuracy of subsequent classification and positioning. In addition, the feature enhancement algorithm is configurable, and users can adjust the enhancement parameters according to actual needs to achieve the best recognition effect;

[0027] The detection head in step g supports detecting small objects in the construction site.

[0028] As a further scheme of the application: judging whether the construction state in the video image is qualified, obtaining a judgment result, comprising:

[0029] The image frames of the video image are convoluted and the image features are extracted by using a neural network algorithm;

[0030] The image features are converted into vector features, and it is compared whether the difference between the vector features and the target construction state is within an error threshold range;

[0031] When the judgment result is that the difference is within the error threshold range, the output result is qualified;

[0032] When the judgment result is that the difference is outside the error threshold range, the output result is unqualified.

[0033] As a further scheme of the application: the vector features include one or two of a first vector feature, a second vector feature, a third vector feature and a fourth vector feature;

[0034] The first vector feature includes a dangerous feature or a safety vector feature, the second vector feature includes a high-altitude construction or a non-high-altitude construction vector feature, the third vector feature includes an incorrect safety belt or a correct safety belt vector, and the fourth vector feature includes an incorrect safety helmet or a correct safety helmet vector;

[0035] The target construction state includes a safety vector feature, no high-altitude construction vector, high-altitude construction and correct safety belt wearing vector, high-altitude construction and correct safety helmet wearing vector.

[0036] As a further scheme of the present application: the neural network element is used to determine whether the construction state in the video image is qualified, and a determination result is obtained, including:

[0037] The first vector feature, the second vector feature, the third vector feature and the fourth vector feature are compared with the target construction state in sequence, respectively, when the target construction state completely contains the vector feature, the vector feature is determined to be qualified, and the output result is qualified;

[0038] Otherwise, it is unqualified.

[0039] As a further scheme of the present application: the video image and the determination result thereof are synthesized into communication data, including:

[0040] Prompt text data corresponding to the output result is generated;

[0041] Corresponding prompt audio data is generated according to the prompt text data;

[0042] An identification box corresponding to the output result is drawn on the video image, and the prompt text data is added to generate new video data. The function used to draw the identification box corresponding to the output result on the video image is the cv2.rectangle function in the OpenCV library, and the function used to add the prompt text data is the cv2.putText function;

[0043] According to the prompt text data, corresponding prompt audio data is generated, including converting the prompt text data into a voice signal, and the voice signal is used as the prompt audio data; the generation of the new video data includes real-time drawing of the identification box and the prompt text data on each frame of the input image to form a continuous video stream.

[0044] As a further scheme of the present application: the communication data is sent to a receiving terminal, including:

[0045] The prompt text data is sent to a remote device by using an asynchronous transceiver interface and terminal equipment;

[0046] The new video data is encoded to generate a network data packet, and is sent to the remote device through a Wi-Fi module.

[0047] The second aspect also provides a multi-stage feedback construction monitoring system, which is used for fusing a first algorithm and a second algorithm to construct a small target detection neural network, collecting video images and inputting the video images into the small target detection neural network, detecting a construction state in the video images, judging whether the construction state in the video images is qualified, obtaining a judgment result, synthesizing the video images and the judgment result into communication data, and sending the communication data to a receiving terminal.

[0048] The system comprises an information processing platform, a portable 4G receiving terminal, a total control video receiving end and an audio emitting assembly; the signal output end of the information processing platform sends signals to the portable 4G receiving terminal and the total control video receiving end;

[0049] The information processing platform is installed in a construction area, the portable 4G receiving terminal is used for receiving the analysis result of the information processing platform in the construction area and issuing a voice warning, and the total control video receiving end is used for receiving data and results transmitted from one or more information processing platforms;

[0050] The receiving terminal is used for receiving text data, audio data and video data and distributing them to the management personnel of the construction area and the total control center. The portable 4G receiving terminal is arranged in the construction area, and the total control video receiving end is arranged in the total control room of the construction area. Therefore, the information processing platform can independently operate and has multiple devices to extend multiple core platforms through physical connection or wireless network, which is used for monitoring the safety situation of the construction area in real time and can collect and analyze the area situation.

[0051] As a further scheme of the application, the information processing platform comprises a platform container and a wide-angle camera, the bottom of the platform container is provided with a first battery power supply, and one side of the platform container is provided with control buttons;

[0052] The platform container is a horizontally arranged rectangular shell structure, the audio emitting assembly is installed in the platform container, the wide-angle camera and a local platform display screen are installed on the top of the platform container; the wide-angle camera is used for environmental monitoring and can feed back visual data of the monitoring area; and the local platform display screen is used for displaying video data processed by the information processing platform;

[0053] The inside of the platform container is fixed with a support, a local loudspeaker, a computing module, a high-power sound card, a Wi-Fi bridge transmitting plate and a 4G-DTU transmitting end are installed in the inside of the support, the computing module obtains video data of the construction site from the wide-angle camera, and the video data is converted into processing results through a compiled video processing program in the module, the compiled video processing program in the module is used for synthesizing new video data, audio data, text data and computer communication data, and is transmitted to the high-power sound card, the Wi-Fi bridge transmitting plate and the 4G-DTU transmitting end through a physical connection line, the high-power sound card receives digital audio signals from the computing module, converts the digital audio signals into analog audio signals, amplifies the signals and sends the signals to the local loudspeaker to emit corresponding audio signals; the Wi-Fi bridge transmitting plate receives video data from the computing module, the video data is compressed and encoded into digital video streams, and then encapsulated into network data packets and converted into radio frequency signals for transmission through a WiFi module; the 4G-DTU transmitting end receives text data from the computing module, encodes the text data into a binary format data frame, and then modulates and processes the data frame through a 4G-DTU to convert the data frame into a radio frequency signal conforming to a 4G wireless communication standard for transmission in a public network; the wide-angle camera is used to collect images of workers and construction types in the construction area, and the computing module is used to evaluate the wearing of safety equipment and the high-altitude construction situation to obtain analysis results to generate text, audio and video data, and the data is distributed to the local loudspeaker, the local platform display screen, the remote loudspeaker and the remote HDMI output interface to form multi-level feedback, and the multi-level feedback includes local construction area voice feedback, management personnel wearing personnel feedback and total control monitoring feedback.

[0054] As a further scheme of the application: the portable 4G receiving terminal includes a remote loudspeaker and a 4G-DTU receiving module, a receiving antenna is installed on the top of the remote loudspeaker, and a second battery power supply is connected to one side of the remote loudspeaker; the 4G-DTU receiving module is used to receive a radio frequency signal conforming to a 4G wireless communication standard from the 4G-DTU transmitting end, then modulate and process the radio frequency signal through the 4G-DTU transmitting end, and convert the radio frequency signal into audio data and play the audio data through the remote loudspeaker to realize the function of playing the processing results at a long distance; the second battery power supply is used to provide a 5V-12V voltage for power supply of the equipment; and the receiving antenna is used to ensure that the signal can be transmitted.

[0055] As a further scheme of the application: the second battery power supply provides a 5V-19V voltage for the information processing platform.

[0056] As a further scheme of the present application: the total control video receiving end comprises a third battery power supply, one side of the third battery power supply is connected with an antenna, and the total control video receiving end is provided with an HDMI output interface and a USB interface on the side; the total control video receiving end is used for receiving compressed video signals sent by a plurality of core platforms, displaying the compressed video signals through the HDMI output interface, and further processing the compressed video signals for an upper computer; and the USB interface can be connected with a mouse and a keyboard to remotely control the information processing platform as the upper computer.

[0057] As a further scheme of the present application: the information processing platform is in a plurality of construction areas, each information processing platform and construction area works independently, a plurality of analysis results are obtained, video data in the plurality of analysis results is sent to the same total control video receiving end, and multi-level dimension feedback is realized, at this time, it is a many-to-one feedback, that is, a plurality of information processing platforms and one total control video receiving end interact information with each other.

[0058] Compared with the prior art, the present application has the following advantages:

[0059] 1. In the present application, by adopting the neural network image recognition method and the multi-level feedback construction monitoring system, the on-site video is collected through the built-in wide-angle camera, the neural network program based on the YOLOv8 algorithm is developed, the convolution feature of the image is extracted and recognized, whether the construction personnel has a safety hidden danger, and the feedback information is transmitted to the local and remote terminal through the sound card module, the Wi-Fi bridge and the 4G-DTU module, respectively, through the real-time identification of the safety state of the construction area, the hierarchical feedback and the remote control, the on-site management efficiency and the accident prevention ability can be improved, and the construction behavior can be standardized, and the safety risk of the construction area can be effectively reduced.

[0060] 2. In the present application, by adopting the distributed architecture comprising an information processing platform, a portable 4G receiving terminal and a total control video receiving end, flexible arrangement and independent operation are supported, the information processing platform collects on-site images through the built-in wide-angle camera, can identify safety hazards such as high-altitude work and personal protective equipment wearing in real time, and generate structured feedback data, the feedback information is transmitted to the local and remote terminal through the sound card, the Wi-Fi and the 4G communication module, audio warning and video monitoring are realized, the portable 4G receiving terminal is provided with a portable warning function, the total control video receiving end centrally manages multi-platform data, supports remote instruction feedback control, realizes single-platform independent monitoring and multi-platform unified management dual mode, has the advantages of rapid response, clear feedback grading and flexible deployment, and is suitable for real-time safety supervision and early warning in various high-risk construction environments. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The neural network image recognition method of the present application is shown in the step diagram.

[0062] Figure 2 Figure of the multi-stage feedback construction monitoring system of the present application;

[0063] Figure 3 Structure diagram of the information processing platform of the present application;

[0064] Figure 4 Structure diagram of the portable 4G receiving terminal of the present application;

[0065] Figure 5 Structure diagram of the total control video receiving end of the present application;

[0066] Figure 6 Structure diagram of the audio emitting assembly of the present application;

[0067] Figure 7 Structure diagram of the local device in the embodiment of the present application;

[0068] Figure 8 Structure diagram of the multi-platform signal flow in the embodiment of the present application;

[0069] Figure 9 Node flow chart of the method embodiment of the present application.

[0070] In the figure: 1, information processing platform; 11, platform container; 12, wide-angle camera; 13, control button; 14, first battery power supply; 15, local platform display screen; 2, portable 4G receiving terminal; 21, remote loudspeaker; 22, 4G-DTU receiving module; 23, receiving antenna; 24, second battery power supply; 3, total control video receiving end; 31, third battery power supply; 32, antenna; 33, remote HDMI output interface; 34, USB interface; 4, audio emitting assembly; 41, support; 42, local loudspeaker; 43, computing module; 44, high-power sound card; 45, Wi-Fi network bridge emitting board; 46, 4G-DTU emitting end. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0072] The steps in the present application are arranged with labels, but are not used to limit the order of the steps, unless the order of the steps is explicitly stated or the execution of a certain step needs other steps as a basis, otherwise the relative order of the steps can be adjusted. It can be understood that the term "and / or" used herein relates to and covers any and all possible combinations of one or more of the associated listed items.

[0073] Embodiment one:

[0074] As shown in the figure, the embodiment provides a neural network image recognition method, comprising the following steps: Figure 1

[0075] S1: Fusion of the first algorithm and the second algorithm to construct a small target detection neural network, wherein the first algorithm is a target detection algorithm, and the second algorithm is a small target detection algorithm;

[0076] S2: Collecting video images and inputting the video images into the small target detection neural network to detect the construction state in the video images, wherein the video images are video images of the construction area;

[0077] S3: Determine whether the construction state in the video images is qualified, and obtain a judgment result;

[0078] S4: Synthesizing the video images and the judgment result into communication data, and sending the communication data to a receiving terminal.

[0079] As shown in the figure, the embodiment provides a neural network image recognition system, comprising a backbone network, a neck network and a detection head, wherein the backbone network comprises a plurality of convolutional layers (Conv) and feature fusion layers (C2f), and finally reaches a spatial pyramid pooling layer; Figure 9 The neck network comprises a plurality of concatenation layers, up-sampling layers and feature fusion layers (C2f), and the processing result of the neck network is output to the detection head, and the detection head comprises a plurality of convolutional layers (Conv).

[0080] Preferably, step S2: detecting the construction state in the video images, comprises:

[0081] a. Feature extraction is performed on the video images to generate four multi-scale feature maps, wherein the size of the i-th feature map is 1 / 2 of the size of the video image i+1 , i≤4; wherein the feature map set is a plurality of feature maps with sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the video image, denoted as {P2, P3, P4, P5};

[0082] b. The fourth feature map in the feature map set is up-sampled to generate four up-sampled feature map sets, and the sampling formula is:

[0083] ​​

[0084] U1 = Upsample(P1, s = 2, mode = nearest),

[0085] U2 = Upsample(P2, s = 2, mode = nearest),

[0086] U3 = Upsample(P3, s = 2, mode = nearest),

[0087] U4 = Upsample(P4, s = 2, mode = nearest), wherein P1 represents a first feature map, P2 represents a second feature map, P3 represents a third feature map, P4 represents a fourth feature map, s represents a sampling multiple, and mode represents an interpolation mode;

[0088] C. a fifth up-sampling feature map U5 is fused with the fourth feature map to generate a fusion feature map F4 by using a fusion formula,

[0089] a fourth up-sampling feature map U4 is fused with the third feature map to generate a fusion feature map F3,

[0090] a third up-sampling feature map U3 is fused with the second feature map to generate a fusion feature map F2,

[0091] a second up-sampling feature map U2 is fused with the first feature map to generate a fusion feature map F1,

[0092] wherein the fusion formula is:

[0093] F4 = Concat(U5, P4, dim = 1),

[0094] F3 = Concat(U4, P3, dim = 1),

[0095] F2 = Concat(U3, P2, dim = 1),

[0096] F1 = Concat(U2, P1, dim = 1), wherein Concat represents concatenation along a channel dimension;

[0097] d. the fusion feature map F4 is subjected to feature enhancement by using a feature enhancement algorithm to generate an enhanced feature map E4;

[0098] the fusion feature map F3 is subjected to feature enhancement by using a feature enhancement algorithm to generate an enhanced feature map E3;

[0099] the fusion feature map F2 is subjected to feature enhancement by using a feature enhancement algorithm to generate an enhanced feature map E2; wherein the feature enhancement algorithm is a C2f algorithm, and the C2f algorithm is:

[0100] where c denotes the number of output channels, N denotes the number of residual units, denotes a 3x3 convolution kernel, BN denotes batch normalization, and ReLU denotes an activation function

[0101] e. Repeating steps b to d to sequentially perform upsampling and feature fusion on the feature maps to generate corresponding enhanced feature maps;

[0102] f. Performing convolution and downsampling on the enhanced feature maps E2, E3, and E4, respectively, to generate multi-scale detection feature maps {D2, D3, D4, D5}, wherein the sampling formula for downsampling is:

[0103] Dk = Conv3x3(Ek, ck, s = 2), k e {2, 3, 4}, where ck denotes the number of output channels, and s denotes the convolution stride;

[0104] g. Using a detection head algorithm to generate detection algorithm configuration parameters from the detection feature maps, and the optimization function of the detection head algorithm is:

[0105] L = λ1L cls + λ2L box + λ3L obj , where L cls , L box , and L obj represent the classification loss, the bounding box regression loss, and the target confidence loss, respectively, and λ1, λ2, and λ3 are weight coefficients.

[0106] Preferably, the feature extraction for the video image comprises:

[0107] The features of the video image are extracted using a backbone network, wherein the backbone network comprises multiple convolution layers and pooling layers;

[0108] Upsampling the Nth feature map in the feature map set comprises:

[0109] The nearest neighbor interpolation method is used to expand the size of the feature map to four times the size of the original feature map, i.e., the size of the feature map is expanded from HxW to 2Hx2W;

[0110] The feature enhancement algorithm in step d includes N = Favored by default (3.0), which enhances the key information extraction capability in the image by adaptively enhancing the feature map. Specifically, the feature enhancement algorithm automatically adjusts the enhancement degree according to the importance of the feature map, so that important features are more significantly represented, while secondary features are relatively weakened. This adaptive enhancement mechanism helps to improve the accuracy of subsequent classification and positioning. In addition, the feature enhancement algorithm is configurable, and users can adjust the enhancement parameters according to actual needs to achieve the best recognition effect;

[0111] The detection head in step g supports detecting small objects in the construction site.

[0112] Preferably, the method comprises judging whether the construction state in the video image is qualified to obtain a judgment result, including:

[0113] The neural network algorithm is used to perform convolution operation on the image frames of the video image and extract image features.

[0114] The image features are converted into vector features, and whether the difference between the vector features and the target construction state is within an error threshold range is compared.

[0115] When the judgment result is that the difference is within the error threshold range, the output result is qualified.

[0116] When the judgment result is that the difference is outside the error threshold range, the output result is unqualified.

[0117] Preferably, the vector features include one or two of a first vector feature, a second vector feature, a third vector feature, and a fourth vector feature.

[0118] The first vector feature includes a dangerous feature or a safety vector feature, the second vector feature includes a high-altitude construction or no high-altitude construction vector feature, the third vector feature includes an incorrect safety belt or a correct safety belt vector, and the fourth vector feature includes an incorrect safety helmet or a correct safety helmet vector.

[0119] The target construction state includes a safety vector feature, no high-altitude construction vector, correct safety belt vector with high-altitude construction, and correct safety helmet vector with high-altitude construction.

[0120] Preferably, the method comprises using a neural network element to judge whether the construction state in the video image is qualified to obtain a judgment result, including:

[0121] The first vector feature, the second vector feature, the third vector feature, and the fourth vector feature are compared with the target construction state in sequence, and when the target construction state completely contains the vector features, the vector features are judged to be qualified, and the output result is qualified.

[0122] Otherwise, it is unqualified.

[0123] In this embodiment, the first vector feature is a safety vector feature, the second vector feature is a high-altitude construction vector feature, the third vector feature is a correct safety belt vector, and the fourth vector feature is a correct safety helmet vector. The first vector feature, the second vector feature, the third vector feature, and the fourth vector feature are respectively contained by the target construction state, and therefore, the output result is qualified.

[0124] Preferably, the video image and its judgment result are synthesized into communication data, including:

[0125] generating prompt text data corresponding to the output result;

[0126] generating corresponding prompt audio data according to the prompt text data;

[0127] drawing a recognition box corresponding to the output result on the video image and adding the prompt text data to generate new video data. The function used to draw the recognition box corresponding to the output result on the video image is the cv2.rectangle function in the OpenCV library, and the function used to add the prompt text data is the cv2.putText function;

[0128] generating corresponding prompt audio data according to the prompt text data includes converting the prompt text data into a voice signal, and the voice signal is used as the prompt audio data; the generation of the new video data includes real-time drawing of the recognition box and the prompt text data on each frame of the input image to form a continuous video stream.

[0129] Preferably, the communication data is sent to a receiving terminal, including:

[0130] sending the prompt text data to a remote device using an asynchronous transceiver interface and terminal equipment;

[0131] encoding the new video data to generate network data packets and sending them to the remote device through a Wi-Fi module.

[0132] As shown in Figures 2-7 The embodiment also provides a multi-stage feedback construction monitoring system. The system is used to collect video images, fuse a first algorithm and a second algorithm to construct a small target detection neural network to detect the video images, collect the video images and input the video images into the small target detection neural network, detect the construction state in the video images, judge whether the construction state in the video images is qualified, obtain a judgment result, synthesize the video images and the judgment result into communication data, and send the communication data to a receiving terminal.

[0133] The system includes an information processing platform 1, a portable 4G receiving terminal 2, a total control video receiving end 3, and an audio transmitting assembly 4. The signal output end of the information processing platform 1 sends signals to the portable 4G receiving terminal 2 and the total control video receiving end 3.

[0134] The information processing platform 1 is installed in a construction area, the portable 4G receiving terminal 2 is used to receive the analysis results of the information processing platform 1 in the construction area and issue voice warnings, and the total control video receiving end 3 is used to receive data and results transmitted from one or more information processing platforms 1.

[0135] The receiving terminal is used for receiving text data, audio data and video data, and distributing to the management personnel of the construction area and the general control center.

[0136] Preferably, the information processing platform 1 comprises a platform container 11 and a wide-angle camera 12, a first battery power supply 14 is installed at the bottom of the platform container 11, and a control button 13 is arranged on one side of the platform container 11.

[0137] The platform container 11 is a horizontally arranged rectangular shell structure, the audio emitting assembly 4 is installed inside the platform container 11, the wide-angle camera 12 and the local platform display screen 15 are installed at the top of the platform container 11; the wide-angle camera 12 is used for environmental monitoring and can feedback visual data of the monitoring area; the local platform display screen 15 is used for displaying video data processed by the information processing platform 1; the information processing platform 1 collects on-site videos through the built-in wide-angle camera 12, uses a neural network program developed based on the YOLOv8 algorithm to extract convolution features of the video images and identify whether the construction personnel have safety hazards, such as whether to perform high-altitude operation, whether to wear safety belts and safety helmets, etc., the identification result is processed by the computing module to generate structured text, audio and video data, and the feedback information is transmitted to the local and remote terminals through the sound card module, the Wi-Fi bridge and the 4G-DTU module, respectively.

[0138] The platform container 11 is fixed with a support 41, the local loudspeaker 42, the computing module 43, the high-power sound card 44, the Wi-Fi bridge transmitting plate 45 and the 4G-DTU transmitting end 46 are installed inside the support 41, the computing module 43 obtains the video data of the construction site from the wide-angle camera 12, and converts the video data into processing results through the compiled video processing program inside the module; the compiled video processing program inside the module is used for synthesizing new video data, audio data, text data and computer communication data, and is transmitted to the high-power sound card 44, the Wi-Fi bridge transmitting plate 45 and the 4G-DTU transmitting end 46 through the physical connection line; the high-power sound card 44 receives the digital audio signal from the computing module 43, converts it into an analog audio signal and amplifies the signal to the local loudspeaker 42 to emit the corresponding audio signal; the Wi-Fi bridge transmitting plate 45 receives the video data from the computing module 43, the video data is compressed and encoded into a digital video stream, and then encapsulated into a network data packet, and then converted into a radio frequency signal through a WiFi module for transmission; the 4G-DTU transmitting end 46 receives the text data from the computing module 43, encodes it into a binary format data frame, and then modulates and processes it through the 4G-DTU to convert it into a radio frequency signal conforming to the 4G wireless communication standard for transmission in the public network.

[0139] Preferably, the portable 4G receiving terminal 2 comprises a remote speaker 21 and a 4G-DTU receiving module 22, the top of the remote speaker 21 is mounted with a receiving antenna 23, and one side of the remote speaker 21 is connected with a second battery power supply 24; the second battery power supply 24 is used to provide 5V-19V voltage for the information processing platform 1; the 4G-DTU receiving module 22 is used to receive the radio frequency signal conforming to the 4G wireless communication standard from the 4G-DTU transmitting end 46, then modulate and process through the 4G-DTU transmitting end 46, and convert into audio data and play through the remote speaker 21, so as to realize the function of playing the processing result remotely; the second battery power supply 24 is used to provide 5V-12V voltage for the equipment power supply; the receiving antenna 23 is used to ensure that the signal can be communicated, and the wide-angle camera 12 is used to collect the images of the workers and the construction types in the construction area, and the calculation module 43 is used to evaluate the wearing of safety equipment and the construction high-altitude operation situation, so as to obtain the analysis result to generate text, audio and video data, and distribute the data to the local speaker 42, the local platform display screen 15, the remote speaker 21 and the remote HDMI output interface 33 to form multi-level feedback, which comprises local construction area voice feedback, management person wearing personnel feedback and total control monitoring feedback.

[0140] Preferably, the total control video receiving end 3 comprises a third battery power supply 31, one side of the third battery power supply 31 is connected with an antenna 32, and the side of the total control video receiving end 3 is provided with a remote HDMI output interface 33 and a USB interface 34; the total control video receiving end 3 is used to receive the compressed video signal sent by the plurality of core platforms, and display through the remote HDMI output interface 33, and further process for the upper computer; the USB interface 34 can connect the mouse and keyboard to remotely control the information processing platform 1 for further control of the upper computer.

[0141] Preferably, the information processing platform 1 is in a plurality of construction areas, each information processing platform 1 and construction area works independently to obtain a plurality of analysis results, the video data in the plurality of analysis results is sent to the same total control video receiving end 3 to realize multi-level dimension feedback, at this time, it is a many-to-one feedback, that is, a plurality of information processing platforms 1 and one total control video receiving end 3 interact information, wherein the information processing platform 1 is a core platform.

[0142] This embodiment is suitable for using a wide-angle camera 12 to monitor construction sites within a 120° viewing angle and a viewing distance of 50 meters. The distance between the master control video receiver 3 and the information processing platforms 1 and 4 must be within 200 meters, preferably with minimal obstructions. The portable 4G receiving terminal 2 has no spatial restrictions. Depending on site needs, warnings can be set for personnel with non-compliant equipment configurations, such as those not wearing a safety helmet, safety harness, or safety vest. Condition alerts can also be set for situations such as high-altitude work or areas with potential safety hazards. These detections are performed entirely within the computing module 43 within the information processing platform.

[0143] Specifically, in this embodiment, a multi-level identification feedback construction monitoring system needs to be configured with at least one core platform. As shown in the information processing platform 1, it has a core platform container 11, a wide-angle camera 12, a first battery power supply 14, and a local platform display screen 15; the core platform container 11 is a horizontally arranged rectangular shell structure, the container can accommodate the platform internal components 4, the wide-angle camera 12 and the local platform display screen 15 can be placed on the top of the container, the second battery power supply 24 can be fixed on the bottom of the container, and the container surface has control buttons 13

[0144] In this embodiment, the core platform container 11 contains multiple components, including a core platform internal bracket 41, a local speaker 42, a computing module 43, a high-power sound card 44, a Wi-Fi bridge transmitter 45, and a 4G-DTU transmitter 46. The detection device in this embodiment is generally composed of one or more cameras, preferably with a wide angle. Figure 1 As shown, the detection device is mounted on the top of the entire device, but this is not limited to the top. Adaptation should be made to special working environments, such as waterproofing, underwater adaptation, and the use of a camera with a light source. The detection device is not limited to using only cameras. Any device that integrates with the camera to expand data sources and provide feedback is considered a detection device, such as laser detection and spectral detection.

[0145] In this embodiment, the local platform display screen 15 is optionally provided at the top of the housing to directly feedback the monitoring situation and can provide sound and light prompts;

[0146] In this embodiment, the battery power supply is optionally provided at the bottom of the housing, and can provide power for the carrying platform, and should provide a voltage of 5V to 19V.

[0147] Specifically, in this embodiment, a multi-level identification and feedback construction monitoring system can be configured with multiple sets of portable 4G receiving terminals. The portable 4G receiving terminal 2 is stacked from bottom to top and includes a remote speaker 21, a 4G-DTU receiving module 22, a receiving antenna 23, and a second battery power supply 24.

[0148] The 4G-DTU receiving module 22 in the embodiment is used for receiving the radio frequency signal conforming to the 4G wireless communication standard from the 4G-DTU transmitting end 46, and then modulating and processing through the 4G-DTU, converting into audio data and playing through the remote loudspeaker 21, so as to realize the function of playing the processing result remotely.

[0149] Specifically, the configurable general control video receiving end 3 in the multi-level identification feedback construction monitoring system in the embodiment comprises a container containing a third battery power supply 31, an antenna 32, a remote HDMI output interface 33 and a USB interface 34. The general control video receiving end 3 is used for receiving the compressed video signal sent by the plurality of core platforms, and displaying through the remote HDMI output interface 33, and further processing as the host computer; the USB interface 34 can connect the mouse and keyboard to remotely control the information processing platform 1 as the host computer for further control.

[0150] It needs to be further explained that:

[0151] The multi-level identification feedback construction monitoring system in the embodiment has the unique information processing platform 1, the unlimited number of portable 4G receiving terminals 2 and the general control video receiving end 3, and the safety monitoring multi-level feedback will be performed.

[0152] As shown in Figure 7 , specifically, there is a construction site, and one information processing platform 1 monitors the site, and the wide-angle camera 12 detects the abnormality to generate processing data by the information processing platform 1, and distribute to the local loudspeaker 42, the portable 4G receiving terminal 2 and the general control video receiving end 3. The general control video receiving end 3 is the receiving end equipped for the construction site, the construction management personnel and the general control center, and the processing data includes text data, audio data and video data.

[0153] As shown in Figure 7 , the portable 4G receiving terminal 2 is arranged in the construction area, and the portable 4G receiving terminal 2 is carried by the wearer to receive the 4G communication signal sent by the information processing platform 1, and realize the audio warning through the local loudspeaker 42. The general control video receiving end 3 is arranged in the on-site general control room, and is used for centrally receiving the video data transmitted by the information processing platform 1, and outputting to the host computer display through the remote HDMI output interface 33, supporting the mouse and keyboard remote operation to return the instruction, realizing the unified management and control of the plurality of platforms. Meanwhile, the information processing platform 1 can independently run and has a plurality of core platforms through physical connection or wireless network expansion, which is used for monitoring the safety situation of the construction area in real time, and can collect and analyze the regional situation.

[0154] Embodiment two:

[0155] As shown in Figure 8As shown, the difference between the embodiment and the above-mentioned embodiment is that the multi-stage identification feedback construction monitoring system in the embodiment has a plurality of information processing platforms 1, a plurality of portable 4G receiving terminals 2 and a unique total control video receiving end 3, the construction site A corresponds to the core platform 1, the construction site B corresponds to the core platform 2, the construction site C corresponds to the core platform 3, and each has a portable 4G receiving terminal 2 carried by a person, and the person is a regional administrator or a specific area construction personnel. The total control video receiving end 3 is unique, and all monitoring video information is collected in the total control video receiving end.

[0156] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacement or change within the technical range disclosed by the present application according to the technical solution and the inventive concept of the present application, which should be covered within the protection scope of the present application.

Claims

1. A neural network image recognition method, characterized in that: include: The first algorithm and the second algorithm are integrated to construct a small target detection neural network, wherein the first algorithm is a target detection algorithm and the second algorithm is a small target detection algorithm; Collecting a video image and inputting the video image into a small target detection neural network to detect a construction status in the video image, wherein the video image is a video image of a construction area; Determine whether the construction status in the video image is qualified and obtain a determination result; The video image and its judgment result are synthesized into communication data, and the communication data is sent to the receiving terminal.

2. The neural network image recognition method according to claim 1, wherein: The detecting the construction status in the video image includes: a. Extract features from the video image and generate N multi-scale feature maps, where the size of the i-th feature map is 1 / 2 of the video image size. (i+1) , i≤N; b. Upsample the Nth feature map in the feature map set to generate M upsampled feature map sets, M≤N, c. Fuse the Mth upsampled feature map with the M-1th feature map to generate a fused feature map; d. Use feature enhancement algorithm to enhance the fused feature map to generate enhanced feature map; e. Repeat steps b to d, upsampling and fusion of the feature maps in turn to generate the corresponding enhanced feature maps; f. Convolve and downsample the enhanced feature map to generate multi-scale detection feature maps; g. Use the detection head algorithm to generate the detection algorithm configuration parameters using the detection feature map.

3. The neural network image recognition method according to claim 2, wherein: The feature extraction of the video image includes: A backbone network is used to extract features of video images, where the backbone network includes multiple convolutional layers and pooling layers; The upsampling of the Nth feature map in the feature map set includes: The nearest neighbor interpolation method is used to expand the feature map size to four times the original feature map size.

4. The neural network image recognition method according to claim 3, wherein: Determine whether the construction status in the video image is qualified and obtain the judgment result, including: A neural network algorithm is used to perform convolution operations on the image frames of the video image and extract image features; Convert the image features into vector features and compare whether the difference between the vector features and the target construction status is within the error threshold; When the judgment result is that the difference is within the error threshold range, the output result is qualified; When the judgment result is that the difference is outside the error threshold range, the output result is unqualified.

5. The neural network image recognition method according to claim 1, wherein: The vector feature includes one or two of a first vector feature, a second vector feature, a third vector feature and a fourth vector feature; The first vector feature includes a danger feature or a safety feature, the second vector feature includes a high-altitude construction feature or an absence of a high-altitude construction feature, the third vector feature includes an incorrect safety belt wearing feature or a correct safety belt wearing feature, and the fourth vector feature includes an incorrect safety helmet wearing feature or a correct safety helmet wearing feature. The target construction state includes a safety vector feature, a non-existence of a high-altitude construction vector, a presence of a high-altitude construction and correct wearing of a safety belt vector, and a presence of a high-altitude construction and correct wearing of a safety helmet vector.

6. The neural network image recognition method according to claim 5, characterized in that: The neural network element is used to determine whether the construction status in the video image is qualified, and obtain the determination result, including: The first vector feature, the second vector feature, the third vector feature, and the fourth vector feature are respectively compared with the target construction state. When the target construction state completely contains the vector feature, the vector feature is judged to be qualified and the output result is qualified. Otherwise it is unqualified.

7. The neural network image recognition method according to claim 6, wherein: The step of synthesizing the video image and the judgment result thereof into communication data includes: Generate prompt text data corresponding to the output result; Generate corresponding prompt audio data according to the prompt text data; Draw a recognition frame corresponding to the output result on the video image, add prompt text data, and generate new video data.

8. The neural network image recognition method according to claim 7, wherein: The sending of communication data to a receiving terminal includes: Using an asynchronous receiver-transmitter interface and a terminal device to send the prompt text data to a remote device; Encode the new video data, generate network data packets, and send them to the remote device via the Wi-Fi module.

9. A multi-level feedback construction monitoring system, characterized by: The system processes video images using the neural network image recognition method according to any one of claims 1 to 8; The system is configured to collect video images, fuse the first algorithm and the second algorithm to construct a small target detection neural network to detect the video images, collect the video images and input the video images into the small target detection neural network, detect the construction status in the video images, and determine whether the construction status in the video images is qualified, obtain a determination result, and synthesize the video images and the determination result into communication data, and send the communication data to a receiving terminal; The system includes an information processing platform, a portable 4G receiving terminal, a master control video receiving terminal and an audio transmitting component; the signal output end of the information processing platform sends signals to the portable 4G receiving terminal and the master control video receiving terminal.

10. The multi-level feedback construction monitoring system according to claim 9, characterized in that: The information processing platform includes a platform container and a wide-angle camera. A first battery power source is installed at the bottom of the platform container, a control button is provided on one side of the platform container, an audio transmission component is installed inside the platform container, and a wide-angle camera and a local platform display are installed on the top of the platform container; A bracket is fixed inside the platform container, and a local speaker, a computing module, a high-power sound card, a Wi-Fi bridge transmitter and a 4G-DTU transmitter are installed inside the bracket. The computing module obtains video data of the construction site from the wide-angle camera and converts it into processing results. The computing module transmits the data to the high-power sound card, the Wi-Fi bridge transmitter and the 4G-DTU transmitter through a physical connection line. The wide-angle camera is used to collect images of workers and construction types in the construction area. The computing module is used to evaluate the wearing of safety equipment and the high-altitude construction operations, and obtain analysis results to generate text, audio and video data.

Images