A method, system, device and medium for on-site target detection and labeling

Abstract

The present application relates to a method, system, device and medium for on-site target detection and labeling, and to the field of image recognition technology, which includes: extracting video frames from the global video screen, and then using the RPN network in the Faster R-CNN algorithm to generate candidate areas in the video frame, and matching and checking the candidate areas with preset feature maps. If they match, the candidate areas are classified and regressed, and the most credible areas are screened out using the NMS technology, thereby outputting the category, location and confidence of the target object. Finally, the Faster R-CNN algorithm is used to perform feature recognition on the local video, and the candidate area screening and output process is performed again. The present application can track different targets in large-scale video data, reduce missed items, and thus realize rapid labeling and annotation of detection videos.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method, system, device and medium for on-site target detection and labeling. Background Art

[0002] In the field of modern construction engineering, quality inspection is a key link to ensure project safety and compliance. With the development of technology, on-site inspection videos have become an important means of recording and analyzing the construction process. The amount of video data is huge, often reaching GB to TB levels, and the recording cycle is long, which poses a challenge to the effective management and analysis of data.

[0003] At present, in order to support the effective playback and traceability of video data, traditional methods mainly rely on manual identification of key information in videos. This method not only requires a large amount of human resources, but also has relatively low output due to the subjectivity and fatigue of manual identification. In addition, it is easy to miss items during the manual identification process, resulting in the omission of key information, affecting the integrity and accuracy of the detection data.

[0004] According to the existing technology, manual recognition has obvious limitations when facing large-scale video data. For example, in large construction sites, there are too many places to monitor, and with the improvement of video resolution and the increase of monitoring equipment, the amount of data from on-site detection videos has increased dramatically. How to solve the problem of tracking different targets in large-scale video data and reduce recognition omissions, and then realize the problem of rapid labeling of detection videos, it is necessary to design a method, system, equipment and medium for on-site target detection labeling. Summary of the invention

[0005] The first purpose of this application is to provide a method for on-site target detection and labeling, which can track different targets in large-scale video data and reduce missed identification items, thereby realizing rapid labeling and annotation of detection videos.

[0006] In the first aspect, the present application provides a method for on-site target detection and labeling, which adopts the following technical solution:

[0007] Obtain global video monitoring of the entire detection area in real time;

[0008] Acquire a global video frame according to the video picture of the global video;

[0009] According to the RPN network in the Faster R-CNN algorithm, a plurality of global candidate regions of the global video frame are generated;

[0010] Projecting the plurality of global candidate regions onto a preset feature map, determining whether the feature representations of the plurality of global candidate regions match the preset feature map of the target object, if so, executing A1, if not, executing A2;

[0011] A1: Obtain the global or local candidate region, classify and regress the feature representation of the candidate region, use NMS to eliminate several overlapping regions, retain the region with the highest confidence, and output the category, position and corresponding confidence detection results of the target object;

[0012] A2: Obtain the feature representation position of the unmatched global candidate region and execute A3;

[0013] A3: Obtain a local video of the monitoring part detection area matching the position of the feature point, and execute A4;

[0014] A4: Perform feature recognition on the local video using the Faster R-CNN algorithm to obtain a candidate region of the local video, and execute A1.

[0015] By adopting the above technical solution, global monitoring is used to obtain global video for feature recognition, which can reduce the amount of video for feature recognition. When the global video monitoring cannot recognize the preset feature map of the target object, the local monitoring video is combined to further refine the monitoring range of feature recognition to perform feature recognition more accurately. According to the matching between the feature representation of the target object and the preset feature map, the detection strategy is flexibly adjusted to achieve efficient tracking and labeling of different targets in large-scale video data, which can effectively reduce missed items and improve the accuracy and comprehensiveness of detection.

[0016] In a preferred example, the present application may be further configured as follows: before projecting the plurality of global candidate regions onto a preset feature map, determining whether the feature representations of the plurality of global candidate regions match the preset feature map of the target object, and if so, executing A1, and if not, executing A2, the step further includes:

[0017] Projecting the global candidate regions onto a feature map of a pile foundation to obtain a pile foundation annotation;

[0018] The area enclosed by every four diagonal pile foundations is divided to obtain the pile foundation area.

[0019] By adopting the above technical solution, according to the area surrounded by the four diagonal piles, when the feature representation of the subsequent global candidate area does not match the feature map of the preset target object, the feature representation position of the current mismatched global candidate area can be accurately located between the piles, and the local video located in the pile area surrounded by the piles is deployed for local feature recognition, thereby improving the comprehensiveness and accuracy of target detection and reducing missed items.

[0020] In a preferred example, the present application can be further configured as follows: A4 includes:

[0021] Acquire a local video frame according to the video picture of the local video located in each pile foundation area;

[0022] According to the RPN network in the Faster R-CNN algorithm, a plurality of local candidate regions of the local video frame are generated;

[0023] Projecting the plurality of local candidate regions onto a preset feature map of the target object to obtain feature representations of the local candidate regions;

[0024] A5: Use the Faster R-CNN algorithm to determine whether the feature representation of the current local video frame matches the preset feature map of the target object. If so, execute A1; if not, execute A6.

[0025] A6: Control the capture device of the partial video to switch the capture angle, obtain the partial video frame of the capture device of the switched capture angle at the next time node, and execute A5.

[0026] By adopting the above technical solution, the RPN network in the Faster R-CNN algorithm is used to generate several local candidate areas of local video frames, and local video frames located in each pile foundation area are obtained, and then feature representations that cannot be recognized in the global video frame are obtained, thereby improving the accuracy of target detection, reducing false detections and missed detections, and realizing refined processing of video data, which can improve the accuracy and efficiency of target detection and reduce the waste of computing resources.

[0027] In a preferred example, the present application may be further configured as follows: before projecting the plurality of global candidate regions onto a preset feature map, determining whether the feature points of the plurality of global candidate regions match the preset feature map, and if so, executing A1, and if not, executing A2, the step further includes:

[0028] Extracting features from the construction drawings through a deep convolutional network and converting them into a first feature map, wherein the first feature map includes pile foundations;

[0029] Through a deep convolutional network, feature extraction is performed on the construction drawing and the result is converted into a second feature map, where the second feature map includes feature maps of equipment and vehicles;

[0030] Through a deep convolutional network, feature extraction is performed on the portrait photo and the construction material library captured at the punch-in point, and the feature map is converted into a third feature map, wherein the third feature map includes the portrait, construction tools and surrounding materials;

[0031] The first feature map, the second feature map and the third feature map are integrated into the feature map.

[0032] By adopting the above technical solution, the deep convolutional network extracts features from the pile foundation, equipment and vehicle feature maps, portrait photos and construction material libraries in the construction drawings, and constructs the first, second and third feature maps. This enables the Faster R-CNN algorithm to fully capture the feature information of different targets, improve the accuracy and comprehensiveness of target detection, and meet the recognition feature requirements of general construction sites.

[0033] In a preferred example, the present application can be further configured as follows: projecting the plurality of global candidate regions onto a preset feature map, determining whether the feature representations of the plurality of global candidate regions match the preset feature map of the target object, and if so, executing step A1, and if not, executing step A2, including:

[0034] Determine whether the feature representations of the pile foundation images of the pile foundation areas match the first feature map, if they match, execute A7; if not, generate the feature representation positions of the unmatched pile foundation areas, and execute A2;

[0035] A7: Determine whether the equipment and vehicles in the pile foundation area match the second feature map. If so, execute A1; if not, generate a feature representation position of the unmatched pile foundation area and execute A2.

[0036] By adopting the above technical solution, the current pile foundation is first identified through the global video, and then handed over to the local video for identification, providing more accurate positioning information to confirm the current pile foundation position, and then by identifying the equipment and vehicles in each pile foundation area, the equipment and vehicles are labeled, effectively coping with the diversity and complexity of targets in large-scale video data, and improving the comprehensiveness and accuracy of detection.

[0037] In a preferred example, the present application can be further configured as follows: before A1, it also includes:

[0038] Determine whether the construction tools and surrounding materials in the pile foundation area match the third feature map. If they match, obtain the construction behavior that each portrait matches by combining the construction tools and surrounding materials.

[0039] If there is no match, the feature representation position of the unmatched pile foundation area is generated and A2 is executed.

[0040] By adopting the above technical solution, it is determined whether the construction tools and surrounding materials in the pile foundation area match the third feature map including the construction tools and surrounding materials, and the construction tools and surrounding materials are combined to achieve accurate identification of construction behavior, improve the accuracy and efficiency of construction behavior detection, and reduce false detections and missed detections.

[0041] In a preferred example, the present application can be further configured as follows: the step of determining whether the construction tools and surrounding materials in the pile foundation area match the third feature map, and if so, combining the construction tools and surrounding materials to obtain the construction behavior that each portrait matches includes:

[0042] Acquire each construction arrangement in a preset construction plan, wherein the construction arrangement includes a collection of images of construction tools and required construction materials for implementing the construction behavior;

[0043] If the feature representation of the portrait photo is detected in the pile foundation area, a plurality of portrait candidate areas are generated through the RPN network in the Faster R-CNN algorithm;

[0044] Determine whether the construction tools and surrounding materials in the candidate portrait areas match the third feature map. If they match, match the construction tools and surrounding materials with each set in the construction arrangement to obtain the construction behavior that each portrait meets.

[0045] By adopting the above technical solution, the candidate area of ​​the RPN network is narrowed, and the characteristic marks of construction tools and surrounding materials are identified in the candidate area around the portrait, which improves the accuracy of matching construction arrangements, and then accurately determines the current construction behavior of the portrait, achieving efficient tracking and rapid labeling.

[0046] In the second aspect, the present application provides a field target detection and labeling system, which adopts the following technical solution:

[0047] A field target detection and labeling system, comprising:

[0048] Video acquisition module: The video acquisition module is used to acquire the global video of the entire detection area in real time;

[0049] Video frame extraction module: the video frame extraction module is used to obtain a global video frame according to the video picture of the global video;

[0050] Global candidate region generation module: The global candidate region generation module is used to generate a number of global candidate regions of the global video frame according to the RPN network in the Faster R-CNN algorithm;

[0051] Global feature map projection module: The global feature map projection module is used to project the global candidate regions onto a preset feature map, and determine whether the feature representations of the global candidate regions match the preset feature map of the target object. If they match, A1 is executed; if not, A2 is executed.

[0052] Candidate region processing module: The candidate region processing module is used to execute A1: obtain the global or local candidate region, classify and regress the feature representation of the candidate region, use NMS to eliminate several overlapping regions, retain the region with the highest confidence, and output the category, position and corresponding confidence detection results of the target object;

[0053] Position acquisition module: the position acquisition module is used to execute A2: obtain the feature representation position of the unmatched global candidate area, and execute A3;

[0054] Local video acquisition module: The local video acquisition module is used to execute A3: obtain a local video of the monitoring part detection area matching the position of the feature point, and execute A4;

[0055] Local feature map projection module: The local feature map projection module is used to execute A4: perform feature recognition on the local video through the Faster R-CNN algorithm, obtain the candidate area of ​​the local video, and execute A1.

[0056] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0057] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned on-site target detection and labeling method are implemented.

[0058] In a fourth aspect, the present application provides a computer storage medium, the following technical solution:

[0059] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned on-site target detection and labeling method are implemented.

[0060] In summary, this application has the following beneficial technical effects:

[0061] This application uses the Faster R-CNN algorithm to first perform feature recognition on the global video of global monitoring, which can reduce the demand for video data for feature recognition. When the feature image of the preset target object cannot be captured in the global video screen, the recognition range of feature recognition is further narrowed by combining the local video of local monitoring, thereby achieving more accurate feature recognition, improving the efficient tracking and labeling of different targets in large-scale video data, reducing recognition omissions, and enhancing the accuracy and comprehensiveness of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a flow chart of a method for on-site target detection and labeling in one embodiment of the present application.

[0063] Figure 2 This is a flowchart of steps added before step S3 in one of the embodiments of the present application.

[0064] Figure 3 It is a sub-step flow chart of step A4 in one embodiment of the present application.

[0065] Figure 4 This is a flowchart of steps added before step S3 in one of the embodiments of the present application.

[0066] Figure 5 It is a sub-step flow chart of step S3 in one embodiment of the present application.

[0067] Figure 6 This is a flowchart of steps added before step A1 in one of the embodiments of the present application.

[0068] Figure 7 It is a sub-step flow chart of step A10 in one embodiment of the present application.

[0069] Figure 8 It is a structural diagram of an on-site target detection and labeling system according to one embodiment of the present application.

[0070] Fig. 9 It is a principle block diagram of an electronic device in one embodiment of the present application.

[0071] Figure numerals: 1. Video acquisition module; 2. Video frame extraction module; 3. Global candidate region generation module; 4. Global feature map projection module; 5. Candidate region processing module; 6. Position acquisition module; 7. Local video acquisition module; 8. Local feature map projection module. DETAILED DESCRIPTION

[0072] The following is combined with Figure 1-9 This application is described in further detail.

[0073] refer to Figure 1 , a method for on-site target detection and labeling, specifically comprising:

[0074] S0: Obtain global video of the entire detection area in real time.

[0075] Specifically, by acquiring global video of the entire detection area in real time, comprehensive coverage and real-time monitoring of the construction site can be achieved, the timeliness and comprehensiveness of target detection can be improved, and the real-time and accuracy of the detection results can be ensured.

[0076] S1: Obtain a global video frame according to a video picture of the global video.

[0077] S2: Based on the RPN network in the Faster R-CNN algorithm, several global candidate regions of the global video frame are generated.

[0078] S3: Projecting several global candidate regions onto a preset feature map, determining whether the feature representations of the several global candidate regions match the preset feature map of the target object, if so, executing A1, if not, executing A2.

[0079] Specifically, when most of the anchor points of the feature representation and the feature map are overlapped, there are still some reasons that affect the video image quality such as occlusion, which affects the matching of the feature representation of the global candidate area with the feature map of the preset target object, resulting in the anchor points of the feature representation and the feature map being not basically consistent, and further feature recognition is needed.

[0080] A1: Obtain global or local candidate regions, classify and regress the feature representations of the candidate regions, use NMS to eliminate several overlapping regions, retain the region with the highest confidence, and output the category, location, and corresponding confidence detection results of the target object.

[0081] Specifically, since RPN may generate multiple overlapping candidate regions, NMS is needed to eliminate these overlapping regions and retain only the regions with the highest confidence, thereby improving the accuracy and robustness of target detection, reducing false detections in overlapping regions, and improving the reliability of detection results.

[0082] A2: Get the feature representation position of the mismatched global candidate region and execute A3.

[0083] A3: Obtain a partial video of the monitoring detection area that matches the feature point position, and execute A4.

[0084] Specifically, since the target object cannot be accurately identified in the current global candidate area, in order to further confirm that the target object is the identified object, feature recognition needs to be performed through the local video.

[0085] A4: Use the Faster R-CNN algorithm to perform feature recognition on the local video, obtain the candidate region of the local video, and execute A1.

[0086] In this embodiment, the Faster R-CNN algorithm is used to select appropriate preset target objects for model training according to specific needs, including steps such as forward propagation, loss calculation, back propagation and updating model parameters, and the rapid labeling of surveillance videos is achieved through continuous adjustment and optimization of the model. Faster R-CNN achieves high-precision object detection performance through the combination of a two-stage network and RPN. This two-stage design enables Faster R-CNN to more accurately locate and classify target objects during the detection process; Faster R-CNN's two-stage network has more advantages in dealing with multi-scale and small target problems, and can effectively identify target objects of different sizes and scales in the image, including those small targets that are difficult to be detected by other algorithms; There are many points that can be optimized in the entire algorithm framework of Faster R-CNN, providing a broad space for algorithm optimization.

[0087] Furthermore, global monitoring is used to obtain global video, and the continuous video stream is converted into a series of static image frames, which provides a basis for subsequent candidate region generation and feature matching for feature recognition, which can reduce the amount of video for feature recognition. When the global video monitoring cannot identify the feature map of the preset target object, the local video of local monitoring is combined to narrow the monitoring range of feature recognition for more accurate feature recognition, achieving efficient tracking and labeling of different targets in large-scale video data, which can effectively reduce the number of missed items.

[0088] refer to Figure 2 Further, in one embodiment, before step S3, steps S30 and S31 are added:

[0089] S30: Projecting several global candidate regions onto the feature map of the pile foundation to obtain the pile foundation annotation.

[0090] S31: Divide the area enclosed by every four diagonal pile foundations to obtain pile foundation areas.

[0091] In this embodiment, based on the area defined by the four diagonal piles, when the feature representation of the global candidate area fails to match the feature map of the preset target object, the feature representation of the candidate area where the mismatch occurs can be accurately located, and then the mismatched feature representation can be confirmed to be located at a specific position between the piles. Subsequently, the local video resources in the pile area are mobilized to perform targeted local feature recognition to improve the comprehensiveness and accuracy of target detection and reduce possible recognition omissions.

[0092] In addition, reference Figure 3 Further, in one embodiment, step A4 is further divided into the following sub-steps:

[0093] A40: Acquire a local video frame according to the video picture of the local video located in each pile foundation area.

[0094] A41: Generate several local candidate regions of the local video frame based on the RPN network in the Faster R-CNN algorithm.

[0095] A42: Project several local candidate regions onto a preset feature map of the target object to obtain feature representations of the local candidate regions.

[0096] A5: Use the Faster R-CNN algorithm to determine whether the feature representation of the current local video frame matches the preset feature map of the target object. If so, execute A1; if not, execute A6.

[0097] A6: Control the capture device of the local video to switch the capture angle, obtain the local video frame of the capture device of the switched capture angle at the next time node, and execute A5.

[0098] Specifically, by switching the viewing angle of the local video capture device, the target object in the local area can be more comprehensively identified, thereby reducing the situation where feature recognition failures due to occlusion of the capture screen of the local video capture device are caused.

[0099] In this embodiment, by applying the RPN network in the Faster R-CNN algorithm, multiple local candidate areas extracted from local video frames in each pile foundation area can be identified. The purpose is to capture feature representations that fail to match the preset feature map in the global video frame, thereby improving the accuracy of target detection and feature recognition, while reducing false detections and missed detections, and reducing the waste of computing resources.

[0100] In addition, reference Figure 4 Further, in one embodiment, before step S3, steps S32, S33, S34, and S35 are added:

[0101] S32: Extract features from the construction drawings through a deep convolutional network and convert them into a first feature map, where the first feature map includes pile foundations.

[0102] S33: Extract features of the construction drawings through a deep convolutional network and convert them into a second feature map, where the second feature map includes feature maps of equipment and vehicles.

[0103] S34: Through a deep convolutional network, feature extraction is performed on the portrait photos and construction material library captured at the punch-in point, and the photos are converted into a third feature map, which includes the portrait, construction tools, and surrounding materials.

[0104] Specifically, constructing the third feature map can provide accurate reference information for subsequent detection of human portraits, construction tools and surrounding materials and identification of construction behaviors.

[0105] S35: Integrate the first feature map, the second feature map, and the third feature map into a feature map.

[0106] In this embodiment, the deep convolutional network constructs feature maps by extracting the feature maps of pile foundations, equipment and vehicles in the construction drawings, as well as the features of portrait photos and the construction material library, so as to ensure that the Faster R-CNN algorithm can comprehensively capture the feature information of various targets, thereby improving the accuracy and comprehensiveness of target detection and meeting the needs of general construction sites for identification features.

[0107] In addition, reference Figure 5 Further, in one embodiment, step S3 is refined into the following sub-steps:

[0108] S36: Determine whether the feature representations of the pile foundation images of the plurality of pile foundation areas match the first feature map. If they match, execute A7; if not, generate feature representation positions of the non-matching pile foundation areas and execute A2.

[0109] A7: Determine whether the equipment and vehicles in the pile foundation areas match the second feature map. If they match, execute A1; if they do not match, generate the feature representation position of the unmatched pile foundation area and execute A2.

[0110] In this embodiment, the current position of the pile foundation is first determined through global video recognition, and then more precise positioning is performed using local video to ensure the accuracy of the pile foundation position as much as possible. Equipment and vehicles are identified and labeled in each pile foundation area, which meets the object requirements of most construction site target identification and effectively copes with the diversity and complexity of targets in large-scale video data, thereby improving the comprehensiveness and accuracy of detection and identification work.

[0111] In addition, reference Figure 6 Further, in one embodiment, before step A1, steps A10 and A11 are added:

[0112] A10: Determine whether the construction tools and surrounding materials in the pile foundation areas match the third feature map. If they match, the construction tools and surrounding materials are combined to obtain the construction behavior that each portrait matches.

[0113] A11: If there is no match, generate the feature representation position of the unmatched pile foundation area and execute A2.

[0114] In this embodiment, the construction tools and surrounding materials in the pile foundation area are identified to determine whether they match the characteristics of the construction tools and surrounding materials contained in the third feature map. By comprehensively analyzing the characteristics of these construction tools and surrounding materials, the construction behavior that matches each portrait can be more accurately identified, thereby improving the accuracy and efficiency of construction behavior detection and reducing the possibility of false detection and missed detection.

[0115] In addition, reference Figure 7 Further, in one embodiment, step A10 is further divided into the following sub-steps:

[0116] A100: Obtain each construction arrangement in a preset construction plan, where the construction arrangement includes a collection of images of construction tools and required construction materials for implementing the construction behavior.

[0117] A101: If the feature representation of the portrait photo is detected in the pile base area, several portrait candidate regions are generated through the RPN network in the Faster R-CNN algorithm.

[0118] A102: Determine whether the construction tools and surrounding materials in the candidate regions of the portraits match the third feature map. If they match, match the construction tools and surrounding materials with each set in the construction arrangement to obtain the construction behavior that each portrait matches.

[0119] In this embodiment, by optimizing and narrowing the candidate area range of the RPN network, we focus on the portrait candidate area around the portrait to identify the features of construction tools and surrounding materials in the portrait candidate area, thereby improving the accuracy of matching with the construction arrangement, and then being able to accurately determine the construction actions of the current portrait, thereby achieving efficient target tracking and rapid labeling.

[0120] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0121] The embodiment of the present application also provides a field target detection and labeling system, which corresponds one-to-one to the field target detection and labeling method in the embodiment.

[0122] refer to Figure 8A field target detection and labeling system includes: a video acquisition module 1, a video frame extraction module 2, a global candidate region generation module 3, a global feature map projection module 4, a candidate region processing module 5, a position acquisition module 6, a local video acquisition module 7, and a local feature map projection module 8. The functional modules are described in detail as follows:

[0123] Video acquisition module 1: Video acquisition module 1 is used to acquire the global video of the entire detection area in real time.

[0124] Video frame extraction module 2: Video frame extraction module 2 is used to obtain a global video frame based on the video picture of the global video.

[0125] Global candidate region generation module 3: The global candidate region generation module 3 is used to generate several global candidate regions of the global video frame according to the RPN network in the Faster R-CNN algorithm.

[0126] Global feature map projection module 4: The global feature map projection module 4 is used to project several global candidate areas onto a preset feature map, and determine whether the feature representations of the several global candidate areas match the preset feature map of the target object. If they match, execute A1; if not, execute A2.

[0127] Candidate region processing module 5: Candidate region processing module 5 is used to execute A1: obtain global or local candidate regions, classify and regress feature representations of candidate regions, use NMS to eliminate several overlapping regions, retain the region with the highest confidence, and output the category, position and corresponding confidence detection results of the target object.

[0128] Position acquisition module 6: Position acquisition module 6 is used to execute A2: obtain the feature representation position of the unmatched global candidate area, and execute A3.

[0129] Local video acquisition module 7: The local video acquisition module 7 is used to execute A3: obtain the local video of the monitoring part detection area that matches the feature point position, and execute A4.

[0130] Local feature map projection module 8: The local feature map projection module 8 is used to execute A4: perform feature recognition on the local video through the Faster R-CNN algorithm, obtain the candidate area of ​​the local video, and execute A1.

[0131] Among them, the video acquisition module 1 captures the global video of the monitoring area in real time, the video frame extraction module 2 extracts key frames from the global video, the global candidate region generation module 3 uses the RPN network in the Faster R-CNN algorithm to identify candidate regions that may contain target objects, and the global feature map projection module 4 compares these candidate regions with the preset feature map to determine the matching situation. If they match, the candidate region processing module 5 will classify and regress these regions, use the non-maximum suppression (NMS) technology to remove overlapping areas, and finally output the category, position and confidence of the target object; if they do not match, the position acquisition module 6 will determine the position of the feature representation and guide the local video acquisition module 7 to obtain the matching local video. The local feature map projection module 8 then performs feature recognition on the local video and executes the steps of the candidate region processing module 5 again to achieve accurate tracking and labeling of the target object.

[0132] For the specific definition of the on-site target detection and labeling system, please refer to the definition of the on-site target detection and labeling method in the context, which will not be repeated here. Each module in the above-mentioned on-site target detection and labeling system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. In one embodiment, an electronic device is provided, which is a user terminal. Reference Fig. 9 , the electronic device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store detection data tables. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for on-site target detection labeling is implemented.

[0133] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0134] S0: Obtain global video of the entire detection area in real time.

[0135] S1: Obtain a global video frame according to a video picture of the global video.

[0136] S2: Based on the RPN network in the Faster R-CNN algorithm, several global candidate regions of the global video frame are generated.

[0137] S3: Projecting several global candidate regions onto a preset feature map, determining whether the feature representations of the several global candidate regions match the preset feature map of the target object, if so, executing A1, if not, executing A2.

[0138] A1: Obtain global or local candidate regions, classify and regress the feature representations of the candidate regions, use NMS to eliminate several overlapping regions, retain the region with the highest confidence, and output the category, location, and corresponding confidence detection results of the target object.

[0139] A2: Get the feature representation position of the mismatched global candidate region and execute A3.

[0140] A3: Obtain a partial video of the monitoring detection area that matches the feature point position, and execute A4.

[0141] A4: Use the Faster R-CNN algorithm to perform feature recognition on the local video, obtain the candidate region of the local video, and execute A1.

[0142] In one embodiment, the steps added after step S3 include:

[0143] S30: Projecting several global candidate regions onto the feature map of the pile foundation to obtain the pile foundation annotation.

[0144] S31: Divide the area enclosed by every four diagonal pile foundations to obtain pile foundation areas.

[0145] In one embodiment, the detailed sub-steps of step A4 include:

[0146] A40: Acquire a local video frame according to the video picture of the local video located in each pile foundation area.

[0147] A41: Generate several local candidate regions of the local video frame based on the RPN network in the Faster R-CNN algorithm.

[0148] A42: Project several local candidate regions onto a preset feature map of the target object to obtain feature representations of the local candidate regions.

[0149] A5: Use the Faster R-CNN algorithm to determine whether the feature representation of the current local video frame matches the preset feature map of the target object. If so, execute A1; if not, execute A6.

[0150] A6: Control the capture device of the local video to switch the capture angle, obtain the local video frame of the capture device of the switched capture angle at the next time node, and execute A5.

[0151] In one embodiment, the steps added before step S3 include:

[0152] S32: Extract features from the construction drawings through a deep convolutional network and convert them into a first feature map, where the first feature map includes pile foundations.

[0153] S33: Extract features of the construction drawings through a deep convolutional network and convert them into a second feature map, where the second feature map includes feature maps of equipment and vehicles.

[0154] S34: Through a deep convolutional network, feature extraction is performed on the portrait photos and construction material library captured at the punch-in point, and the photos are converted into a third feature map, which includes the portrait, construction tools, and surrounding materials.

[0155] S35: Integrate the first feature map, the second feature map, and the third feature map into a feature map.

[0156] In one embodiment, the detailed sub-steps of step S3 include:

[0157] S36: Determine whether the feature representations of the pile foundation images of the plurality of pile foundation areas match the first feature map. If they match, execute A7; if not, generate feature representation positions of the non-matching pile foundation areas and execute A2.

[0158] A7: Determine whether the equipment and vehicles in the pile foundation areas match the second feature map. If they match, execute A1; if they do not match, generate the feature representation position of the unmatched pile foundation area and execute A2.

[0159] In one embodiment, the steps added before step S1 include:

[0160] A10: Determine whether the construction tools and surrounding materials in the pile foundation areas match the third feature map. If they match, the construction tools and surrounding materials are combined to obtain the construction behavior that each portrait matches.

[0161] A11: If there is no match, generate the feature representation position of the unmatched pile foundation area and execute A2.

[0162] In one embodiment, the sub-steps of step A10 include:

[0163] A100: Obtain each construction arrangement in a preset construction plan, where the construction arrangement includes a collection of images of construction tools and required construction materials for implementing the construction behavior.

[0164] A101: If the feature representation of the portrait photo is detected in the pile base area, several portrait candidate regions are generated through the RPN network in the Faster R-CNN algorithm.

[0165] A102: Determine whether the construction tools and surrounding materials in the candidate regions of the portraits match the third feature map. If they match, match the construction tools and surrounding materials with each set in the construction arrangement to obtain the construction behavior that each portrait matches.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0167] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A method for on-site target detection and labeling, characterized in that: include: Obtain global video monitoring of the entire detection area in real time; Acquire a global video frame according to the video picture of the global video; According to the RPN network in the Faster R-CNN algorithm, a plurality of global candidate regions of the global video frame are generated; Projecting the plurality of global candidate regions onto a preset feature map, determining whether the feature representations of the plurality of global candidate regions match the preset feature map of the target object, if so, executing A1, if not, executing A2; A1: Obtain the global or local candidate region, classify and regress the feature representation of the candidate region, use NMS to eliminate several overlapping regions, retain the region with the highest confidence, and output the category, position and corresponding confidence detection results of the target object; A2: Obtain the feature representation position of the unmatched global candidate region and execute A3; A3: Obtain a local video of the monitoring part detection area matching the feature representation position, and execute A4; A4: Perform feature recognition on the local video using the Faster R-CNN algorithm to obtain a candidate region of the local video, and execute A1.

2. The method according to claim 1, characterized in that Before projecting the plurality of global candidate regions onto a preset feature map, determining whether the feature representations of the plurality of global candidate regions match the preset feature map of the target object, and if so, executing A1, and if not, executing A2, the method further includes: Projecting the global candidate regions onto a feature map of a pile foundation to obtain a pile foundation annotation; The area enclosed by every four diagonal pile foundations is divided to obtain the pile foundation area.

3. The method according to claim 2, characterized in that The A4 includes: Acquire a local video frame according to the video picture of the local video located in each pile foundation area; According to the RPN network in the Faster R-CNN algorithm, a plurality of local candidate regions of the local video frame are generated; Projecting the plurality of local candidate regions onto a preset feature map of the target object to obtain feature representations of the local candidate regions; A5: Use the Faster R-CNN algorithm to determine whether the feature representation of the current local video frame matches the preset feature map of the target object. If so, execute A1; if not, execute A6. A6: Control the capture device of the partial video to switch the capture angle, obtain the partial video frame of the capture device of the switched capture angle at the next time node, and execute A5.

4. The method according to claim 2, characterized in that: Before projecting the plurality of global candidate regions onto a preset feature map, determining whether the feature representations of the plurality of global candidate regions match the preset feature map, and if so, executing A1, and if not, executing A2, the method further includes: Extracting features from the construction drawings through a deep convolutional network and converting them into a first feature map, wherein the first feature map includes pile foundations; Through a deep convolutional network, feature extraction is performed on the construction drawing and the result is converted into a second feature map, where the second feature map includes feature maps of equipment and vehicles; Through a deep convolutional network, feature extraction is performed on the portrait photos and construction material library captured at the punch-in point, and the features are converted into a third feature map, where the third feature map includes the portrait, construction tools, and surrounding materials; The first feature map, the second feature map and the third feature map are integrated into the feature map.

5. The method according to claim 4, characterized in that The step of projecting the plurality of global candidate regions onto a preset feature map, determining whether the feature representations of the plurality of global candidate regions match the preset feature map of the target object, and if so, executing A1, and if not, executing A2, comprises: Determine whether the feature representations of the pile foundation images of the pile foundation areas match the first feature map, if they match, execute A7; if not, generate the feature representation positions of the unmatched pile foundation areas, and execute A2; A7: Determine whether the equipment and vehicles in the pile foundation area match the second feature map. If so, execute A1; if not, generate a feature representation position of the unmatched pile foundation area and execute A2.

6. The method according to claim 4, characterized in that Before A1, it also includes: Determine whether the construction tools and surrounding materials in the pile foundation area match the third feature map. If they match, obtain the construction behavior that each portrait matches by combining the construction tools and surrounding materials. If there is no match, the feature representation position of the unmatched pile foundation area is generated and A2 is executed.

7. The method according to claim 6, characterized in that The step of determining whether the construction tools and surrounding materials in the pile foundation area match the third feature map, and if so, combining the construction tools and surrounding materials to obtain the construction behavior that each portrait matches, includes: Acquire each construction arrangement in a preset construction plan, wherein the construction arrangement includes a collection of images of construction tools and required construction materials for implementing the construction behavior; If the feature representation of the portrait photo is detected in the pile foundation area, a plurality of portrait candidate areas are generated through the RPN network in the Faster R-CNN algorithm; Determine whether the construction tools and surrounding materials in the candidate portrait areas match the third feature map. If they match, match the construction tools and surrounding materials with each set in the construction arrangement to obtain the construction behavior that each portrait meets.

8. A field target detection and labeling system, characterized in that: include: Video acquisition module (1): the video acquisition module (1) is used to acquire a global video of the entire detection area in real time; Video frame extraction module (2): the video frame extraction module (2) is used to obtain a global video frame according to the video picture of the global video; A global candidate region generation module (3): the global candidate region generation module (3) is used to generate a plurality of global candidate regions of the global video frame according to the RPN network in the Faster R-CNN algorithm; A global feature map projection module (4): The global feature map projection module (4) is used to project the plurality of global candidate regions onto a preset feature map, and determine whether the feature representations of the plurality of global candidate regions match the preset feature map of the target object; if they match, execute A1; if not, execute A2; Candidate region processing module (5): the candidate region processing module (5) is used to execute A1: obtain the global or local candidate region, classify and regress the feature representation of the candidate region, use NMS to eliminate several overlapping regions, retain the region with the highest confidence, and output the category, position and corresponding confidence detection results of the target object; Position acquisition module (6): the position acquisition module (6) is used to execute A2: obtain the feature representation position of the unmatched global candidate area, and execute A3; Local video acquisition module (7): the local video acquisition module (7) is used to execute A3: obtain a local video of the monitoring part detection area matching the feature representation position, and execute A4; Local feature map projection module (8): The local feature map projection module (8) is used to execute A4: perform feature recognition on the local video using the Faster R-CNN algorithm, obtain a candidate area of ​​the local video, and execute A1.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes a method for on-site target detection and labeling as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and executes the on-site target detection and labeling method according to any one of claims 1 to 7.

Images