A construction machinery detection method based on large model

Abstract

The present application relates to the field of digital image processing technology, and in particular to a construction machinery detection method based on a large model. In the embodiment of the present application, a small target detection model is used to determine whether there is a construction machinery in each image frame to be detected in the video. If the second number of target image frames containing construction machinery in the first number of consecutive image frames to be detected is greater than a quantity threshold, the target image frame is detected based on a multimodal large model to determine whether there is a construction machinery in the target image frame. If so, an alarm message is output. Only when the small target detection model detects multiple target image frames containing construction machinery in the first number of consecutive image frames to be detected, the multimodal model is used to identify the target image frame, which can effectively filter out accidental false detections of the small target detection model, thereby improving the accuracy of construction machinery detection.

Description

Technical Field

[0001] The present application relates to the technical field of digital image processing, and in particular to a construction machinery detection method based on a large model. Background Art

[0002] In the process of modern railway maintenance and surrounding environment construction, construction machinery such as excavators, bulldozers, road rollers, cranes, etc. play an important role. However, the operation and operation process of these equipment may affect the safety of train operation. For example, excavators may damage railway infrastructure during excavation, bulldozers may change the terrain around the railway during bulldozing, road rollers may put pressure on railway tracks when compacting the road surface, and cranes may pose a threat to railway electrical facilities when lifting heavy objects. These may lead to safety problems in train operation, such as derailment, collision, power failure, etc. Therefore, the management of construction machinery requires strict regulations and monitoring to ensure the safety of train operation.

[0003] In the current field of construction machinery inspection, the main inspection methods still rely on manual inspection and small model inspection. However, both methods have some obvious problems and challenges. First, manual inspection is inefficient, inaccurate, and time-consuming, which may become a bottleneck for efficiency and safety in large-scale construction projects. Secondly, although small model inspection can be automated through machine learning and artificial intelligence technology, the noise of images taken by high-speed trains may be relatively large, and small models may cause false detection due to image noise.

[0004] Therefore, how to improve the accuracy of construction machinery detection has become an urgent problem to be solved. Summary of the invention

[0005] The embodiment of the present application provides a large model-based construction machinery detection method to solve the problem of low construction machinery detection accuracy in the prior art.

[0006] The present application provides a large-model-based construction machinery detection method, the method comprising:

[0007] Determine whether there is construction machinery in each image frame to be detected in the video based on the target detection model;

[0008] If the second number of target image frames containing construction machinery among the first number of consecutive image frames to be detected is greater than the quantity threshold, the target image frames are detected based on the multimodal large model to determine whether there is construction machinery in the target image frames. If so, an alarm message is output.

[0009] The above technical solution determines whether there is construction machinery in each image frame to be detected in the video based on the target detection small model. If the second number of target image frames with construction machinery in the first number of consecutive image frames to be detected is greater than the number threshold, the target image frame is detected based on the multimodal large model to determine whether there is construction machinery in the target image frame. If so, an alarm message is output. Only when the target detection small model detects multiple target image frames with construction machinery in the first number of consecutive image frames to be detected, the multimodal model is used to identify the target image frame, which can effectively filter out accidental false detections of the target detection small model, thereby improving the accuracy of construction machinery detection.

[0010] Furthermore, the method further comprises:

[0011] If the second number of target image frames containing construction machinery in the first number of consecutive image frames to be detected is not greater than the number threshold, a false detection prompt message is output and the detection is terminated.

[0012] In the above technical scheme, if the second number of target image frames containing construction machinery in the first number of consecutive image frames to be detected by the target detection small model is not greater than the number threshold, it is considered that a false detection has occurred in the target detection small model, and a false detection prompt message can be output, and the detection is terminated, thereby improving the accuracy of construction machinery detection.

[0013] Furthermore, the detecting the target image frame based on the multimodal large model to determine whether there is a construction machine in the target image frame includes:

[0014] Obtaining the confidence level of the existence of the construction machinery in each of the target image frames determined by the target detection mini-model;

[0015] The target image frame corresponding to the maximum confidence is input into the multimodal large model to determine whether there is a construction machine in the target image frame corresponding to the maximum confidence.

[0016] Since there may be multiple target image frames, it will take a long time to input each target image frame into the multimodal large model. Therefore, the above technical solution inputs the target image frame corresponding to the maximum confidence into the multimodal large model to determine whether there is construction machinery in the target image frame corresponding to the maximum confidence, thereby improving the efficiency of construction machinery detection.

[0017] Furthermore, after obtaining the confidence that a construction machine exists in each target image frame determined by the target detection small model, and before inputting the target image frame corresponding to the maximum confidence into the multimodal large model, the method further includes:

[0018] If the maximum confidence is greater than the confidence threshold, and the area of ​​the construction machinery in the target image frame corresponding to the maximum confidence is greater than the first area threshold, then continue to execute the subsequent step of inputting the target image frame corresponding to the maximum confidence into the multimodal large model.

[0019] In the above technical solution, if the target detection small model believes that there is construction machinery, but the area of ​​the construction machinery in the target image frame is not within a reasonable range, the multimodal large model is used for re-inspection, thereby improving the accuracy of construction machinery detection.

[0020] Furthermore, the method further comprises:

[0021] If the maximum confidence is greater than the confidence threshold and the area is greater than the second area threshold and not greater than the first area threshold, it is determined that there is construction machinery in the target image frame corresponding to the maximum confidence, and an alarm message is output, wherein the second area threshold is smaller than the first area threshold.

[0022] In the above technical solution, if the target detection small model believes that there is construction machinery and the area of ​​the construction machinery in the target image frame is within a reasonable range, it is not necessary to call the multimodal large model for re-inspection and directly output the alarm information, thereby further improving the accuracy of construction machinery detection.

[0023] Furthermore, if the multimodal large model determines that there is no construction machinery in the target image frame, it is determined based on the image-text matching model whether the description text output by the multimodal large model matches the target image frame; if so, it is determined that there is no construction machinery in the target image frame.

[0024] In the above technical scheme, when the detection results of the small model and the large model are different, that is, the small model believes that there is construction machinery in the target image frame, while the large model believes that there is no construction machinery in the target image frame, a graphic-text matching model is used to determine whether the description text output by the multimodal large model matches the target image frame, and the accuracy and reliability of the fusion judgment of the large and small models are improved by calculating the graphic-text matching degree.

[0025] Further, the multimodal large model includes a Mamba encoder, a multilayer perceptron, a text encoder and a Mamba model;

[0026] The detecting the target image frame based on the multimodal large model to determine whether there is a construction machine in the target image frame includes:

[0027] Inputting the target image frame into the Mamba encoder for feature extraction to obtain a first feature matrix;

[0028] Inputting the first feature matrix into the multilayer perceptron for feature mapping to obtain a second feature matrix;

[0029] Inputting the prompt text into the text encoder for feature extraction to obtain a third feature matrix, wherein the prompt text is used to prompt the multimodal large model to process the target image frame;

[0030] The second feature matrix and the third feature matrix are fused and input into the Mamba model to obtain a description text for describing whether a construction machine exists in the target image frame.

[0031] In the above technical solution, the Mamba model is a new type of deep learning model, and its biggest feature is that it has linear computational complexity. When the Mamba model is introduced into a large multimodal model, the computation time of the Mamba model will maintain linear growth regardless of the size of the input data, thereby greatly improving the computational efficiency.

[0032] Furthermore, the process of the Mamba encoder extracting features from the target image frame includes:

[0033] Dividing the target image frame into a plurality of sub-images according to a preset resolution;

[0034] sorting the plurality of sub-images according to different sorting strategies respectively to obtain a plurality of sub-image sequences, wherein any sub-image sequence includes all the sub-images;

[0035] Perform feature extraction on each sub-image sequence respectively to obtain a fourth feature matrix;

[0036] The fourth characteristic matrix corresponding to each sub-image sequence is fused to obtain the first characteristic matrix.

[0037] The above technical solution arranges each sub-image according to different sorting strategies to obtain multiple sub-image sequences, and extracts features from each sub-image sequence to obtain the fourth feature matrix. By combining all sub-images in different arrangement methods, the image information can be considered from different angles, thereby extracting richer and more accurate image features.

[0038] Furthermore, the multimodal large model is obtained by fine-tuning training based on the instruction data set;

[0039] The construction process of the instruction data set includes:

[0040] Get a sample image;

[0041] Recognize the sample image based on the pre-trained multimodal large model to obtain a global description;

[0042] Based on the small target detection model, the location information of the construction machinery included in the sample image is identified; the construction machinery is identified in the sample image according to the location information to obtain a labeled image; based on the pre-trained multimodal large model, the labeled image is identified to obtain a first local description;

[0043] Cutting out a sub-image including the construction machinery from the sample image, and identifying the sub-image based on the pre-trained multimodal large model to obtain a second local description;

[0044] The sample image, as well as the input data and output data each time the pre-trained multimodal large model is called are correspondingly saved, the input data including the image and the prompt text, and the output data including the global description and the local description.

[0045] The above technical solution, a method for constructing a multi-level and multi-scale instruction dataset, improves the detection effect of the multimodal large model on construction machinery by fine-tuning training based on the instruction dataset constructed in this way.

[0046] Furthermore, a sub-image including the construction machinery is cropped from the sample image in at least one of the following ways:

[0047] Determine a marking frame of the construction machine in the sample image according to the position information, and cut out an image area corresponding to the marking frame in the sample image to obtain the sub-image;

[0048] A marking box of the construction machine in the sample image is determined according to the position information, and a coverage area of ​​the marking box in the sample image is enlarged by a set multiple and cropped to obtain the sub-image.

[0049] The above technical solution constructs instruction data sets of different levels based on sub-images of different sizes of the same machine, which effectively enriches the instruction data set. The multimodal large model trained based on the rich instruction data set can perform detection more accurately, thereby improving the robustness of the multimodal large model.

[0050] The present application provides a detection device for construction machinery based on a large model, the device comprising:

[0051] A first detection module is used to determine whether there is a construction machine in each image frame to be detected in the video based on a small target detection model;

[0052] The second detection module is used to detect the target image frames based on the multimodal large model to determine whether there are construction machinery in the target image frames if the second number of target image frames containing construction machinery in the consecutive first number of image frames to be detected is greater than the number threshold, and output an alarm message if so.

[0053] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned large-model-based construction machinery detection methods when executing a computer program stored in a memory.

[0054] The present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned large-model-based construction machinery detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 A schematic diagram of a flow chart of a large-model-based construction machinery detection process provided in an embodiment of the present application;

[0057] Figure 2a A schematic diagram of the detection results of the pre-trained small model provided in the embodiment of the present application;

[0058] Figure 2b A schematic diagram of the detection results of the small target detection model after class enhancement provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a large-model-based detection process of a construction machinery provided in an embodiment of the present application;

[0060] Figure 4a A schematic diagram of target image frame segmentation provided in an embodiment of the present application;

[0061] Figure 4b A schematic diagram of a first method for sorting sub-images provided in an embodiment of the present application;

[0062] Figure 4c A schematic diagram of a second method for sorting sub-images provided in an embodiment of the present application;

[0063] Figure 4d A schematic diagram of a third method for sorting sub-images provided in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of the architecture of a multimodal large model provided in an embodiment of the present application;

[0065] Figure 6aA schematic diagram of a sample image provided in an embodiment of the present application;

[0066] Figure 6b A schematic diagram of annotated image provided in an embodiment of the present application;

[0067] Figure 6c A schematic diagram of a sub-image provided in an embodiment of the present application;

[0068] Figure 6d A schematic diagram of a sub-image provided in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of the structure of a detection device for construction machinery based on a large model provided in an embodiment of the present application;

[0070] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0072] An embodiment of the present application provides a construction machinery detection method based on a large model. In this method, it is determined whether there is construction machinery in each image frame to be detected in the video based on a small target detection model; if a second number of target image frames in which construction machinery exists in a first number of consecutive image frames to be detected is greater than a quantity threshold, the target image frames are detected based on the multimodal large model to determine whether there is construction machinery in the target image frames, and if so, an alarm message is output.

[0073] Figure 1 A flow chart of a large-scale model-based construction machinery detection process provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the process includes the following steps:

[0074] S101: Determine whether there is construction machinery in each image frame to be detected in the video based on the target detection model.

[0075] The large-model-based construction machinery detection method provided in the present application is applied to electronic equipment, which may be a server, a PC, a mobile terminal, and the like.

[0076] If all the image frames in the video are processed using a large multimodal model alone, it will take a long time. In order to improve the efficiency of construction machinery detection, in an embodiment of the present application, it is possible to determine whether there is construction machinery in each image frame to be detected in the video based on a small target detection model. The video may be a video captured by an image acquisition device installed on the locomotive. For example, during the movement of the train, the image acquisition device may capture a video of the road section that the train passes through, and the video may clearly record the environmental information around the road section that the train passes through. The video may be an offline video or an online video.

[0077] In the embodiment of the present application, each image frame in the video can be determined as an image frame to be detected, or part of the image frames in the video can be determined as image frames to be detected. For example, different frame extraction strategies can be formulated in advance for different vehicle speeds, wherein the frame extraction strategy can be to extract an image frame in the video as an image frame to be detected at every interval of a set time length, the faster the vehicle speed, the shorter the set time length, and the slower the vehicle speed, the longer the set time length.

[0078] In order to improve the accuracy of the small target detection model, in an embodiment of the present application, the small target detection model may be a model obtained after class enhancement. In an embodiment of the present application, the pre-trained model may be fine-tuned based on a private data set, and the pre-trained model may be obtained based on training of a general data set. Among them, the private data set may be images containing specific targets, such as construction machinery such as excavators, bulldozers, and tower cranes. Exemplarily, the original deep learning model pre-trained on the COCO data set is used to detect 80 types of targets. Through fine-tuning training, this model can be transformed into a model that only detects 3 types of targets, such as excavators, bulldozers, and tower cranes. Among them, the deep learning model can be understood as a small model.

[0079] S102: If a second number of target image frames containing construction machinery among a first number of consecutive image frames to be detected is greater than a quantity threshold, the target image frames are detected based on a multimodal large model to determine whether there is a construction machinery in the target image frames, and if so, an alarm message is output.

[0080] The small model for target detection may have some false detections in actual detection due to its simple structure. In the continuous image frames to be detected, if an object really exists, it should be continuously detected in multiple image frames to be detected. On the contrary, if it only appears in a single image frame to be detected, it is likely to be a false detection caused by image noise or instantaneous picture changes. Therefore, in order to improve the accuracy of construction machinery detection, in an embodiment of the present application, a post-processing logic is proposed. After obtaining the results of the detection of the small model for target detection, it can be determined whether the second number of target image frames containing construction machinery in the continuous first number of image frames to be detected is greater than the quantity threshold. Among them, the first number and the quantity threshold can be any positive integers, and the quantity threshold is less than the first number.

[0081] If it is determined that the second number is greater than the number threshold, it can be considered that there is indeed construction machinery in the target image frame. In order to further determine whether there is construction machinery in the target image frame, the target image frame can be detected based on the multimodal large model to determine whether there is construction machinery in the target image frame. If so, an alarm message is output to facilitate relevant staff to go to the scene for processing based on the alarm message.

[0082] Specifically, if there are 2 target image frames containing construction machinery among 5 consecutive image frames to be detected, then it can be determined that construction machinery does exist, and the multimodal large model can be further called to detect the target image frames to determine whether there is construction machinery in the target image frames. In other words, the target detection small model must detect construction machinery twice in 5 consecutive image frames to be detected before inputting the corresponding target image frames into the multimodal large model for detection. Such rules help reduce misjudgments caused by noise in a single frame image or instantaneous picture changes. In this way, the detection of construction machinery can be ensured to be more stable and reliable.

[0083] In the embodiment of the present application, it is determined whether there is construction machinery in each image frame to be detected in the video based on the target detection small model. If the second number of target image frames with construction machinery in the first number of consecutive image frames to be detected is greater than the number threshold, the target image frame is detected based on the multimodal large model to determine whether there is construction machinery in the target image frame. If so, an alarm message is output. Only when the target detection small model detects multiple target image frames with construction machinery in the first number of consecutive image frames to be detected, the multimodal model is used to identify the target image frame, which can effectively filter out accidental false detections of the target detection small model, thereby improving the accuracy of construction machinery detection.

[0084] The following is a description of the class strengthening process of the target detection model in conjunction with a specific embodiment. The existing pre-trained model weights usually include a large number of categories, including large construction machinery such as trucks. In specific images, these pre-trained models can already detect some targets such as excavators, bulldozers, road rollers and cranes, but because the models are not specifically trained for these targets, their detection effects are often poor. Therefore, it is necessary to fine-tune the pre-trained model based on a private training set. The detection effect of this fine-tuning method mainly depends on three factors: the number of private data sets, the richness of image data, and the experience of the fine-tuning personnel. First, the more private data sets there are, the more target features the model can learn, thereby improving the accuracy of detection. Secondly, the richness of image data will also affect the detection effect. If the image data contains the state of the target in various environments and conditions, the model can better adapt to various detection scenarios. Finally, the experience of the fine-tuning personnel is also very important. Experienced fine-tuning personnel can better understand the working principle of the model and adjust the parameters of the model more accurately, thereby improving the detection effect. Therefore, in order to improve the effect of object detection fine-tuning methods, we need to spend a lot of time and money on data collection, and the experience of the fine-tuning personnel is also a decisive factor.

[0085] Therefore, we can fine-tune the model by adding more images of construction machinery and a few images of other categories. In this way, the model can maintain its original detection capabilities while improving the detection effect on specific target categories. Specifically, taking the Yolov9 detection model pre-trained on the COCO dataset as an example, since the 80 categories detected by the Yolov9 model include the truck category, the original Yolov9 model can detect some images of construction machinery, but the precision and recall rates are low. Figure 2a As shown, Figure 2a A schematic diagram of the detection results of the pre-trained small model provided in the embodiment of the present application, Figure 2a There are two excavators on the rails, but based on the pre-trained small model, only Figure 2a The location information of the excavator identified in the figure is relatively low, with a confidence level of only 0.26. The other excavator was not identified because it overlapped with the roadside guardrail.

[0086] In order to fine-tune the pre-trained small model, more construction machinery images can be collected and annotated, such as 2,000 construction machinery images, 1,000 difficult samples, and 1,000 images without construction machinery. Among them, difficult samples can be understood as images containing construction machinery with special postures. The model can easily detect the construction machinery in these images as other objects, such as roofs, dams, etc. For example, when an excavator is working, the excavating arm may travel at an angle of 90°. For construction machinery with this posture, the model can easily identify the construction machinery as a roof. After collecting all the images, these images can be added to the training data set so that the model can learn more construction machinery features during the training process. At the same time, a small number of images of other categories can also be added to enhance the generalization ability of the model so that it can have better detection effects when facing unseen targets. Figure 2b A schematic diagram of the detection results of the small target detection model after the class enhancement provided in the embodiment of the present application, such as Figure 2b As shown in the figure, for the same image, the small target detection model after class enhancement can identify both construction machines in the image with relatively high confidence.

[0087] In order to further improve the accuracy of construction machinery detection, based on the above embodiment, in the embodiment of the present application, the method further includes:

[0088] If the second number of target image frames containing construction machinery in the first number of consecutive image frames to be detected is not greater than the number threshold, a false detection prompt message is output and the detection is terminated.

[0089] If the second number of target image frames containing construction machinery in the first number of consecutive image frames to be detected is not greater than the number threshold, a false detection prompt message is output and the detection is terminated.

[0090] Specifically, if the construction machinery is not detected multiple times in any five consecutive image frames to be detected among all the image frames to be detected, it can be considered as a false detection, a false detection prompt message can be output, and the detection is terminated.

[0091] In order to further improve the efficiency of construction machinery detection, on the basis of the above embodiments, in the embodiment of the present application, the target image frame is detected based on the multimodal large model to determine whether there is a construction machinery in the target image frame, including:

[0092] Obtaining the confidence level of the existence of the construction machinery in each of the target image frames determined by the target detection mini-model;

[0093] The target image frame corresponding to the maximum confidence is input into the multimodal large model to determine whether there is a construction machine in the target image frame corresponding to the maximum confidence.

[0094] Since there may be many target image frames of construction machinery, it will take a lot of time to detect all of these target image frames using a large multimodal model. Therefore, in order to further improve the efficiency of construction machinery detection, in an embodiment of the present application, one target image frame can be selected from multiple target image frames for processing.

[0095] In an embodiment of the present application, when the target detection model outputs the position information of the construction machinery in each target image frame, it can also output the confidence level of the existence of the construction machinery in each target image frame, that is, the possibility that the target in the corresponding position information of the output target image frame is indeed a construction machinery.

[0096] After obtaining the confidence corresponding to each target image frame, the target image frame corresponding to the maximum confidence can be input into the multimodal large model, so that the multimodal large model determines whether there is a construction machine in the target image frame corresponding to the maximum confidence.

[0097] In order to further improve the efficiency of construction machinery detection, on the basis of the above embodiments, in the embodiment of the present application, after obtaining the confidence that a construction machinery exists in each of the target image frames determined by the target detection small model, and before inputting the target image frame corresponding to the maximum confidence into the multimodal large model, the method further includes:

[0098] If the maximum confidence is greater than the confidence threshold, and the area of ​​the construction machinery in the target image frame corresponding to the maximum confidence is greater than the first area threshold, then continue to execute the subsequent step of inputting the target image frame corresponding to the maximum confidence into the multimodal large model.

[0099] Since the construction machinery will not be very close to the railway or on the rails during the construction process near the railway, it is found based on this feature that if the construction machinery is close to the image acquisition device, the area of ​​the construction machinery in the captured image will be too large. Therefore, a first area threshold is set in the embodiment of the present application. Among them, the confidence threshold can be 1 / 250 of the area of ​​the target image frame, that is, the first surface threshold is dynamically determined with the resolution of the target image frame, and the first area threshold can also be a fixed value, such as 8000 pixels. After obtaining the confidence that the construction machinery exists in each target image frame determined by the target detection model, it can be determined whether the maximum confidence is greater than the confidence threshold. If so, it can be considered that the target detection model believes that the possibility of the construction machinery existing in the target image frame corresponding to the maximum confidence is greater. In order to further determine the detection result, it can also be further determined whether the area of ​​the construction machinery in the target image frame corresponding to the maximum confidence is greater than the first area threshold. Exemplarily, the number of pixels included in the image area corresponding to the construction machinery in the target image frame can be counted to determine whether the number of pixels is greater than the first area threshold of 8000 pixels. That is to say, the area can be identified by the number of pixels included in the corresponding image area. If it is determined that the area of ​​the construction machinery in the target image frame is greater than the first area threshold, it means that the area of ​​the construction machinery is too large, and it may be a false detection or it may not be a construction machine, but the target detection small model believes that there is a high possibility that construction machinery exists in the target image frame. Then, in an embodiment of the present application, when the area of ​​the construction machinery in the target image frame corresponding to the maximum confidence is greater than the first area threshold, the subsequent step of inputting the target image frame corresponding to the maximum confidence into the multimodal large model can be continued, so that the multimodal large model can further verify whether there is construction machinery in the target image frame.

[0100] In order to further improve the efficiency of construction machinery detection, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0101] If the maximum confidence is greater than the confidence threshold and the area is greater than the second area threshold and not greater than the first area threshold, it is determined that there is construction machinery in the target image frame corresponding to the maximum confidence, and an alarm message is output, wherein the second area threshold is smaller than the first area threshold.

[0102] Since there are not many construction machines near the railway, if there are construction machines near the railway, then the construction machines are also at a certain distance from the rails on which the train runs. Therefore, in the embodiment of the present application, a second area threshold is also pre-configured, and the second area threshold is less than the first area threshold. The area threshold interval composed of the first area threshold and the second area threshold can be considered to be a reasonable range. If the maximum confidence is greater than the confidence threshold, and the area of ​​the construction machine in the target image frame corresponding to the maximum confidence is greater than the second area threshold and not greater than the first area threshold, it means that the size of the construction machine in the target image frame is reasonable, then it can be determined that there is a construction machine in the target image frame corresponding to the maximum confidence, and the alarm information can be directly output. In other words, when the size of the construction machine in the target image frame corresponding to the maximum confidence is reasonable, it can be considered that there is indeed a construction machine in the target image frame.

[0103] In order to further improve the efficiency of construction machinery detection, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0104] If the multimodal large model determines that there is no construction machinery in the target image frame, then based on the image-text matching model, it is determined whether the description text output by the multimodal large model matches the target image frame. If they match, it is determined that there is no construction machinery in the target image frame.

[0105] Although the detection accuracy of the multimodal large model is high, there may still be the possibility of misjudgment for some complex scenes. The purpose of this application to process the target image frame using the multimodal large model is to further determine whether there is construction machinery in the target image frame. If the multimodal large model determines that there is no construction machinery in the target image frame, but the target detection small model believes that there is construction machinery in the target image frame, in order to verify whether the detection result of the multimodal large model is correct, in an embodiment of the present application, it can be determined based on the image-text matching model whether the description text output by the multimodal large model matches the target image frame. If the image-text matching model determines a match, it means that the description text output by the multimodal large model is correct, and it can be determined that there is no construction machinery in the target image frame.

[0106] In the embodiment of the present application, a closed-loop control method is proposed, that is, when the detection results of the small model and the large model are different, that is, when the small model believes that there is a construction machine in the target image frame, and the multimodal large model believes that there is no construction machine in the target image frame, the matching degree between the conclusion output by the multimodal large model and the target image frame can be determined based on the image-text matching model, thereby judging whether the output of the multimodal large model is reliable. In this way, the accuracy and reliability of the large and small model fusion judgment are improved.

[0107] The process of construction machinery detection is described below with reference to a specific embodiment. Figure 3 A schematic diagram of a detection process of a construction machinery based on a large model provided in an embodiment of the present application, such as Figure 3 As shown, when performing construction machinery inspection, a video can be acquired, the video can be decoded, and a corresponding target frame extraction strategy can be determined according to the vehicle speed and the video frame rate, and the image frame to be inspected can be extracted from the video based on the target frame extraction strategy.

[0108] After obtaining each image frame to be detected, the target detection model with class enhancement is used to analyze and judge each image frame to be detected, and based on the post-processing logic, it is determined whether the detection result of the target detection model is a false detection. Among them, the post-processing logic is: at least 2 target image frames with construction machinery are detected in 5 consecutive image frames to be detected. If the post-processing logic is not satisfied, it can be determined that there is no construction machinery in the image frame to be detected, and the target detection model can be continued to be called to continue other detections.

[0109] If the post-processing logic is satisfied, the confidence of the existence of construction machinery in each target image frame determined by the target detection model is obtained. If the maximum confidence is greater than the set threshold, such as 0.85, and the area of ​​the construction machinery in the target image frame is within the second area threshold and the first area threshold, for example, the number of pixel points of the image area corresponding to the construction machinery in the target image frame is within 1600-8000 pixels, it can be determined that the construction machinery exists in the target image frame corresponding to the maximum confidence, and an alarm message is output.

[0110] If the area of ​​the construction machinery in the target image frame is greater than the first area threshold, that is, the number of pixels in the image area corresponding to the construction machinery in the target image frame is greater than 8,000 pixels, it can be considered as some false detections, and the multimodal large model can be used for more in-depth analysis and judgment. If the result output by the multimodal large model believes that there is construction machinery in the target image frame, it means that the detection results of the large and small models are consistent, and the result can be directly output and saved. If the result output by the multimodal large model believes that there is no construction machinery in the target image frame, it means that the detection results of the large and small models are inconsistent. The image-text matching model can be called to determine whether the description text output by the multimodal large model matches the target image frame. If it matches, the output result of the multimodal large model can be trusted to determine that there is no construction machinery in the target image frame, and subsequent detection can be continued. If it does not match, it can be considered that there is indeed construction machinery in the target image frame, and an alarm message can be output.

[0111] If the maximum confidence is not greater than the set threshold, it means that the features of the construction machinery in the target image frame are not obvious, and the multimodal large model can be directly used for secondary judgment. Since the fine-tuned multimodal large model usually has a stronger ability to understand images, it can handle more complex tasks and identify construction machinery that is difficult for small models to identify. The output of the multimodal large model can be directly used as the final result.

[0112] In order to further improve the accuracy of construction machinery detection, on the basis of the above embodiments, in the embodiment of the present application, the multimodal large model includes a Mamba encoder, a multilayer perceptron, a text encoder and a Mamba model;

[0113] The detecting the target image frame based on the multimodal large model to determine whether there is a construction machine in the target image frame includes:

[0114] Inputting the target image frame into the Mamba encoder for feature extraction to obtain a first feature matrix;

[0115] Inputting the first feature matrix into the multilayer perceptron for feature mapping to obtain a second feature matrix;

[0116] Inputting the prompt text into the text encoder for feature extraction to obtain a third feature matrix, wherein the prompt text is used to prompt the multimodal large model to process the target image frame;

[0117] The second feature matrix and the third feature matrix are fused and input into the Mamba model to obtain a description text for describing whether a construction machine exists in the target image frame.

[0118] At present, the multimodal large model is mainly based on the structure of Transformer, and the Transformer model has the problem of low computational efficiency when processing large-scale data. This is because the self-attention mechanism of the Transformer model needs to calculate the relationship between all pairs of elements in the input sequence, and its computational complexity and memory complexity are both the square level of the input length, which will cause significant computing resource consumption when processing long sequence data. In order to solve this problem, the Mamba model can be introduced into the multimodal large model. The Mamba model is a new type of deep learning model, and its biggest feature is that it has linear computational complexity. This means that no matter how the scale of the input data grows, the calculation time of the Mamba model will keep linear growth, thereby greatly improving the computational efficiency. In an embodiment of the present application, the multimodal large model can include a Mamba encoder, a multilayer perceptron, a text encoder and a Mamba model.

[0119] When the target image frame is detected based on the multimodal large model to determine whether there is a construction machine in the target image frame, the target image frame can be input into the Mamba encoder for feature extraction to obtain a first feature matrix.

[0120] After obtaining the first feature matrix, the first feature matrix can be input into a multilayer perceptron (MLP) for feature mapping to obtain a second feature matrix. In other words, the multilayer perceptron performs feature mapping on the first feature matrix according to the input requirements of the Mamba model to obtain a second feature matrix.

[0121] When the target image frame is processed by the multimodal large model, not only the target image frame needs to be input, but also the corresponding prompt text needs to be input, so that the multimodal large model can process the received image based on the requirements of the prompt text. Therefore, in an embodiment of the present application, the prompt text received when the image is received can also be input into the text encoder, and the text encoder processes the received prompt text to obtain a third feature matrix. The prompt text can be received together with the video, or it can be pre-saved in a preset position, and the prompt text can be used to prompt the multimodal large model to process the target image frame.

[0122] After obtaining the second feature matrix and the third feature matrix, the second feature matrix and the third feature matrix can be fused and input into the Mamba model, and the Mamba model processes the second feature matrix based on the third feature matrix of the prompt text to obtain a description text for describing whether there is construction machinery in the target image frame.

[0123] In order to further improve the accuracy of construction machinery detection, on the basis of the above embodiments, in the embodiment of the present application, the process of the Mamba encoder extracting features from the target image frame includes:

[0124] Dividing the target image frame into a plurality of sub-images according to a preset resolution;

[0125] sorting the plurality of sub-images according to different sorting strategies respectively to obtain a plurality of sub-image sequences, wherein any sub-image sequence includes all the sub-images;

[0126] Perform feature extraction on each sub-image sequence respectively to obtain a fourth feature matrix;

[0127] The fourth characteristic matrix corresponding to each sub-image sequence is fused to obtain the first characteristic matrix.

[0128] In visual tasks, the Mamba encoder mainly adopts the method of scanning sub-images in the image row by row or column by column. This method has an obvious disadvantage, that is, it will cause the splitting of tokens that were originally adjacent in 2D space. This splitting increases the difficulty of extracting local area features, because in actual visual scenes, adjacent pixels usually have similar color and texture features, which are very important for understanding the semantic information of the image. In order to solve this problem, the embodiment of the present application proposes a local plus global scanning mechanism to enhance the feature extraction capability of the visual Mamba encoder. In the embodiment of the present application, when the Mamba encoder extracts features from the target image frame, it can divide the target image frame into multiple sub-images according to a preset resolution. For example, the target image frame is divided into 5*5 sub-images.

[0129] After obtaining each sub-image, all sub-images can be excluded according to different sorting strategies to obtain multiple sub-image sequences. Among them, any sub-image sequence includes all sub-images. In other words, all sub-images are sorted a set number of times, and the sorting strategy used in each sorting is different.

[0130] Specifically, Figure 4a A target image frame segmentation schematic diagram provided in an embodiment of the present application is as follows: Figure 4a As shown in , the target sub-image is divided into 12 sub-images. Figure 4a Each sub-image is labeled, and the labels corresponding to the four sub-images in the first row of the target image frame are 1, 2, 3, and 4, respectively, the labels corresponding to the four sub-images in the second row are 4, 6, 7, and 8, respectively, and the labels corresponding to the four sub-images in the third row are 9, 10, 11, and 12. When sorting all the sub-images, they can be sorted in the following three ways.

[0131] Method 1: for each row of sub-images in the target image frame, the sub-images corresponding to the row are added to the sub-image sequence in order from left to right until all sub-images in all rows are added to the sub-image sequence. Figure 4b A schematic diagram of a first method for sorting sub-images provided in an embodiment of the present application, such as Figure 4b As shown, when sorting each row, each sub-image is sorted from left to right in the direction of the arrow, and the obtained sub-image sequence is: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12.

[0132] Method 2: for each column of sub-images in the target image frame, the sub-images corresponding to the column are added to the sub-image sequence in order from top to bottom until all sub-images in all columns are added to the sub-image sequence. Figure 4cA schematic diagram of a second method for sorting sub-images provided in an embodiment of the present application, such as Figure 4c As shown, when sorting each column, each sub-image is sorted from top to bottom in the direction of the arrow, and the obtained sub-image sequence is: 1, 5, 9, 2, 6, 10, 3, 7, 11, 4, 8, 12.

[0133] Method three, the sub-images in the target image frame are grouped to obtain multiple image groups, and any image group at least includes sub-images located in different rows and columns in the target image frame. For each image group, the sub-images included in the image group are sorted, and the sorted sub-images are added to the sub-image sequence accordingly. When sorting the sub-images in the image group, a snake-like sorting can be performed starting from any sub-image, as long as it can be ensured that each sub-image in the image group has been sorted. For example, Figure 4d A schematic diagram of a third method for sorting sub-images provided in an embodiment of the present application is shown in FIG. Figure 4d As shown, sub-images 1, 2, 5, and 6 are divided into an image group, called image group 1; sub-images 3, 4, 7, and 8 are divided into an image group, called image group 2; and sub-images 9, 10, 11, and 12 are divided into an image group, called image group 3. When sorting image group 1, sub-image 1 can be used as the starting point for serpentine sorting, and the sorting is: 1, 2, 6, and 5. Of course, sub-image 2, sub-image 6, or sub-image 5 can also be used as the starting point for serpentine sorting. When sorting image group 2, sub-image 3 can be used as the starting point for serpentine sorting, and the sorting is: 3, 4, 8, and 7. Of course, sub-image 4, sub-image 8, or sub-image 7 can also be used as the starting point for serpentine sorting. When sorting image group 3, sub-image 12 can be used as the starting point for sorting. Since the sub-images included in the image group are all located in the same row in the target image frame, serpentine sorting cannot be performed when sorting, and linear sorting can be performed directly. After sorting, the sub-images are: 12, 11, 10, 9. Of course, sub-image 9 can also be used as the starting point for sorting, but the sorting order of 9, 10, 11, 12 in the above method 1 has already appeared, so it is better to use sub-image 12 as the starting point. After sorting each image group, the sub-image sequence can be obtained: 1, 2, 6, 5, 3, 4, 8, 7, 12, 11, 10, 9.

[0134] After each sub-image sequence is obtained based on each sorting strategy, for each sub-image sequence, the sub-images of the sub-image sequence may be spliced ​​in the sorting order, and features may be extracted from the spliced ​​images to obtain a fourth feature matrix corresponding to the sub-image sequence.

[0135] In the embodiment of the present application, since the method three groups the sub-images and scans each image group separately, the method three can be regarded as a local scan, while the methods one and two can be regarded as a global scan.

[0136] After obtaining the fourth characteristic matrix corresponding to each sub-image sequence, all the fourth characteristic matrices can be fused to obtain the first characteristic matrix. When performing the fusion process, each fourth characteristic matrix can be spliced ​​in sequence to obtain the first characteristic matrix. For example, the dimension of the fourth characteristic matrix corresponding to sub-image sequence 1 is A*M, the dimension of the fourth characteristic matrix corresponding to sub-image sequence 2 is B*M, and the dimension of the fourth characteristic matrix corresponding to sub-image sequence 3 is C*M. Then, the dimension of the first characteristic matrix after the fusion process is (A+B+C)*M.

[0137] In an embodiment of the present application, the image is first divided into multiple independent windows, and then scanned within the window. The advantage of this is that it can ensure that tokens belonging to the same local feature can be close to each other for feature extraction. Among them, the independent window can be understood as the above-mentioned image group. Then a global scanning strategy is adopted, including scanning sub-images row by row and column by column, to supplement the connection between sub-images. By adopting a local plus global scanning method, while significantly improving the model's ability to capture local details, the model's understanding of the global features of the image is improved.

[0138] The following describes a specific embodiment of the process of processing a received image by a multimodal large model. Figure 5 The following is a schematic diagram of the architecture of a multi-modal large model provided in an embodiment of the present application. Figure 5 As shown, first, the prompt text can be segmented and vectorized. This step is to convert the natural language prompt into a form that can be understood by the multimodal large model. After vectorization, the vector corresponding to the prompt text can be input into the text encoder for feature extraction, and each word can be converted into a high-dimensional third feature matrix. This third feature matrix can capture the semantic information of the word.

[0139] The received image is input to the Mamba encoder, which extracts features from the input image based on the strategy of local scanning plus global scanning to obtain the first feature matrix. This strategy effectively captures local dependencies by dividing the image into different windows while maintaining Mamba's original global dependency capabilities. In this way, the global and local information of the image can be considered at the same time, thereby extracting richer and more accurate image features.

[0140] After obtaining the first feature matrix, the first feature matrix can be input into a multilayer perceptron for feature mapping, thereby being aligned with the third feature matrix.

[0141] Finally, the third feature matrix and the second feature matrix are input into the Mamba model to obtain a description text for describing whether there is a construction machine in the target image frame. In the embodiment of the present application, the Transformer-based backbone large language model is replaced with a pre-trained Mamba model. The Mamba model has a linear computational complexity and is therefore more efficient when processing large-scale data.

[0142] In an embodiment of the present application, a local plus global scanning strategy based on the Mamba model is used to enhance the model's ability to capture image features. At the same time, since the Mamba model has a linear computational complexity characteristic, the multimodal large model adopts the Mamba structure, which can greatly improve the computational efficiency of the model.

[0143] In order to further improve the accuracy of construction machinery detection, on the basis of the above embodiments, in the embodiment of the present application, the multimodal large model is obtained by fine-tuning and training based on the instruction data set;

[0144] The construction process of the instruction data set includes:

[0145] Get a sample image;

[0146] Recognize the sample image based on the pre-trained multimodal large model to obtain a global description;

[0147] Based on the small target detection model, the location information of the construction machinery included in the sample image is identified; the construction machinery is identified in the sample image according to the location information to obtain a labeled image; based on the pre-trained multimodal large model, the labeled image is identified to obtain a first local description;

[0148] Cutting out a sub-image including the construction machinery from the sample image, and identifying the sub-image based on the pre-trained multimodal large model to obtain a second local description;

[0149] The sample image, as well as the input data and output data each time the multimodal large model is called are correspondingly saved, the input data including the image and the prompt text, and the output data including the global description and the local description.

[0150] When the open source multimodal large model is processing image recognition tasks, it does not perform well when detecting whether the image contains construction machinery. Since the open source multimodal large model does not conduct deep learning on the characteristics of construction machinery in a targeted manner, the accuracy is low when identifying such specific objects. In addition, the open source multimodal large model also has certain difficulties in explaining its recognition results. Therefore, for the open source multimodal large model, we need to further optimize and improve it to improve its performance in specific tasks, and also enhance its interpretability. Therefore, in an embodiment of the present application, the open source multimodal large model can be fine-tuned. When fine-tuning the open source multimodal large model, fine-tuning training can be performed based on the instruction data set. The construction process of the instruction data set is explained below.

[0151] In the embodiment of the present application, a detailed instruction data set can be constructed by multi-level and multi-scale fusion. First, a sample image can be obtained, and the sample image can be collected in advance by relevant staff.

[0152] After obtaining the sample image, the sample image can be identified based on the pre-trained multimodal large model to obtain a global description. In order to select a pre-trained multimodal large model with a high recognition accuracy for construction machinery, a relatively accurate global description can be obtained. An image containing construction machinery is input into the original multimodal large model and the GPT4O model, and the following different test results are obtained:

[0153] For the original multimodal large model:

[0154] Input an image containing construction machinery;

[0155] The question is: Please describe the image in detail;

[0156] Answer: The picture shows a village with railway tracks and many houses next to the tracks.

[0157] For the GPT4O model:

[0158] Input an image containing construction machinery;

[0159] The question is: Please describe the image in detail;

[0160] Answer: On April 11, 2033 at 17:05, it was a cloudy day and a train was passing through a village. The picture contained many yellow houses, green grass and a yellow excavator under construction.

[0161] From the above experimental results, it can be seen that GPT4O has good ability in construction machinery detection. Therefore, the pre-trained multimodal large model can be GPT4O. Of course, it can also be other large models, and technicians in this field can configure them as needed.

[0162] In the embodiment of the present application, other models can also be used to determine whether there is a construction machine in the sample image. Specifically, based on the small target detection model, the location information of the construction machine included in the sample image can be identified, and the construction machine can be identified in the sample image according to the location information to obtain a labeled image, in which the construction machine can be annotated in the form of a labeled box. After obtaining the labeled image, the labeled image can be identified based on the pre-trained multimodal large model to obtain a first local description.

[0163] In order to construct an instruction data set containing different levels, in an embodiment of the present application, a sub-image including construction machinery can be cropped out from the sample image, and the sub-image can be identified based on the pre-trained multimodal large model to obtain a second local description.

[0164] After obtaining the second local description, the sample image, as well as the input data and output data for each call to the pre-trained multimodal large model can be saved in the instruction data set accordingly. The input data includes the image and prompt text, and the output data includes the global description and the local description, wherein the local description includes the first local description and the second local description.

[0165] In order to further improve the accuracy of construction machinery detection, on the basis of the above embodiments, in the embodiment of the present application, a sub-image including the construction machinery is cropped from the sample image by at least one of the following methods:

[0166] Determine a marking frame of the construction machine in the sample image according to the position information, and cut out an image area corresponding to the marking frame in the sample image to obtain the sub-image;

[0167] A marking box of the construction machine in the sample image is determined according to the position information, and a coverage area of ​​the marking box in the sample image is enlarged by a set multiple and cropped to obtain the sub-image.

[0168] Since the focus of the pre-trained multimodal large model is different when images of different sizes are input into the pre-trained multimodal large model for processing, in an embodiment of the present application, sub-images of different sizes including construction machinery can be cropped out.

[0169] In a possible implementation, a marking box of the construction machine in the sample image may be determined according to the position information of the construction machine, and an image area corresponding to the marking box may be cropped out of the sample image to obtain a sub-image.

[0170] In a possible implementation, a marking box of the construction machinery in the sample image may be determined according to the position information, and the coverage area of ​​the marking box in the sample image may be enlarged by a set multiple and cropped to obtain a sub-image.

[0171] The construction process of instruction data is described below in conjunction with a specific embodiment.

[0172] 1. From the perspective of global description, GPT4O can be used to Figure 6a The image shown is processed, Figure 6a A schematic diagram of a sample image provided for an embodiment of the present application. Specifically, when inputting the image, the prompt text input together may be: Please describe the content in the image in detail? The reply information of GPT4O may be: At 17:05 on April 11, 2033, it was cloudy, and a train was passing through a village. The image contained many yellow houses, green grass, and a yellow excavator under construction. The reply information is the global description.

[0173] 2. Describe the local angle, which can be achieved by combining large and small models. Figure 6a First, a small model, such as the Yolo target detection series model, is used to detect the suspected large construction machinery in the image, and the target is framed in the image to obtain a labeled image. Figure 6b As shown, Figure 6b A schematic diagram of annotated image provided for an embodiment of the present application. Then ask GPT4O whether there is a large construction machine in the red box in the picture. Answer: There is a large yellow construction machine in the picture, and there is an arm-like hook at the front end, so it is an excavator. The reply information is the first partial description.

[0174] 3. Crop the image area corresponding to the annotation box in the annotated image and the surrounding image area of ​​448*448 pixels to obtain a sub-image. Figure 6c As shown, Figure 6c A schematic diagram of a sub-image provided in an embodiment of the present application. Figure 6cThe sub-image shown is input into the GPT4O model, and the question is: Are there large construction machines such as excavators, bulldozers, road rollers, and cranes in the picture? The GPT4O model outputs the following answer: According to the environment around the target machine, the target machine is at a construction site, it is a large yellow construction machine, and there is a hook like an arm in front of the machine, so it is judged to be an excavator. This answer is the second partial description.

[0175] 4. Crop the image area corresponding to the annotation box in the annotated image to obtain a sub-image. Figure 6d As shown, Figure 6d A schematic diagram of a sub-image provided in an embodiment of the present application. Figure 6d The sub-image shown is input into the GPT4O model, and the question is: Are there any large construction machinery such as excavators, bulldozers, road rollers, and cranes in the image? The GPT4O model outputs the answer: Based on the hook at the front end of the target object in the image, it is judged to be an excavator. This answer is the second partial description.

[0176] Based on the above sample images and the input data and output data each time the multimodal large model is called, the following multi-round dialogue can be constructed to guide the fine-tuning training of the original multimodal large model.

[0177]

[0178]

[0179] The detection method for construction machinery provided in this application mainly makes up for the problems of poor detection accuracy and complex training of small models for construction machinery from the aspects of small model training, large model effect improvement and large and small model fusion strategy. This detection scheme can still detect construction machinery events for the image of the same construction machinery even in different operating environments, which meets the characteristics of reproducibility in the trustworthy characteristics; this detection scheme can determine the detection degree of construction machinery through user-set hyperparameters (such as confidence threshold, etc.), and the degree of attention and adoption of event detection results can be intervened by users, which meets the characteristics of controllability in the trustworthy characteristics; in addition to the above two trustworthy characteristics, the construction machinery detection method based on large and small model fusion proposed in this scheme can greatly improve the detection efficiency and accuracy of construction machinery around railways.

[0180] Based on the same inventive concept, the embodiment of the present application also provides a detection device for construction machinery based on a large model. Figure 7 A schematic diagram of a detection device for construction machinery based on a large model provided in an embodiment of the present application, the device comprising:

[0181] The first detection module 701 is used to determine whether there is a construction machine in each image frame to be detected in the video based on the target detection model;

[0182] The second detection module 702 is used to detect the target image frames based on the multimodal large model to determine whether there are construction machinery in the target image frames if the second number of target image frames containing construction machinery in a first number of consecutive image frames to be detected is greater than a quantity threshold, and output an alarm message if so.

[0183] In a possible implementation, the second detection module 702 is further configured to output a false detection prompt message and terminate the detection if a second number of target image frames of construction machinery in a first number of consecutive image frames to be detected is not greater than the number threshold.

[0184] In a possible implementation, the second detection module 702 is specifically used to obtain the confidence that construction machinery exists in each of the target image frames determined by the target detection small model; the target image frame corresponding to the maximum confidence is input into the multimodal large model to determine whether construction machinery exists in the target image frame corresponding to the maximum confidence.

[0185] In a possible implementation, the second detection module 702 is also used to continue to execute the subsequent step of inputting the target image frame corresponding to the maximum confidence into the multimodal large model if the maximum confidence is greater than the confidence threshold and the area of ​​the construction machinery in the target image frame corresponding to the maximum confidence is greater than the first area threshold.

[0186] In a possible implementation, the second detection module 702 is also used to determine that there is construction machinery in the target image frame corresponding to the maximum confidence if the maximum confidence is greater than the confidence threshold and the area is greater than the second area threshold and not greater than the first area threshold, and output an alarm message, wherein the second area threshold is smaller than the first area threshold.

[0187] In a possible implementation, the second detection module 702 is also used to determine whether the description text output by the multimodal large model matches the target image frame based on a graphic-text matching model if the multimodal large model determines that there is no construction machinery in the target image frame; if so, determining that there is no construction machinery in the target image frame.

[0188] In one possible implementation, the multimodal large model includes a Mamba encoder, a multilayer perceptron, a text encoder, and a Mamba model;

[0189] The second detection module 702 is specifically used to input the target image frame into the Mamba encoder for feature extraction to obtain a first feature matrix; input the first feature matrix into the multi-layer perceptron for feature mapping to obtain a second feature matrix; input the prompt text into the text encoder for feature extraction to obtain a third feature matrix, and the prompt text is used to prompt the multimodal large model to process the target image frame; and fuse the second feature matrix and the third feature matrix and input them into the Mamba model to obtain a description text for describing whether there is construction machinery in the target image frame.

[0190] In a possible implementation, the second detection module 702 is specifically used to divide the target image frame into multiple sub-images according to a preset resolution; sort the multiple sub-images according to different sorting strategies to obtain multiple sub-image sequences, and any sub-image sequence includes all the sub-images; extract features from each sub-image sequence to obtain a fourth feature matrix; and fuse the fourth feature matrices corresponding to each sub-image sequence to obtain the first feature matrix.

[0191] In a possible implementation, the multimodal large model is obtained by fine-tuning training based on the instruction data set;

[0192] The device also includes:

[0193] Module 703 is constructed to obtain a sample image; recognize the sample image based on a pre-trained multimodal large model to obtain a global description; recognize the location information of the construction machinery included in the sample image based on the target detection small model; identify the construction machinery in the sample image according to the location information to obtain a labeled image; recognize the labeled image based on the pre-trained multimodal large model to obtain a first local description; crop a sub-image including the construction machinery from the sample image, recognize the sub-image based on the pre-trained multimodal large model to obtain a second local description; save the sample image, as well as the input data and output data corresponding to each time the pre-trained multimodal large model is called, the input data including the image and prompt text, and the output data including the global description and the local description.

[0194] In a possible implementation, the construction module 703 is specifically used to crop a sub-image including the construction machinery in the sample image in at least one of the following ways: determining a marking box of the construction machinery in the sample image based on the position information, and cropping an image area corresponding to the marking box in the sample image to obtain the sub-image; determining a marking box of the construction machinery in the sample image based on the position information, expanding the coverage area of ​​the marking box in the sample image by a set multiple and cropping it to obtain the sub-image.

[0195] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, it includes: a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804;

[0196] The memory 803 stores a computer program. When the program is executed by the processor 801, the processor 801 performs the following steps:

[0197] Determine whether there is construction machinery in each image frame to be detected in the video based on the target detection model;

[0198] If the second number of target image frames containing construction machinery among the first number of consecutive image frames to be detected is greater than the quantity threshold, the target image frames are detected based on the multimodal large model to determine whether there is construction machinery in the target image frames. If so, an alarm message is output.

[0199] In a possible implementation, the method further includes:

[0200] If the second number of target image frames containing construction machinery in the first number of consecutive image frames to be detected is not greater than the number threshold, a false detection prompt message is output and the detection is terminated.

[0201] In a possible implementation, the detecting the target image frame based on the multimodal large model to determine whether there is a construction machine in the target image frame includes:

[0202] Obtaining the confidence level of the existence of the construction machinery in each of the target image frames determined by the target detection mini-model;

[0203] The target image frame corresponding to the maximum confidence is input into the multimodal large model to determine whether there is a construction machine in the target image frame corresponding to the maximum confidence.

[0204] In a possible implementation, after obtaining the confidence that a construction machine exists in each of the target image frames determined by the target detection small model, and before inputting the target image frame corresponding to the maximum confidence into the multimodal large model, the method further includes:

[0205] If the maximum confidence is greater than the confidence threshold, and the area of ​​the construction machinery in the target image frame corresponding to the maximum confidence is greater than the first area threshold, then continue to execute the subsequent step of inputting the target image frame corresponding to the maximum confidence into the multimodal large model.

[0206] In a possible implementation, the method further includes:

[0207] If the maximum confidence is greater than the confidence threshold and the area is greater than the second area threshold and not greater than the first area threshold, it is determined that there is construction machinery in the target image frame corresponding to the maximum confidence, and an alarm message is output, wherein the second area threshold is smaller than the first area threshold.

[0208] In a possible implementation, the method further includes:

[0209] If the multimodal large model determines that there is no construction machinery in the target image frame, then based on the image-text matching model, it is determined whether the description text output by the multimodal large model matches the target image frame. If they match, it is determined that there is no construction machinery in the target image frame.

[0210] In one possible implementation, the multimodal large model includes a Mamba encoder, a multilayer perceptron, a text encoder, and a Mamba model;

[0211] The detecting the target image frame based on the multimodal large model to determine whether there is a construction machine in the target image frame includes:

[0212] Inputting the target image frame into the Mamba encoder for feature extraction to obtain a first feature matrix;

[0213] Inputting the first feature matrix into the multilayer perceptron for feature mapping to obtain a second feature matrix;

[0214] Inputting the prompt text into the text encoder for feature extraction to obtain a third feature matrix, wherein the prompt text is used to prompt the multimodal large model to process the target image frame;

[0215] The second feature matrix and the third feature matrix are fused and input into the Mamba model to obtain a description text for describing whether a construction machine exists in the target image frame.

[0216] In a possible implementation manner, the process of the Mamba encoder extracting features from the target image frame includes:

[0217] Dividing the target image frame into a plurality of sub-images according to a preset resolution;

[0218] sorting the plurality of sub-images according to different sorting strategies respectively to obtain a plurality of sub-image sequences, wherein any sub-image sequence includes all the sub-images;

[0219] Perform feature extraction on each sub-image sequence respectively to obtain a fourth feature matrix;

[0220] The fourth characteristic matrix corresponding to each sub-image sequence is fused to obtain the first characteristic matrix.

[0221] In a possible implementation, the multimodal large model is obtained by fine-tuning training based on the instruction data set;

[0222] The construction process of the instruction data set includes:

[0223] Get a sample image;

[0224] Recognize the sample image based on the pre-trained multimodal large model to obtain a global description;

[0225] Based on the small target detection model, the location information of the construction machinery included in the sample image is identified; the construction machinery is identified in the sample image according to the location information to obtain a labeled image; based on the pre-trained multimodal large model, the labeled image is identified to obtain a first local description;

[0226] Cutting out a sub-image including the construction machinery from the sample image, and identifying the sub-image based on the pre-trained multimodal large model to obtain a second local description;

[0227] The sample image, as well as the input data and output data each time the pre-trained multimodal large model is called are correspondingly saved, the input data including the image and the prompt text, and the output data including the global description and the local description.

[0228] In a possible implementation manner, a sub-image including the construction machinery is cropped from the sample image in at least one of the following ways:

[0229] Determine a marking frame of the construction machine in the sample image according to the position information, and cut out an image area corresponding to the marking frame in the sample image to obtain the sub-image;

[0230] A marking box of the construction machine in the sample image is determined according to the position information, and a coverage area of ​​the marking box in the sample image is enlarged by a set multiple and cropped to obtain the sub-image.

[0231] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 802 is used for communication between the above electronic device and other devices. The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0232] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0233] On the basis of the above-mentioned embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the large-model-based construction machinery detection method described in the above-mentioned embodiments.

[0234] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0235] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0236] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0237] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0238] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A construction machinery detection method based on a large model, characterized in that: The method comprises: Determine whether there is construction machinery in each image frame to be detected in the video based on the target detection model; If the second number of target image frames containing construction machinery among the first number of consecutive image frames to be detected is greater than the quantity threshold, the target image frames are detected based on the multimodal large model to determine whether there is construction machinery in the target image frames. If so, an alarm message is output.

2. The method according to claim 1, characterized in that: The method further comprises: If the second number of target image frames containing construction machinery in the first number of consecutive image frames to be detected is not greater than the number threshold, a false detection prompt message is output and the detection is terminated.

3. The method according to claim 1, characterized in that The detecting the target image frame based on the multimodal large model to determine whether there is a construction machine in the target image frame includes: Obtaining the confidence level of the existence of the construction machinery in each of the target image frames determined by the target detection mini-model; The target image frame corresponding to the maximum confidence is input into the multimodal large model to determine whether there is a construction machine in the target image frame corresponding to the maximum confidence.

4. The method according to claim 3, characterized in that After obtaining the confidence that the construction machinery exists in each target image frame determined by the target detection small model, and before inputting the target image frame corresponding to the maximum confidence into the multimodal large model, the method further includes: If the maximum confidence is greater than the confidence threshold, and the area of ​​the construction machinery in the target image frame corresponding to the maximum confidence is greater than the first area threshold, then continue to execute the subsequent step of inputting the target image frame corresponding to the maximum confidence into the multimodal large model.

5. The method according to claim 4, characterized in that The method further comprises: If the maximum confidence is greater than the confidence threshold and the area is greater than the second area threshold and not greater than the first area threshold, it is determined that there is construction machinery in the target image frame corresponding to the maximum confidence, and an alarm message is output, wherein the second area threshold is smaller than the first area threshold.

6. The method according to claim 1, characterized in that The method further comprises: If the multimodal large model determines that there is no construction machinery in the target image frame, then based on the image-text matching model, it is determined whether the description text output by the multimodal large model matches the target image frame. If they match, it is determined that there is no construction machinery in the target image frame.

7. The method according to claim 1, characterized in that The multimodal large model includes a Mamba encoder, a multilayer perceptron, a text encoder and a Mamba model; The detecting the target image frame based on the multimodal large model to determine whether there is a construction machine in the target image frame includes: Inputting the target image frame into the Mamba encoder for feature extraction to obtain a first feature matrix; Inputting the first feature matrix into the multilayer perceptron for feature mapping to obtain a second feature matrix; Inputting the prompt text into the text encoder for feature extraction to obtain a third feature matrix, wherein the prompt text is used to prompt the multimodal large model to process the target image frame; The second feature matrix and the third feature matrix are fused and input into the Mamba model to obtain a description text for describing whether a construction machine exists in the target image frame.

8. The method according to claim 7, characterized in that The process of the Mamba encoder extracting features from the target image frame includes: Dividing the target image frame into a plurality of sub-images according to a preset resolution; sorting the plurality of sub-images according to different sorting strategies respectively to obtain a plurality of sub-image sequences, wherein any sub-image sequence includes all the sub-images; Perform feature extraction on each sub-image sequence respectively to obtain a fourth feature matrix; The fourth characteristic matrix corresponding to each sub-image sequence is fused to obtain the first characteristic matrix.

9. The method according to any one of claims 1 to 8, characterized in that: The multimodal large model is obtained by fine-tuning training based on the instruction data set; The construction process of the instruction data set includes: Get a sample image; Recognize the sample image based on the pre-trained multimodal large model to obtain a global description; Based on the small target detection model, the location information of the construction machinery included in the sample image is identified; the construction machinery is identified in the sample image according to the location information to obtain a labeled image; based on the pre-trained multimodal large model, the labeled image is identified to obtain a first local description; Cutting out a sub-image including the construction machinery from the sample image, and identifying the sub-image based on the pre-trained multimodal large model to obtain a second local description; The sample image, as well as the input data and output data each time the pre-trained multimodal large model is called are correspondingly saved, the input data including the image and the prompt text, and the output data including the global description and the local description.

10. The method according to claim 9, characterized in that A sub-image including the construction machinery is cut out from the sample image in at least one of the following ways: Determine a marking frame of the construction machine in the sample image according to the position information, and cut out an image area corresponding to the marking frame in the sample image to obtain the sub-image; A marking box of the construction machine in the sample image is determined according to the position information, and a coverage area of ​​the marking box in the sample image is enlarged by a set multiple and cropped to obtain the sub-image.

Images