Excavator loading confirmation method, system and equipment based on video recognition and medium

Abstract

The invention belongs to the technical field of equipment monitoring, and particularly discloses an excavator loading confirmation method, system and equipment based on video recognition and a medium. The method disclosed by the invention comprises the following steps: firstly, calling a shot excavator loading video; then, the loading moment of the excavator is obtained through a vibration sensor or an excavator loading video; secondly, frame extraction processing is carried out on the loading video of the excavator based on the loading moment, and a loading picture is extracted; thirdly, analyzing and calculating the picture after frame extraction through an OCR (Optical Character Recognition) algorithm, identifying the number labeled on the excavator, and pairing the excavator with the truck; and finally, the pairing information and the photos are uploaded to a remote management platform for storage and counting. According to the invention, the matching of the excavator and the truck during each loading is carried out through video recording and analysis, so that the recording of the truck and the excavator for loading the truck during each loading is realized. The method can be widely applied to loading pairing recording of the excavator and the truck.

Description

Technical Field

[0001] The present invention belongs to the technical field of equipment monitoring, and particularly relates to a method, system, device and medium for confirming the loading of an excavator based on video recognition. Background Art

[0002] Loading an excavator's soil into a truck is a very common scenario. People often use the method of manual recording to record the loading volume of the excavator and the transportation volume of the truck. However, manual recording takes a long time and has low efficiency. To improve efficiency, wireless communication devices for short-range communication are installed on the excavator and the truck. If the wireless communication device on the excavator establishes a communication connection with the wireless communication device on a certain truck, it means that the two are very close, indicating that the excavator is loading the truck. The monitoring system will automatically determine the truck loaded by the excavator, and thus automatically record the loading volume of the excavator and the number of trips of the truck.

[0003] However, in the actual large-scale excavator loading scenario, the excavator closest to the truck is not necessarily the one loading the current truck. Using distance as the confirmation index for the excavator to load the truck often results in many errors. Therefore, how to accurately determine which truck the excavator is loading is an urgent problem to be solved in the construction field. Summary of the Invention

[0004] The object of the present invention is to provide a method for confirming the loading of an excavator based on video recognition, which pairs the excavator and the truck during each loading through video recording and analysis, so as to record the truck and the excavator loading the truck during each loading; The second object of the present invention is to provide a system for confirming the loading of an excavator based on video recognition, which is used to implement a method for confirming the loading of an excavator based on video recognition; The third object of the present invention is to provide a terminal device, which can implement a method for confirming the loading of an excavator based on video recognition when executing its own program; The fourth object of the present invention is to provide a computer-readable storage medium, which is used to store the corresponding computer program of a method for confirming the loading of an excavator based on video recognition.

[0005] The technical solutions adopted by the present invention to achieve the above objects are as follows: A method for confirming the loading of an excavator based on video recognition includes the following steps carried out in sequence: S1. Retrieve the video of the excavator loading the truck taken by the camera on the truck; S2. Obtain the loading moment of the excavator through the vibration sensor on the truck, or obtain the loading moment of the excavator through the video of the excavator loading the truck; S3. Perform frame extraction on the excavator loading video based on the loading time, and extract photos of the excavator loading the truck; S4. Parse and calculate the frame-extracted photos through the OCR recognition algorithm to identify the number on the label of the excavator; S5. Pair the excavator with the truck according to the excavator number identified in step S4; S6. Upload the paired excavator and truck information and the photo identifying the excavator number to the remote management platform, and the remote management platform counts the number of times the excavator loads the truck and the number of trips the truck transports.

[0006] As a limitation, the obtaining of the loading time of the excavator through the excavator loading video in step S2 includes the following steps carried out in sequence: S21. Manually extract the picture of the excavator bucket unloading soil above the truck carriage from the video and label the picture; S22. Train the labeled pictures using the YOLO algorithm to generate a loading time recognition model; S23. Put the excavator loading video into the loading time recognition model to extract the loading time of the excavator.

[0007] As the second limitation, when performing frame extraction in step S3, extract the video content 20 seconds before and 20 seconds after the loading time in the excavator loading video, extract one frame per second, and each frame corresponds to one photo.

[0008] As the third limitation, the label on the excavator is pasted on the bucket handle, boom, fuselage and counterweight at the tail of the excavator.

[0009] An excavator loading confirmation system based on video recognition for implementing the above-mentioned excavator loading confirmation method based on video recognition, including a shooting module with a camera, a vibration sensing module with a vibration sensor, a loading confirmation module and a remote management platform; The shooting module is used to shoot the loading picture of the excavator for the truck and transmit the shot excavator loading video to the loading confirmation module; The vibration sensing module is used to sense the vibration of the truck carriage after it stops moving and transmit the generated vibration information to the loading confirmation module; The loading confirmation module is used to receive the excavator loading video transmitted by the shooting module and the vibration information transmitted by the vibration sensing module, generate paired excavator and truck information and photos identifying the excavator number after processing, and upload them to the remote management platform; The remote management platform is used to receive and store the paired excavator and truck information and the photos identifying the excavator number transmitted by the loading confirmation module, and count the number of times the excavator loads the truck and the number of trips the truck transports.

[0010] As a limitation, the vibration sensing module further includes a truck motion sensor, and the vibration sensor starts to work only after the truck motion sensor detects that the truck has stopped moving.

[0011] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-described method for confirming the loading of an excavator based on video recognition is implemented.

[0012] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the above-described method for confirming the loading of an excavator based on video recognition.

[0013] Due to the adoption of the above technical solutions, compared with the prior art, the technical progress achieved by the present invention lies in: (1) The method of the present invention records and analyzes videos to pair the excavator and the truck during each loading, realizing the recording of the truck and the excavator loading the truck during each loading, facilitating the management of the excavator loading. (2) In the method of the present invention, the OCR recognition algorithm is used to recognize the numbers on the excavator, which is efficient, fast, and has high recognition accuracy. (3) In the method of the present invention, the YOLO algorithm is used to recognize the loading moment of the excavator. The multi-scale processing ability of the YOLO algorithm can ensure that the excavator can be accurately recognized at different positions and distances, thus enabling the recognition of the loading moment with high accuracy. (4) In the method of the present invention, videos 20 seconds before and 20 seconds after the loading moment are extracted during the frame extraction process, ensuring that the excavator captured in the photos obtained by frame extraction is the excavator loading the truck, improving the accuracy of pairing the excavator and the truck. (5) In the system of the present invention, the vibration sensing module includes a vibration sensor and a truck motion sensor. After the truck motion sensor detects that the truck has stopped moving, the vibration sensor starts to detect the vibration of the carriage, avoiding misjudgment of the carriage vibration caused by the movement of the truck.

[0014] The present invention belongs to the technical field of equipment monitoring. Through video recording and analysis, it is possible to record the truck and the excavator loading the truck during each loading. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention.

[0016] In the drawings: Figure 1It is the processing flow chart of Embodiment 1 of the present invention; Figure 2 It is the marked picture of the excavator bucket unloading soil in Embodiment 1 of the present invention; Figure 3 It is the processing flow chart of Embodiment 2 of the present invention; Figure 4 It is the system structure diagram of Embodiment 3 of the present invention. Specific Embodiments

[0017] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for the purpose of illustrating and explaining the present invention, and are not used to limit the present invention.

[0018] Embodiment 1 A method for confirming the loading of an excavator based on video recognition As Figure 1 shown, this embodiment is a method for confirming the loading of an excavator based on video recognition, including the following steps carried out in sequence: S1. Retrieve the video of the excavator loading the truck taken by the camera on the truck.

[0019] In this embodiment, the camera on the truck is located at the top of the truck cab, and the lens of the camera faces the truck carriage.

[0020] In this embodiment, the loading of the excavator is confirmed by analyzing the video of the excavator loading the truck. Therefore, it is necessary to first retrieve the video of the excavator loading the truck taken by the camera on the truck.

[0021] S2. Obtain the loading moment of the excavator from the video of the excavator loading the truck.

[0022] Specifically, it includes the following steps carried out in sequence: S21. Manually extract the picture of the excavator bucket unloading soil above the truck carriage from the video, and mark the picture, as Figure 2 shown.

[0023] S22. Train the marked picture using the YOLO algorithm to generate a loading moment recognition model.

[0024] S23. Put the video of the excavator loading the truck into the loading moment recognition model to extract the loading moment of the excavator.

[0025] The YOLO (You Only Look Once) algorithm is a real-time object detection algorithm based on deep learning. The algorithm regards the object detection problem as a regression problem and completes the recognition and localization of objects by looking at the image once. The algorithm uses a single neural network to predict the image, directly outputting the coordinates of the bounding box, class probability, and confidence, achieving an end-to-end detection process. The training process of the algorithm requires a large amount of labeled data. Through training, the model can learn how to accurately achieve the automatic recognition and localization of the excavator loading behavior. In this embodiment, the loading moment is when a large amount of falling material is detected under the excavator bucket.

[0026] S3. Perform frame extraction on the excavator loading video based on the loading moment to extract photos of the excavator loading.

[0027] After confirming the loading moment, it is necessary to find the excavator at the loading moment. Therefore, when performing frame extraction, extract the video content 20 seconds before and 20 seconds after the loading moment in the excavator loading video, extract one frame per second, and each frame corresponds to a photo.

[0028] S4. Use the OCR recognition algorithm to parse and calculate the framed photos to identify the number on the label of the excavator.

[0029] The OCR (Optical Character Recognition) algorithm is an algorithm that can recognize and convert the text content in an image into editable text. The algorithm uses computer vision and pattern recognition technologies to automatically recognize and parse the text in the image. First, the algorithm will use image preprocessing technologies, such as denoising, binarization, and image enhancement, to improve the image quality and make the text more clearly distinguishable. Then, the algorithm will use a specific text recognition model to scan and recognize the preprocessed image word by word or line by line, and finally convert the recognized text into an editable text format. For the convenience of OCR algorithm recognition, labels containing numbers are pasted on the boom, arm, body, and counterweight of the excavator's tail. The label is a label with strong reflectivity, and when necessary, the label can be powered to make the number on the label glow.

[0030] S5. Pair the excavator with the truck according to the excavator number recognized in step S4.

[0031] Since the camera is bound to the truck and the number of the truck is known, after obtaining the number of the excavator, the excavator and the truck that are loading can be paired.

[0032] S6. Upload the paired excavator and truck information and photos identifying the excavator number to the remote management platform, and the remote management platform counts the number of excavator loadings and truck transportation trips.

[0033] In this embodiment, the algorithm for extracting the loading time of the excavator from the video and identifying the number on the excavator through the OCR recognition algorithm can be performed using an AI box. The AI ​​box can be installed on a truck and powered by the truck, which is convenient and practical. When uploading the excavator and truck pairing information and photos to the remote management platform, wireless communication network is used for transmission.

[0034] In summary, this embodiment pairs the excavator and the truck each time the vehicle is loaded through video recording and analysis, thereby realizing the recording of the truck and the excavator loading the truck each time the vehicle is loaded, and facilitating the management of the excavator loading.

[0035] Example 2 A method for confirming loading of an excavator based on video recognition like Figure 3 As shown, this embodiment is a method for confirming loading of an excavator based on video recognition, comprising the following steps performed in sequence: A1. Retrieve the video of the excavator loading the truck captured by the camera on the truck.

[0036] In this embodiment, the camera on the truck is located on the top of the truck cab, and the lens of the camera faces the truck compartment.

[0037] This embodiment analyzes the excavator loading video to confirm the excavator loading, so it is necessary to first retrieve the excavator loading video shot by the camera on the truck.

[0038] A2. Obtain the loading time of the excavator through the vibration sensor on the truck.

[0039] The truck is equipped with vibration sensors and truck motion sensors. The vibration sensor detects the vibration of the truck compartment only after the truck motion sensor detects that the truck has stopped moving, thus avoiding misjudgment of the vibration of the compartment caused by the movement of the truck. When the excavator is loading into the truck compartment, it will cause the truck compartment to vibrate violently. The vibration sensor collects the vibration of the compartment and uses it as the loading time of the excavator.

[0040] A3. Frame extraction is performed on the excavator loading video based on the loading time to extract photos of the excavator loading.

[0041] After confirming the loading time, it is necessary to find the excavator at the loading time. Therefore, when performing frame extraction processing, the video content of the excavator loading video 20 seconds before and 20 seconds after the loading time is extracted, and one frame is extracted per second, and each frame corresponds to a photo.

[0042] A4. Parse and calculate the photos after frame extraction through the OCR recognition algorithm to identify the number on the label of the excavator.

[0043] For the convenience of OCR algorithm recognition, labels containing numbers are pasted on the boom, arm, fuselage, and counterweight of the excavator. The labels are highly reflective, and when necessary, the labels can be powered to make the numbers on the labels glow.

[0044] A5. Pair the excavator with the truck according to the excavator number identified in step A4.

[0045] Since the camera is bound to the truck and the number of the truck is known, after obtaining the number of the excavator, the excavator and the truck during loading can be paired.

[0046] A6. Upload the paired excavator and truck information and the photo of the identified excavator number to the remote management platform, and the remote management platform counts the number of times the excavator is loaded onto the truck and the number of trips the truck makes.

[0047] In this embodiment, the algorithm for identifying the number on the excavator through the OCR recognition algorithm can be performed using an AI box. The AI box can be installed on the truck and powered by the truck, which is convenient and practical. When uploading the paired excavator and truck information and photos to the remote management platform, a wireless communication network is used for transmission.

[0048] In summary, in this embodiment, the sensor is combined with video recording and analysis to pair the excavator and the truck during each loading, realizing the recording of the truck and the excavator loading the truck during each loading, which facilitates the management of excavator loading.

[0049] Embodiment 3 An Excavator Loading Confirmation System Based on Video Recognition This embodiment is an excavator loading confirmation system based on video recognition, used to implement Embodiment 1 or Embodiment 2, including a shooting module containing a camera, a vibration sensing module containing a vibration sensor, a loading confirmation module, and a remote management platform.

[0050] The shooting module is used to shoot the loading picture of the excavator loading the truck and transmit the shot excavator loading video to the loading confirmation module. The shooting module contains a camera, and the camera is located on the top of the truck cab, facing the truck carriage.

[0051] The vibration sensing module is used to sense the vibration of the truck carriage after the truck stops moving and transmit the generated vibration information to the loading confirmation module.

[0052] The vibration sensing module includes a truck motion sensor in addition to the vibration sensor. The vibration sensor starts to work only after the truck motion sensor detects that the truck has stopped moving.

[0053] The loading confirmation module is used to receive the loading video of the excavator transmitted by the shooting module and the vibration information transmitted by the vibration sensing module. After processing, it generates paired excavator and truck information and a photo for identifying the excavator number, and uploads them to the remote management platform.

[0054] The remote management platform is used to receive and store the paired excavator and truck information and the photo for identifying the excavator number transmitted by the loading confirmation module, and count the number of excavator loading times and the number of truck transportation trips.

[0055] Embodiment 4 A terminal device This embodiment is a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it is used to implement a method for confirming the loading of an excavator based on video recognition according to Embodiment 1 or Embodiment 2.

[0056] Embodiment 5 A computer-readable storage medium This embodiment is a computer-readable storage medium. The computer program stored in this computer-readable storage medium is used to implement a method for confirming the loading of an excavator based on video recognition according to Embodiment 1 or Embodiment 2 when executed by a processor.

[0057] Among them, the computer-readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general or special-purpose computer. For example, the computer-readable storage medium is coupled to the processor, so that the processor can read information from the computer-readable storage medium and write information to the computer-readable storage medium. Of course, the computer-readable storage medium can also be a component of the processor. The processor and the computer-readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in the user equipment. Of course, the processor and the computer-readable storage medium can also exist as discrete components in the communication device. Specifically, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk or an optical disc, etc. The storage medium can be any available medium that can be accessed by a general or special-purpose computer.

Claims

1. An excavator loading confirmation method based on video recognition, characterized in that, It includes the following steps carried out in sequence: S1. Retrieve the video of the excavator loading the truck taken by the camera on the truck; S2. Obtain the loading moment of the excavator through the vibration sensor on the truck, or obtain the loading moment of the excavator through the excavator loading video; S3. Perform frame extraction on the excavator loading video based on the loading moment, and extract the photos of the excavator loading; S4. Parse and calculate the framed photos through the OCR recognition algorithm to identify the number on the label of the excavator; S5. Pair the excavator with the truck according to the excavator number identified in step S4; S6. Upload the paired excavator and truck information and the photo identifying the excavator number to the remote management platform, and the remote management platform counts the number of excavator loadings and the number of truck transportation trips.

2. The method for confirming the loading of an excavator based on video recognition according to claim 1, wherein The step of obtaining the loading moment of the excavator through the excavator loading video in step S2 includes the following steps carried out in sequence: S21. Manually extract the picture of the excavator bucket unloading soil above the truck carriage from the video and label the picture; S22. Train the labeled pictures using the YOLO algorithm to generate a loading moment recognition model; S23. Put the excavator loading video into the loading moment recognition model to extract the loading moment of the excavator.

3. A method for confirming the loading of an excavator based on video recognition according to claim 1, characterized in that, When performing frame extraction in step S3, extract the video content 20s before and 20s after the loading moment in the excavator loading video, extract one frame per second, and each frame corresponds to a photo.

4. A method for confirming the loading of an excavator based on video recognition according to claim 1, characterized in that, The label on the excavator is pasted on the bucket handle, boom, fuselage and counterweight at the tail of the excavator.

5. An excavator loading confirmation system based on video recognition, which is used to implement an excavator loading confirmation method based on video recognition as described in claims 1 to 4, and is characterized in that, It includes a shooting module with a camera, a vibration sensing module with a vibration sensor, a loading confirmation module and a remote management platform; The shooting module is used to shoot the loading picture of the excavator for the truck and transmit the taken excavator loading video to the loading confirmation module; The vibration sensing module is used to sense the vibration of the truck carriage after the truck stops moving and transmit the generated vibration information to the loading confirmation module; The loading confirmation module is used to receive the excavator loading video transmitted by the shooting module and the vibration information transmitted by the vibration sensing module, generate paired excavator and truck information and the photo identifying the excavator number after processing, and upload them to the remote management platform; The remote management platform is used to receive and store the paired excavator and truck information and the photo identifying the excavator number transmitted by the loading confirmation module, and count the number of excavator loadings and the number of truck transportation trips.

6. The system for confirming the loading of an excavator based on video recognition according to claim 5, characterized in that, The vibration sensing module further includes a truck motion sensor, and the vibration sensor starts to work only after the truck motion sensor detects that the truck has stopped moving.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a method for confirming excavator loading based on video recognition as described in claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it is used to implement a method for confirming excavator loading based on video recognition as described in claims 1 to 4.

Images