Material yard feeding detection method, electronic equipment and storage medium

Abstract

The invention discloses a material yard feeding detection method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first video image frame from the monitoring data of a target material yard, the trained recognition model is used for recognizing the bucket state of the loader used for conveying materials in the video image frame, and first position information of the loader in the target stock ground is determined; determining material loaded in the bucket based on the bucket status and the first position information; a second video image frame is obtained from the monitoring data of the target stock ground, the bucket state of the loader is obtained based on the second video image frame, and second position information of the loader in the target stock ground is determined; according to the state of the bucket and the second position information, whether the loader needs to carry out material feeding or not is judged; and when it is determined that the loader needs to conduct material feeding and the second position allows storage of the materials loaded in the bucket, it is determined that the loader conducts normal feeding. In this way, the risk of wrong batching in the stock ground can be avoided.

Description

Technical Field

[0001] The present application relates to the technical field of automatic material loading, and in particular to a material loading detection method, electronic equipment and storage medium. Background Art

[0002] Concrete aggregates include various sand and gravel materials, which are the raw materials with the largest usage in concrete. Aggregates in traditional mixing plants are generally piled up and stored in the material yard, and are manually batched using loaders. After the loader loads the raw materials with a bucket, it is transferred to the top of the batching machine for dumping and loading. Because the materials in the material yard are piled up in a disorderly manner, similar materials are not easy to distinguish, and the loading hoppers of different materials in the batching machine are closely distributed side by side, lacking identification guidance, so the loader is prone to the risk of batching errors during the manual loading process. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a material yard loading detection method, electronic equipment and storage medium, which can avoid the risk of incorrect material batching in the material yard.

[0004] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions: In a first aspect, an embodiment of the present application provides a material loading detection method in a material yard, comprising: Acquire a first video image frame from monitoring data of a target material yard, use a trained recognition model to identify a bucket state of a loader used to transport materials in the video image frame, and determine first position information of the loader in the target material yard; determining a material loaded in the bucket based on the bucket state and the first position information; Acquire a second video image frame from monitoring data of the target material yard, acquire a bucket state of the loader based on the second video image frame, and determine second position information of the loader in the target material yard; According to the bucket state and the second position information, it is determined whether the loader needs to load the material; When it is determined that the loader is to load materials and the second position allows the material loaded in the bucket to be stored, it is determined that the loader is loading materials normally.

[0005] In one embodiment, before obtaining the first video image frame from the monitoring data of the target material yard, the method also includes: obtaining a panoramic image of the target material yard, and marking each shoveling area and loading area of ​​the target material yard, as well as the loading area allowed for the materials stored in each shoveling area based on the panoramic image.

[0006] In one embodiment, the training process of the recognition model includes: obtaining pictures of the loader's bucket in an upward state and in a downward state, and assigning classification labels to the pictures of the bucket in an upward state and in a downward state, respectively; inputting the pictures and corresponding classification labels into the recognition model to train the recognition.

[0007] In one embodiment, a recognition model is used to determine the position information of a loader in a target material yard, including: determining a reference frame size for locating the loader in a monitoring video image frame through aggregate training; determining the size of a bounding box of the loader predicted by the recognition model based on a grid coordinate offset and a reference frame offset output by the recognition model; and determining the center position of the predicted frame as the position of the loader's bucket in the material yard.

[0008] In one embodiment, the material loaded in the bucket is determined based on the first bucket state and the first position information, including: when the distance between the first position and any shoveling area is less than a first preset distance threshold, and the bucket state is in an upward state, determining that the material in the loader bucket is material stored in any shoveling area.

[0009] In one embodiment, whether the loader is going to load materials is determined based on the bucket status and the second position information, including: when the distance range between the second position and any loading area is less than a second preset distance range, and the bucket is in an upward state, it is determined that the loader is about to load materials into the loading area corresponding to the second position.

[0010] In one embodiment, when it is determined that the loader is to load materials and the second position allows the material loaded in the bucket to be stored, before determining that the loader is loading normally, the method also includes: judging whether the material loaded in the bucket is always the material in the shoveling area corresponding to the first position when the loader reaches the second position from the first position.

[0011] In one embodiment, determining whether the material loaded in the bucket is always the material in the shoveling area corresponding to the first position when the loader arrives at the second position from the first position includes: determining, based on the first video image frame and the second video image frame, whether the bucket remains in an upward position when the loader reaches the second position from the first position; If so, it is determined that the material loaded in the bucket is always the material in the shoveling area corresponding to the first position.

[0012] In a second aspect, the present application provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute instructions to implement the steps of the material yard loading detection method as described in any one of the first aspects.

[0013] In a third aspect, the present application provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor, the steps of the material yard loading detection method as described in any one of the first aspects are implemented.

[0014] The material yard loading detection method, electronic device and storage medium provided by the present application include: obtaining a first video image frame from the monitoring data of the target material yard, using the trained recognition model to identify the bucket state of the loader used to transport materials in the video image frame and determine the first position information of the loader in the target material yard; determining the material loaded in the bucket based on the bucket state and the first position information; obtaining a second video image frame from the monitoring data of the target material yard, obtaining the bucket state of the loader based on the second video image frame, and determining the second position information of the loader in the target material yard; judging whether the loader is to load materials according to the bucket state and the second position information; determining that the loader is loading materials normally when it is determined that the loader is to load materials and the second position allows the storage of the materials loaded in the bucket. Therefore, the material yard loading detection method, electronic device and storage medium provided by the present application, by obtaining the monitoring data of the target material yard, determine the position and bucket state of the loader according to the monitoring data, thereby determining the materials loaded in the bucket and the target position of the material delivery. In this way, the risk of incorrect batching in the material yard can be avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic flow chart of a material loading detection method provided for an embodiment of the present application; Figure 2 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0017] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this way can be interchanged where appropriate so that the embodiments of the present application are implemented in a sequence other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0018] See also Figure 1 , which is a flow chart of a material yard loading detection method provided in an embodiment of the present application. The material yard loading detection method can be executed by an electronic device provided in an embodiment of the present application. The electronic device can be implemented in software and / or hardware, such as a server or a computer.

[0019] In this embodiment, the execution subject of the material yard feeding detection method is a computer processor as an example. The material yard feeding detection method provided by this embodiment includes: Step S1, obtaining a first video image frame from monitoring data of a target material yard, using a trained recognition model to identify a bucket state of a loader used to transport materials in the video image frame and determine first position information of the loader in the target material yard.

[0020] It is understandable that a fixed camera is installed in the material yard to monitor the activity status of the loader within a fixed range. In this embodiment, the position and status of the loader can be determined by acquiring the monitoring data of the target material yard.

[0021] In one embodiment, before obtaining the first video image frame from the monitoring data of the target material yard, the method also includes: obtaining a panoramic image of the target material yard, and marking each shoveling area and loading area of ​​the target material yard, as well as the loading area allowed for the materials stored in each shoveling area based on the panoramic image.

[0022] Specifically, the "shoveling area" is located in front of the aggregate pile, where the loader bucket shovels materials. The calibration process includes calibrating the coordinates of the shoveling area in the actual material field and the raw material information corresponding to the current shoveling area. Similarly, the "loading area" is located in front of the loading hopper, that is, the area where the loader bucket stops when unloading raw materials into the loading hopper; the calibration process includes calibrating the coordinates of the loading area in the actual material field and the raw material information corresponding to the current loading area.

[0023] In one embodiment, the training process of the recognition model includes: obtaining pictures of the loader's bucket in an upward state and in a downward state, and assigning classification labels to the pictures of the bucket in an upward state and in a downward state, respectively; inputting the pictures and corresponding classification labels into the recognition model to train the recognition.

[0024] It can be understood that the recognition model receives the input image data of the loader bucket and performs feature extraction and analysis based on the image content. Based on the features learned during training, the model can distinguish whether the bucket is in an upward or downward state. It should be noted that the performance of the model depends on the quality and diversity of the training data and the adequacy of the training process. If the training data covers a variety of different scenes, angles, lighting conditions, etc., the model is more likely to have a stronger generalization ability and can accurately identify the state of the bucket in practical applications. Therefore, in the process of training the recognition model, the embodiment of the present application uses as many pictures as possible to feed the recognition model to improve the generalization ability of the model, so as to accurately identify the state of the bucket.

[0025] Specifically, the recognition model in the present application may be YOLOv5 (You Only Look Once version 5), which is a target detection algorithm based on a deep neural network, designed to recognize and locate multiple objects in real time in an image or video. Its main features are fast speed and high accuracy, and it can achieve fast target detection in real-time scenarios. YOLOv5 treats target detection as a regression problem, directly predicts bounding boxes and category probabilities in a single network, and can achieve end-to-end fast detection.

[0026] Step S2, determining the material loaded in the bucket based on the bucket state and the first position information.

[0027] Specifically, the material loaded in the bucket is determined based on the first bucket state and the first position information, including: when the distance between the first position and any shoveling area is less than a first preset distance threshold, and the bucket state is in an upward state, determining that the material in the loader bucket is material stored in any shoveling area.

[0028] Here, the "ray algorithm" can be used to calculate whether the current position of the bucket falls within the pre-set range area ("shoveling area", "loading area"), and combined with the bucket status, the bucket's working intention and the loaded raw materials can be analyzed.

[0029] It should be noted that, when the distance between the position of the loader and multiple shoveling areas is less than a first preset distance threshold, the material stored in the shoveling area closest to the loader is determined as the target shoveling area of ​​the loader.

[0030] Step S3, obtaining a second video image frame from the monitoring data of the target material yard, obtaining the bucket state of the loader based on the second video image frame, and determining second position information of the loader in the target material yard.

[0031] Here, the manner of determining the bucket state and the second position information of the loader through the second video image frame is the same as the manner of determining the bucket state and the first position information of the loader through the first video image frame in step S1.

[0032] In addition, the first video image frame and the second video image frame may be acquired by extracting frame by frame, or by setting a fixed interval for extraction.

[0033] Step S4, judging whether the loader needs to load materials according to the bucket state and the second position information.

[0034] Specifically, based on the bucket status and the second position information, it is determined whether the loader is going to load materials, including: when the distance range between the second position and any loading area is less than the second preset distance range, and the bucket is in the raised state, it is determined that the loader is about to load materials into the loading area corresponding to the second position.

[0035] Similarly, when the distance between the position of the loader and multiple loading areas is less than a first preset distance threshold, the material stored in the loading area closest to the loader is determined as the target loading area of ​​the loader.

[0036] If the bucket is in the raised state and the current position coordinates fall into a certain "loading area", it can be preliminarily determined that the bucket is about to unload materials into a certain loading hopper. At the same time, the raw materials allowed by the current loading hopper can be determined to ensure that the materials are accurately delivered to the correct loading area.

[0037] In one embodiment, a recognition model is used to determine the position information of a loader in a target material yard, including: determining a reference frame size for locating the loader in a monitoring video image frame through aggregate training; determining the size of a bounding box of the loader predicted by the recognition model based on a grid coordinate offset and a reference frame offset output by the recognition model; and determining the center position of the predicted frame as the position of the loader's bucket in the material yard.

[0038] Here, the method of positioning the loader is applicable to step S1 and step S3.

[0039] Understandably, when the YOLOv5 object detection algorithm outputs coordinate information, it focuses on how the final output information of the network is mapped back to the precise coordinates of the original image. The output coordinate information contains four dimensions: x, y, w, and h, where x and y represent the offset of the center of the prediction box relative to the center of the grid, and w and h represent the offset of the width of the height prediction box relative to the anchor box. This information, combined with the coordinates of the grid and the image size, can calculate the precise coordinates of the prediction box on the original image.

[0040] Specifically, in the YOLOv5 algorithm, each grid is responsible for predicting a set of anchor boxes (i.e., reference boxes) on the image. Each anchor box has a fixed size, which is obtained by aggregating the training level (kmeans). The x, y values ​​output by YOLOv5 are an offset based on the grid coordinates. The sigmoid activation function makes the output value between [0, 1], and then adds it to the grid coordinates to get the coordinates of the center of the predicted box. The w, h values ​​are combined with the anchor box size and the original image size to determine the size of the predicted box. When calculating the center coordinates of the predicted box, the grid coordinates are used as the basis. For example, if the grid coordinates are 3 and the predicted cx is 0.5, the center coordinates of the predicted box are 3.5. This coordinate needs to be mapped according to the sampling rate of the original image size to ensure that the coordinate value corresponds to the actual image position. By multiplying the feature map size and the original image size by the corresponding sampling rate, the exact position of the predicted box on the original image can be obtained.

[0041] Step S5, when it is determined that the loader is to load materials and the second position allows the storage of the materials loaded in the bucket, it is determined that the loader is loading materials normally.

[0042] According to the above content, if the type of material loaded in the loader bucket is known, the loading area corresponding to the second position, and the type of material allowed to be stored in the loading area are known, it can be determined whether the material loaded in the loader bucket can be unloaded to the loading area of ​​the second position.

[0043] In one embodiment, when it is determined that the loader is to load materials and the second position allows the material loaded in the bucket to be stored, before determining that the loader is loading normally, the method also includes: judging whether the material loaded in the bucket is always the material in the shoveling area corresponding to the first position when the loader reaches the second position from the first position.

[0044] Furthermore, determining whether the material loaded in the bucket is always the material in the shoveling area corresponding to the first position when the loader reaches the second position from the first position includes: determining, based on the first video image frame and the second video image frame, whether the bucket remains in an upward state when the loader reaches the second position from the first position; if so, determining that the material loaded in the bucket is always the material in the shoveling area corresponding to the first position.

[0045] It can be understood that when the position coordinates of the loader fall into a certain "shoveling area" and the bucket is in an upward position, it can be preliminarily determined that the bucket is loaded with raw materials for the shoveling area; the current analysis result can be used as the "detection starting point" of the entire process; if the bucket remains in an "upward position" in the subsequent recognition result analysis, it means that the raw materials in the bucket have been loaded with raw materials and have not been unloaded. If the bucket remains in an upward position from the "detection starting point" and the current position coordinates fall into a certain "loading area", it can be preliminarily determined that the bucket is about to unload a certain loading bucket, and the raw materials allowed by the current loading bucket can be determined; then the raw materials in the bucket can be compared with the raw materials corresponding to the current loading port to see if they match, which can be used as the "detection end point" of the entire process.

[0046] In one embodiment, the material yard loading detection method provided in the present application also includes: when it is detected that the material unloaded by the loader does not match the raw materials allowed to be stored in the current loading area, a warning is given through an external device (light, sound) to prompt the material yard staff.

[0047] The material yard loading detection method provided by the present application obtains image frames of monitoring data of the target material yard, and then inputs these image frames one by one into the recognition model for analysis and recognition, and obtains the state and position of the loader bucket according to the recognition results returned by the recognition model, and at the same time compares the various pre-calibrated areas in the image, comprehensively analyzes the loader behavior, and determines that the loader puts the material into the correct loading area. In this way, the risk of incorrect batching in the material yard can be avoided.

[0048] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0049] Figure 2 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 2 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0050] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0051] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0052] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a data processing device at the logical level. The processor executes the program stored in the memory and is specifically used to execute the long-length material field feeding detection method provided in this embodiment.

[0053] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0054] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by a portable electronic device including multiple application programs, the portable electronic device can execute the material yard loading detection method provided by the present application.

[0055] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0056] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0057] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0058] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0059] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A material loading detection method in a material field, characterized in that: include: Acquire a first video image frame from monitoring data of a target material yard, use a trained recognition model to identify a bucket state of a loader used to transport materials in the video image frame, and determine first position information of the loader in the target material yard; determining a material loaded in the bucket based on the bucket state and the first position information; Acquire a second video image frame from monitoring data of the target material yard, acquire a bucket state of the loader based on the second video image frame, and determine second position information of the loader in the target material yard; Determining whether the loader is to load materials according to the bucket state and the second position information; When it is determined that the loader is to load materials and the second position allows the material loaded in the bucket to be stored, it is determined that the loader is loading materials normally.

2. The method according to claim 1, characterized in that Before acquiring the first video image frame from the monitoring data of the target material yard, the method further includes: A panoramic image of the target material yard is acquired, and based on the panoramic image, each shoveling area and loading area of ​​the target material yard, as well as an allowed loading area for materials stored in each shoveling area, are marked.

3. The method according to claim 1, characterized in that The training process of the recognition model includes: Obtain pictures of the bucket of the loader in an upward state and in a downward state, and assign classification labels to the pictures of the bucket in an upward state and in a downward state respectively; The images and corresponding classification labels are input into the recognition model to train the recognition.

4. The method according to claim 1, characterized in that: Determining the position information of the loader in the target material yard using the recognition model includes: Determine the reference frame size for positioning the loader in the monitoring video image frame through aggregate training; Determining the size of the bounding box of the loader predicted by the recognition model based on the grid coordinate offset and the reference frame offset output by the recognition model; The center position of the prediction frame is determined as the position of the bucket of the loader in the material yard.

5. The method according to claim 2 or 3, characterized in that: The determining the material loaded in the bucket based on the first bucket state and the first position information includes: When the distance between the first position and any shoveling area is less than a first preset distance threshold, and the bucket state is in an upward state, it is determined that the material in the loader bucket is the material stored in any shoveling area.

6. The method according to claim 2 or 3, characterized in that: The step of judging whether the loader is to load materials according to the bucket state and the second position information includes: When the distance range between the second position and any loading area is less than a second preset distance range, and the bucket is in an upward state, it is determined that the loader is about to load materials in the loading area corresponding to the second position.

7. The method according to claim 1, characterized in that When it is determined that the loader is to load materials and the second position allows the material loaded in the bucket to be stored, before determining that the loader is loading materials normally, the method further includes: It is determined whether the material loaded in the bucket is always the material in the shoveling area corresponding to the first position when the loader reaches the second position from the first position.

8. The method according to claim 7, characterized in that The determining whether the material loaded in the bucket is always the material in the shoveling area corresponding to the first position when the loader arrives at the second position from the first position includes: determining, according to the first video image frame and the second video image frame, whether the bucket remains in an upward position when the loader reaches the second position from the first position; If so, it is determined that the material loaded in the bucket is always the material in the shoveling area corresponding to the first position.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the steps of the material yard loading detection method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor, the steps of the material yard loading detection method according to any one of claims 1 to 8 are implemented.

Images