A method and system for non-fluid spill detection based on scene structure information

Abstract

This invention provides a non-fluid overflow detection method and system based on scene structure information, comprising the following steps: real-time acquisition of scene structure information of the loading scene, the scene structure information including RGB image and depth image of the loading scene; performing loading container entry detection based on scene structure information, and obtaining edge contour information of the loading container through edge detection after the loading container has entered the scene, the edge contour information including contour coordinates and depth values; during loading, obtaining edge contour information of non-fluid loads inside the loading container based on scene structure information; comparing the edge contour information of the loading container and the edge contour information of non-fluid loads, and determining that non-fluid loads overflow when the difference between the depth value at a certain contour coordinate of the non-fluid edge contour inside the container and the depth value at the corresponding contour coordinate of the loading container edge contour is less than a set threshold.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a non-fluid overflow detection method and system based on scene structure information. Background Technology

[0002] With the rapid pace of social development, non-fluid materials such as cement, gravel, and coal have become indispensable materials or resources for urban construction and development. Utilizing transportation to load and transport these materials is a crucial step in ensuring their use in urban construction. However, during the mechanized loading of non-fluid materials, spillage frequently occurs, polluting the loading environment and wasting resources.

[0003] Currently, to prevent non-fluid overflows during loading, cameras are used to record the loading process, which is then observed by video surveillance personnel. The loading equipment is moved based on the video footage to avoid overflows. This process requires the video surveillance personnel to maintain a high level of concentration at all times, resulting in high labor costs and a high risk of false detections. Therefore, there is an urgent need for an intelligent non-fluid overflow detection method to replace manual labor and reduce loading costs.

[0004] In order to solve the above problems, people have been seeking an ideal technological solution. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a non-fluid overflow detection method and system based on scene structure information, thereby achieving automated and intelligent overflow detection of non-fluid loading processes.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a non-fluid overflow detection method based on scene structure information, comprising the following steps:

[0007] The scene structure information of the loading scene is acquired in real time, including the RGB image and depth image of the loading scene;

[0008] The loading container is detected to enter the scene based on the scene structure information, and the edge contour information of the loading container is obtained by edge detection after the loading container has entered the scene. The edge contour information includes contour coordinates and depth values.

[0009] During the loading process, the edge contour information of the non-fluid load inside the loading container is obtained based on the scene structure information;

[0010] By comparing the edge contour information of the loading container with the edge contour information of the non-fluid loading material, if the difference between the depth value at a certain contour coordinate of the non-fluid edge contour inside the container and the depth value at the corresponding contour coordinate of the loading container edge contour is less than a set threshold, it is determined that the non-fluid loading material has overflowed.

[0011] The present invention also provides a non-fluid overflow detection system based on scene structure information, comprising:

[0012] The acquisition module is configured to acquire scene structure information of the loading scene in real time, the scene structure information including RGB image and depth image of the loading scene;

[0013] The loading container entry detection module is configured to perform loading container entry detection based on scene structure information, and obtain the edge contour information of the loading container through edge detection after the loading container has entered the scene. The edge contour information includes contour coordinates and depth values.

[0014] The overflow detection module is configured to obtain the edge contour information of the non-fluid load inside the loading container based on the scene structure information during the loading process; compare the edge contour information of the loading container with the edge contour information of the non-fluid load, and determine that the non-fluid load overflows when the difference between the depth value at a certain contour coordinate of the non-fluid edge contour inside the container and the depth value at the corresponding contour coordinate of the loading container edge contour is less than a set threshold.

[0015] This invention has outstanding substantive features and significant progress compared to the prior art. Specifically, this invention identifies the loading container by using the scene structure information of the loading scene, and further extracts the outline of the loading container. Based on the outline of the loading container and the edge outline information of the non-fluid load inside the container, it automatically determines whether there is an overflow of non-fluid load, which improves the accuracy of loading container identification and non-fluid load overflow detection, and greatly saves manpower. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the present invention.

[0017] Figure 2 It involves labeling the contours and the inside of the contours on the depth map. Detailed Implementation

[0018] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0019] Example 1

[0020] like Figure 1 As shown, this embodiment provides a non-fluid overflow detection method based on scene structure information, including the following steps:

[0021] Step S1: Real-time acquisition of scene structure information of the loading scene, the scene structure information including RGB image and depth image of the loading scene;

[0022] A device capable of simultaneously acquiring RGB and depth images of the loading scene, such as consumer RGB-D cameras, is placed in a non-fluid loading scene at a location where the loading container can be captured. The data acquisition device is used to obtain RGB and depth images of the loading scene. In a specific implementation, the device is preferably installed on top of the loading container.

[0023] Step S2: Detect the entry of the loading container based on the scene structure information, and obtain the edge contour information of the loading container through edge detection after the loading container has entered the scene. The edge contour information includes contour coordinates and depth values.

[0024] Specifically, the inter-frame differences of the RGB images of the loading scene acquired at different times are compared, where the RGB image of the loading scene acquired at time T is I. T The RGB image of the loading scene acquired at the next moment is I. T+1 , when I T with I T+1 When a significant difference occurs between them, the loading container is determined to begin entering the site. When I T with I T+1 When the difference between them tends to be stable or zero, it is determined that the loading container has entered the site.

[0025] At this point, edge detection of the loading container is performed on the RGB image, such as... Figure 2 As shown; extract the contour coordinates of its edge contour, and find the corresponding depth value in the depth image at the corresponding time based on the contour coordinates;

[0026] Specifically, the steps for finding the depth value of the pixel in the depth image at the corresponding time based on the contour coordinates are as follows: after extracting the contour coordinates of the edge contour, the pixel on the RGB image can be found based on the contour coordinates, and the depth value of the pixel is found in the depth image at the corresponding time. This depth value is the depth value corresponding to the coordinates.

[0027] It is understandable that after edge detection, point cloud registration of RGB and depth images can be performed directly. Specifically, the RGB image is scaled down to the size of the depth image and cropped according to the edge contour of the loading container; the same cropping operation is performed on the depth image, and the RGB point cloud is returned with the geometric center of the depth sensor as the specified origin. Point cloud data registration is performed to obtain the contour coordinates and depth values ​​of each pixel in the edge contour of the loading container.

[0028] Step S3: During the loading process, obtain the edge contour information of the non-fluid load inside the loading container based on the scene structure information;

[0029] Specifically, during the loading process, the edge contour E of the loading container is scaled inward proportionally to obtain at least one inner closed area edge contour, such as... Figure 2 As shown in the white markings inside the container, calculate the contour coordinates of its edge outline, where / =k i ; The depth value at the corresponding position in the depth image is extracted based on the contour coordinates. The contour coordinates and the depth value constitute the edge contour information of the non-fluid load inside the loading container; where i is the i-th internal closed area edge contour obtained by scaling the edge contour E inwards proportionally.

[0030] It is understandable that during the loading of non-fluid loads, the depth value on the internal contour coordinates of the container will change in real time as the non-fluid loads are loaded.

[0031] Step S4: Compare the edge contour information of the loading container with the edge contour information of the non-fluid loading material to determine whether the non-fluid loading material has overflowed.

[0032] Specifically, the depth value at a certain contour coordinate of the non-fluid edge contour inside the container is subtracted from the depth value at the corresponding contour coordinate of the loading container edge contour to obtain the difference M between the two; the relationship between the difference M and the set threshold P is used to determine whether the non-fluid loading has overflowed. Specifically, when the difference M is less than the set threshold P, it is determined that the non-fluid loading has overflowed; otherwise, it is determined that the non-fluid loading has not overflowed.

[0033] In actual operation, the inner closed area edge contour is scaled inward based on the edge contour of the loading container, and the number of inner closed area edge contours is N=3, and the threshold is P=0. When the difference between the depth value corresponding to any contour coordinate of any inner closed area edge contour and the depth value at the corresponding contour coordinate on the edge contour of the loading container is less than the set threshold, it can be determined that there is an overflow.

[0034] It is understandable that the closer the edge contour of the internal closed area is to the edge contour of the loading container, the more accurate the detection result. Based on the above principle, users can set the scaling ratio of the edge contour of the internal closed area according to their own needs.

[0035] When a user wants to increase the loading rate of non-fluid loads, the scaling ratio of the inner closed area edge contour is set to be smaller, so that the inner closed area edge contour is close to the edge contour of the loading container. In this way, when it is detected that non-fluid loads overflow on one side of the loading container, and the non-fluid loads on one side of the loading container are much larger than the depth value of the loading container edge contour, the loading container can be moved according to the position of the non-fluid load overflow, so that the loading port of the non-fluid loads faces the side where no non-fluid loads overflow, thereby ensuring that the non-fluid loads are loaded evenly. When it is further detected that the non-fluid loads are evenly loaded and overflow occurs globally, the loading of non-fluid loads is stopped, thus obtaining the loading rate of non-fluid loads.

[0036] When users want to prevent non-fluid loads from overflowing, they can set a larger scaling ratio for the inner closed area edge contour so that the inner closed area edge contour is far away from the edge contour of the loading container. This will trigger an alarm before the non-fluid load overflows, thus preventing the non-fluid load from overflowing.

[0037] Furthermore, when it is determined that there is a non-fluid overflow, the overflow part structure information is further output. The overflow part structure information includes the RGB image and depth image of the overflow part at the current moment, as well as the global RGB image and depth image at the current moment. Among them, the local area extracted from the global RGB image and depth image at the current moment with the container's outer contour coordinates and container's inner contour coordinates corresponding to the overflow position as the center is used as the RGB image and depth image of the overflow part.

[0038] In summary, this invention identifies loading containers by utilizing the scene structure information of the loading scenario, and further extracts the outline of the loading container. Based on the outline of the loading container and the edge outline information of the non-fluid load inside the container, it automatically determines whether there is an overflow of non-fluid load, thereby improving the accuracy of loading container identification and non-fluid load overflow detection, and greatly saving manpower.

[0039] Example 2

[0040] The difference between this embodiment and Embodiment 1 is that: when performing container entry detection based on scene structure information, the inter-frame differences of the depth images of the loading scene acquired at different times are compared, where the depth image of the loading scene acquired at time T is S. T The depth image of the loading scene acquired at the next moment is S. T+1 When S T With S T+1 When a significant difference occurs between them, the loading container is determined to begin entering the site. When S T With S T+1 When the difference between them tends to be stable or zero, it is determined that the loading container has entered the site.

[0041] Example 3

[0042] The difference between this embodiment and Embodiments 1 and 2 is that: when performing container entry detection based on scene structure information, the inter-frame differences between the RGB image and depth image of the loading scene acquired at different times are compared, where the RGB image of the loading scene acquired at time T is I. T The depth image is S T The RGB image of the loading scene acquired at the next moment is I. T+1 The depth image is S T+1 , when I T with I T+1 and / or S T With S T+1 When a significant difference occurs between them, the loading container is determined to begin entering the site. When I T with I T+1 and / or S T With S T+1 When the difference between them tends to be stable or zero, it is determined that the loading container has entered the site.

[0043] Example 4

[0044] The difference between this embodiment and embodiment 1 is that when obtaining the edge contour information of the loading container through edge detection, the edge detection of the loading container is performed directly in the depth image, and the contour coordinates and depth values ​​of its edge contour are extracted.

[0045] Example 5

[0046] The difference between this embodiment and embodiment 1 is that: when obtaining the edge contour information of the non-fluid load inside the loading container based on the scene structure information, edge detection of the non-fluid load inside the loading container is performed in the depth image, and the contour coordinates and depth values ​​of its edge contour are extracted. The contour coordinates and the depth values ​​constitute the edge contour information of the non-fluid load inside the loading container.

[0047] It is understandable that the above method can accurately obtain the edge contour information of non-fluid loads inside the loading container, but the computation is complex.

[0048] Example 6

[0049] This embodiment provides a non-fluid overflow detection system based on scene structure information, including:

[0050] The acquisition module is configured to acquire scene structure information of the loading scene in real time, the scene structure information including RGB image and depth image of the loading scene;

[0051] Specifically, the acquisition module is a device capable of simultaneously acquiring RGB images and depth images of the loading scene, such as an RGB-D camera; the RGB-D camera is placed in a non-fluid loading scene at a location where the loading container can be captured, such as above the non-fluid loading inlet; the RGB-D camera takes pictures at set time intervals to acquire RGB images and depth images of the loading scene in real time;

[0052] The loading container entry detection module is configured to perform loading container entry detection based on scene structure information, and obtain the edge contour information of the loading container through edge detection after the loading container has entered the scene. The edge contour information includes contour coordinates and depth values.

[0053] The overflow detection module is configured to obtain the edge contour information of the non-fluid load inside the loading container based on the scene structure information during the loading process; compare the edge contour information of the loading container with the edge contour information of the non-fluid load, and determine that the non-fluid load overflows when the difference between the depth value at a certain contour coordinate of the non-fluid edge contour inside the container and the depth value at the corresponding contour coordinate of the loading container edge contour is less than a set threshold.

[0054] In practice, the scaling ratio of the edge contour of the inner closed area can be set according to user needs.

[0055] When a user wants to increase the loading rate of non-fluid loads, the scaling ratio of the inner closed area edge contour is set to be smaller, so that the inner closed area edge contour is close to the edge contour of the loading container. In this way, when it is detected that non-fluid loads overflow on one side of the loading container, and the non-fluid loads on one side of the loading container are much larger than the depth value of the loading container edge contour, the loading container can be moved according to the position of the non-fluid load overflow, so that the loading port of the non-fluid loads faces the side where no non-fluid loads overflow, thereby ensuring that the non-fluid loads are loaded evenly. When it is further detected that the non-fluid loads are evenly loaded and overflow occurs globally, the loading of non-fluid loads is stopped, thus obtaining the loading rate of non-fluid loads.

[0056] When users want to prevent non-fluid loads from overflowing, they can set a larger scaling ratio for the inner closed area edge contour so that the inner closed area edge contour is far away from the edge contour of the loading container. This will trigger an alarm before the non-fluid load overflows, thus preventing the non-fluid load from overflowing.

[0057] It is understood that the present invention may also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the non-fluid overflow detection method based on scene structure information as described in any one of the foregoing embodiments 1-5.

[0058] Of course, the present invention may also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the non-fluid overflow detection method based on scene structure information as described in any one of the foregoing embodiments 1-5.

[0059] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0060] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This application is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0063] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0064] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0065] The above provides a detailed description of a data transmission method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A non-fluid overflow detection method based on scene structure information, characterized in that, Includes the following steps: The scene structure information of the loading scene is acquired in real time, including the RGB image and depth image of the loading scene; The loading container is detected to enter the scene based on the scene structure information, and the edge contour information of the loading container is obtained by edge detection after the loading container has entered the scene. The edge contour information includes contour coordinates and depth values. This involves comparing the inter-frame differences of the RGB images of the loading scene acquired at different times, where the RGB image of the loading scene acquired at time T is I. T The RGB image of the loading scene acquired at the next moment is I. T+1 , when I T with I T+1 When a significant difference occurs between them, the loading container is determined to begin entering the site. When I T with I T+1 When the difference between them tends to be stable or zero, it is determined that the loading container has entered the site. During the loading process, the edge contour information of the non-fluid load inside the loading container is obtained based on the scene structure information; During the loading process, the edge contour of the loading container is scaled inward to obtain at least one edge contour of the internal closed area. The contour coordinates of the edge contour are calculated, and the depth value at the corresponding position in the depth image is extracted based on the contour coordinates. The contour coordinates and the depth value constitute the edge contour information of the non-fluid loading material inside the loading container. By comparing the edge contour information of the loading container and the edge contour information of the non-fluid loading material, if the difference between the depth value at a certain contour coordinate of the non-fluid edge contour inside the container and the depth value at the corresponding contour coordinate of the loading container edge contour is less than a set threshold, it is determined that the non-fluid loading material has overflowed.

2. The non-fluid overflow detection method based on scene structure information according to claim 1, characterized in that, The specific steps for container entry detection based on scene structure information are as follows: Compare the inter-frame differences of the depth images of the loading scene acquired at different times, where the depth image of the loading scene acquired at time T is S. T The depth image of the loading scene acquired at the next moment is S. T+1 When S T With S T+1 When a significant difference occurs between them, the loading container is determined to begin entering the site. When S T With S T+1 When the difference between them tends to be stable or zero, it is determined that the loading container has entered the site.

3. The non-fluid overflow detection method based on scene structure information according to claim 1, characterized in that: Compare the inter-frame differences between the RGB and depth images of the loading scene acquired at different times, where the RGB image of the loading scene acquired at time T is I. T The depth image is S T The RGB image of the loading scene acquired at the next moment is I. T+1 The depth image is S T+1 , when I T with I T+1he 、 and / or S T With S T+1 When a significant difference occurs between them, the loading container is determined to begin entering the site. When I T with I T+1 and / or S T With S T+1 When the difference between them tends to be stable or zero, it is determined that the loading container has entered the site.

4. The non-fluid overflow detection method based on scene structure information according to claim 1, characterized in that: Edge detection of the loading container is performed on the RGB image, the contour coordinates of its edge contour are extracted, and the depth value at the corresponding coordinate position in the depth image is extracted based on the contour coordinates.

5. The non-fluid overflow detection method based on scene structure information according to claim 1, characterized in that: Edge detection of the loading container is performed in the depth image, and the contour coordinates and depth values ​​of its edge contour are extracted.

6. The non-fluid overflow detection method based on scene structure information according to claim 1, characterized in that: During the loading process, edge detection of non-fluid loads inside the loading container is performed in the depth image, and the contour coordinates and depth values ​​of their edge contours are extracted. The contour coordinates and the depth values ​​constitute the edge contour information of the non-fluid loads inside the loading container.

7. The non-fluid overflow detection method based on scene structure information according to claim 3, characterized in that: When it is determined that there is a non-fluid overflow, the overflow part structure information is further output. The overflow part structure information includes the RGB image and depth image of the overflow part at the current moment, as well as the global RGB image and depth image at the current moment. Among them, the local area extracted from the global RGB image and depth image at the current moment with the container's outer contour coordinates and container's inner contour coordinates corresponding to the overflow position as the center is used as the RGB image and depth image of the overflow part.

8. A non-fluid overflow detection system based on scene structure information, characterized in that, include: The acquisition module is configured to acquire scene structure information of the loading scene in real time, the scene structure information including RGB image and depth image of the loading scene; The loading container entry detection module is configured to perform loading container entry detection based on scene structure information, and after the loading container has entered the scene, obtain the edge contour information of the loading container through edge detection. The edge contour information includes contour coordinates and depth values. Specifically, it compares the inter-frame differences of the RGB images of the loading scene acquired at different times, where the RGB image of the loading scene acquired at time T is I. T The RGB image of the loading scene acquired at the next moment is I. T+1 , when I T with I T+1 When a significant difference occurs between them, the loading container is determined to begin entering the site. When I T with I T+1 When the difference between them tends to be stable or zero, it is determined that the loading container has entered the site. The overflow detection module is configured to obtain the edge contour information of the non-fluid load inside the loading container based on the scene structure information during the loading process; compare the edge contour information of the loading container with the edge contour information of the non-fluid load, and determine that the non-fluid load overflows when the difference between the depth value at a certain contour coordinate of the non-fluid edge contour inside the container and the depth value at the corresponding contour coordinate of the loading container edge contour is less than a set threshold. During the loading process, the edge contour of the loading container is scaled inward to obtain at least one edge contour of the internal closed area. The contour coordinates of the edge contour are calculated, and the depth value at the corresponding position in the depth image is extracted based on the contour coordinates. The contour coordinates and the depth value constitute the edge contour information of the non-fluid loading material inside the loading container.

Images