Loading rate measurement method, device, system, equipment, storage medium and product

Abstract

The present application relates to the field of machine vision detection, and discloses a method, device, system, equipment, storage medium and product for measuring loading rate, which includes: collecting a current visible light image in a cargo loading scene and collecting target depth information of multiple pixel points in the current visible light image; performing image segmentation and background modeling on the current visible light image to determine the objects to which the multiple pixel points belong respectively; determining interference points and failure points among the multiple pixel points; deleting the target depth information of the interference point, and updating the target depth information of the failure point based on the target depth information of the pixel points around the failure point that belong to the same object as the failure point; determining the cargo volume change based on the target depth information of the pixel points in the current visible light image and the target depth information of the pixel points in the historical visible light image, and determining the current cargo loading rate based on the cargo volume change. The method can effectively improve the accuracy of the determined current cargo loading rate.

Description

Technical Field

[0001] The present application relates to the field of machine vision detection technology, and in particular to a loading rate measurement method, device, system, equipment, storage medium and product. Background Art

[0002] In the field of logistics, cargo loading rate is a very important parameter. The cargo loading rate refers to the degree of filling of cargo in the cargo compartment. When the cargo loading rate is too high, it may cause the cargo to shift or tip over during transportation, while when the cargo loading rate is too low, it will cause space waste and increase transportation costs. Therefore, logistics companies will make reasonable arrangements based on the nature and packaging of the cargo to achieve the best loading effect of the cargo. Therefore, in the actual loading process, the measurement of the cargo loading rate is essential.

[0003] In the related art, a binocular camera is generally used to measure the distance between the camera and multiple points of the cargo and the cargo compartment, and the cargo loading rate is determined based on the measured multiple distances. However, due to the influence of light conditions and the measurement distance, some of the measured distances are inaccurate, which leads to inaccurate cargo loading rates determined based on the distances. Summary of the invention

[0004] The main purpose of this application is to provide a loading rate measurement method, device, system, equipment, storage medium and product, which can effectively improve the accuracy of the measured cargo loading rate.

[0005] To achieve the above object, the present application provides a loading rate measurement method, the loading rate measurement method comprising:

[0006] Collecting a current visible light image in a cargo loading scene and collecting target depth information of a plurality of pixel points in the current visible light image, wherein the target depth information indicates spatial positions of object points corresponding to the plurality of pixel points;

[0007] Performing image segmentation and background modeling on the current visible light image to determine the objects to which the multiple pixels in the current visible light image respectively belong;

[0008] Determine an interference point and a failure point among the plurality of pixels, wherein the interference point is a pixel point corresponding to a loader, and the failure point is a pixel point where target depth information has a deviation;

[0009] Deleting the target depth information of the interference point, and updating the target depth information of the failure point based on the target depth information of the pixel points around the failure point that belong to the same object as the failure point;

[0010] Based on the target depth information of the pixel points in the current visible light image and the target depth information of the pixel points in the historical visible light image, the cargo volume change is determined, and the current cargo loading rate is determined based on the cargo volume change, wherein the historical visible light image is collected in the cargo loading scenario before the current visible light image.

[0011] Optionally, the collecting target depth information of a plurality of pixel points in the current visible light image includes:

[0012] Using a depth camera, collecting first depth information of the plurality of pixel points by using laser ranging;

[0013] Using a multi-camera, collecting second depth information of the plurality of pixel points using binocular parallax;

[0014] The first depth information of the plurality of pixels is fused with the second depth information to obtain target depth information of the plurality of pixels.

[0015] Optionally, the fusing the first depth information of the multiple pixel points with the second depth information to obtain target depth information of the multiple pixel points includes:

[0016] For any pixel point among the multiple pixel points, when a depth value in the second depth information is less than a reference threshold, using the second depth information as the target depth information;

[0017] When the depth value in the second depth information is not less than the reference threshold, using the first depth information as the target depth information;

[0018] The depth value indicates the distance between the object point corresponding to the pixel point and the camera.

[0019] Optionally, the fusing the first depth information of the multiple pixel points with the second depth information to obtain target depth information of the multiple pixel points includes:

[0020] For any pixel point among the multiple pixel points, performing weighted averaging on the depth value in the first depth information and the depth value in the second depth information to obtain the depth value in the target depth information;

[0021] The depth value indicates the distance between the object point corresponding to the pixel point and the camera.

[0022] Optionally, the collecting of the current visible light image in the cargo loading scene and the collecting of target depth information of a plurality of pixel points in the current visible light image include:

[0023] Acquiring a plurality of reference images, wherein the plurality of reference images include the current visible light image and adjacent frames of the current visible light image, and the reference images include the plurality of pixel points;

[0024] Acquiring reference depth information of the multiple pixel points in the multiple reference images;

[0025] For any pixel point among the multiple pixel points, multiple reference depth information of the pixel point are fused to obtain target depth information of the pixel point.

[0026] Optionally, updating the target depth information of the failure point based on the target depth information of pixels around the failure point that belong to the same object as the failure point includes:

[0027] The target depth information of a plurality of pixel points around the failure point and belonging to the same object as the failure point is fused to obtain updated target depth information of the failure point.

[0028] In addition, to achieve the above purpose, the present application also proposes a loading rate measuring device, the loading rate measuring device comprising:

[0029] A data acquisition module, used to acquire a current visible light image in a cargo loading scene and acquire target depth information of a plurality of pixel points in the current visible light image, wherein the target depth information indicates spatial positions of object points corresponding to the plurality of pixel points;

[0030] An image processing module, used for performing image segmentation and background modeling on the current visible light image to determine the objects to which the multiple pixels in the current visible light image respectively belong;

[0031] A pixel point determination module, used to determine interference points and failure points among the plurality of pixel points, wherein the interference points are pixel points corresponding to the loader, and the failure points are pixel points where target depth information has deviations;

[0032] An information updating module, configured to delete the target depth information of the interference point, and update the target depth information of the failure point based on the target depth information of the pixel points around the failure point that belong to the same object as the failure point;

[0033] A loading rate determination module is used to determine a change in cargo volume based on target depth information of pixel points in the current visible light image and target depth information of pixel points in historical visible light images, and to determine a current cargo loading rate based on the change in cargo volume, wherein the historical visible light image is collected in the cargo loading scenario before the current visible light image.

[0034] Optionally, the data acquisition module includes:

[0035] A first acquisition submodule is used to acquire first depth information of the plurality of pixel points by using a depth camera and laser ranging;

[0036] A second acquisition submodule is used to acquire second depth information of the plurality of pixel points by using a multi-camera and binocular parallax;

[0037] The fusion submodule is used to fuse the first depth information of the multiple pixel points with the second depth information to obtain target depth information of the multiple pixel points.

[0038] Optionally, the fusion submodule is used to, for any pixel point among the multiple pixel points, use the second depth information as the target depth information when the depth value in the second depth information is less than a reference threshold; and use the first depth information as the target depth information when the depth value in the second depth information is not less than the reference threshold; wherein the depth value indicates the distance between the object point corresponding to the pixel point and the camera.

[0039] Optionally, the fusion submodule is used to perform a weighted average of the depth value in the first depth information and the depth value in the second depth information for any pixel point among the multiple pixel points to obtain a depth value in the target depth information; wherein the depth value indicates the distance between the object point corresponding to the pixel point and the camera.

[0040] Optionally, the data acquisition module is used to acquire multiple reference images, the multiple reference images include the current visible light image and adjacent frames of the current visible light image, and the reference images include the multiple pixel points; acquire reference depth information of the multiple pixel points in the multiple reference images; for any pixel point among the multiple pixel points, fuse the multiple reference depth information of the pixel point to obtain the target depth information of the pixel point.

[0041] Optionally, the information updating module is used to fuse target depth information of a plurality of pixel points around the failure point that belong to the same object as the failure point, to obtain updated target depth information of the failure point.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a loading rate measurement device, which includes a memory, a processor, and a loading rate measurement program stored in the memory and executable on the processor, and the loading rate measurement program is configured to implement the loading rate measurement method described above.

[0043] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, on which a load rate measurement program is stored. When the load rate measurement program is executed by a processor, the load rate measurement method as described above is implemented.

[0044] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a loading rate measurement program, and when the loading rate measurement program is executed by a processor, it implements the loading rate measurement method described above.

[0045] One or more technical solutions proposed in this application have at least the following technical effects:

[0046] In the solution provided in the present application, the current visible light image under the cargo loading scene and the target depth information of multiple pixels in the current visible light image are collected. Since the target depth information indicates the spatial position of the object points corresponding to the multiple pixels, the current cargo loading rate can be determined based on the target depth information. Considering that there are interference points and failure points in the multiple pixels collected, the interference points are the pixels corresponding to the loaders, and the failure points are the pixels with deviations in the target depth information. Whether it is an interference point or a failure point, its target depth information will introduce errors to the calculation of the current cargo loading rate. Therefore, the current visible light image is segmented and the background modeling is performed to determine the objects to which the multiple pixels in the current visible light image belong respectively. In this way, not only can the interference points corresponding to the loaders in the multiple pixels be known, thereby deleting the target depth information of the interference points, but also the target depth information of the failure point can be updated based on the target depth information of the pixels around the failure point that belong to the same object as the failure point. Since there will be no large jump between the target depth information of the pixels around the failure point that belong to the same object as the failure point and the target depth information of the failure point, the target depth information of the failure point is updated based on the target depth information of the pixels around the failure point that belong to the same object as the failure point, which can ensure the accuracy of the updated target depth information. In this way, based on the target depth information of the pixels in the current visible light image and the target depth information of the pixels in the historical visible light image, the change in the volume of the cargo can be accurately determined, and the current cargo loading rate can be determined based on the change in the volume of the cargo. Therefore, this solution can effectively improve the accuracy of the current cargo loading rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 A schematic diagram of a loading rate measurement system provided for an exemplary embodiment of the present application;

[0050] Figure 2 A schematic diagram of a visual device provided as an exemplary embodiment of the present application;

[0051] Figure 3 A schematic diagram of another visual device provided as an exemplary embodiment of the present application;

[0052] Figure 4 A schematic diagram of another visual device provided as an exemplary embodiment of the present application;

[0053] Figure 5 This is a flow chart of the first embodiment of the method for measuring loading rate of the present application;

[0054] Figure 6 A schematic diagram of a cargo loading rate measurement scenario provided for an exemplary embodiment of the present application;

[0055] Figure 7 This is a flow chart of a second embodiment of the method for measuring loading rate of the present application;

[0056] Figure 8 A schematic diagram of a loading rate measurement process provided by an exemplary embodiment of the present application;

[0057] Fig. 9 A schematic diagram of another loading rate measurement process provided by an exemplary embodiment of the present application;

[0058] Fig.10 A schematic diagram of another loading rate measurement process provided by an exemplary embodiment of the present application;

[0059] Fig.11 This is a schematic diagram of the module structure of the loading rate measurement device according to an embodiment of the present application;

[0060] Fig.12 Schematic diagram of the device structure of the hardware operating environment involved in the load rate measurement method in the embodiment of the present application.

[0061] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0063] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0064] Figure 1 Schematic diagram of a loading rate measurement system provided in an embodiment of the present application. Figure 1 The loading rate measurement system includes a visual device 101 and a measuring device 102. The visual device 101 is electrically connected to the measuring device 102. Exemplarily, the visual device 101 and the measuring device 102 are connected via a wireless or wired network. The visual device 101 is set at a position with a preset distance from the cargo loading area, and its field of view covers the cargo loading area.

[0065] Exemplarily, the visual device 101 is arranged near the cargo loading platform and has a preset distance from the cargo loading platform to shoot the cargo loading scene. For example, a shooting rod is arranged in front of the cargo loading platform, and the visual device 101 is mounted above the shooting rod and faces the direction of the cargo loading platform.

[0066] The visual device 101 includes a visible light camera and a depth camera. Exemplarily, the measuring device 102 includes a computer, a mobile phone, a tablet computer or other types of measuring devices.

[0067] In one possible implementation, reference Figure 2 , the visual device 101 in the loading rate measurement system includes a depth camera and a visible light camera. The visible light camera is used to collect visible light images in the cargo loading scene, and the depth camera is used to collect target depth information of multiple pixels in the visible light image, and the target depth information indicates the spatial position of the object points corresponding to the multiple pixels. The measuring device 102 is used to perform image segmentation and background modeling on the current visible light image to determine the objects to which the multiple pixels in the current visible light image belong respectively; determine the interference points and failure points among the multiple pixels, the interference points are the pixels corresponding to the loading personnel, and the failure points are the pixels with deviations in the target depth information; delete the target depth information of the interference points, and update the target depth information of the failure points based on the target depth information of the pixels around the failure points that belong to the same object as the failure points; determine the change in cargo volume based on the target depth information of the pixels in the current visible light image and the target depth information of the pixels in the historical visible light images, and determine the current cargo loading rate based on the change in cargo volume. Among them, the historical visible light image is collected in the cargo loading scene before the current visible light image. The cargo loading rate is the volume ratio of cargo in the cargo compartment, that is, the ratio of the volume of cargo to the volume of cargo compartment.

[0068] One point that needs to be explained is that, when the visual device 101 in the loading rate measurement system includes a depth camera and a visible light camera, the depth camera and the visible light camera in the visual device 101 can constitute a binocular camera. Accordingly, the depth camera is used to collect the first depth information of multiple pixels in the current visible light image using laser ranging. The binocular camera composed of the depth camera and the visible light camera is used to collect the second depth information of multiple pixels using binocular parallax, that is, the parallax between the depth camera and the visible light camera. The measuring device 102 is used to fuse the first depth information of multiple pixels with the second depth information to obtain the target depth information of multiple pixels. Among them, the depth camera includes a laser and a detector. The depth camera emits laser through a laser, and then detects the echo signal through a detector to obtain the depth information of the pixels in the visible light image under the cargo loading scene.

[0069] In another possible implementation, the visual device 101 in the loading rate measurement system includes a depth camera and a multi-eye camera. The multi-eye camera includes at least two visible light cameras. Accordingly, any visible light camera in the multi-eye camera is used to collect the current visible light image in the cargo loading scene. The depth camera is used to collect the first depth information of multiple pixel points in the current visible light image using laser ranging; the multi-eye camera is used to collect the second depth information of multiple pixel points using binocular parallax; the measuring device 102 is used to fuse the first depth information of multiple pixel points with the second depth information to obtain the target depth information of multiple pixel points.

[0070] refer to Figure 3 The visual device 101 in the loading rate measurement system includes a depth camera and a binocular camera, and the binocular camera is composed of two visible light cameras. The binocular camera can determine the second depth information of the pixel points in the visible light image under the cargo loading scene by using binocular parallax. In this way, the visual device 101 has two sources of depth information, namely the depth camera and the binocular camera, so that more accurate depth information can be obtained by using the depth information from these two sources.

[0071] Exemplary, reference Figure 4, the visual device 101 in the loading rate measurement system includes a depth camera and a trinocular camera, and the trinocular camera is composed of three visible light cameras. Exemplarily, the focal lengths of the three visible light cameras are different, and accordingly, each visible light camera is responsible for capturing images within different distance ranges. For example, the first visible light camera is provided with a wide-angle lens for observing close-range environments and providing a wide viewing angle. The second visible light camera is provided with a standard lens for observing at medium distances, providing a balanced viewing angle and distance measurement capability. The third visible light camera is provided with a telephoto lens for long-distance observation, which helps to detect distant objects. The configuration of the trinocular camera provides a wider viewing angle and more accurate distance measurement capability than a monocular camera or a binocular camera. Since there is parallax between the three visible light cameras of the trinocular camera, in an embodiment of the present application, the visual device 101 can determine the second depth information of the pixel points in the visible light image under the cargo loading scene through the parallax between any two visible light cameras in the trinocular camera.

[0072] Figure 5 This is a flow chart of the first embodiment of the method for measuring the loading rate of the present application. Figure 5 , taking the execution subject as a loading rate measurement system as an example, the loading rate measurement method includes:

[0073] Step S10: collecting a current visible light image in a cargo loading scene and collecting target depth information of a plurality of pixel points in the current visible light image, wherein the target depth information indicates spatial positions of object points corresponding to the plurality of pixel points.

[0074] Visible light images are also called RGB (Red Green Blue) images, that is, full-color images. Visible light images have abundant pixels, and each pixel has a color, so the image is clear and easy to identify objects. Visible light images are collected by visible light cameras in the loading rate measurement system. Visible light cameras are also called RGB cameras, which use three independent sensors or filters to capture red, green, and blue light respectively, thereby generating full-color images.

[0075] The cargo loading scene includes a variety of objects, such as cargo and cargo compartments. The current visible light image of the cargo loading scene is collected, and the objects in the cargo loading scene are recorded through the pixels in the visible light image. Each pixel in the visible light image corresponds to an object point in the actual space. In the cargo loading scene, the object point has an actual spatial position, and the target depth information of the pixel point reflects the spatial position of the object point corresponding to the pixel point.

[0076] Optionally, the target depth information includes the depth value and angle information of the pixel point. The depth value is the distance between the object point corresponding to the pixel point and the camera. The angle information describes the direction of the object point corresponding to the pixel point. For example, the angle information is the angle between the line between the object point corresponding to the pixel point and the camera relative to the reference line. The present application does not limit the position and direction of the reference line. Optionally, the target depth information includes the spatial coordinates of the object point corresponding to the pixel point. For example, after the depth value and angle information of multiple pixel points are collected by a depth camera, a point cloud is drawn based on the acquired depth value and angle information, and the point cloud includes the three-dimensional coordinates of the object points corresponding to the multiple pixel points.

[0077] Optionally, the loading rate measurement system includes a depth camera, such as a TOF (Time Of Flight) camera, which measures distance by the time of light propagation. Accordingly, the target depth information of multiple pixels in the visible light image is collected by the depth camera. The specific implementation method is as follows: the loading rate measurement system collects the current visible light image through the visible light camera therein, and collects the depth image at the same time through the depth camera, and aligns the current depth image with the current visible light image based on the image registration parameters between the visible light camera and the depth camera. Then, the target depth information of multiple pixels in the current depth image is determined as the target depth information of multiple pixels in the current visible light image.

[0078] One thing that needs to be explained is that before collecting images, the visible light camera and depth camera in the load rate measurement system must be calibrated with RGBD (Red Green Blue Depth, color image and depth information), that is, the visible light camera and depth camera must be calibrated. Specifically, it involves accurately measuring and adjusting the internal and external parameters of the camera to ensure that the collected visible light image and depth image can accurately reflect the spatial relationship of the real world. The calibration process includes the following steps:

[0079] Intrinsic calibration: Determine the camera’s focal length, principal point coordinates and other parameters, which determine how the image is projected onto the imaging plane.

[0080] Extrinsic calibration: Determine the spatial relationship between the visible light camera and the depth camera, including rotation and translation matrices, to ensure that the two images are correctly aligned.

[0081] Depth calibration: Since the depth information of the depth image may have errors, these errors need to be corrected through calibration to obtain more accurate depth information.

[0082] Image registration: Align the depth image with the visible light image to ensure that they are in the same coordinate system.

[0083] Optionally, the loading rate measurement system includes a binocular camera consisting of two visible light cameras. The target depth information of multiple pixel points in the visible light image is collected by the binocular camera. The specific implementation method is: the loading rate measurement system collects visible light images respectively through two visible light cameras in the binocular camera, and determines the target depth information of multiple pixel points in any visible light image based on the position deviation between corresponding points of the two visible light images.

[0084] Optionally, the target depth information of multiple pixels in the current visible light image is determined by combining the previous and next frame images. Accordingly, the current visible light image in the cargo loading scene and the target depth information of multiple pixels in the current visible light image are collected, including: collecting multiple reference images, the multiple reference images include the current visible light image and adjacent frames of the current visible light image, wherein the reference images include the above-mentioned multiple pixels; collecting reference depth information of multiple pixels in the multiple reference images; for any pixel among the multiple pixels, fusing the multiple reference depth information of the pixel to obtain the target depth information of the pixel.

[0085] There is no restriction on the number of reference images collected. For example, the current visible light image and two frames before the current visible light image are collected as reference images, or the current visible light image and one frame before and after the current visible light image are collected as reference images. In addition, there is no restriction on the way to fuse multiple reference depth information of pixels. For example, multiple reference depth information are weighted averaged to obtain the target depth information of the pixel.

[0086] In the embodiment of the present application, considering that the camera's acquisition frame rate is relatively fast and the depth information of pixels in adjacent images does not change much, the target depth information of multiple pixels in the current visible light image is determined by combining the previous and next frame images, that is, multiple reference images including the current visible light image are collected, and reference depth information of multiple pixels in the multiple reference images is collected. For any pixel, the multiple reference depth information of the pixel are fused to obtain the target depth information of the pixel, which can greatly improve the accuracy of the target depth information.

[0087] Step S20: performing image segmentation and background modeling on the current visible light image to determine the objects to which the multiple pixels in the current visible light image belong respectively.

[0088] After the loading rate measurement system collects the current visible light image in the cargo loading scene and the target depth information of multiple pixel points in the current visible light image through the visual device, the visible light image and the target depth information of multiple pixel points in the visible light image are sent to the measuring device, and step S20 and subsequent steps are executed through the measuring device.

[0089] Image segmentation refers to the process of dividing an image into multiple image regions, each of which corresponds to an object with similar attributes in the image. Exemplarily, the implementation method of image segmentation of the current visible light image is: multiple object labels are set for the cargo loading scene, and the current visible light image is segmented based on the multiple object labels to obtain multiple image regions and object labels corresponding to each image region. Among them, the pixel points located in the same image region among the multiple pixel points belong to the same object. This application does not limit the algorithm used for image segmentation.

[0090] Through image segmentation, the pixels belonging to different objects in multiple pixels have been distinguished, and then the pixels corresponding to the objects in motion are determined through background modeling. Background modeling refers to the establishment of a mathematical model representing a static background, which is used to distinguish between moving objects and background. Among them, this application does not limit the algorithm used for background modeling. Since the loaders are in motion in the cargo loading scenario, and background modeling can determine the objects in motion in the image, therefore, the pixels corresponding to the loaders in the current visible light image can be accurately determined through background modeling.

[0091] Step S30: determining interference points and failure points among the plurality of pixels, where the interference points are pixels corresponding to the loader, and the failure points are pixels where the target depth information has deviations.

[0092] Considering that the calculation of cargo loading rate mainly refers to the depth information of cargo and cargo compartment, and the depth information of loading personnel is interference information for the calculation of cargo loading rate, it is necessary to determine the interference points from multiple pixels, that is, the pixels corresponding to loading personnel, so as to eliminate the interference caused by the depth information of loading personnel in the future. Through the background modeling above, the pixels corresponding to loading personnel have been determined, and the pixels corresponding to loading personnel can be determined as interference points.

[0093] There are two ways to determine the failure point among multiple pixels:

[0094] The first method is to determine the failure point where the target depth information has a deviation by setting a depth information reference range. The implementation method is: for each pixel among multiple pixels, the target depth information of the pixel is compared with the depth information reference range. When the target depth information of the pixel is outside the depth information reference range, the pixel is determined as a failure point. Among them, the depth information reference range can be set according to the distance between the cargo and the camera in the cargo loading scenario and the direction of the cargo relative to the camera. For example, the depth value range in the depth information reference range is set to 2-17 meters.

[0095] In an embodiment of the present application, considering that in a cargo loading scenario, the distance between the cargo and the camera and the direction relative to the camera can be determined, therefore, a depth information reference range is set based on the distance between the cargo and the camera and the direction of the cargo relative to the camera. If the target depth information of any pixel point is not within the depth information reference range, it indicates that there is a deviation in the measurement of its target depth information, and the pixel point is determined as a failure point. In this way, the failure point can be determined quickly and accurately.

[0096] The second method is to determine the failure point where the target depth information has deviations through the target depth information of the surrounding pixels. The implementation method is: for each pixel among multiple pixels, the depth values ​​of the surrounding pixels are weighted averaged to obtain a reference depth value. The absolute difference between the depth value of the pixel and the reference depth value is determined. When the absolute difference is greater than a preset reference threshold, the pixel is determined as a failure point. Among them, the reference threshold can be set to any value as needed.

[0097] In the embodiment of the present application, considering that the pixel point and the pixels around it are very likely to belong to the same object, or even if they do not belong to the same object, they may be adjacent objects. Therefore, the depth information of the pixel point is similar to that of the surrounding pixels. In this case, the depth values ​​of the surrounding pixels are weighted averaged to obtain a reference depth value. Based on the absolute difference between the depth value of the pixel point and the reference depth value, it is judged whether there is a deviation in the depth value of the pixel point, and the failure point where the target depth information has a deviation can be accurately determined.

[0098] Step S40: deleting the target depth information of the interference point, and updating the target depth information of the failure point based on the target depth information of the pixel points around the failure point that belong to the same object as the failure point.

[0099] Deleting the target depth information of the interference point, that is, deleting the target depth information of the pixel point corresponding to the loader, can avoid the interference caused by the target depth information corresponding to the loader, thereby improving the accuracy of the determined cargo loading rate.

[0100] Through the image segmentation in the above text, the objects to which the multiple pixels in the current visible light image belong have been determined. Therefore, in this step, the pixels around the failure point that belong to the same object as the failure point can be determined based on the image segmentation result, so that the target depth information of the failure point can be updated based on the target depth information of the pixels around the failure point that belong to the same object as the failure point.

[0101] Optionally, updating the target depth information of the failure point based on the target depth information of the pixel points around the failure point that belong to the same object as the failure point includes: determining the target depth information of any pixel point around the failure point that belongs to the same object as the failure point, and using the target depth information as the updated target depth information of the failure point, so that the target depth information of the failure point can be determined simply and quickly.

[0102] Optionally, updating the target depth information of the failure point based on the target depth information of the pixel points around the failure point that belong to the same object as the failure point includes: fusing the target depth information of multiple pixel points around the failure point that belong to the same object as the failure point to obtain the updated target depth information of the failure point. For example, the depth values ​​of multiple pixel points around the failure point that belong to the same object as the failure point are averaged to obtain the depth value of the failure point.

[0103] In the embodiment of the present application, the target depth information of multiple pixel points around the failure point that belong to the same object as the failure point is fused to obtain the updated target depth information of the failure point, which can further reduce the error and improve the accuracy of the target depth information of the failure point.

[0104] Step S50: Determine the change in cargo volume based on target depth information of pixel points in the current visible light image and target depth information of pixel points in historical visible light images, and determine the current cargo loading rate based on the change in cargo volume, wherein the historical visible light image is collected in a cargo loading scenario before the current visible light image.

[0105] The cargo loading rate is the volume ratio of the cargo in the cargo compartment, that is, the ratio of the volume of the cargo to the volume of the cargo compartment. The cargo volume change refers to the change in the cargo volume when the current visible light image is collected relative to the cargo volume when the historical visible light image is collected.

[0106] Exemplarily, determining the change in cargo volume based on target depth information of pixel points in a current visible light image and target depth information of pixel points in historical visible light images includes: determining a depth difference value based on target depth information of pixel points in the current visible light image and target depth information of pixel points in historical visible light images, the depth difference value representing the depth of newly loaded cargo; and multiplying the depth difference value by the width and height of the cargo to obtain the change in cargo volume.

[0107] One point that needs to be explained is that in the cargo loading scenario, as the amount of cargo loaded in the cargo compartment continues to increase, the distance between the cargo and the camera continues to decrease, so the depth value detected by the camera will continue to decrease, then the depth difference value of the pixel point of the current visible light image relative to the historical visible light image can represent the depth of the newly loaded cargo. Exemplarily, in an embodiment of the present application, the width and height of the cargo compartment are used as the width and height of the cargo to improve the calculation efficiency of the cargo volume change, thereby improving the calculation efficiency of the cargo loading rate. Alternatively, based on the target depth information of the pixel points in the current visible light image, the width and height of the loaded cargo are determined, and the cargo volume change is determined based on the currently determined width and height, thereby improving the accuracy of the cargo loading rate.

[0108] Exemplarily, determining the current cargo loading rate based on the cargo volume change includes: dividing the cargo volume change by the cargo compartment volume to obtain the loading rate change, and determining the current cargo loading rate as the sum of the loading rate change and the historical cargo loading rate.

[0109] The historical cargo loading rate is stored in the measuring device so that it can be called at any time. Exemplarily, the historical cargo loading rate is also obtained and stored by the method provided by the present application. Optionally, the historical visible light image is a previous frame image adjacent to the current visible light image. Of course, the historical visible light image and the current visible light image are not limited to two adjacent frames of images.

[0110] Figure 6 Schematic diagram of cargo loading rate measurement scenario. Figure 6 , the visual device is set in front of the cargo compartment, and the depth camera detects the depth value L1 of object point A, the angle θ, and the depth value d1 of object point B. Based on the depth value L1, the depth value d1 and the angle θ, the width x1 of the cargo compartment can be determined. Similarly, the height and depth of the cargo compartment can also be measured. Then, based on the height, depth and width x1 of the cargo compartment, the volume of the cargo compartment can be obtained.

[0111] Optionally, after determining the current cargo loading rate, the measuring device displays the current cargo loading rate on the interface, so that the user can check the current cargo loading rate at any time. Alternatively, the measuring device can also display the loading rate change on the interface, so that the user can understand the loading efficiency more intuitively through the loading rate change.

[0112] In the embodiment of the present application, the cargo volume change is determined based on the target depth information of the pixel points in the current visible light image and the target depth information of the pixel points in the historical visible light image. Since the cargo volume change is the change in the cargo volume when the current visible light image is collected relative to the cargo volume when the historical visible light image is collected, the current cargo loading rate can be accurately determined based on the cargo volume change and the historical cargo loading rate corresponding to the collection time of the historical visible light image.

[0113] In the solution provided in the present application, the current visible light image under the cargo loading scene and the target depth information of multiple pixels in the current visible light image are collected. Since the target depth information indicates the spatial position of the object points corresponding to the multiple pixels, the current cargo loading rate can be determined based on the target depth information. Considering that there are interference points and failure points in the multiple pixels collected, the interference points are the pixels corresponding to the loaders, and the failure points are the pixels with deviations in the target depth information. Whether it is an interference point or a failure point, its target depth information will introduce errors to the calculation of the current cargo loading rate. Therefore, the current visible light image is segmented and the background modeling is performed to determine the objects to which the multiple pixels in the current visible light image belong respectively. In this way, not only can the interference points corresponding to the loaders in the multiple pixels be known, thereby deleting the target depth information of the interference points, but also the target depth information of the failure point can be updated based on the target depth information of the pixels around the failure point that belong to the same object as the failure point. Since there will be no large jump between the target depth information of the pixels around the failure point that belong to the same object as the failure point and the target depth information of the failure point, the target depth information of the failure point is updated based on the target depth information of the pixels around the failure point that belong to the same object as the failure point, which can ensure the accuracy of the updated target depth information. In this way, based on the target depth information of the pixels in the current visible light image and the target depth information of the pixels in the historical visible light image, the change in the volume of the cargo can be accurately determined, and the current cargo loading rate can be determined based on the change in the volume of the cargo. Therefore, this solution can effectively improve the accuracy of the current cargo loading rate.

[0114] Figure 7 This is a flow chart of the second embodiment of the method for measuring the loading rate of the present application. Figure 5 The first embodiment of the loading rate measurement method shown in the figure proposes a second embodiment of the loading rate measurement method of the present application. Taking the execution subject as the loading rate measurement system as an example, in the second embodiment, the above step S10 includes:

[0115] Step S101: collecting a current visible light image in a cargo loading scene.

[0116] Step S102: using a depth camera and laser ranging to collect first depth information of a plurality of pixel points in the current visible light image.

[0117] Step S103: using a multi-camera and binocular parallax to collect second depth information of a plurality of pixel points in the current visible light image.

[0118] Exemplarily, the multi-eye camera is a binocular camera or a trinocular camera. Exemplarily, the binocular camera is composed of a visible light camera and a depth camera, or the binocular camera is composed of two visible light cameras. Exemplarily, the trinocular camera is composed of three visible light cameras.

[0119] Step S104: Fusing the first depth information of the plurality of pixels with the second depth information to obtain target depth information of the plurality of pixels.

[0120] Optionally, the first depth information of multiple pixels is fused with the second depth information to obtain target depth information of the multiple pixels, including: for any pixel among the multiple pixels, when the depth value in the second depth information is less than a reference threshold, the second depth information is used as the target depth information; when the depth value in the second depth information is not less than the reference threshold, the first depth information is used as the target depth information.

[0121] Among them, the depth value indicates the distance between the object point corresponding to the pixel point and the camera. The above reference threshold can be set according to the accuracy of the distance measurement of the binocular camera and the depth camera. For example, if the measurement accuracy of the binocular camera within 3 meters is higher than that of the depth camera, the reference threshold is set to 3 meters. In this way, for any pixel point, if the depth value measured by the binocular camera is less than 3 meters, the second depth information measured by the binocular camera is used as the target depth information of the pixel point. If the depth value measured by the binocular camera is not less than 3 meters, the first depth information measured by the depth camera is used as the target depth information of the pixel point.

[0122] In the embodiment of the present application, taking into account the different principles of distance measurement, in the scenario of close-range measurement, the binocular camera has higher accuracy, and in the scenario of long-range measurement, the depth camera has higher accuracy. Therefore, a reference threshold is set. For any pixel point, when the depth value measured by the binocular camera is less than the reference threshold, the second depth information measured by the binocular camera is selected as the target depth information of the pixel point. When the depth value measured by the binocular camera is not less than the reference threshold, the first depth information measured by the depth camera is selected as the target depth information of the pixel point. This depth information fusion method greatly improves the accuracy of the determined target depth information.

[0123] Optionally, the first depth information of the plurality of pixels is fused with the second depth information to obtain the target depth information of the plurality of pixels, including: for any pixel of the plurality of pixels, a depth value in the first depth information is weighted averaged with a depth value in the second depth information to obtain a depth value in the target depth information. This method is simple and fast, and improves the efficiency of determining the depth value in the target depth information while ensuring the accuracy of the depth value in the target depth information.

[0124] The present application does not restrict the weight set for the depth value of the first depth information and the weight set for the depth value of the second depth information. For example, in a close-range scene, the weight of the depth value of the second depth information is greater than the weight of the depth value of the first depth information. In a long-range scene, the weight of the depth value of the first depth information is greater than the weight of the depth value of the second depth information.

[0125] In an embodiment of the present application, it is necessary to obtain depth information from two different sources, namely, first depth information of multiple pixel points collected by a depth camera, and second depth information of multiple pixel points collected by a binocular camera, and the first depth information is obtained using the laser ranging principle, and the second depth information is obtained using the binocular parallax principle. Then, the first depth information of the multiple pixel points and the second depth information are fused to obtain target depth information of the multiple pixel points, which can compensate for the errors introduced by the two ranging methods respectively and obtain more accurate target depth information.

[0126] Figure 8 A schematic diagram of a loading rate measurement process provided by an exemplary embodiment of the present application. Figure 8 , the depth information of the cargo compartment is collected by the depth camera, and the volume of the cargo compartment is determined based on the depth information of the cargo compartment. The current visible light image is collected by the visible light camera, and the current visible light image is segmented and background modeled to realize object recognition, that is, to determine the objects to which each pixel in the current visible light image belongs. After the pixel corresponding to the loader is determined by background modeling, interference must be removed, that is, the target depth information of the pixel corresponding to the loader is deleted. Then point cloud filtering and point cloud completion are performed. Point cloud filtering refers to determining the failure point where the target depth information of the pixel of the current visible light image has a deviation. Point cloud completion refers to updating the target depth information of the failure point based on the target depth information of the pixel around the failure point that belongs to the same object as the failure point. After that, the change in cargo volume can be determined based on the target depth information of the historical visible light image and the target depth information of the current visible light image. By comparing the change in cargo volume with the cargo compartment volume, the change in loading rate can be obtained, and the cargo loading rate can be determined based on the change in loading rate.

[0127] Fig. 9A schematic diagram of another loading rate measurement process provided by an exemplary embodiment of the present application. Fig. 9 , and the above Figure 8 The loading rate measurement process in the above is different from Figure 8 The measurement process shown only uses the depth information collected by the depth camera, while Fig. 9 The measurement process shown uses depth information from two different sources. One is the depth information collected by the depth camera, and the other is the depth information collected by the binocular camera by forming a binocular camera with a depth camera and a visible light camera. Specifically, after the visible light camera collects the current visible light image and the depth camera collects the grayscale image, binocular stereo vision ranging is realized based on the position deviation between the corresponding points of the two images. Then the depth information from the two different sources is fused to obtain the target depth information of the current visible light image. Then point cloud filtering and point cloud completion are performed, and then the change in cargo volume is determined to obtain the cargo loading rate.

[0128] Fig.10 A schematic diagram of another loading rate measurement process provided by an exemplary embodiment of the present application. Fig.10 , and the above Fig. 9 The loading rate measurement process in the above is different from Fig. 9 The binocular camera in the image is composed of a depth camera and a visible light camera. Fig.10 The binocular camera in the figure is composed of two visible light cameras. Correspondingly, after the two visible light cameras respectively collect the current visible light images, the binocular stereo vision distance measurement is realized based on the position deviation between the corresponding points of the two visible light images. The subsequent loading rate measurement process is similar to Fig. 9 The process shown is similar and will not be repeated here.

[0129] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the loading rate measurement method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0130] This application also provides a loading rate measurement device, please refer to Fig.11 , the loading rate measuring device comprises:

[0131] A data acquisition module 10 is used to acquire a current visible light image in a cargo loading scene and acquire target depth information of a plurality of pixel points in the current visible light image, wherein the target depth information indicates a spatial position of an object point corresponding to the plurality of pixel points;

[0132] An image processing module 20 is used to perform image segmentation and background modeling on the current visible light image to determine the objects to which the multiple pixels in the current visible light image belong respectively;

[0133] A pixel point determination module 30 is used to determine interference points and failure points among a plurality of pixel points, wherein the interference points are pixel points corresponding to the loader, and the failure points are pixel points where the target depth information has deviations;

[0134] An information updating module 40, configured to delete the target depth information of the interference point, and update the target depth information of the failure point based on the target depth information of the pixel points around the failure point that belong to the same object as the failure point;

[0135] The loading rate determination module 50 is used to determine the change in cargo volume based on the target depth information of the pixel points in the current visible light image and the target depth information of the pixel points in the historical visible light image, and determine the current cargo loading rate based on the change in cargo volume, wherein the historical visible light image is collected in the cargo loading scene before the current visible light image.

[0136] Optionally, the data acquisition module 10 includes:

[0137] A first acquisition submodule is used to acquire first depth information of a plurality of pixel points by using a depth camera and laser ranging;

[0138] A second acquisition submodule is used to acquire second depth information of a plurality of pixel points by using a multi-camera and binocular parallax;

[0139] The fusion submodule is used to fuse the first depth information of the multiple pixel points with the second depth information to obtain the target depth information of the multiple pixel points.

[0140] Optionally, a fusion submodule is used to use the second depth information as target depth information for any pixel among multiple pixels when the depth value in the second depth information is less than a reference threshold; and use the first depth information as target depth information when the depth value in the second depth information is not less than the reference threshold; wherein the depth value indicates the distance between the object point corresponding to the pixel and the camera.

[0141] Optionally, a fusion submodule is used to perform a weighted average of a depth value in the first depth information and a depth value in the second depth information for any pixel among multiple pixels to obtain a depth value in the target depth information; wherein the depth value indicates the distance between the object point corresponding to the pixel and the camera.

[0142] Optionally, the data acquisition module 10 is used to acquire multiple reference images, the multiple reference images include a current visible light image and adjacent frames of the current visible light image, and the reference images include multiple pixels; acquire reference depth information of multiple pixels in the multiple reference images; for any pixel among the multiple pixels, fuse the multiple reference depth information of the pixel to obtain the target depth information of the pixel.

[0143] Optionally, the information updating module 40 is used to fuse the target depth information of multiple pixel points around the failure point that belong to the same object as the failure point, to obtain updated target depth information of the failure point.

[0144] The loading rate measurement device provided by the present application adopts the loading rate measurement method in the above embodiment to measure the cargo loading rate, which can solve the technical problem of inaccurate cargo loading rate measurement in the related art. Compared with the related art, the beneficial effects of the loading rate measurement device provided by the present application are the same as the beneficial effects of the loading rate measurement method provided by the above embodiment, and other technical features in the loading rate measurement device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0145] The present application provides a loading rate measurement device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the loading rate measurement method in the above-mentioned embodiment one.

[0146] Reference below Fig.12 , which shows a schematic diagram of the structure of a loading rate measuring device suitable for implementing the embodiment of the present application. The loading rate measuring device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The loading rate measurement device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0147] like Fig.12As shown, the load rate measurement device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a ROM (Read Only Memory) 1002 or a program loaded from a storage device 1003 to a RAM (Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the load rate measurement device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the load rate measurement device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a load rate measurement device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0148] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0149] The loading rate measurement device provided by the present application adopts the loading rate measurement method in the above embodiment to measure the cargo loading rate, which can solve the technical problem of inaccurate cargo loading rate measurement in the related art. Compared with the related art, the beneficial effects of the loading rate measurement device provided by the present application are the same as the beneficial effects of the loading rate measurement method provided by the above embodiment, and the other technical features in the loading rate measurement device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0150] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0151] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0152] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the loading rate measurement method in the above-mentioned embodiment.

[0153] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), optical fiber, CD-ROM (CD-Read Only Memory, portable compact disk read-only memory), optical storage device, magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0154] The computer-readable storage medium may be included in the load factor measurement device; or may exist independently without being assembled into the load factor measurement device.

[0155] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the loading rate measuring device, the loading rate measuring device: collects the current visible light image in the cargo loading scene and collects the target depth information of multiple pixel points in the current visible light image; performs image segmentation and background modeling on the current visible light image to determine the objects to which the multiple pixel points in the current visible light image respectively belong; determines interference points and failure points among the multiple pixel points, the interference points are the pixel points corresponding to the loaders, and the failure points are the pixel points with deviations in the target depth information; deletes the target depth information of the interference points, and updates the target depth information of the failure points based on the target depth information of the pixel points around the failure points that belong to the same object as the failure points; determines the cargo volume change based on the target depth information of the pixel points in the current visible light image and the target depth information of the pixel points in the historical visible light images, and determines the current cargo loading rate based on the cargo volume change, wherein the historical visible light images are collected in the cargo loading scene before the current visible light images.

[0156] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0158] The modules described in the embodiments of the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0159] The readable storage medium provided by the present application is a computer-readable storage medium, and the computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned loading rate measurement method, which can solve the technical problem of inaccurate measurement of the cargo loading rate in the related art. Compared with the related art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the loading rate measurement method provided by the above embodiments, and will not be elaborated here.

[0160] The present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the loading rate measurement method as described above.

[0161] The computer program product provided by the present application can solve the technical problem of inaccurate measurement of the cargo loading rate in the related art. Compared with the related art, the beneficial effects of the computer program product provided by the present application are the same as those of the loading rate measurement method provided by the above embodiments, and will not be elaborated here.

[0162] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made under the technical concept of the present application by using the content of the specification and drawings of the present application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A method for measuring a loading rate, characterized in that: The loading rate measurement method comprises: Collecting a current visible light image in a cargo loading scene and collecting target depth information of a plurality of pixel points in the current visible light image, wherein the target depth information indicates spatial positions of object points corresponding to the plurality of pixel points; Performing image segmentation and background modeling on the current visible light image to determine objects to which a plurality of pixels in the current visible light image respectively belong, and objects in motion in the visible light image; Determine an interference point and a failure point among the multiple pixels, the interference point being a pixel point corresponding to an object in motion in the visible light image, the object in motion being a loader, and the failure point being a pixel point where target depth information has a deviation; Deleting the target depth information of the interference point, and updating the target depth information of the failure point based on the target depth information of the pixel points around the failure point that belong to the same object as the failure point; Based on the target depth information of the pixel points in the current visible light image and the target depth information of the pixel points in the historical visible light image, determining the cargo volume change, dividing the cargo volume change by the cargo compartment volume to obtain the loading rate change, determining the sum of the loading rate change and the historical cargo loading rate as the current cargo loading rate, and displaying the current cargo loading rate and the loading rate change on an interface, wherein the historical visible light image is collected in the cargo loading scene before the current visible light image; The collecting of target depth information of a plurality of pixel points in the current visible light image includes: Using a depth camera, collecting first depth information of the plurality of pixel points by using laser ranging; Using a multi-camera, collecting second depth information of the plurality of pixel points using binocular parallax; For any pixel point among the multiple pixel points, when the depth value in the second depth information is less than a reference threshold, the second depth information is used as the target depth information; when the depth value in the second depth information is not less than the reference threshold, the first depth information is used as the target depth information, the reference threshold is set based on the accuracy of distance measurement between the multi-eye camera and the depth camera, within the reference threshold range, the accuracy of the multi-eye camera is higher than that of the depth camera, and the depth value indicates the distance between the object point corresponding to the pixel point and the camera.

2. The method for measuring the loading rate according to claim 1, characterized in that: The collecting of the current visible light image in the cargo loading scene and the collecting of target depth information of a plurality of pixel points in the current visible light image include: Acquiring a plurality of reference images, wherein the plurality of reference images include the current visible light image and adjacent frames of the current visible light image, and the reference images include the plurality of pixel points; Acquiring reference depth information of the multiple pixel points in the multiple reference images; For any pixel point among the multiple pixel points, multiple reference depth information of the pixel point are fused to obtain target depth information of the pixel point.

3. The method for measuring the loading rate according to claim 1, characterized in that: The updating of the target depth information of the failure point based on the target depth information of the pixel points around the failure point and belonging to the same object as the failure point comprises: The target depth information of a plurality of pixel points around the failure point and belonging to the same object as the failure point is fused to obtain updated target depth information of the failure point.

4. A loading rate measuring device, characterized in that: The loading rate measuring device comprises: A data acquisition module, used to acquire a current visible light image in a cargo loading scene and acquire target depth information of a plurality of pixel points in the current visible light image, wherein the target depth information indicates spatial positions of object points corresponding to the plurality of pixel points; An image processing module, used to perform image segmentation and background modeling on the current visible light image to determine objects to which multiple pixels in the current visible light image belong, and objects in motion in the visible light image; a pixel point determination module, used to determine interference points and failure points among the plurality of pixel points, wherein the interference points are pixel points corresponding to objects in motion in the visible light image, wherein the objects in motion are loaders, and the failure points are pixel points where target depth information has deviations; An information updating module, configured to delete the target depth information of the interference point, and update the target depth information of the failure point based on the target depth information of the pixel points around the failure point that belong to the same object as the failure point; a loading rate determination module, configured to determine a cargo volume change based on target depth information of pixel points in the current visible light image and target depth information of pixel points in historical visible light images, divide the cargo volume change by the cargo compartment volume to obtain a loading rate change, and determine the current cargo loading rate as the sum of the loading rate change and the historical cargo loading rate, wherein the historical visible light image is collected in the cargo loading scene before the current visible light image; Wherein, the data acquisition module includes: A first acquisition submodule is used to acquire first depth information of the plurality of pixel points by using a depth camera and laser ranging; A second acquisition submodule is used to acquire second depth information of the plurality of pixel points by using a multi-camera and binocular parallax; a fusion submodule, configured to, for any pixel point among the multiple pixel points, use the second depth information as the target depth information when the depth value in the second depth information is less than a reference threshold; and use the first depth information as the target depth information when the depth value in the second depth information is not less than the reference threshold, wherein the reference threshold is set based on the accuracy of distance measurement between the multi-eye camera and the depth camera, within the reference threshold range, the accuracy of the multi-eye camera is higher than that of the depth camera, and the depth value indicates the distance between the object point corresponding to the pixel point and the camera; The device also includes a module for displaying the current cargo loading rate and the loading rate change on an interface.

5. A loading rate measurement system, characterized in that: The loading rate measurement system comprises: a visual device and a measuring device, wherein the visual device is electrically connected to the measuring device; the visual device is arranged at a position with a preset distance from the cargo loading area, and its field of view covers the cargo loading area; The visual device includes a depth camera and a multi-eye camera; the multi-eye camera includes at least two visible light cameras; any visible light camera in the multi-eye camera is used to collect the current visible light image in the cargo loading scene; The depth camera is used to collect first depth information of multiple pixel points in the current visible light image by using laser ranging; the multi-eye camera is used to collect second depth information of the multiple pixel points by using binocular parallax; the measuring device is used to use the second depth information as target depth information for any pixel point among the multiple pixel points when the depth value in the second depth information is less than a reference threshold; and use the first depth information as the target depth information when the depth value in the second depth information is not less than the reference threshold, wherein the reference threshold is set based on the accuracy of distance measurement between the multi-eye camera and the depth camera, and within the reference threshold range, the accuracy of the multi-eye camera is higher than that of the depth camera, the depth value indicates the distance between the object point corresponding to the pixel point and the camera, and the target depth information indicates the spatial position of the object point corresponding to the multiple pixel points; The measuring device is also used to perform image segmentation and background modeling on the current visible light image to determine the objects to which the multiple pixels in the current visible light image respectively belong, and the objects in motion in the visible light image; determine interference points and failure points among the multiple pixels, wherein the interference points are pixels corresponding to the objects in motion in the visible light image, the objects in motion are loaders, and the failure points are pixels whose target depth information has deviations; delete the target depth information of the interference points, and update the target depth information of the failure points based on the target depth information of the pixels around the failure points that belong to the same object as the failure points; determine the change in cargo volume based on the target depth information of the pixels in the current visible light image and the target depth information of the pixels in the historical visible light images, divide the change in cargo volume by the cargo compartment volume to obtain the change in loading rate, determine the sum of the change in loading rate and the historical cargo loading rate as the current cargo loading rate, and display the current cargo loading rate and the change in loading rate on an interface, wherein the historical visible light images are collected in the cargo loading scene before the current visible light image.

6. A loading rate measuring device, characterized in that: The load rate measurement device comprises: a memory, a processor, and a load rate measurement program stored in the memory and executable on the processor, wherein the load rate measurement program implements the load rate measurement method according to any one of claims 1 to 3 when executed by the processor.

7. A storage medium, characterized in that: The storage medium stores a load factor measurement program, and when the load factor measurement program is executed by the processor, the load factor measurement method according to any one of claims 1 to 3 is implemented.

8. A computer program product, characterized in that The computer program product comprises a loading rate measurement program, and when the loading rate measurement program is executed by a processor, the loading rate measurement method according to any one of claims 1 to 3 is implemented.

Images