Detection method, device and equipment for ship unloading operation and medium

Abstract

The invention provides a ship unloading operation detection method and device, equipment and a medium, and the method comprises the steps: obtaining a first image and a second image of a material pile in a cabin, the first image and the second image are two adjacent frames of images, and the first image is the previous frame of image of the second image; determining an optical flow vector of each pixel point in the second image and a corresponding pixel point in the first image; determining a target optical flow vector in the optical flow vectors, wherein the target optical flow vector is an optical flow vector representing downward movement of a pixel point; and under the condition that the target optical flow vector set is distributed in a target area, determining that the material pile has a material collapse risk in the target area. According to the method, the movement trend of each pixel point in the current frame image relative to the previous frame can be obtained based on the optical flow vector analysis, whether the material collapse risk exists or not can be further determined, the material collapse risk does not need to be judged manually, and the material collapse risk sensing efficiency can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of ship unloading management, and in particular, relates to a detection method, device, equipment and medium for ship unloading operations. Background Art

[0002] Ship unloading refers to the process of unloading the cargo carried by the ship from the ship at the port. During the unloading process, whether the pile of materials in the cabin collapses has a certain impact on the material grabbing. Specifically, if there is a risk of collapse in the cabin, the material grabbing head may be buried by the collapsed material. In the related art, the specific location of the collapsed material in the cabin is usually determined based on manual experience, resulting in low efficiency. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a detection method, device, equipment and medium for ship unloading operations to solve the above-mentioned problems.

[0004] The detection method of ship unloading operation provided in this application includes:

[0005] Acquire a first image and a second image of a material pile in a cabin, wherein the first image and the second image are two adjacent frames of images, and the first image is a previous frame of image of the second image;

[0006] Determine an optical flow vector between each pixel in the second image and a corresponding pixel in the first image;

[0007] Determine a target optical flow vector among the optical flow vectors, wherein the target optical flow vector is an optical flow vector representing downward movement of a pixel point;

[0008] In the case that the target optical flow vectors are concentrated in the target area, it is determined that the material pile has a risk of collapse in the target area.

[0009] In one embodiment of the present application, when the first image and the second image are captured by a motion-based camera, determining an optical flow vector between each pixel in the second image and a corresponding pixel in the first image includes:

[0010] Extracting first feature points in the first image and second feature points in the second image;

[0011] Performing feature point matching processing on the first feature point and the second feature point to obtain a feature point matching result;

[0012] Based on the feature point matching result, determining the posture change information of the second image acquired by the camera relative to the first image acquired;

[0013] Performing motion compensation processing on the second image based on the posture change information;

[0014] Determine an optical flow vector between each pixel point in the second image after motion compensation and a corresponding pixel point in the first image.

[0015] In one embodiment of the present application, when the target optical flow vectors are concentrated in a target area, after determining that the material pile has a risk of collapse in the target area, the method further includes:

[0016] The severity of the material collapse corresponding to the target area is determined based on the average amplitude of the target optical flow vector in the target area and the preset ranges corresponding to each severity of the material collapse.

[0017] In one embodiment of the present application, the method further includes:

[0018] When the chain bucket grabs the material in the material pile, acquiring a fourth image of the chain bucket;

[0019] Performing image segmentation processing on the fourth image to obtain a bucket wall image of the chain bucket and a surface image of the material in the chain bucket;

[0020] Based on the bucket wall image of the chain bucket, the surface image of the material in the chain bucket, and a pre-calibrated first correlation coefficient, a first volume of the material in the chain bucket is determined.

[0021] In one embodiment of the present application, the method further includes:

[0022] When the chain bucket grabs the material in the material pile, obtaining first point cloud data of the chain bucket;

[0023] Performing registration and alignment processing on the first point cloud data and the second point cloud data to obtain third point cloud data, wherein the second point cloud data is point cloud data when the chain bucket is empty;

[0024] Respectively performing voxelized three-dimensional reconstruction on the third point cloud data and the second point cloud data to obtain a first voxelized grid of the third point cloud data and a second voxelized grid of the second point cloud data;

[0025] Performing forward projection processing on the first voxelized grid and the second voxelized grid respectively to obtain a first voxel matrix of the first voxelized grid and a second voxel matrix of the second voxelized grid;

[0026] Based on the height ratio between the first voxel matrix and the second voxel matrix and a pre-calibrated second correlation coefficient, a second volume of the material in the paternoster is determined.

[0027] In one embodiment of the present application, the method further includes:

[0028] In the case that the first volume is inconsistent with the second volume, a weighted average process is performed based on the weight of the first volume and the weight of the second volume to obtain a third volume, and the third volume is the final volume of the material in the chain bucket.

[0029] In one embodiment of the present application, the method further includes:

[0030] When the material pile in the target area has a risk of collapse, and the material loading amount in the chain bucket for grabbing the material is greater than a preset threshold, a control alarm is activated and the chain bucket is controlled to be lifted.

[0031] The detection device for ship unloading operation provided in this application includes:

[0032] A first acquisition module, used to acquire a first image and a second image of a material pile in the cabin, wherein the first image and the second image are two adjacent frames of images, and the first image is a previous frame of image of the second image;

[0033] A first determination module, used to determine an optical flow vector between each pixel point in the second image and a corresponding pixel point in the first image;

[0034] A second determination module is used to determine a target optical flow vector among the optical flow vectors, wherein the target optical flow vector is an optical flow vector representing the downward movement of a pixel point;

[0035] The third determination module is used to determine that the material pile has a risk of collapse in the target area when the target optical flow vectors are concentrated in the target area.

[0036] In one embodiment of the present application, the first image and the second image are captured based on a moving camera, and the second determination module is specifically configured to:

[0037] Extracting first feature points in the first image and second feature points in the second image;

[0038] Performing feature point matching processing on the first feature point and the second feature point to obtain a feature point matching result;

[0039] Based on the feature point matching result, determining the posture change information of the second image acquired by the camera relative to the first image acquired;

[0040] Performing motion compensation processing on the second image based on the posture change information;

[0041] The fourth determination module is used to determine the optical flow vector of each pixel point in the second image after motion compensation processing and the corresponding pixel point in the first image.

[0042] In one embodiment of the present application, the device further includes:

[0043] The fifth determination module is used to determine the severity of the material collapse corresponding to the target area based on the average amplitude of the target optical flow vector in the target area and the preset ranges corresponding to each severity of the material collapse.

[0044] In one embodiment of the present application, the device further includes:

[0045] A second acquisition module, configured to acquire a fourth image of the chain bucket when the chain bucket grabs the material in the material pile;

[0046] An image processing module, used for performing image segmentation processing on the fourth image to obtain a bucket wall image of the chain bucket and a surface image of the material in the chain bucket;

[0047] The sixth determination module is used to determine a first volume of the material in the chain bucket based on the bucket wall image of the chain bucket, the surface image of the material in the chain bucket, and a pre-calibrated first correlation coefficient.

[0048] In one embodiment of the present application, the device further includes:

[0049] A third acquisition module is used to acquire first point cloud data of the chain bucket when the chain bucket grabs the material in the material pile;

[0050] A registration and alignment module, used for performing registration and alignment processing on the first point cloud data and the second point cloud data to obtain third point cloud data, wherein the second point cloud data is the point cloud data when the chain bucket is empty;

[0051] A three-dimensional reconstruction module, used to perform voxelized three-dimensional reconstruction on the third point cloud data and the second point cloud data respectively, to obtain a first voxelized grid of the third point cloud data and a second voxelized grid of the second point cloud data;

[0052] a projection processing module, configured to perform forward projection processing on the first voxelized grid and the second voxelized grid respectively, to obtain a first voxel matrix of the first voxelized grid and a second voxel matrix of the second voxelized grid;

[0053] The seventh determination module is used to determine a second volume of the material in the chain bucket based on a height ratio between the first voxel matrix and the second voxel matrix and a pre-calibrated second correlation coefficient.

[0054] In one embodiment of the present application, the device further includes:

[0055] The weighted average processing module is used to perform weighted average processing based on the weight of the first volume and the weight of the second volume to obtain a third volume when the first volume is inconsistent with the second volume. The third volume is the final volume of the material in the chain bucket.

[0056] In one embodiment of the present application, the device further includes:

[0057] The control module is used to start an alarm and control the chain bucket to lift when the material pile in the target area has a risk of collapse and the material loading amount in the chain bucket used to grab the material is greater than a preset threshold.

[0058] The electronic device provided by the present application includes:

[0059] one or more processors;

[0060] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the detection method of the ship unloading operation.

[0061] The computer-readable storage medium provided in the present application stores a computer program thereon, and when the computer program is executed by a processor of a computer, the computer is enabled to execute the method for detecting the ship unloading operation.

[0062] Beneficial effects of the present application: In the present application, the first image and the second image are acquired, and the optical flow vector of each pixel point in the second image and the corresponding pixel point in the first image is determined. Based on the optical flow vector, the movement trend of the pixel point from the first image to the second image can be judged. Based on this, the present application can determine the target optical flow vector in the optical flow vector that represents the downward movement of the pixel point; when the target optical flow vector is concentrated in the target area, it can be determined that the material pile in the target area has a risk of collapse. It can be seen that the present application can obtain the movement trend of each pixel point in the current frame image compared with the previous frame based on the optical flow vector analysis, and further determine whether there is a risk of collapse, without the need to manually judge the risk of collapse, which is conducive to improving the efficiency of collapse risk detection.

[0063] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0065] Figure 1 is a flow chart of a method for detecting a ship unloading operation shown in an exemplary embodiment of the present application;

[0066] Figure 2 is a schematic diagram of detecting a collapsed material area shown in an exemplary embodiment of the present application;

[0067] Figure 3 is a schematic diagram of the setting position of the intelligent sensing device shown in an exemplary embodiment of the present application;

[0068] Figure 4 is a structural schematic diagram of a detection device for ship unloading operations shown in an exemplary embodiment of the present application;

[0069] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0070] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0071] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0072] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0073] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for detecting a ship unloading operation shown in an exemplary embodiment of the present application. Figure 1 As shown, in an exemplary embodiment, the method for detecting the ship unloading operation includes at least steps S110 to S140, which are described in detail as follows:

[0074] Step S110, acquiring a first image and a second image of a material pile in a cabin, wherein the first image and the second image are two adjacent frames of images, and the first image is a previous frame of image of the second image;

[0075] Step S120, determining an optical flow vector between each pixel point in the second image and a corresponding pixel point in the first image;

[0076] Step S130, determining a target optical flow vector in the optical flow vectors, wherein the target optical flow vector is an optical flow vector representing the downward movement of a pixel point;

[0077] Step S140, when the target optical flow vectors are concentrated in the target area, determining that the material pile has a risk of collapse in the target area.

[0078] In step S110, the first image is the previous frame image of the second image. During the unloading operation, the camera can continuously collect images of the material pile based on a certain frequency. The second image can be the current frame image, and the first image can be the previous frame image. By acquiring the current frame image and the previous frame image, the movement trend of each pixel in the current frame image compared with the previous frame image can be further analyzed.

[0079] In step S120, based on the optical flow analysis algorithm, the optical flow vector of each pixel in the second image and the corresponding pixel in the first image can be determined. The optical flow vector is used to describe the movement of pixels in the image between consecutive frames. It is a basic component of the optical flow field and is used to characterize the displacement or motion information of a single pixel in the current frame compared to the previous frame. The above optical flow vector in the embodiment of the present application can be calculated by a variety of algorithms, such as the Horn-Schunck optical flow method or the Farneback method.

[0080] In step S130, a target optical flow vector representing the downward movement of the pixel is determined. The optical flow vector can usually be represented by a line segment with an arrow, the length of the arrow represents the size (amplitude) of the vector, and the direction of the arrow represents the direction of the vector. The direction of the vector is the direction of movement of the pixel. Based on the direction of the vector, the target optical flow vector can be easily determined.

[0081] In order to more clearly understand how to determine the target optical flow vector, the present application takes determining the target optical flow vector based on the angle of the optical flow vector as an example and explains it in detail.

[0082] In some embodiments, the horizontal direction can be set as the x-axis, the right direction is the positive direction of the x-axis, the vertical direction is the y-axis, and the downward direction is the positive direction of the y-axis. When the direction of the optical flow vector is 90° with the x-axis, the direction of the optical flow vector is the same as the positive direction of the y-axis, both of which are vertically downward. Based on this, the angle range (ang) representing the downward movement of the pixel point can be pre-set, for example, ang is set to [45°, 135°] or [π / 5, 4π / 5]. When the angle of the optical flow vector meets the above angle range ang, it can be determined that the movement direction of the pixel point represented is downward.

[0083] In other embodiments, in addition to setting the angle of the target optical flow vector to meet the above-mentioned angle range conditions of [45°, 135°] or [π / 5, 4π / 5], the amplitude of the target optical flow vector can also be set to be greater than a preset amplitude threshold (min_magnitude, referred to as mag) to ignore optical flow with too small movement. The above-mentioned mag can be 8.0 by default and can also be modified based on demand. In this embodiment, when the angle of the optical flow vector meets the angle range ang and the amplitude of the optical flow vector is greater than mag, it is determined to be the target optical flow vector.

[0084] In step S140, concentrated distribution can be understood as the proportion of the target optical flow vector in the target area exceeds a preset proportion threshold, or the number of the target optical flow vector in the target area exceeds a preset number, wherein the target area can be a pre-defined area of ​​a specific size.

[0085] For a clearer understanding, the following combination Figure 2 An example is given of how to determine an area with a risk of landslide.

[0086] a. Divide the second image into N small blocks of size region_size x region_size, where N is a positive integer greater than 1;

[0087] b. Determine the optical flow vector of each pixel in each block and the corresponding pixel in the first image. The representation of the optical flow vector can be found in Figure 2 The green marked part in;

[0088] c. Screen the optical flow vector of each pixel in each block to determine the target optical flow vector in each block;

[0089] d. In any block a of the N blocks above, if the proportion of the target optical flow vector exceeds a preset threshold (e.g., 50%), it is determined that the area corresponding to block a (i.e., the target area) has a risk of collapse. Figure 2 The above-mentioned small block a is marked with a red frame.

[0090] It is worth mentioning that the order of the above steps is not fixed. In some embodiments, the optical flow vector of each pixel can be calculated first and then the image can be divided into N small blocks; in some embodiments, the optical flow vector of each pixel can be calculated first, and the target optical flow vector can be determined before the image is divided into N small blocks.

[0091] The embodiment of the present application can obtain the movement trend of each pixel in the current frame image compared with the previous frame based on optical flow vector analysis, and can further determine whether there is a risk of material collapse. There is no need to manually judge the risk of material collapse, which is conducive to improving the efficiency of material collapse risk perception.

[0092] In one embodiment of the present application, when the first image and the second image are captured by a motion-based camera, determining an optical flow vector between each pixel in the second image and a corresponding pixel in the first image includes:

[0093] Extracting first feature points in the first image and second feature points in the second image;

[0094] Performing feature point matching processing on the first feature point and the second feature point to obtain a feature point matching result;

[0095] Based on the feature point matching result, determining the posture change information of the second image acquired by the camera relative to the first image acquired;

[0096] Performing motion compensation processing on the second image based on the posture change information;

[0097] Determine an optical flow vector between each pixel point in the second image after motion compensation and a corresponding pixel point in the first image.

[0098] In this implementation, when the first image and the second image are captured based on a moving camera, there may be optical flow errors in the first image and the second image caused by the camera movement. Based on this, before determining the optical flow vector, this embodiment first performs motion compensation processing on the second image to eliminate the optical flow errors caused by the camera's own movement. The camera for capturing the moving image of the material pile can be set on the side of the bucket elevator facing the cabin. For details, see Figure 3 , Figure 3 The intelligent sensing equipment on the center left includes a camera that captures the material pile.

[0099] The above-mentioned extracted feature points can be determined by adopting a feature point detection algorithm. Commonly used feature point detection algorithms include Harris corner detection, FAST corner detection, SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features), etc. These algorithms can find pixels with significant local features in the image, such as corner points and edge points in the image.

[0100] The feature point matching process described above matches the feature points extracted from the previous frame image with the feature points in the current frame image, so as to determine their corresponding relationship in different frames. The feature point matching process can be determined by using a commonly used feature point matching algorithm. Among them, a random sampling consistency algorithm can also be used to eliminate erroneous matching points to obtain a reliable matching pair (i.e., the feature point matching result described above).

[0101] Based on the above feature point matching results, camera motion estimation can be performed, that is, the posture change information of the camera capturing the second image relative to the first image can be calculated, and motion compensation can be performed on each pixel in the second image based on the posture change information.

[0102] In the embodiment of the present application, when the first image and the second image are captured by a motion-based camera, the above method can eliminate the optical flow error caused by the movement of the camera itself, thereby improving the accuracy of collapsed area recognition.

[0103] In one embodiment of the present application, when the target optical flow vectors are concentrated in a target area, after determining that the material pile has a risk of collapse in the target area, the method further includes:

[0104] The severity of the material collapse corresponding to the target area is determined based on the average amplitude of the target optical flow vector in the target area and the preset ranges corresponding to each severity of the material collapse.

[0105] In this embodiment, each collapse severity can be divided according to needs, for example, divided into two or more severity levels. For example, it can be divided into three levels with increasing severity, namely collapse severity 1, collapse severity 2 and collapse severity 3, and the numerical range of each collapse severity increases with the increasing level.

[0106] In the embodiment of the present application, by further analyzing the amplitude of the target optical flow vector and judging the severity of the material collapse, it is helpful to take corresponding measures in time.

[0107] In some embodiments, after determining that the target area has the risk of material collapse, it can be reported in real time; after determining the severity of the material collapse in the target area, it can also be reported in real time.

[0108] In one embodiment of the present application, the method further includes:

[0109] When the chain bucket grabs the material in the material pile, acquiring a fourth image of the chain bucket;

[0110] Performing image segmentation processing on the fourth image to obtain a bucket wall image of the chain bucket and a surface image of the material in the chain bucket;

[0111] Based on the bucket wall image of the chain bucket, the surface image of the material in the chain bucket, and a pre-calibrated first correlation coefficient, a first volume of the material in the chain bucket is determined.

[0112] In this embodiment, the fourth image can be collected based on a camera facing the chain bucket, and the camera facing the chain bucket can also be set on the above-mentioned BE tube. Figure 3 As shown, the intelligent sensing device on the right includes a camera facing the bucket.

[0113] The above image segmentation process can adopt the Mask R-CNN deep learning method to improve the segmentation accuracy through model training. In the model application, the fourth image is input into the model to obtain the bucket wall image of the chain bucket and the surface image of the material in the chain bucket output by the model.

[0114] Based on the bucket wall image of the chain bucket, the surface image of the material in the chain bucket, and the pre-calibrated first correlation coefficient, the first volume of the material in the chain bucket can be determined. The first correlation coefficient is a parameter obtained by fitting the field measurement and image data, which can be specifically understood as: in the historical period: the parameter obtained by fitting the field measured material volume with the corresponding bucket wall image data and material surface data. For ease of understanding, the following formula is provided to explain how to determine the first volume.

[0115] V1=a×λ+b

[0116] Wherein, V1 represents the first volume, λ and b are the first correlation coefficients, and a is the ratio of the area of ​​the material surface area to the area of ​​the bucket wall area, which can be determined based on the ratio of the detection frame size of the material surface image to the detection frame size of the bucket wall image.

[0117] In the embodiment of the present application, when the chain bucket grabs material in the material pile, obtaining an image of the chain bucket and determining the volume of the material based on the image is conducive to taking further measures based on the volume of the material, such as continuing loading or stopping loading.

[0118] In one embodiment of the present application, the method further includes:

[0119] When the chain bucket grabs the material in the material pile, obtaining first point cloud data of the chain bucket;

[0120] Performing registration and alignment processing on the first point cloud data and the second point cloud data to obtain third point cloud data, wherein the second point cloud data is point cloud data when the chain bucket is empty;

[0121] Respectively performing voxelized three-dimensional reconstruction on the third point cloud data and the second point cloud data to obtain a first voxelized grid of the third point cloud data and a second voxelized grid of the second point cloud data;

[0122] Performing forward projection processing on the first voxelized grid and the second voxelized grid respectively to obtain a first voxel matrix of the first voxelized grid and a second voxel matrix of the second voxelized grid;

[0123] Based on the height ratio between the first voxel matrix and the second voxel matrix and a pre-calibrated second correlation coefficient, a second volume of the material in the paternoster is determined.

[0124] In this embodiment, the first point cloud data can be collected based on a point cloud collection device facing the chain bucket, and the point cloud collection device can also be set on the above-mentioned BE cylinder. Figure 3 As shown, the intelligent sensing device on the right includes the above-mentioned point cloud acquisition device.

[0125] When the above-mentioned first point cloud data is collected and the fourth image is collected, the actual volume of the material in the chain bucket must not change. The first point cloud data and the fourth image can be collected at the same time, and can be collected in real time after grabbing the material.

[0126] After collecting the first point cloud data, the first point cloud data is aligned with the second point cloud data of the predictive modeling to ensure that the real-time point cloud data is aligned with the model. The second point cloud data of the predictive modeling is the point cloud data when the bucket is empty. The algorithm for the alignment processing can adopt the iterative closest point algorithm (ICP).

[0127] The first point cloud data (i.e., the third point cloud data) and the second point cloud data after registration and alignment are voxelized and three-dimensionally reconstructed and divided into regular three-dimensional grids to obtain a first voxelized grid and a second voxelized grid. Each voxel is a cube, indicating whether a certain area is occupied by point cloud data.

[0128] The first voxelized grid and the second voxelized grid are respectively subjected to orthographic projection processing, that is, projected onto an orthographic plane on an xz plane to obtain a first voxel matrix and a second voxel matrix. The projection matrix projects the three-dimensional voxelized data onto an orthographic plane on an xz plane, thereby simplifying the analysis of volume differences.

[0129] Based on the height ratio of the first voxel matrix to the second voxel matrix and the pre-calibrated second correlation coefficient, the second volume of the material in the chain bucket can be determined. The above second correlation coefficient can map the geometric volume to the physical volume. The second correlation coefficient is pre-calibrated based on the measured physical volume of the material in the chain bucket during the historical period and the point cloud data. For a clearer understanding, a specific formula is provided below to illustrate how to calculate the physical volume based on the height ratio.

[0130] V2=m×(V_geo)^k+c

[0131] Wherein, V2 is the above-mentioned second volume (i.e., physical volume); V_geo is the height of the first voxel matrix ÷ the height of the second voxel matrix; m, k, and c are all second correlation coefficients. Among them, m is the calibration scale factor, which mainly reflects the global proportional relationship between the geometric volume and the actual material volume; b is the nonlinear index, which controls the degree of nonlinearity between the geometric volume and the actual volume; c is the bias term, which is used to correct the geometric volume noise when empty. Considering the nonlinear relationship between the geometric volume and the actual material volume in a specific scene. For example, when the chain bucket is empty, the geometric volume is not exactly zero (there will be a small amount of noise or the influence of different structures between the chain buckets), and the fact that the actual volume is empty needs to be corrected. When the chain bucket is empty, the point cloud data will still detect some geometric volumes (errors of different chain bucket structures), resulting in V_geo! = 0. At this time, the actual material volume V2 should be zero.

[0132] In an embodiment of the present application, when the chain bucket grabs material in a material pile, the point cloud data of the chain bucket can be obtained in real time, and the volume of the material can be determined based on the real-time point cloud data and the point cloud data when the chain bucket is empty, which is conducive to taking further measures based on the volume of the material, such as continuing loading or stopping loading.

[0133] In some embodiments, when the first volume and the second volume are consistent, the physical volume of the material in the chain bucket can be directly output.

[0134] In some other embodiments, the method further comprises:

[0135] In the case that the first volume is inconsistent with the second volume, a weighted average process is performed based on the weight of the first volume and the weight of the second volume to obtain a third volume, and the third volume is the final volume of the material in the chain bucket.

[0136] The weight of the first volume and the weight of the second volume are both preset weights. In the embodiment of the present application, when the first volume and the second volume are inconsistent, weighted fusion is performed based on their respective weights, which is conducive to improving the accuracy of physical volume detection.

[0137] In one embodiment of the present application, the method further includes:

[0138] When the material pile in the target area has a risk of collapse, and the material loading amount in the chain bucket for grabbing the material is greater than a preset threshold, a control alarm is activated and the chain bucket is controlled to be lifted.

[0139] In this implementation, the material loading amount in the chain bucket may be the physical volume of the material determined by the method of the above-mentioned embodiment, and the preset threshold is a preset volume threshold.

[0140] In the embodiment of the present application, when there is a risk of material collapse in the target area and the material loading amount in the chain bucket used to grab the material is greater than a preset threshold, the control alarm is activated and the chain bucket is controlled to be lifted, which can effectively prevent the material grabbing head from being buried by the collapsed material.

[0141] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0142] Figure 4 FIG. 1 is a block diagram of a detection device for ship unloading operation shown in an exemplary embodiment of the present application. Figure 4 As shown, the exemplary detection device for ship unloading operation includes:

[0143] A first acquisition module 410 is used to acquire a first image and a second image of a material pile in a cabin, wherein the first image and the second image are two adjacent frames of images, and the first image is a previous frame of image of the second image;

[0144] A first determination module 420, configured to determine an optical flow vector between each pixel in the second image and a corresponding pixel in the first image;

[0145] A second determination module 430 is used to determine a target optical flow vector among the optical flow vectors, wherein the target optical flow vector is an optical flow vector representing the downward movement of a pixel point;

[0146] The third determination module 440 is configured to determine that the material pile has a risk of collapse in the target area when the target optical flow vectors are concentrated in the target area.

[0147] In one embodiment of the present application, the first image and the second image are captured based on a moving camera, and the second determination module 430 is specifically configured to:

[0148] Extracting first feature points in the first image and second feature points in the second image;

[0149] Performing feature point matching processing on the first feature point and the second feature point to obtain a feature point matching result;

[0150] Based on the feature point matching result, determining the posture change information of the second image acquired by the camera relative to the first image acquired;

[0151] Performing motion compensation processing on the second image based on the posture change information;

[0152] The fourth determination module is used to determine the optical flow vector of each pixel point in the second image after motion compensation processing and the corresponding pixel point in the first image.

[0153] In one embodiment of the present application, the device further includes:

[0154] The fifth determination module is used to determine the severity of the material collapse corresponding to the target area based on the average amplitude of the target optical flow vector in the target area and the preset ranges corresponding to each severity of the material collapse.

[0155] In one embodiment of the present application, the device further includes:

[0156] A second acquisition module, configured to acquire a fourth image of the chain bucket when the chain bucket grabs the material in the material pile;

[0157] An image processing module, used for performing image segmentation processing on the fourth image to obtain a bucket wall image of the chain bucket and a surface image of the material in the chain bucket;

[0158] The sixth determination module is used to determine a first volume of the material in the chain bucket based on the bucket wall image of the chain bucket, the surface image of the material in the chain bucket, and a pre-calibrated first correlation coefficient.

[0159] In one embodiment of the present application, the device further includes:

[0160] A third acquisition module is used to acquire first point cloud data of the chain bucket when the chain bucket grabs the material in the material pile;

[0161] A registration and alignment module, used for performing registration and alignment processing on the first point cloud data and the second point cloud data to obtain third point cloud data, wherein the second point cloud data is the point cloud data when the chain bucket is empty;

[0162] A three-dimensional reconstruction module, used to perform voxelized three-dimensional reconstruction on the third point cloud data and the second point cloud data respectively, to obtain a first voxelized grid of the third point cloud data and a second voxelized grid of the second point cloud data;

[0163] a projection processing module, configured to perform forward projection processing on the first voxelized grid and the second voxelized grid respectively, to obtain a first voxel matrix of the first voxelized grid and a second voxel matrix of the second voxelized grid;

[0164] The seventh determination module is used to determine a second volume of the material in the chain bucket based on a height ratio between the first voxel matrix and the second voxel matrix and a pre-calibrated second correlation coefficient.

[0165] In one embodiment of the present application, the device further includes:

[0166] The weighted average processing module is used to perform weighted average processing based on the weight of the first volume and the weight of the second volume to obtain a third volume when the first volume is inconsistent with the second volume. The third volume is the final volume of the material in the chain bucket.

[0167] In one embodiment of the present application, the device further includes:

[0168] The control module is used to start an alarm and control the chain bucket to lift when the material pile in the target area has a risk of collapse and the material loading amount in the chain bucket used to grab the material is greater than a preset threshold.

[0169] It should be noted that the detection device for ship unloading operation provided in the above embodiment and the detection method for ship unloading operation provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the detection device for ship unloading operation provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0170] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements the detection method for ship unloading operations provided in the above-mentioned embodiments.

[0171] Figure 5 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 5 The computer system 500 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0172] like Figure 5As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503, such as executing the method described in the above embodiment. In the RAM 503, various programs and data required for system operation are also stored. The CPU 501, the ROM 502 and the RAM 503 are connected to each other through the bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0173] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

[0174] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 509, and / or installed from a removable medium 511. When the computer program is executed by a central processing unit (CPU) 501, various functions defined in the system of the present application are executed.

[0175] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0176] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0177] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0178] Another aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the above-mentioned method for detecting ship unloading operations. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0179] Another aspect of the present application also provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the detection method of the ship unloading operation provided in each of the above embodiments.

[0180] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for detecting ship unloading operations, characterized in that: include: Acquire a first image and a second image of a material pile in a cabin, wherein the first image and the second image are two adjacent frames of images, and the first image is a previous frame of image of the second image; Determine an optical flow vector between each pixel in the second image and a corresponding pixel in the first image; Determine a target optical flow vector among the optical flow vectors, wherein the target optical flow vector is an optical flow vector representing downward movement of a pixel point; In the case that the target optical flow vectors are concentrated in the target area, it is determined that the material pile has a risk of collapse in the target area.

2. The method according to claim 1, characterized in that: In the case where the first image and the second image are captured by a motion-based camera, determining an optical flow vector of each pixel point in the second image and a corresponding pixel point in the first image includes: Extracting first feature points in the first image and second feature points in the second image; Performing feature point matching processing on the first feature point and the second feature point to obtain a feature point matching result; Based on the feature point matching result, determining the posture change information of the second image acquired by the camera relative to the first image acquired; Performing motion compensation processing on the second image based on the posture change information; Determine an optical flow vector between each pixel point in the second image after motion compensation and a corresponding pixel point in the first image.

3. The method according to claim 1, characterized in that In the case where the target optical flow vectors are concentrated in the target area, after determining that the material pile has a risk of material collapse in the target area, the method further includes: The severity of the material collapse corresponding to the target area is determined based on the average amplitude of the target optical flow vector in the target area and the preset ranges corresponding to each severity of the material collapse.

4. The method according to claim 1, characterized in that The method further comprises: When the chain bucket grabs the material in the material pile, acquiring a fourth image of the chain bucket; Performing image segmentation processing on the fourth image to obtain a bucket wall image of the chain bucket and a surface image of the material in the chain bucket; Based on the bucket wall image of the chain bucket, the surface image of the material in the chain bucket, and a pre-calibrated first correlation coefficient, a first volume of the material in the chain bucket is determined.

5. The method according to claim 4, characterized in that The method further comprises: When the chain bucket grabs the material in the material pile, obtaining first point cloud data of the chain bucket; Performing registration and alignment processing on the first point cloud data and the second point cloud data to obtain third point cloud data, wherein the second point cloud data is point cloud data when the chain bucket is empty; Respectively performing voxelized three-dimensional reconstruction on the third point cloud data and the second point cloud data to obtain a first voxelized grid of the third point cloud data and a second voxelized grid of the second point cloud data; Performing forward projection processing on the first voxelized grid and the second voxelized grid respectively to obtain a first voxel matrix of the first voxelized grid and a second voxel matrix of the second voxelized grid; Based on the height ratio between the first voxel matrix and the second voxel matrix and a pre-calibrated second correlation coefficient, a second volume of the material in the paternoster is determined.

6. The method according to claim 5, characterized in that The method further comprises: In the case that the first volume is inconsistent with the second volume, a weighted average process is performed based on the weight of the first volume and the weight of the second volume to obtain a third volume, and the third volume is the final volume of the material in the chain bucket.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: When the material pile in the target area has a risk of collapse, and the material loading amount in the chain bucket for grabbing the material is greater than a preset threshold, a control alarm is activated and the chain bucket is controlled to be lifted.

8. A detection device for ship unloading operation, characterized in that: include: A first acquisition module, used to acquire a first image and a second image of a material pile in the cabin, wherein the first image and the second image are two adjacent frames of images, and the first image is a previous frame of image of the second image; A first determination module, used to determine an optical flow vector between each pixel point in the second image and a corresponding pixel point in the first image; A second determination module is used to determine a target optical flow vector among the optical flow vectors, wherein the target optical flow vector is an optical flow vector representing the downward movement of a pixel point; The third determination module is used to determine that the material pile has a risk of collapse in the target area when the target optical flow vectors are concentrated in the target area.

9. A device, characterized in that: include: one or more processors and memory, A computer program is stored in the memory, and when the one or more processors execute the computer program, the device executes the method for detecting ship unloading operations according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when executed by one or more processors, the device executes the method for detecting ship unloading operations as described in any one of claims 1 to 7.

Images