A method for estimating warehouse material volume

Abstract

This disclosure provides an estimation method, apparatus, electronic device, and storage medium. The method includes: acquiring a video stream of warehouse materials, decomposing it frame by frame to obtain a first set of frame images; identifying the warehouse materials in each frame of the first set of frame images, and performing depth detection on the warehouse materials in each frame to determine the depth information of the warehouse materials; acquiring the height information of the warehouse materials, and calculating the volume of the warehouse materials based on the depth information and height information; and performing fusion learning on the volumes of the warehouse materials determined from the periodically acquired video streams, and using the fused volume as the volume of the warehouse materials.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of material volume estimation, and in particular to an estimation method based on material stacking depth. BACKGROUND

[0002] In the engineering field, in order to improve the level of intelligent factory and supply chain management, there is an urgent need to estimate the volume or mass of large material stacks. Through technical means, automatic, fast and accurate acquisition of raw material information can improve work efficiency and reduce the consumption of manpower and material resources.

[0003] At present, the estimation methods of large material stack volume mainly include multi-view stereo vision, joint detection of projector and camera, laser radar, three-dimensional laser scanner, ultrasonic ranging and other technical solutions. In the measurement fields of grain storage, coal mine, cement and landslide, they can be practically applied.

[0004] The existing technical solutions all rely on expensive equipment such as laser scanners, depth cameras, or require additional auxiliary devices such as projection devices. In order to be practically applied, expensive new devices need to be equipped. At the same time, the visual solution or point cloud solution based on convolutional neural network has high resource consumption, so the system hardware performance requirement is also higher, and expensive servers need to be configured.

[0005] At the same time, there is a lack of related optimization for warehouse stacks with low result update frequency. The result of video conversion to frame image processing is that the detection result fluctuates greatly, which does not conform to the real situation. SUMMARY

[0006] The present disclosure provides a warehouse material volume estimation method to at least solve the foregoing technical problems.

[0007] According to a first aspect of the present disclosure, a warehouse material volume estimation method is provided, which comprises:

[0008] Obtaining a first video stream of the warehouse material, frame decomposing the first video stream to obtain a first group of frame pictures;

[0009] Identifying the warehouse material in each frame picture in the first group of frame pictures, and performing depth detection on the warehouse material in each frame picture to determine the depth information of the warehouse material;

[0010] Obtaining the height information of the warehouse material, and calculating the volume of the warehouse material based on the depth information and the height information of the warehouse material;

[0011] Fusing the volumes of the warehouse material determined by the periodically obtained video streams, and taking the fused volume as the volume of the warehouse material.

[0012] Its beneficial effect is, through the combination of perspective principle, signal processing, rule learning and deep learning, the warehouse material can be detected simply and quickly, and the warehouse stack volume can be estimated by using a single ordinary camera combined with the actual situation on site.

[0013] In some embodiments, before identifying the warehouse material in each frame picture in the first set of frame pictures, further comprising:

[0014] Performing green screen inspection on each frame picture in the first set of frame pictures;

[0015] The green screen inspection includes: inspecting the average value of the pixels of each frame picture in the first set of frame pictures, and screening out the frame picture average value that meets the preset frame picture average value standard.

[0016] Performing strip inspection on each frame picture in the first set of frame pictures;

[0017] The strip inspection includes: inspecting the standard deviation of the longitudinal pixels of each frame picture in the first set of frame pictures, and screening out the longitudinal pixel standard deviation that meets the preset frame picture longitudinal pixel standard deviation.

[0018] Performing blur inspection on each frame picture in the first set of frame pictures;

[0019] The blur inspection includes: inspecting the standard deviation of the overall pixels of each frame picture in the first set of frame pictures, and screening out the overall pixel standard deviation that meets the preset frame picture overall standard deviation.

[0020] Performing light inspection on each frame picture in the first set of frame pictures;

[0021] The light inspection includes: counting the high-light pixel points in each frame picture in the first set of frame pictures, and screening out the high-light pixel points that meet the preset high-light pixel point proportion in the frame picture.

[0022] Performing vehicle inspection on each frame picture in the first set of frame pictures;

[0023] The vehicle inspection includes: using a target detection model to inspect the first set of frame pictures, and identifying the pictures containing vehicles in the first set of frame pictures and the number of vehicles in the pictures.

[0024] Its beneficial effect is that by screening the obtained frame pictures, the possibility of affecting the estimation result due to the defects of the pictures themselves is reduced.

[0025] In some embodiments, wherein the warehouse material in each frame picture in the first set of frame pictures is identified, and the depth of the warehouse material in each frame picture is detected to determine the depth information of the warehouse material, comprising:

[0026] annotating positions of four vertices of the warehouse bottom edge in each frame picture in the first group of frame pictures in the frame picture;

[0027] According to the annotated positions of the four vertices of the warehouse bottom edge, a warehouse bottom surface part in the frame picture is intercepted to obtain a second group of pictures;

[0028] An affine transformation is performed on the second group of pictures to convert the second group of pictures into rectangles to obtain a third group of pictures;

[0029] Pixels of each picture in the third group of pictures are convoluted with a normal distribution, and the third group of pictures after the convolution are all converted into gray scale pictures corresponding to the third group of pictures;

[0030] According to pixel information of the gray scale pictures corresponding to the third group of pictures, the third group of pictures are converted into one-dimensional signals corresponding to the third group of pictures;

[0031] According to a corresponding relationship between the pixel points in the third group of pictures and the warehouse bottom edge size, a determined peak position in the one-dimensional signal is taken as depth information of the warehouse material.

[0032] In some embodiments, the volume of the warehouse material determined according to the periodically acquired video stream is fused and learned, and the volume after the fusion and learning is taken as the volume of the warehouse material, including:

[0033] The detection results of each frame in the first group of frame pictures are sorted into a first queue, and the detection results include whether the current frame picture passes the green screen test, the pull bar test, the blur test, the light test, vehicle information in the current frame picture, and a specific value of the volume of the warehouse material in the current frame picture;

[0034] The detection results in the first queue that do not pass the green screen test, the pull bar test, the blur test, and the light test are replaced with the detection results of the previous frame to generate a second queue;

[0035] When the frame image corresponding to the last detection result in the first queue does not identify a vehicle in the vehicle test, the volume of the warehouse material in the last detection result in the second queue is taken as the volume of the warehouse material obtained by this fusion and learning;

[0036] When the frame image corresponding to the last result in the first queue identifies a vehicle in the vehicle test and the number of vehicles is 1, the detection results in the second queue are filtered from back to front until a detection result in which there is no vehicle information is filtered out, the specific value of the volume of the warehouse material in the filtered detection result in the second queue is compared with the specific value of the volume of the warehouse material in the last detection result in the first queue, and a difference value between the specific value of the volume of the warehouse material in the filtered detection result in the second queue and the specific value of the volume of the warehouse material in the last detection result in the first queue is calculated;

[0037] When the frame image corresponding to the last result in the first queue identifies a vehicle in the vehicle inspection, and the number of vehicles is greater than 1, stop this fusion learning, and continue to screen the inspection results of the subsequent frame pictures decomposed from the video stream until the detection result in which there is no vehicle information is screened out;

[0038] The specific value of the warehouse material volume in the inspection results of the last 10 frame pictures of the screened inspection results in the inspection results of the subsequent frame pictures decomposed from the video stream is calculated by the median method;

[0039] The median calculation result is taken as the warehouse material volume obtained by this fusion learning.

[0040] In some embodiments, the specific value of the warehouse material volume in the detection result screened by the second queue is compared with the specific value of the warehouse material volume in the last detection result in the first queue, and the comparison further includes:

[0041] If the warehouse material volume in the detection result screened by the second queue is greater than the warehouse material volume in the last detection result in the first queue, the difference calculation result is compared with the material volume scooped up by one forklift;

[0042] If the difference calculation result does not exceed the material volume scooped up by one forklift, the warehouse material volume in the last detection result in the first queue is taken as the warehouse material volume obtained by this fusion learning, and if the difference calculation result exceeds the material volume scooped up by one forklift, the specific value of the warehouse material volume in the detection result screened by the second queue is subtracted by the material volume scooped up by one forklift as the warehouse material volume obtained by this fusion learning;

[0043] If the warehouse material volume in the detection result screened by the second queue is less than the warehouse material volume in the last detection result in the first queue, the difference calculation result is compared with the material volume increased by one truck;

[0044] If the difference calculation result does not exceed the material volume increased by one truck, the warehouse material volume in the last detection result in the first queue is taken as the warehouse material volume obtained by this fusion learning, and if the difference calculation result exceeds the material volume increased by one truck, the specific value of the warehouse material volume in the detection result screened by the second queue is added by the material volume increased by one truck as the warehouse material volume obtained by this fusion learning.

[0045] According to the first aspect of the present disclosure, a warehouse material volume estimation device is also provided, wherein the device comprises:

[0046] A camera unit for shooting a warehouse video stream;

[0047] An image acquisition unit for decomposing the warehouse video stream to obtain a first group of frame pictures;

[0048] the information processing unit is configured to perform depth detection on the first set of images, and obtain a warehouse material volume corresponding to each frame of image in the first set of images according to a detection result;

[0049] the signal processing unit is further configured to perform fusion learning on the warehouse material volume corresponding to each frame of image;

[0050] the signal receiving unit is configured to receive image information transmitted by the image acquisition unit to the information processing unit;

[0051] the signal receiving unit is further configured to output the material volume after fusion learning.

[0052] In some embodiments, further comprising:

[0053] the image inspection unit is configured to perform a first inspection on the first set of images obtained by the image acquisition unit;

[0054] The first inspection includes:

[0055] light inspection on the first set of images;

[0056] quality inspection on the first set of images;

[0057] obstacle inspection on the first set of images;

[0058] the storage unit is configured to store a warehouse video stream photographed by the camera unit;

[0059] The storage unit is further configured to store the material volume after fusion learning.

[0060] According to the first aspect of the present disclosure, there is also provided an electronic device, comprising:

[0061] at least one processor; and

[0062] a memory in communication with the at least one processor; wherein

[0063] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of the above.

[0064] According to the first aspect of the present disclosure, there is also provided a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to perform the method of any one of the above.

[0065] The present application combines the characteristics of the warehouse in the engineering field, converts the estimation of the volume of the stack into the estimation of the depth of the stack in the warehouse, and achieves the detection purpose by detecting the change of light of the stack and the ground. At the same time, by constructing a fusion learning method, the stability and accuracy of the detection result are improved. In the existing scheme, the spatial information is mainly obtained in detail to achieve a good prediction effect, and the spatial information is obtained by means of laser and the like. The present application only uses a single ordinary camera combined with the actual situation on the spot to meet the warehouse material volume estimation demand, and the supporting hardware is simple, without GPU server, and through information fusion at different times, a good prediction effect is achieved.

[0066] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0067] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will be more apparent from the following detailed description read in conjunction with the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example, and not limitation. In the drawings:

[0068] In the drawings, identical or corresponding reference numerals indicate identical or corresponding parts.

[0069] Figure 1 A flowchart of a method for estimating the volume of warehouse materials according to an embodiment of the present disclosure is shown Figure One ;

[0070] Figure 2 A flowchart of a method for estimating the volume of warehouse materials according to an embodiment of the present disclosure is shown Figure Two ;

[0071] Figure 3 A flowchart of a method for estimating the volume of warehouse materials according to an embodiment of the present disclosure is shown

[0072] Figure 4 A flowchart of a method for estimating the volume of warehouse materials according to an embodiment of the present disclosure is shown

[0073] Figure 5 A one-dimensional signal graph converted from a gray-scale graph according to an embodiment of the present disclosure is shown

[0074] Figure 6 A flowchart of a method for estimating the volume of warehouse materials according to an embodiment of the present disclosure is shown DETAILED DESCRIPTION

[0075] In order to make the purposes, characteristics and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0076] The essence of the technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings.

[0077] Figure 1 The flowchart of the visual estimation method for the volume of warehouse materials in the embodiments of the present application Figure One As shown in the flowchart of the visual estimation method for the volume of warehouse materials in the embodiments of the present application Figure 1 As shown in the flowchart of the visual estimation method for the volume of warehouse materials in the embodiments of the present application Figure One The method comprises the following steps:

[0078] Step 101, obtaining a first video stream for warehouse materials, performing frame decomposition on the first video stream to obtain a first group of frame pictures.

[0079] In the present embodiment, preferably, the first video stream is a video stream transmitted by a camera in real time. The source of the video stream is not limited to a monitoring camera, but can also be any product with a shooting function, or any storage medium that can output video resources.

[0080] In the present embodiment, each frame picture in the first group of frame pictures corresponds to an independent result, and these results are not affected by each other. These results are finally fused and learned in step 104.

[0081] Step 102, identifying the warehouse materials in each frame picture in the first group of frame pictures, and performing depth detection on the warehouse materials in each frame picture to determine the depth information of the warehouse materials.

[0082] In the present embodiment, the depth of the warehouse materials identified in each frame picture is detected by a depth detection method. The depth detection method identifies the pixel information of the boundary line between the materials and the bottom edge of the warehouse in the picture to the critical position of the materials, and determines the actual depth size of the warehouse materials according to the corresponding relationship between the pixel information in the picture and the actual size of the warehouse.

[0083] Step 103, obtaining the height information of the warehouse materials, and calculating the volume of the warehouse materials based on the depth information and the height information of the warehouse materials.

[0084] In this embodiment, the height information of the warehouse material is related to the material itself. Different materials have different stacking heights. Since the materials are stacked in the warehouse according to the specification, the stacking height is a fixed empirical value. The material stacking width in the warehouse is the same as the warehouse width. By obtaining the height empirical value of the material stacking of the corresponding material and the warehouse material stacking depth information, the volume of the warehouse material can be calculated by combining the fixed material stacking width.

[0085] In step 104, the volumes of the warehouse materials determined by the periodically obtained video stream are fused and learned, and the fused and learned volume is taken as the volume of the warehouse material.

[0086] In this embodiment, the warehouse material estimated volume obtained in step 103 corresponds to each frame picture in step 101. Since the material update frequency in the warehouse is low, in order to ensure the stability and accuracy of the estimation result, the warehouse material estimated volume corresponding to each frame picture in the first group of frame pictures is fused and learned. Preferably, after the warehouse material estimated volumes in the first group of frame pictures are fused and learned, a single result is output as the result of this fusion learning, and this result is taken as the final estimation result of the volume of the material stacked in the warehouse in the first video stream.

[0087] Figure 2 The flowchart of the warehouse material volume estimation method of the embodiment of the present application Figure Two As shown in Figure 2 The flowchart of the warehouse material volume estimation method of the embodiment of the present application Figure Two includes the following steps:

[0088] In step 201, a video stream for warehouse material is obtained, and is decomposed frame by frame.

[0089] In this embodiment, the obtained warehouse camera video stream is decomposed into a group of frame pictures, and each picture after decomposition is individually executed in the subsequent steps to obtain an independent result. In step 205, the results of multiple frames are fused to output the result.

[0090] In this embodiment, since the collected video stream is updated in real time, the decomposed frame pictures of the updated video stream are also updated in real time. However, the material update frequency of the warehouse is low. Preferably, the estimated volume can be updated every three hours.

[0091] In step 202, it is verified whether the frame picture passes the quality inspection and light inspection.

[0092] In this embodiment, the frame picture may have different quality problems, which affect the judgment of the algorithm. Therefore, quality verification is needed before processing. Preferably, quality detection can be performed according to several common image quality problems of the picture, such as green screen, strip, blur, etc.

[0093] Figure 3 The flowchart for detecting the quality of the frame image in step 202 of the embodiment is shown in FIG. 2, and the quality detection of the frame image according to the embodiment of the application includes the following steps: Figure 3

[0094] Step 301, green screen detection. The green screen detection in the embodiment can be performed by detecting the average value of the pixels of the frame image. Whether the average value of the pixels of the frame image meets the standard is determined to determine whether the quality of the detected frame image meets the standard.

[0095] Step 302, bar detection. The bar detection in the embodiment can be performed by detecting the standard deviation of the vertical pixels of the frame image. Whether the standard deviation of the frame image is within the specified range is determined to determine whether the quality of the detected frame image meets the standard.

[0096] Step 303, blur detection. The blur detection in the embodiment can be performed by detecting the standard deviation of the overall frame image. Whether the standard deviation of the frame image is within the specified range is determined to determine whether the quality of the detected frame image meets the standard.

[0097] The green screen detection, bar detection, and blur detection can be sequentially detected in order or simultaneously detected, and the embodiment is not limited in this regard.

[0098] In the embodiment, the quality of the frame image collected by the light at some special moments may be affected, for example, the sunlight shines on the stacked material in the warehouse with the window in the morning, which will affect the subsequent estimation method detection. Preferably, the frame image meeting the above detection steps is subjected to light detection. In the embodiment, the detection method can be to count the number of highlight pixel points existing in the frame image. If the proportion of the existing highlight pixel points in the total pixel points of the image is small, the light detection is passed.

[0099] Preferably, the average value of the pixels of the frame image, the standard deviation of the vertical pixels, the standard deviation of the overall pixels, and the proportion of the highlight pixel points can be preset by the user according to the pixel point information of the expected image before detection.

[0100] In the embodiment, if the frame image passes the above detection, the subsequent step is continued, otherwise, the result fusion output is directly entered in step 205.

[0101] Step 203, detecting whether the frame image passes the working vehicle detection.

[0102] ​In this embodiment, there may be vehicles in the collected pictures, thereby affecting the detection of the estimation method. For example, when there is unloading or loading operation, the warehouse materials will change, and these operations need vehicles to work. Therefore, it is necessary to detect whether there is a vehicle in the collected frame picture. Preferably, a target detection model can be used to detect whether there is a truck or a forklift. For example, a YOLO or other deep learning single-stage detection model is selected, and with the aid of the pre-trained model in the COCO dataset, the image can be quickly and accurately detected to determine whether it contains a vehicle and the number of vehicles.

[0103] If a vehicle is detected, the current frame picture is not subjected to subsequent detection, and directly enters step 205 for result fusion output, but the subsequent frame image is subjected to working vehicle detection.

[0104] Step 204, identifying the warehouse materials in the picture and estimating the volume of the warehouse materials.

[0105] Figure 4 A flowchart for estimating the volume of warehouse materials in step 204 of this embodiment is shown in FIG. 8. Figure 4 As shown in FIG. 8, the estimation of the volume of warehouse materials in this embodiment includes the following steps:

[0106] Step 401, obtaining a complete warehouse bottom edge screenshot.

[0107] In this embodiment, the camera and the warehouse are in a relatively fixed position, so the positions of the four vertices of the warehouse bottom edge can be labeled on the first set of collected pictures, and the information of the four points of the complete bottom edge is saved in the configuration. According to the labeling information of the positions of the four points of the bottom edge, the frame picture is intercepted, and the intercepted picture is a quadrilateral picture containing only the bottom edge of the warehouse.

[0108] Step 402, modifying the perspective direction of the screenshot.

[0109] In this embodiment, since the intercepted warehouse bottom edge screenshot is not a standard rectangle, there is perspective, so the perspective direction of the screenshot needs to be changed for the subsequent steps. Preferably, the bottom edge image part after interception can be subjected to affine transformation, and the transformation result is a standard rectangle without perspective view angle. According to the pixel point information of the transformed screenshot and the actual size of the warehouse bottom edge, the correspondence between the pixels of the transformed screenshot and the actual size can be obtained.

[0110] Step 403, converting the transformed screenshot into a one-dimensional signal.

[0111] In this embodiment, after the transformed screenshot is obtained, the image can be preferably Gaussian blurred to eliminate noise and reduce the level of detail. The image is converted into a grayscale image to calculate the segmentation position of the material in the picture and the bottom edge of the warehouse. The average value of the pixels in the width direction of the grayscale image is calculated, and a 5-pixel window is preferably used in the depth direction to reduce the noise effect. The grayscale image is converted into a one-dimensional signal for processing.

[0112] Step 404, determine the peak position of the one-dimensional signal.

[0113] Figure 5 A one-dimensional signal graph converted from the grayscale image in step 403 of this embodiment is shown, where the horizontal coordinate of the one-dimensional signal graph is the pixel value in the depth direction of the intercepted picture, and the vertical coordinate is the sliding difference value of the pixel. In this embodiment, according to the corresponding relationship between the pixel point information and the actual size of the warehouse bottom edge, the sliding difference value of each 1 meter corresponding pixel of the one-dimensional signal is calculated. According to the obtained difference value, the possible position of the segmentation point in the one-dimensional signal is determined, where the segmentation point represents the boundary position of the stacked material and the bottom edge of the warehouse. The segmentation point will appear as a wave in the one-dimensional signal, and the sliding difference value corresponding to the segmentation point is the position of the wave peak, i.e. a maximum value of the signal.

[0114] Step 405, verify the peak position.

[0115] In this embodiment, due to light and specification problems, the material accumulation in the warehouse will reflect multiple different wave peaks in the one-dimensional signal. To ensure that the wave peak position determined in the one-dimensional signal in the above step is appropriate, 10 pictures of each material are preferably selected and marked with a boundary line. The sliding difference value of the marked position is calculated, and the average value of the sliding difference values of the 10 pictures is taken as the threshold value.

[0116] In this embodiment, according to the above threshold value, the minimum position x1 of the sliding difference value of the one-dimensional signal of each new picture that exceeds the threshold value is compared. According to x1, the wave peak position x2 can be determined within a certain range backward, and the starting position x0 of the fluctuation can be determined within a certain range forward.

[0117] If there is a value greater than x1 before the fluctuation position x0 of the one-dimensional signal, it is determined that there is a fluctuation at a position less than x0, and the new wave peak is recursively searched forward until the frontmost wave peak is determined. The pixel point position corresponding to the wave peak position is taken as the depth position of the material in the intercepted warehouse bottom edge image.

[0118] In the embodiment, after determining the last wave peak of the one-dimensional signal, the wavelength is determined. If the wavelength is within a reasonable range, it is determined that the picture effect is good, and the corresponding wave peak position information is output. If the wavelength is too large, it proves that the stacking state of the materials in the picture is not good, and a signal that the picture does not meet the depth detection requirement is output.

[0119] In step 406, the stacking volume of the warehouse materials is determined.

[0120] In the embodiment, since the demarcation line between the warehouse material stacking and the bottom edge of the warehouse is presented in the form of a wave through the one-dimensional signal, the wave peak position obtained in step 405 is the embodiment of the depth of the warehouse material stacking in the picture. Since the picture is a mapping of the complete bottom edge of the warehouse, the wave peak position can be directly converted into the actual depth of the material stacking in the warehouse before the perspective is modified.

[0121] Therefore, after obtaining the output wave peak position information, the pixel position corresponding to the wave peak position is converted into the actual size of the position of the warehouse material stacking depth according to the corresponding relationship between the pixel point information of the intercepted warehouse bottom edge image and the actual size of the warehouse bottom edge, and the estimated warehouse material stacking depth is obtained. Preferably, the warehouse width is a fixed width, and in actual application, the stacking height of the warehouse materials is generally fixed, so the height can be set as an empirical value. Therefore, after determining the warehouse material stacking depth, the estimated volume of the material stacking is calculated according to the fixed width and the empirical value of the height.

[0122] In step 205, the fusion learning result is output.

[0123] In the embodiment, since the warehouse update frequency is low, the fusion learning method can be used to fuse multiple frames of results, which can improve the accuracy and stability of the results and avoid errors caused by the truck just after unloading the materials without standardizing the material stacking.

[0124] In the embodiment, two empty queues and a vehicle counter are set. The queue length at least contains the estimation result of each frame in the video stream. The first queue is used as an actual queue to save the actual estimation result of each frame and the information of the above quality inspection, light inspection, and vehicle inspection. The results corresponding to the frame pictures that do not meet the requirements of the previous quality inspection and light inspection are discarded, and the result of the previous frame is replaced with the discarded result. The replaced queue is used as a fusion queue to occupy the second empty queue.

[0125] In the embodiment, when the information of the vehicle inspection in the last result of the actual queue is 0, that is, the vehicle is not contained in the corresponding frame picture, the result of the fusion learning is that the last result in the fusion queue is output as the result of the fusion learning. Since the fusion queue corresponds to the actual queue, when the last result in the actual queue detects the vehicle working state, that is, the vehicle information is detected in the frame picture corresponding to the last result of the actual queue, and the vehicle information is 1, the detection results are filtered from back to front in the fusion queue until the detection result in which the vehicle information does not exist is filtered out. In order to ensure the accuracy and stability of the output estimated volume, the detection results in the fusion queue except the filtered detection result are emptied, and the vehicle counter becomes 1.

[0126] In the embodiment, when the last result in the actual queue is in the normal state, that is, the vehicle information is not detected in the frame picture corresponding to the last result of the actual queue, the current fusion queue is detected. If the fusion queue is just emptied and only contains the fusion learning result of the last time, and the vehicle counter is 1, it is proved that the vehicle unloading or loading occurred in the last fusion learning, that is, the vehicle working occurred. At this time, the result in the fusion queue is compared with the last result in the actual queue. If the last result in the actual queue is larger than the result in the fusion queue, it is proved that the unloading working exists. The difference obtained by comparison is evaluated. If the difference exceeds the material volume of one vehicle unloading, it is indicated that the forklift does not standardize the material after unloading. The result of the fusion queue is added by the reasonable unloading volume of one vehicle to be output as the volume of the fusion learning.

[0127] If the last result in the actual queue is smaller than the result in the fusion queue, the loading working exists. The difference obtained by comparison is evaluated. If the difference exceeds the material volume of one forklift loading, it is indicated that the material is not standardized after loading. The result of the fusion queue is subtracted by the reasonable loading volume of one forklift to be output as the result of the fusion learning. If the difference does not exceed the material volume of one forklift loading, it is indicated that the material is standardized after loading. The last result in the actual queue is output as the result of the fusion learning.

[0128] Preferably, since the unloading volume of the vehicle is an empirical value, if the vehicle working is detected in the previous frame, the estimated volume of the material corresponding to the picture before the vehicle working is compared with the estimated volume of the material after the vehicle working. When the comparison difference is too large, for example, more than 5 trucks unloading, it is indicated that the forklift does not work to make the material pile standard after the vehicle unloading. Therefore, the estimated volume information of the material after the vehicle working is not adopted, and only one truck carrying material volume is added on the basis of the estimated volume of the material before the vehicle working.

[0129] In this embodiment, when multiple vehicles are unloading or loading, that is, the actual queue last result detects the vehicle working state, and the frame picture corresponding to the result is identified as multiple vehicles working at the same time during vehicle inspection, then stop this fusion learning, the estimation system continues to identify the subsequent video stream, and when identifying that the forklift completes the flattening operation, the estimation results of the last 10 frame pictures after the frame picture are calculated by the median, and the calculated result is output as the final estimation result of this video stream.

[0130] Figure 6 The composition structure schematic diagram of the device for the warehouse material volume estimation method of the embodiment is shown in FIG. 1. Figure 6 The composition structure schematic diagram of the device for the warehouse material volume estimation method of the embodiment is shown in FIG. 1.

[0131] The camera unit 601 is used for shooting the warehouse video stream. The image acquisition unit 602 is used for decomposing the warehouse video stream to obtain a first group of frame pictures. The information processing unit 603 is used for performing depth detection on the first group of pictures, and obtaining the warehouse material volume corresponding to each frame picture in the first group of pictures according to the detection result; wherein the signal processing unit is also used for performing fusion learning on the warehouse material volume corresponding to each frame picture. The signal receiving unit 604 is used for receiving the picture information transmitted by the image acquisition unit to the information processing unit; wherein the signal receiving unit is also used for outputting the material volume after fusion learning. The image inspection unit 605 is used for performing the first inspection on the first group of pictures obtained by the image acquisition unit; wherein the first inspection includes: light inspection on the first group of pictures; quality inspection on the first group of pictures; obstacle inspection on the first group of pictures. The storage unit 606 is used for storing the warehouse video stream shot by the camera unit; wherein the storage unit is also used for storing the output material volume after fusion learning.

[0132] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuitry, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0133] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0134] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0136] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0137] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0138] In another aspect, the present application provides a computer readable storage medium storing a set of computer executable instructions, when executed, for performing the estimation method of any of the above.

[0139] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0140] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0141] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of estimating the volume of warehouse material, characterized by, The method comprises: acquiring a first video stream of warehouse materials, frame decomposing the first video stream to obtain a first group of frame pictures; identifying the warehouse materials in each frame picture in the first group of frame pictures, and performing depth detection on the warehouse materials in each frame picture to determine depth information of the warehouse materials; acquiring height information of the warehouse materials, and calculating a volume of the warehouse materials based on the depth information and the height information of the warehouse materials; performing fusion learning on the volumes of the warehouse materials determined by periodically acquired video streams, and taking the fusion-learned volume as the volume of the warehouse materials; wherein the fusion learning on the volumes of the warehouse materials determined by periodically acquired video streams comprises: sorting detection results of each frame of the first group of frame pictures into a first queue, the detection results including: whether the current frame picture passes the green screen test, the pull bar test, the blur test, and the light test, vehicle information in the current frame picture, and a specific value of the volume of the warehouse materials in the current frame picture; replacing detection results in the first queue that do not pass the green screen test, the pull bar test, the blur test, and the light test with detection results of a previous frame to generate a second queue; when a frame image corresponding to the last detection result in the first queue does not identify a vehicle in the vehicle test, taking the volume of the warehouse materials in the last detection result in the second queue as the volume of the warehouse materials obtained by this fusion learning; when a frame image corresponding to the last detection result in the first queue identifies a vehicle in the vehicle test, and the number of vehicles is 1, then filtering the detection results of the second queue from back to front until a detection result without vehicle information is filtered out, comparing a specific value of the volume of the warehouse materials in the filtered detection result of the second queue with a specific value of the volume of the warehouse materials in the last detection result of the first queue, and performing difference calculation on the specific values of the volumes of the warehouse materials in the filtered detection result of the second queue and the last detection result of the first queue; when a frame image corresponding to the last detection result in the first queue identifies a vehicle in the vehicle test, and the number of vehicles is greater than 1, stopping this fusion learning, and continuing to filter the detection results of frame pictures decomposed from subsequent video streams until a detection result without vehicle information is filtered out; performing median calculation on specific values of the volumes of the warehouse materials in the detection results of the last 10 frame pictures in the filtered detection results of the detection results of the frame pictures decomposed from the subsequent video streams; taking the median calculation result as the volume of the warehouse materials obtained by this fusion learning.

2. The method of claim 1, before identifying the warehouse materials in each frame picture in the first group of frame pictures, the method further comprises: performing a green screen test on each frame picture in the first group of frame pictures; the green screen test comprises: testing the average pixel value of each frame picture in the first group of frame pictures, and filtering out frame pictures with an average pixel value meeting a preset frame picture average value standard.

3. The method of claim 1, before identifying the warehouse material in each frame picture of the first set of frame pictures, the method further comprises: performing a strip test on each frame picture of the first set of frame pictures; the strip test comprises: performing a standard deviation test on the longitudinal pixels of each frame picture of the first set of frame pictures, and screening out the frame pictures with a longitudinal pixel standard deviation that meets a preset frame picture longitudinal pixel standard deviation.

4. The method of claim 1, before identifying the warehouse material in each frame picture of the first set of frame pictures, the method further comprises: performing a blur test on each frame picture of the first set of frame pictures; the blur test comprises: performing a standard deviation test on the overall pixels of each frame picture of the first set of frame pictures, and screening out the frame pictures with an overall pixel standard deviation that meets a preset frame picture overall standard deviation.

5. The method of claim 1, before identifying the warehouse material in each frame picture of the first set of frame pictures, the method further comprises: performing a light test on each frame picture of the first set of frame pictures; The light inspection includes: counting the high-light pixel points in each frame picture of the first set of frame pictures, and screening out the frame pictures with a high-light pixel point proportion that is lower than a preset high-light pixel point proportion in the frame pictures.

6. The method of claim 1, before identifying the warehouse material in each frame picture of the first set of frame pictures, the method further comprises: performing a vehicle test on each frame picture of the first set of frame pictures; The vehicle inspection includes: using a target detection model to test the first set of frame pictures, and identifying the frame pictures containing vehicles and the number of vehicles in the frame pictures.

7. The method of claim 1, wherein, the identifying the warehouse material in each frame picture of the first set of frame pictures, and the depth detection of the warehouse material in each frame picture to determine the depth information of the warehouse material comprises: labeling the positions of the four vertices of the warehouse bottom edge in the frame pictures in each frame picture of the first set of frame pictures; according to the labeled positions of the four vertices of the warehouse bottom edge, cutting the warehouse bottom surface part in the frame pictures to obtain a second set of pictures; performing an affine transformation on the second set of pictures to convert the second set of pictures into rectangles and obtain a third set of pictures; performing convolution processing on the pixels of each picture of the third set of pictures and a normal distribution, and converting all the third set of pictures after convolution processing into gray scale pictures corresponding to the third set of pictures; according to the pixel information of the gray scale pictures corresponding to the third set of pictures, converting the third set of pictures into one-dimensional signals corresponding to the third set of pictures; according to the correspondence between the pixel points in the third set of pictures and the warehouse bottom edge size, taking the determined wave peak position in the one-dimensional signal as the depth information of the warehouse material.

8. The method of claim 1, comparing the specific volume of the warehouse material in the detection result of the second queue screening with the specific volume of the warehouse material in the last detection result of the first queue, the method further comprises: if the volume of the warehouse material in the detection result of the second queue screening is greater than the volume of the warehouse material in the last detection result of the first queue, comparing the result of the difference value calculation with the volume of the material that can be carried by a forklift. If the result of the difference calculation does not exceed the material volume scooped up by one forklift, the warehouse material volume in the last detection result in the first queue is taken as the warehouse material volume obtained by the fusion learning this time; if the result of the difference calculation exceeds the material volume scooped up by one forklift, the specific value of the warehouse material volume in the detection result screened in the second queue is subtracted by the material volume scooped up by one forklift, and the result is taken as the warehouse material volume obtained by the fusion learning this time; If the warehouse material volume in the detection result screened in the second queue is less than the warehouse material volume in the last detection result in the first queue, the result of the difference calculation is compared with the material volume added by one truck; If the result of the difference calculation does not exceed the material volume added by one truck, the warehouse material volume in the last detection result in the first queue is taken as the warehouse material volume obtained by the fusion learning this time; if the result of the difference calculation exceeds the material volume added by one truck, the specific value of the warehouse material volume in the detection result screened in the second queue is added by the material volume added by one truck, and the result is taken as the warehouse material volume obtained by the fusion learning this time.

9. A warehouse material volume estimation device, characterized by, The device comprises: a camera unit for shooting a warehouse video stream; an image acquisition unit for decomposing the warehouse video stream to obtain a first group of frame pictures; an information processing unit for performing depth detection on the first group of pictures and obtaining the warehouse material volume corresponding to each frame picture in the first group of pictures according to the detection result; the information processing unit is further configured to perform fusion learning on the warehouse material volume corresponding to each frame picture; a signal receiving unit for receiving the picture information transmitted by the image acquisition unit to the information processing unit; the signal receiving unit is further configured to output the material volume after the fusion learning. The information processing unit is further configured to sort the detection results of each frame of the first group of frame pictures into a first queue, the detection results including whether the current frame picture passes the green screen test, the strip test, the blur test, the light test, vehicle information in the current frame picture, and a specific value of the warehouse material volume in the current frame picture; replace the detection results that do not pass the green screen test, the strip test, the blur test, and the light test in the first queue with the detection results of the previous frame to generate a second queue; when the last detection result in the first queue corresponds to a frame image that does not identify a vehicle in the vehicle test, take the warehouse material volume in the last detection result in the second queue as the warehouse material volume obtained through this time of fusion learning; when the last result in the first queue corresponds to a frame image that identifies a vehicle in the vehicle test and the number of vehicles is 1, filter the detection results of the second queue from back to front until a detection result that does not have vehicle information is filtered out, compare the specific value of the warehouse material volume in the filtered detection result of the second queue with the specific value of the warehouse material volume in the last detection result of the first queue, and calculate the difference between the specific value of the warehouse material volume in the filtered detection result of the second queue and the specific value of the warehouse material volume in the last detection result of the first queue; when the last result in the first queue corresponds to a frame image that identifies a vehicle in the vehicle test and the number of vehicles is greater than 1, stop this time of fusion learning and continue to filter the detection results of the frame pictures decomposed from the subsequent video stream until a detection result that does not have vehicle information is filtered out; perform median calculation on the specific values of the warehouse material volumes in the detection results of the last 10 frame pictures of the filtered detection results of the detection results of the frame pictures decomposed from the subsequent video stream; and take the result of the median calculation as the warehouse material volume obtained through this time of fusion learning.

10. The apparatus of claim 9, further comprising: an image inspection unit; configured to perform a first inspection on the first group of pictures obtained by the image acquisition unit; wherein the first inspection comprises: performing a light test on the first group of pictures; performing a quality test on the first group of pictures; performing an obstacle test on the first group of pictures; a storage unit configured to store the warehouse video stream captured by the image acquisition unit; wherein the storage unit is further configured to store the output material volume obtained through fusion learning.

11. An electronic device, comprising: comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

12. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are configured to enable the computer to perform the method of any one of claims 1-8.

Images