Method, equipment and computer program product for acquiring volume information of logistics parcel

Abstract

The embodiment of the invention discloses a method and device for acquiring volume information of a logistics package, a computer readable storage medium, electronic equipment and a computer program product. The method is used for estimating the volume of a logistics package in a conveying motion state through a warehousing system conveying line, the logistics package is provided with a high-transparency packaging material, and the method comprises the following steps: obtaining an image pair collected in a logistics package conveying process, the image pair comprises a depth image and a visible light image which are acquired in the height direction of the logistics package at the same time; performing transparent texture feature detection on the visible light image, and determining a part with transparent texture features as a target area of the packaging material in the visible light image; obtaining a target image obtained by performing image matching on the depth image and the visible light image, determining a to-be-processed area corresponding to the target area from the target image, and obtaining associated available depth information; and depth compensation processing is carried out on the to-be-processed area based on the available depth information, and logistics package volume information is calculated.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a method and device for obtaining volume information of logistics packages, a method and device for deep information processing, a computer-readable storage medium, an electronic device, and a computer program product. Background Art

[0002] The intelligent warehousing system can automatically and intelligently complete the storage, sorting, distribution and management of items in the warehouse, which can improve warehouse productivity and efficiency.

[0003] Usually, the sorted items are placed in a container, such as a rectangular plastic frame containing items, and sent to a designated location via a conveyor line. In this process, there may be a need to detect the volume of the conveyed items in combination with the usage in a specific application scenario. Specifically, the depth point cloud data of the items on the conveyor line can be collected by a depth camera, and the volume of the items can be calculated by point cloud integration.

[0004] Under such a detection scheme, if the surface texture of the inspected object is relatively special, for example, the surface of the object is a special material with high transmittance and high reflectivity such as transparent packaging, then the depth point cloud data collected by the depth camera will be obviously missing, resulting in inaccurate volume calculation.

[0005] How to solve the problem of missing depth point cloud data and ensure the accuracy of object volume calculation has become a technical problem that needs to be solved by technical personnel in this field. Summary of the invention

[0006] The present application provides a method and device for obtaining volume information of logistics packages, a depth information processing method and device, a computer-readable storage medium, an electronic device, and a computer program product, which help to solve the problem of missing depth information and thereby improve the accuracy of volume calculation.

[0007] This application provides the following solutions:

[0008] A method for obtaining volume information of a logistics package, for estimating the volume of a logistics package in a state of being transported through a conveyor line of a warehousing system, wherein the logistics package has a packaging material with high transparency, and the method comprises:

[0009] Obtaining an image pair acquired during the delivery of the logistics package, the image pair comprising a depth image and a visible light image acquired at the same time in the height direction of the logistics package;

[0010] Performing transparent texture feature detection on the visible light image, and determining a portion having the transparent texture feature as a target area of ​​the packaging material corresponding to the visible light image;

[0011] Obtaining a target image obtained by image matching the depth image and the visible light image, and determining, from the target image, a region to be processed corresponding to the target region and obtaining available depth information associated with the region to be processed;

[0012] The missing depth information of the area to be processed is compensated based on the available depth information, and the volume information of the logistics package is calculated according to the compensated target image.

[0013] The available depth information is at least one of the following depth information:

[0014] Depth information collected at the edge of the outer contour of the logistics package;

[0015] Depth information collected from the label area of ​​the logistics package;

[0016] Depth information collected from the graphic printing area of ​​the packaging material;

[0017] Depth information collected from the wrinkles of the packaging material.

[0018] The storage system is equipped with depth cameras and industrial cameras that can cover the designated area of ​​the conveyor line, as well as inductive sensors.

[0019] The obtaining of the image pair collected during the logistics package transmission process includes:

[0020] The induction sensor generates a trigger signal when detecting the passage of the logistics package, so that the depth camera captures the depth image under the action of the trigger signal, and the industrial camera captures the visible light image under the action of the trigger signal, thereby obtaining the image pair captured at the same time.

[0021] The installation height of the depth camera is determined according to the following requirements:

[0022] The depth image captured by the depth camera includes a panoramic view of the logistics package, and does not include other packages being transported adjacently on the conveyor line.

[0023] Wherein, the storage system is provided with a plurality of depth cameras covering the designated area of ​​the conveyor line from different angles.

[0024] The depth image is obtained as follows:

[0025] Obtaining initial depth images of the logistics package taken by different depth cameras at different viewing angles at the same time;

[0026] Point cloud fusion processing is performed on the multiple initial depth images, and the fused image is determined as the depth image of the logistics package.

[0027] The transparent texture feature detection for the visible light image includes:

[0028] Obtaining a transparent object detection model for transparent texture feature detection;

[0029] The visible light image is input into the transparent object detection model to obtain a detection result output by the model, wherein the detection result is used to identify a target area existing in the visible light image.

[0030] Wherein, the visible light image is collected by an industrial camera that can cover the specified area of ​​the conveyor line.

[0031] The step of obtaining a transparent object detection model for transparent texture feature detection includes:

[0032] Obtaining a plurality of sample images captured by the industrial camera in a designated area of ​​the conveyor line, and marking transparent areas included in the plurality of sample images respectively;

[0033] An initial model capable of detecting transparent texture features is obtained, and the initial model is fine-tuned through the sample image and the corresponding annotated transparent area to obtain the transparent object detection model.

[0034] Wherein, the logistics package is placed in a container and is transported on the conveyor line through the container.

[0035] The obtaining of a target image obtained by performing image matching between the depth image and the visible light image comprises:

[0036] Obtaining a plurality of available matching point information determined according to the shape characteristics of the container;

[0037] Locating matching positions that match the plurality of available matching point information in the depth image and the visible light image respectively;

[0038] According to the matching position, an affine transformation is performed on the depth image or the visible light image to achieve matching and fusion between the depth image and the visible light image to obtain the target image.

[0039] Wherein, the logistics package is placed in a container and is transported on the conveyor line through the container, and the method further includes:

[0040] The full load rate of the container is calculated according to the volume information of the logistics parcel contained in the container.

[0041] Wherein, the method further comprises:

[0042] When the full load rate is lower than a preset value, the placement quantity information and / or placement method information in the container associated with the package type to which the logistics package belongs is adjusted.

[0043] A depth information processing method, the method comprising:

[0044] Obtaining an image pair associated with a logistics package, the image pair comprising a depth image and a visible light image acquired at the same time in a height direction of the logistics package;

[0045] Performing transparent texture feature detection on the visible light image, and when the visible light image includes a target area having transparent texture features, obtaining a target image obtained by performing image matching between the depth image and the visible light image;

[0046] Determining a to-be-processed area corresponding to the target area from the target image, and obtaining available depth information associated with the to-be-processed area;

[0047] Compensation processing is performed on the missing depth information of the area to be processed based on the available depth information.

[0048] A device for obtaining volume information of a logistics package, used for estimating the volume of a logistics package in a state of being transported through a conveyor line of a warehousing system, wherein the logistics package has a packaging material with high transparency, and the device comprises:

[0049] An image pair obtaining unit, configured to obtain an image pair acquired during the transport of the logistics package, wherein the image pair includes a depth image and a visible light image acquired at the same time in the height direction of the logistics package;

[0050] a target area determination unit, configured to perform transparent texture feature detection on the visible light image, and determine a portion having transparent texture features as a target area of ​​the packaging material corresponding to the visible light image;

[0051] A target image obtaining unit, used to obtain a target image obtained by performing image matching between the depth image and the visible light image;

[0052] A depth information determining unit, configured to determine, from the target image, a region to be processed corresponding to the target region and obtain available depth information associated with the region to be processed;

[0053] A volume information calculation unit is used to compensate for the missing depth information of the area to be processed based on the available depth information, and calculate the volume information of the logistics package according to the compensated target image.

[0054] A depth information processing device, the device comprising:

[0055] An image pair obtaining unit, used to obtain an image pair associated with a logistics package, wherein the image pair includes a depth image and a visible light image collected at the same time in a height direction of the logistics package;

[0056] A feature detection unit, used for performing transparent texture feature detection on the visible light image;

[0057] a target image obtaining unit, configured to obtain a target image obtained by image matching between the depth image and the visible light image when the visible light image includes a target area having a transparent texture feature;

[0058] A depth information determining unit, configured to determine, from the target image, a region to be processed corresponding to the target region, and obtain available depth information associated with the region to be processed;

[0059] A compensation processing unit is used to perform compensation processing on the missing depth information of the to-be-processed area based on the available depth information.

[0060] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described above.

[0061] An electronic device, comprising:

[0062] one or more processors; and

[0063] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the aforementioned methods.

[0064] A computer program product comprises a computer program / computer executable instructions, wherein the computer program / computer executable instructions implement the steps of any of the aforementioned methods when executed by a processor in an electronic device.

[0065] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0066] In an embodiment of the present application, depth images and visible light images of a logistics package in motion being transported through a conveyor line can be collected, and the two types of images can be fused and registered in the time and space dimensions to compensate and repair the missing depth information on the outer surface of the package due to the high transparency of the packaging material of the logistics package, thereby improving the accuracy of the logistics package volume estimation.

[0067] Among them, the fusion of the time dimension is mainly used to determine the image pair of the depth image and the visible light image collected at the same time, and the fusion of the space dimension is mainly used to realize the spatial alignment and integration of the corresponding points on the two images included in the image pair, so that the points in the target image obtained include both the information of the depth image and the information of the visible light image. In this way, when the target area with transparent texture features is identified from the visible light image included in the image pair, the area to be processed corresponding to the position of the target area can be matched from the target image, and then the available depth information associated with the area to be processed is used to compensate for the missing depth information of the area to be processed, thereby solving the problem of missing depth information.

[0068] It can be understood that the target image after compensation processing can include complete depth information of the logistics package. Calculating the volume of the logistics package based on this can also help improve the accuracy of volume calculation.

[0069] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 is a flow chart of a method for obtaining logistics package volume information provided by an embodiment of the present application;

[0072] Figure 2 is a schematic diagram of a device for obtaining logistics package volume information provided in an embodiment of the present application;

[0073] Figure 3 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0075] At present, common volume detection is mostly for regularly packaged items, and these items have uniform packaging box materials and shapes, such as common cardboard packaging boxes. When the depth information of the item is collected by a depth camera, relatively complete point cloud data can usually be obtained. Therefore, combined with other information such as the width of the item determined by the passing time of the item on the conveyor line, the volume of the item can be calculated. Among them, the depth information is also the height information, which can represent the distance between the surface of the item and the depth camera, usually in the point cloud xyz format. The specific implementation process of calculating the volume of an item based on depth point cloud data can be referred to the relevant technology.

[0076] In actual applications, there are a large number of irregularly packaged items, especially in the smart warehousing system that provides merchandise sorting and transportation services for offline physical stores in new retail scenarios. The packaging shapes of logistics packages are diverse and the materials used are also different, resulting in complex surface textures of the packages, which increases the difficulty of volume detection. Especially for packaging materials such as transparent packaging, which are made of special materials with high penetration and high reflectivity, there will be obvious point cloud data missing when collecting depth information of logistics packages, which affects volume calculation.

[0077] Correspondingly, the embodiment of the present application provides a tool that can be used for depth information compensation processing. For logistics packages with transparent texture features on the surface, the depth image and visible light image of the logistics package can be fused and registered in two dimensions of time and space, based on which the missing depth information can be compensated and repaired.

[0078] In actual applications, the tool can exist in the form of a client, and be deployed locally in the warehouse that needs to perform depth information compensation processing, and compensate for the missing depth information of the logistics packages that need to be sorted and transported in the warehouse. Alternatively, the tool can exist in the form of a server, deployed on a cloud server, and provide services to warehouses that need to perform depth information compensation processing.

[0079] Specifically, the depth information compensation processing process can be reflected as follows: obtaining an image pair associated with a logistics package, the image pair including a depth image and a visible light image collected at the same time in the height direction of the logistics package; performing transparent texture feature detection on the visible light image, and when the visible light image includes a target area with transparent texture features, obtaining a target image obtained by image matching the depth image and the visible light image; determining a to-be-processed area corresponding to the target area from the target image, and obtaining available depth information associated with the to-be-processed area; and compensating for the missing depth information of the to-be-processed area based on the available depth information.

[0080] It can be understood that the target image after compensation processing can include complete depth information of the logistics package. Calculating the volume of the logistics package based on the complete depth information helps to improve the accuracy of volume calculation.

[0081] The following is an explanation of the implementation process of the method for obtaining the volume information of logistics packages provided in the embodiment of the present application with reference to specific examples. Based on this, the volume of logistics packages in the state of being transported through the conveyor line of the warehousing system can be estimated. Figure 1 The flowchart shown may include:

[0082] S101: Obtain an image pair captured during the transmission of the logistics package, wherein the image pair includes a depth image and a visible light image captured at the same time in the height direction of the logistics package.

[0083] As an example, the embodiment of the present application can be applied to an intelligent warehousing system to estimate the volume of logistics packages dynamically transmitted on a conveyor line. In particular, when the logistics packages have highly transparent packaging materials, such as vegetables packaged in soft plastic bags, fresh-cut fruits packaged in hard plastic boxes, fragile items packaged in plastic inflatable bags, etc., the accuracy of volume estimation can be improved after depth information compensation processing based on the embodiment of the present application.

[0084] In the embodiment of the present application, the depth image can be collected by a depth camera that can cover the specified area of ​​the conveyor line. Among them, the conveyor line can also be called an assembly line, which is mainly used to realize the delivery task of logistics packages. According to the use requirements, the camera required for image acquisition can be arranged in the specified area of ​​the conveyor line. The embodiment of the present application does not specifically limit the specific arrangement position of the camera.

[0085] At present, the depth cameras commonly used for depth detection in indoor scenes include: visual binocular depth camera, structured light depth camera, ToF (Time-of-Flight) depth camera, single-line laser depth camera, and multi-line laser depth camera. The inventors analyzed and found that: visual binocular depth camera and structured light depth camera cannot achieve good accuracy on moving objects, and are not suitable for depth detection of items transported on conveyor lines; single-line laser depth camera has blind spots and occlusion problems, and has difficulty in performing well on rugged surfaces, and is not suitable for depth detection of special-shaped products with variable outer packaging shapes; multi-line laser depth camera is highly complex and costly.

[0086] Taking the sorting and transportation of goods in the new retail scenario as an example, there are many logistics packages with transparent outer packaging. As mentioned above, these outer packaging may be hard plastic or soft plastic attached to the outer surface of the goods in a vacuum form. In order to ensure the accuracy of the depth image taken and to collect the depth information of the wrinkled surface as comprehensively as possible, the embodiment of the present application may preferably use a ToF depth camera to capture the depth image of the logistics package.

[0087] The embodiment of the present application does not specifically limit the installation position of the depth camera. The camera field of view can cover the conveyor line in a specified area, and in the vertical height direction, it can collect the depth information of the logistics packages transmitted on the conveyor line.

[0088] Preferably, the depth image taken for logistics packages can meet the following requirements:

[0089] 1. The depth image includes a panoramic view of the logistics package. After depth information compensation, the complete depth information of the logistics package can be obtained.

[0090] 2. The depth image does not include other packages that are transported nearby on the conveyor line. This can prevent other packages from interfering with the depth information of the logistics package, and avoid the consumption of computing resources caused by logistics package identification, image cropping and other processing.

[0091] It can be understood that if a logistics package is placed in a container and is transported on a conveyor line through the container, for example, the container is a rectangular plastic frame, then the depth image may be required to include a panoramic view of the container and not include other containers that are transported adjacently on the conveyor line, that is, to avoid capturing multiple containers in one image. In other words, the container where the logistics package is located can be used as the basic acquisition object, and the acquired depth image includes both the depth information of the logistics package and the depth information of the container.

[0092] In practical applications, the installation height of the depth camera can be determined according to the above requirements. For example, for a 20cm high plastic frame, when the conveying speed of the conveyor line is 2m / s, if the exposure frame rate and depth range of the depth camera are adjusted to determine that the accuracy for packages within 62.5cm is good, the depth camera can be installed at a height of 60cm from the conveyor line to ensure that the camera is facing vertically downward to capture the depth information of the logistics package.

[0093] In an embodiment of the present application, visible light images can be collected by an industrial camera that can cover a specified area of ​​the conveyor line to obtain clear visible light images of the logistics package.

[0094] Alternatively, considering that infrared cameras have good imaging capabilities for transparent plastic surfaces, infrared cameras can also be selected to capture visible light images of logistics packages according to usage requirements. Specifically, the camera for capturing visible light images can be determined in combination with the usage requirements of the depth detection scene.

[0095] Preferably, the camera used to capture visible light images can meet the following requirements:

[0096] 1. It can be triggered by high frequency. For example, when the plastic frames are transported at high speed and densely, it is necessary to collect visible light images at intervals of several milliseconds or tens of milliseconds.

[0097] 2. Sufficient exposure time. If the light environment in the warehouse is dark, insufficient exposure time may result in darker images and poor clarity, which will affect texture feature recognition and target image fusion, and ultimately affect the processing effect of depth compensation.

[0098] The embodiment of the present application does not specifically limit the installation position of the industrial camera. The camera field of view can cover the conveyor line in a specified area, and in the vertical height direction, it only needs to collect visible light images of the logistics packages transmitted on the conveyor line.

[0099] Corresponding to the example of the depth camera installation height given above, the industrial camera can be installed at a height of 120 cm from the conveyor line, ensuring that the camera is facing vertically downward to capture visible light images of the logistics packages.

[0100] In the embodiment of the present application, the visible light image can be fused on the basis of the depth image, and the depth information can be compensated by spatiotemporal fusion, wherein the fusion of the time dimension can be reflected in the fusion processing of the depth image and the visible light image taken at the same time.

[0101] Specifically, the imaging time of the image taken for the logistics package can be obtained, and image fusion in the time dimension is performed based on it to determine at least one image pair that is synchronized in time.

[0102] In one example, the depth camera and the industrial camera can continuously capture images in real time and record the imaging time corresponding to each of the captured images.

[0103] In another example, the depth camera and the industrial camera can acquire images under the action of a trigger signal and record the timestamp of the trigger signal as the imaging time. The depth camera and the industrial camera can be triggered regularly by configuring a preset trigger time interval.

[0104] Alternatively, sensors can be deployed in the warehouse system, and when the sensors detect the logistics packages being transported on the conveyor line, a trigger signal is generated and sent to the depth camera and the industrial camera. In this way, the depth camera takes a depth image of the logistics package under the action of the trigger signal, and the industrial camera takes a visible light image of the logistics package under the action of the trigger signal. The two can be determined as an image pair collected at the same time, using the timestamp of the sensor trigger as the imaging time.

[0105] As an example, the sensor that detects the logistics package and generates the trigger signal can be a photoelectric sensor. When the logistics package transmitted on the conveyor line passes through the photoelectric sensor, a trigger signal can be generated and sent to the depth camera and the industrial camera.

[0106] Optionally, in order to obtain more detailed depth information, the embodiment of the present application can deploy multiple depth cameras that can cover the conveyor line area from different angles in the warehousing system. In this way, after obtaining the initial depth images of the logistics package taken by different depth cameras at different viewing angles at the same time, the multiple initial depth images can be subjected to point cloud fusion processing, and the fused image is determined as the depth image of the logistics package, and is spatially and temporally fused with the visible light image of the logistics package.

[0107] Taking the triggering of photoelectric sensors as an example, when it is determined that a logistics package has passed, the trigger signal generated by the sensor can be sent to multiple depth cameras at the same time, triggering each depth camera to take the initial depth image of the logistics package from different perspectives. These initial depth images have the same imaging time. Multiple initial depth images can be first point cloud fused, and the point cloud data collected from different perspectives can be aligned to obtain more and more accurate depth information of each point on the surface of the logistics package. In this way, by fusing the initial depth images under different perspectives, some missing depth information can be supplemented and improved to a certain extent, which helps to reduce the processing volume of subsequent depth compensation; in addition, the subsequent spatiotemporal fusion processing based on the depth image including more accurate depth information can also improve the accuracy of the depth information in the fused target image, thereby improving the processing effect of subsequent depth compensation.

[0108] S102: Perform transparent texture feature detection on the visible light image, and determine a portion having transparent texture features as a target area of ​​the packaging material corresponding to the visible light image.

[0109] In the embodiment of the present application, the detection and identification of the target area can be realized by a transparent object detection model. Specifically, after obtaining a transparent object detection model for transparent texture feature detection, the visible light image can be input into the model to obtain the detection result output by the model, and the detection result is used to identify the target area existing in the visible light image.

[0110] In one example, the detection result output by the model can be reflected as a binary image output for the visible light image, where the pixel points that meet the transparent texture feature are outputted as "1", and the pixel points that do not meet the transparent texture feature are outputted as "0". In this way, the area marked as "1" can be identified as the target area of ​​the packaging material in the visible light image. Of course, if the model output result is all "0", it can be determined that there is no target area with transparent texture features in the visible light image.

[0111] In the embodiment of the present application, model training can be performed based on traditional deep learning methods. For example, the network topology can be determined based on convolutional neural networks (CNN), an initial model can be constructed, and then model training can be performed based on a large number of training samples and a preset loss function to obtain a transparent object detection model that meets the use requirements.

[0112] Alternatively, an existing general model that can perform transparent texture feature detection, such as the EBLNet model, a pre-trained large model, etc., can be determined as the initial model, and then based on fine-tuning technology, a transparent object detection model that meets the usage requirements can be obtained through a small amount of training samples.

[0113] Specifically, multiple sample images captured by an industrial camera in a specified area of ​​a conveyor line can be obtained, and the transparent areas included in the multiple sample images can be respectively annotated; an initial model for transparent texture feature detection is obtained, and the initial model is fine-tuned through the sample images and the corresponding annotated transparent areas to obtain a transparent object detection model of an embodiment of the present application.

[0114] Preferably, the sample images used for model fine-tuning can be collected by industrial cameras deployed in a warehousing system that requires depth compensation processing. Model fine-tuning based on sample images that are more in line with the actual application environment can help improve the recognition accuracy of the transparent object detection model.

[0115] When performing model fine-tuning training, the sample image can be used as the model input, and the predicted transparent area identified by the model can be used as the output, which is then compared with the transparent area marked in the sample image. When the degree of matching between the predicted transparent area and the corresponding marked transparent area meets the requirements of the preset loss function, the model fine-tuning process can be ended to obtain the transparent object detection model of the embodiment of the present application.

[0116] S103: Obtain a target image obtained by performing image matching between the depth image and the visible light image.

[0117] In the embodiment of the present application, after obtaining an image pair associated with a logistics package through fusion in the time dimension, the two images in the image pair can be used to match the points on the surface of the logistics package in the spatial dimension to obtain a target image. In one example, the target image can be embodied as an RGB-D image that integrates the color information and depth information of the logistics package.

[0118] Specifically, when matching in the spatial dimension, the depth image and the visible light image can be analyzed separately to locate the position that is easy to match, and then the image is affine transformed based on this (these) position to achieve the fusion of the two images. Among them, the position that is easy to match can be determined based on the characteristics of the logistics package itself, or based on the characteristics of the container in which the logistics package is located.

[0119] Taking into account that the container has a fixed shape and the material used can usually ensure that the complete depth information of the container can be collected, therefore, for the use scenario of placing the logistics package in the container for transmission, the embodiment of the present application can provide the following image matching solution for obtaining the target image, which helps to simplify the matching process: obtain multiple available matching point information determined according to the shape characteristics of the container; locate the matching positions that are consistent with the multiple available matching point information in the depth image and the visible light image respectively; according to the matching positions, perform an affine transformation on the depth image or the visible light image to achieve matching and fusion between the depth image and the visible light image, and obtain the target image.

[0120] Taking a rectangular plastic frame as an example, the easy-to-match positions determined by its shape characteristics can be reflected as the four corners of the plastic frame, so the features of the four corners can be extracted as available matching point information. In this way, after obtaining the image pair of the logistics package, the four corners of the plastic frame can be located from the depth image and the four corners of the plastic frame can be located from the visible light image based on the available matching point information. After the position correspondence between the depth image and the visible light image is achieved through the four corners as the matching positions, one of the images can be fixed and the other image can be spatially transformed to achieve registration and superposition between the two. For example, the depth image can be fixed, and the visible light image can be affine transformed, and the visible light image can be superimposed on the depth image by translation, scaling, etc., to achieve spatial consistency of the corresponding points on the two images. In this way, the depth information of each point of the depth image corresponding to the visible light image can be clearly understood.

[0121] In practical applications, the lack of depth information may be caused by transparent packaging, so the embodiment of the present application can be as follows Figure 1As shown, S102 is first executed to identify whether the visible light image includes a target area with transparent texture features. On the one hand, it can be clarified whether depth compensation is required, and on the other hand, the specific location where depth compensation is required can be clarified, so that when the target area is identified, S103 is triggered to perform a spatiotemporal fusion processing process for the depth image and the visible light image to obtain the target image; alternatively, spatiotemporal fusion can be performed first to obtain the target image, and then the target area can be identified for the visible light image; alternatively, the above two steps can be performed simultaneously, and the embodiment of the present application does not specifically limit the execution order of the two.

[0122] It can be understood that when the embodiment of the present application is executed in the order shown in the figure, if the target area is not identified in the visible light image, the subsequent processing process can be stopped, which helps to save the computing resources consumed by spatiotemporal fusion. In addition, the depth image can be analyzed first to determine whether there is a lack of depth information, and then trigger the execution of the target area identification and target image fusion steps. Through the target area identification, it can be further clarified whether the reason for the lack of depth information is the transparent packaging material of the outer packaging of the logistics package, or the irregular shape of the package, etc., which causes this part itself to have no depth information.

[0123] S104: Determine, from the target image, a region to be processed corresponding to the target region and obtain available depth information associated with the region to be processed.

[0124] After image fusion is completed, the target area included in the visible light image can be clearly identified, and the corresponding position in the target image can be used as the area to be processed in the embodiment of the present application. Because the target area has transparent texture features, the depth information corresponding to this part of the area may be missing, so targeted depth compensation processing can be performed on this.

[0125] As described above, the points in the target image include both the information of the depth image and the information of the visible light image. Therefore, after the area to be processed is determined, the available depth information associated with the area to be processed can be determined. The available depth information can be specifically embodied as at least one of the following depth information:

[0126] 1. Depth information collected at the outer edge of the logistics package;

[0127] 2. Depth information collected from the label area of ​​the logistics package;

[0128] 3. Depth information collected from the graphic printing area of ​​the packaging material;

[0129] 4. Depth information collected from the folds of packaging materials.

[0130] That is to say, in one case, the area corresponding to the entire logistics package on the target image can be determined as the area to be processed, and compensation processing can be performed based on the depth information collected at the edge of the package's outer contour. Generally, the depth camera has a high accuracy in collecting the depth information of the outer contour edge, and determining it as usable depth information helps improve the processing effect of depth compensation.

[0131] In another case, the depth information collected from a special area on the surface of the outer packaging of the logistics package can be determined as the available depth information. For example, the outer packaging of the package is usually affixed with a label related to its contents, and compensation processing can be performed based on the depth information collected from the label area; if there is a graphic printing area on the packaging material, compensation processing can also be performed based on the depth information collected from the graphic printing area; and so on.

[0132] In another case, due to the particularity of the plastic packaging material, partial wrinkles may form on the outer surface of the logistics package. Therefore, depth compensation processing can also be performed based on the depth information collected at the wrinkles of the packaging material.

[0133] In practical applications, the depth information collected in the above multiple areas may be determined as available depth information, and depth compensation processing may be performed by integrating multiple types of available depth information.

[0134] S105: Compensating for the missing depth information of the area to be processed based on the available depth information, and calculating the volume information of the logistics package according to the compensated target image.

[0135] In the embodiment of the present application, interpolation can be performed in the processing area according to the available depth information to achieve compensation for the missing depth information. According to the use requirements, the missing depth information can be supplemented by linear interpolation or other interpolation methods. The embodiment of the present application does not specifically limit the supplementation process.

[0136] In actual applications, affected by factors such as camera coverage and conveyor line transmission speed, multiple images of logistics packages may be captured and multiple image pairs may be obtained. The embodiment of the present application can perform fusion processing and interpolation processing on each image pair separately, and combine the compensation processing results of each image pair to obtain the final compensated target image.

[0137] In summary, for logistics packages transmitted on the conveyor line, depth images and visible light images can be collected, and the two types of images can be fused and registered in the time and space dimensions to compensate and repair the missing depth information on this basis.

[0138] Among them, the fusion of the time dimension is mainly used to determine the image pair of the depth image and the visible light image collected at the same time, and the fusion of the space dimension is mainly used to realize the spatial alignment and integration of the corresponding points on the two images included in the image pair, so that the points in the target image obtained include both the information of the depth image and the information of the visible light image. In this way, when the target area with transparent texture features is identified from the visible light image included in the image pair, the area to be processed corresponding to the position of the target area can be matched from the target image, and then the available depth information associated with the area to be processed is used to compensate for the missing depth information of the area to be processed. After solving the problem of missing depth information, the complete depth information of the logistics package can be used to estimate the volume and obtain more accurate volume information of the logistics package.

[0139] In one example, a more accurate volume estimation for logistics packages can help improve the accuracy of the full load rate estimation of the container containing the logistics packages. In particular, when hard plastic boxes, plastic air bags, etc. are used as the outer packaging of goods, the volume of the outer packaging is much larger than the fresh-cut fruits, fragile items, etc. as the contents. That is, the overall volume of the logistics package mainly depends on the volume of the outer packaging. If the volume is estimated based on the depth information of the contents, it will inevitably affect the accuracy of the full load rate calculation.

[0140] That is to say, in the use scenario of transporting logistics packages through containers, considering the different sizes of packages and the different operating methods of operators (such as the way the packages are placed in the container), the containers may not be fully utilized, resulting in a low container full load rate, which will increase the number of containers required in the warehouse and increase the operating costs in the warehouse. Correspondingly, the embodiment of the present application can also calculate the full load rate of the container where the logistics package is located, that is, the proportion of the package to the container volume, based on the estimated volume information of the logistics package.

[0141] Optionally, it is also possible to identify low-full-rate containers, i.e. containers with a full-load rate lower than a preset value, and adjust the placement quantity information and / or placement method information in the container associated with the package type to which the logistics package belongs, so as to achieve efficient use of container resources and save material costs in the warehouse. Among them, the placement quantity information refers to the maximum number of packages of a certain type that can be contained in the container. As an example, the package type can be divided according to the category of the package content and / or the specification and size of the package.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0143] Corresponding to the aforementioned method embodiment, the embodiment of the present application further provides a device for obtaining volume information of a logistics package, which is used to estimate the volume of a logistics package in a state of being transported through a conveyor line of a warehousing system, wherein the logistics package has a packaging material with high transparency. Figure 2 , the device may include:

[0144] An image pair obtaining unit 201 is used to obtain an image pair collected during the transmission of the logistics package, wherein the image pair includes a depth image and a visible light image collected at the same time in the height direction of the logistics package;

[0145] A target region determining unit 202 is configured to perform transparent texture feature detection on the visible light image, and determine a portion having transparent texture features as a target region of the packaging material corresponding to the visible light image;

[0146] A target image obtaining unit 203 is used to obtain a target image obtained by performing image matching between the depth image and the visible light image;

[0147] A depth information determining unit 204 is used to determine, from the target image, a region to be processed corresponding to the target region and obtain available depth information associated with the region to be processed;

[0148] The volume information calculation unit 205 is used to compensate for the missing depth information of the area to be processed based on the available depth information, and calculate the volume information of the logistics package according to the compensated target image.

[0149] The available depth information is at least one of the following depth information:

[0150] Depth information collected at the edge of the outer contour of the logistics package;

[0151] Depth information collected from the label area of ​​the logistics package;

[0152] Depth information collected from the graphic printing area of ​​the packaging material;

[0153] Depth information collected from the wrinkles of the packaging material.

[0154] The storage system is equipped with depth cameras and industrial cameras that can cover the designated area of ​​the conveyor line, as well as inductive sensors.

[0155] The image pair acquisition unit can be specifically used to: generate a trigger signal when the induction sensor detects the passage of the logistics package, so that the depth camera shoots the depth image under the action of the trigger signal, and the industrial camera shoots the visible light image under the action of the trigger signal, to obtain the image pair collected at the same time.

[0156] The installation height of the depth camera is determined according to the following requirements:

[0157] The depth image captured by the depth camera includes a panoramic view of the logistics package, and does not include other packages being transported adjacently on the conveyor line.

[0158] Wherein, the storage system is provided with a plurality of depth cameras covering the designated area of ​​the conveyor line from different angles.

[0159] The image pair acquisition unit can also be used to: obtain initial depth images of the logistics package taken by different depth cameras at different perspectives at the same time; perform point cloud fusion processing on multiple initial depth images, and determine the fused image as the depth image of the logistics package.

[0160] The target area determination unit may be specifically configured to:

[0161] Obtaining a transparent object detection model for transparent texture feature detection;

[0162] The visible light image is input into the transparent object detection model to obtain a detection result output by the model, wherein the detection result is used to identify a target area existing in the visible light image.

[0163] Wherein, the visible light image is collected by an industrial camera that can cover the specified area of ​​the conveyor line, and the target area determination unit can also be used to:

[0164] Obtaining a plurality of sample images captured by the industrial camera in a designated area of ​​the conveyor line, and marking transparent areas included in the plurality of sample images respectively;

[0165] An initial model capable of detecting transparent texture features is obtained, and the initial model is fine-tuned through the sample image and the corresponding annotated transparent area to obtain the transparent object detection model.

[0166] Wherein, the logistics package is placed in a container and is transported on a conveyor line through the container, and the target image acquisition unit can be specifically used for:

[0167] Obtaining a plurality of available matching point information determined according to the shape characteristics of the container;

[0168] Locating matching positions that match the plurality of available matching point information in the depth image and the visible light image respectively;

[0169] According to the matching position, an affine transformation is performed on the depth image or the visible light image to achieve matching and fusion between the depth image and the visible light image to obtain the target image.

[0170] Wherein, the logistics package is placed in a container and transported on a conveyor line through the container, and the device also includes:

[0171] The full load rate calculation unit is used to calculate the full load rate of the container according to the volume information of the logistics package contained in the container.

[0172] Wherein, the device further comprises:

[0173] The information adjustment unit is used to adjust the placement quantity information and / or placement method information in the container associated with the package type to which the logistics package belongs when the full load rate is lower than a preset value.

[0174] Corresponding to the foregoing method embodiment, the embodiment of the present application further provides a depth information processing device, which may include:

[0175] An image pair obtaining unit, used to obtain an image pair associated with a logistics package, wherein the image pair includes a depth image and a visible light image collected at the same time in a height direction of the logistics package;

[0176] A feature detection unit, used for performing transparent texture feature detection on the visible light image;

[0177] a target image obtaining unit, configured to obtain a target image obtained by image matching between the depth image and the visible light image when the visible light image includes a target area having a transparent texture feature;

[0178] A depth information determining unit, configured to determine, from the target image, a region to be processed corresponding to the target region, and obtain available depth information associated with the region to be processed;

[0179] A compensation processing unit is used to perform compensation processing on the missing depth information of the to-be-processed area based on the available depth information.

[0180] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0181] And an electronic device, comprising:

[0182] one or more processors; and

[0183] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0184] In addition, an embodiment of the present application also provides a computer program product, including a computer program / computer executable instructions, which, when executed by a processor in an electronic device, implements the steps of any of the methods described in the aforementioned method embodiments.

[0185] in, Figure 3 The architecture of an electronic device is exemplarily shown. For example, device 300 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, an aircraft, etc.

[0186] Reference Figure 3 , device 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output (I / O) interface 312 , a sensor component 314 , and a communication component 316 .

[0187] The processing component 302 generally controls the overall operation of the device 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the method provided by the technical solution of the present disclosure. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.

[0188] The memory 304 is configured to store various types of data to support operations on the device 300. Examples of such data include instructions for any application or method operating on the device 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0189] The power supply component 306 provides power to the various components of the device 300. The power supply component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 300.

[0190] The multimedia component 308 includes a screen that provides an output interface between the device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0191] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the device 300 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.

[0192] The input / output (I / O) interface 312 provides an interface between the processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0193] The sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for the device 300. For example, the sensor assembly 314 can detect the open / closed state of the device 300, the relative positioning of components, such as the display and keypad of the device 300, and the sensor assembly 314 can also detect the position change of the device 300 or a component of the device 300, the presence or absence of user contact with the device 300, the orientation or acceleration / deceleration of the device 300, and the temperature change of the device 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0194] The communication component 316 is configured to facilitate wired or wireless communication between the device 300 and other devices. The device 300 can access a wireless network based on a communication standard, such as WiFi, or a mobile communication network such as 2G, 3G, 4G / LTE, 5G, etc. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0195] In an exemplary embodiment, the device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0196] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, and the above instructions can be executed by a processor 320 of the device 300 to complete the method provided by the technical solution of the present disclosure. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0197] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0198] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0199] The above is a detailed introduction to the scheme provided by the present application. The principles and implementation methods of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for obtaining volume information of logistics packages, characterized in that: The method is used to estimate the volume of a logistics package in a state of being transported through a conveyor line of a warehousing system, wherein the logistics package has a packaging material with high transparency, and the method comprises: Obtaining an image pair acquired during the delivery of the logistics package, the image pair comprising a depth image and a visible light image acquired at the same time in the height direction of the logistics package; Performing transparent texture feature detection on the visible light image, and determining a portion having the transparent texture feature as a target area of ​​the packaging material corresponding to the visible light image; Obtaining a target image obtained by image matching the depth image and the visible light image, and determining, from the target image, a region to be processed corresponding to the target region and obtaining available depth information associated with the region to be processed; The missing depth information of the area to be processed is compensated based on the available depth information, and the volume information of the logistics package is calculated according to the compensated target image.

2. The method according to claim 1, characterized in that The available depth information is at least one of the following depth information: Depth information collected at the edge of the outer contour of the logistics package; Depth information collected from the label area of ​​the logistics package; Depth information collected from the graphic printing area of ​​the packaging material; Depth information collected from the wrinkles of the packaging material.

3. The method according to claim 1, characterized in that The storage system is equipped with depth cameras and industrial cameras that can cover the designated area of ​​the conveyor line, as well as inductive sensors. The obtaining of the image pair collected during the logistics package transmission process includes: The induction sensor generates a trigger signal when detecting the passage of the logistics package, so that the depth camera captures the depth image under the action of the trigger signal, and the industrial camera captures the visible light image under the action of the trigger signal, thereby obtaining the image pair captured at the same time.

4. The method according to claim 3, characterized in that The installation height of the depth camera is determined according to the following requirements: The depth image captured by the depth camera includes a panoramic view of the logistics package, and does not include other packages being transported adjacently on the conveyor line.

5. The method according to claim 1, characterized in that The storage system is equipped with a plurality of depth cameras covering the designated area of ​​the conveyor line from different angles. The depth image is obtained as follows: Obtaining initial depth images of the logistics package taken by different depth cameras at different viewing angles at the same time; Point cloud fusion processing is performed on the multiple initial depth images, and the fused image is determined as the depth image of the logistics package.

6. The method according to claim 1, characterized in that The transparent texture feature detection for the visible light image includes: Obtaining a transparent object detection model for transparent texture feature detection; The visible light image is input into the transparent object detection model to obtain a detection result output by the model, wherein the detection result is used to identify a target area existing in the visible light image.

7. The method according to claim 6, characterized in that The visible light image is collected by an industrial camera that can cover the specified area of ​​the conveyor line. The step of obtaining a transparent object detection model for transparent texture feature detection includes: Obtaining a plurality of sample images captured by the industrial camera in a designated area of ​​the conveyor line, and marking transparent areas included in the plurality of sample images respectively; An initial model capable of detecting transparent texture features is obtained, and the initial model is fine-tuned through the sample image and the corresponding annotated transparent area to obtain the transparent object detection model.

8. The method according to claim 1, characterized in that The logistics package is placed in a container and is transported on the conveyor line through the container. The obtaining of a target image obtained by performing image matching between the depth image and the visible light image comprises: Obtaining a plurality of available matching point information determined according to the shape characteristics of the container; Locating matching positions that match the plurality of available matching point information in the depth image and the visible light image respectively; According to the matching position, an affine transformation is performed on the depth image or the visible light image to achieve matching and fusion between the depth image and the visible light image to obtain the target image.

9. The method according to any one of claims 1 to 8, characterized in that: The logistics package is placed in a container and is transported on the conveyor line through the container. The method further includes: The full load rate of the container is calculated according to the volume information of the logistics parcel contained in the container.

10. The method according to claim 9, characterized in that The method further comprises: When the full load rate is lower than a preset value, the placement quantity information and / or placement method information in the container associated with the package type to which the logistics package belongs is adjusted.

11. A depth information processing method, characterized in that: The method comprises: Obtaining an image pair associated with a logistics package, the image pair comprising a depth image and a visible light image acquired at the same time in a height direction of the logistics package; Performing transparent texture feature detection on the visible light image, and when the visible light image includes a target area having transparent texture features, obtaining a target image obtained by performing image matching between the depth image and the visible light image; Determining a to-be-processed area corresponding to the target area from the target image, and obtaining available depth information associated with the to-be-processed area; Compensation processing is performed on the missing depth information of the area to be processed based on the available depth information.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented.

13. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the steps of the method according to any one of claims 1 to 11 are executed.

14. A computer program product comprising a computer program / computer executable instructions, characterized in that: When the computer program / computer executable instructions are executed by a processor in an electronic device, the steps of the method according to any one of claims 1 to 11 are implemented.

Images