Package breakage detection method and program product

Abstract

The invention discloses a package breakage detection method and a program product. The method comprises the steps that parcel video information of a plurality of parcels located on a conveying belt is collected through a binocular shooting device, and the parcel video information comprises a plurality of video frames; determining a plurality of target video frames corresponding to the same parcel, a first detection frame of the parcel in the target video frames and type information of the parcel from the plurality of video frames according to the parcel video information; for a single package, determining a first damage detection result of the package according to a plurality of target video frames corresponding to the package and a damage detection model; determining a second damage detection result of the package according to a plurality of target video frames corresponding to the package and the type information corresponding to the package; and determining a target damage detection result of the package according to the first damage detection result and the second damage detection result of the package. According to the technical scheme, package damage can be automatically detected, the manual inspection cost is reduced, and the package detection efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of express package detection, and in particular to a package damage detection method and program product. BACKGROUND

[0002] In a modern logistics system, package mailing involves multiple links such as warehousing, transportation, sorting, etc., and is prone to hidden or explicit damage due to factors such as squeezing, collision, falling, etc., which not only causes loss of goods value, but also may cause complaints and disputes and logistics reputation risk. Therefore, rapid and accurate detection of the damage state of a package is a key link to guarantee the quality of logistics services and reduce operating costs.

[0003] In related technologies, current package damage detection mainly relies on manual visual inspection, which has significant limitations: on the one hand, manual detection is inefficient and difficult to meet the rapid processing needs of large-scale logistics scenarios; on the other hand, the detection results are easily affected by factors such as personnel experience, fatigue, subjective judgment, etc., resulting in high rates of missed detection and false detection, and long-term labor cost is high, which cannot meet the technical requirements of intelligent logistics development. SUMMARY

[0004] The present application provides a package damage detection method and program product to solve the technical problems of insufficient detection capability and poor reliability of the presence of damage in a large number of express packages in related technologies.

[0005] According to an aspect of the present application, a package damage detection method is provided, which comprises:

[0006] Collecting package video information of a plurality of packages located on a conveyor belt by means of a binocular shooting device, wherein the package video information comprises a plurality of video frames;

[0007] According to the package video information, determining a plurality of target video frames corresponding to the same package, a first detection box of the package in the target video frame, and type information of the package from the plurality of video frames;

[0008] For a single package, determining a first damage detection result of the package according to a plurality of target video frames corresponding to the package and a damage detection model;

[0009] Determining a second damage detection result of the package according to a plurality of target video frames corresponding to the package and the type information corresponding to the package;

[0010] Determining a target damage detection result of the package according to the first damage detection result and the second damage detection result of the package.

[0011] According to another aspect of the present application, there is provided a package damage detection device, comprising:

[0012] a package video collection module configured to collect package video information of a plurality of packages on a conveyor belt by means of a binocular shooting device, wherein the package video information comprises a plurality of video frames;

[0013] a package information determination module configured to determine, according to the package video information, a plurality of target video frames corresponding to a same package, a first bounding box of the package in the target video frames, and type information of the package from the plurality of video frames;

[0014] a first damage detection module configured to determine, according to a plurality of target video frames corresponding to a single package and a damage detection model, a first damage detection result of the package;

[0015] a second damage detection module configured to determine, according to a plurality of target video frames corresponding to the package and the type information of the package, a second damage detection result of the package;

[0016] a damage detection determination module configured to determine a target damage detection result of the package according to the first damage detection result and the second damage detection result of the package.

[0017] According to another aspect of the present application, there is provided an electronic device, comprising:

[0018] at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform a package damage detection method according to any one of the embodiments of the present application.

[0019] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to perform a package damage detection method according to any one of the embodiments of the present application.

[0020] According to another aspect of the present application, the embodiments of the present disclosure further provide a computer program product comprising a computer program, which, when executed by a processor, implements a package damage detection method according to any one of the embodiments of the present disclosure.

[0021] The technical scheme of the embodiment of the present application is that a binocular shooting device is used to collect package video information of a plurality of packages on a conveying belt, wherein the package video information comprises a plurality of video frames; and a plurality of video frame images of the packages are obtained for subsequent package damage analysis. A plurality of target video frames corresponding to the same package, a first detection frame of the package in the target video frame and type information of the package are determined from the plurality of video frames according to the package video information; the target video frame, the first detection frame and the type information of the package are determined, so that the package is detected in all directions to ensure the accuracy of the package damage detection. For a single package, a first damage detection result of the package is determined according to a plurality of target video frames corresponding to the package and a damage detection model; the damage detection model is used to analyze a plurality of video frame images, so that efficient package damage detection is realized. A second damage detection result of the package is determined according to a plurality of target video frames corresponding to the package and the type information of the package corresponding to the package; different damage detection methods are adopted for different types of packages, so that more accurate damage information of different types of packages is obtained, so as to ensure the accuracy and wide applicability of the package damage detection method. Finally, a target damage detection result of the package is determined according to the first damage detection result and the second damage detection result of the package. By comparing the damage detection results determined in different ways, the authenticity and reliability of the final damage identification and detection result can be ensured. The technical scheme can effectively improve the ability of package damage identification and detection and ensure the reliability of the final package damage detection result.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0024] Figure 1 is a flowchart of a package damage detection method provided by the first embodiment of the present application;

[0025] Figure 2 is a flowchart of a package damage detection method provided by the second embodiment of the present application;

[0026] Figure 3is a flow chart of a package damage detection method according to an embodiment of the present application;

[0027] Figure 4 is a structural schematic diagram of a package damage detection device according to an embodiment of the present application;

[0028] Figure 5 is a structural schematic diagram of an electronic device for implementing a package damage detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative rather than limiting, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between the multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0033] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means in accordance with relevant laws and regulations.

[0034] For example, in response to receiving an active request of a user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can autonomously select whether to provide personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium performing the operation of the technical solution of the present disclosure according to the prompt information.

[0035] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending prompt information to the user may, for example, be a pop-up window manner, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry selection controls for the user to select “agree” or “disagree” to provide personal information to the electronic device.

[0036] It can be understood that the above notification and obtaining user authorization process is only illustrative and does not limit the implementation manner of the present disclosure, and other manners meeting relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0037] It can be understood that the data (including but not limited to the data itself, the obtaining or use of the data) involved in the technical solution should comply with the requirements of relevant laws and regulations and relevant provisions.

[0038] Embodiment one

[0039] Figure 1 A flowchart of a package damage detection method is provided for the first embodiment of the present application. The present embodiment can be applied to the scenario of identifying and detecting packages in a mail processing center. The method can be executed by a package damage detection device, which can be realized in the form of hardware and / or software. Optionally, the package damage detection device can be realized by an electronic device, which can be a mobile terminal, a PC terminal or a server, etc. As shown in Figure 1 The method can specifically include the following steps.

[0040] S110, collecting package video information of a plurality of packages located on a conveying belt by a binocular shooting device, wherein the package video information includes a plurality of video frames.

[0041] The binocular shooting device refers to a shooting system obtained by placing two cameras or cameras at a certain interval, which can be used to calculate and obtain depth information of a target object. The package refers to each express, mail, cargo and the like. The package video information refers to a sequence of dynamic images containing at least one package, i.e. a plurality of image video frames.

[0042] Specifically, the parcel video information of multiple parcels on the conveying belt is acquired by a binocular shooting device installed above the conveying belt, so that image information of the mail parcels from multiple angles is obtained, and the complete and comprehensive parcel image can be obtained, so that the accuracy of parcel recognition and detection is improved.

[0043] In S120, multiple target video frames corresponding to the same parcel, a first detection box of the parcel in the target video frame, and type information of the parcel are determined from the multiple video frames according to the parcel video information.

[0044] In the embodiment of the present application, the target video frame can be a video frame in which the parcel is located at a preset position in the field of view. For example, the target video frame can include, but is not limited to, at least one of a video frame in which the parcel completely enters the field of view of the binocular shooting device (such as the first time when the parcel appears in the field of view of the binocular shooting device), a video frame in which the parcel is located at the center of the field of view of the binocular shooting device (such as being located directly below the binocular shooting device), and a video frame in which the parcel is about to leave the field of view of the binocular shooting device (such as the last time when the parcel appears in the field of view of the binocular shooting device). The target video frame can also be a video frame acquired through a preset time interval. The first detection box is used to indicate the display area of the parcel in the video frame. The type information of the parcel can be determined according to one or more information such as the volume, shape, and packaging material of the parcel. For example, the type information of the parcel can include at least one of a parcel type, a carton type, a soft parcel type, a document type, and a special-shaped part type.

[0045] In S130, for a single parcel, a first damage detection result of the parcel is determined according to the multiple target video frames corresponding to the parcel and a damage detection model.

[0046] The damage detection model specifically refers to a computer vision model for detecting the damage degree of the parcel, which can be obtained by training sample parcel images containing labels and corresponding damage annotation information. The first damage detection result can include at least one of the damage state (whether damage occurs), the damage degree, the damage area, and the damage picture of the parcel in multiple video frames. The damage degree can include at least one of a parcel in good condition, slight damage, and damage, for example. Slight damage can mean that the damage area of the parcel is less than or equal to 10 square centimeters, and damage can mean that the damage area of the parcel is greater than 10 square centimeters.

[0047] In an embodiment, the determining the first damage detection result of the package according to the plurality of target video frames corresponding to the package and the damage detection model comprises: for a single target video frame, providing the target video frame corresponding to the package to the damage detection model to obtain a third damage detection result corresponding to the target video frame; in response to the third damage detection result including a second detection box, determining a fourth damage detection result of the target video frame according to the second detection box and the first detection box of the package in the target video frame; wherein the second detection box is used to indicate an image region where the damaged package is detected; and determining the first damage detection result of the package according to the fourth damage detection results of the plurality of target video frames corresponding to the package.

[0048] The third damage detection result specifically comprises a detection box. The fourth damage detection result comprises at least one of the damage state (whether damage occurs), the damage degree, the damage area, and the damage picture of the video frame.

[0049] Specifically, for a single target video frame, the single target video frame of the target package is input into the damage detection model, so as to obtain the second detection box output by the damage detection model. The image region of the target package in the second detection box and the first detection box is matched, and the damage area, picture, and other data of the package in the single target video frame are obtained, so as to determine the fourth damage detection result of the single target video frame. Finally, the first damage detection result of the package is determined according to the fourth damage detection results of the plurality of target video frames corresponding to the package. The technical solution can obtain a more comprehensive damage detection result of the package by inputting the plurality of target videos at different positions of the package into the damage detection model, thereby guaranteeing the damage detection accuracy of the package.

[0050] In an embodiment, the determining the fourth damage detection result of the target video frame according to the second detection box and the first detection box of the plurality of packages in the target video frame comprises: determining the coincidence degree of the second detection box and the first detection box of the package in the target video frame, and determining the fourth damage detection result of the target video frame according to the coincidence degree.

[0051] The coincidence degree can refer to the coincidence proportion of a plurality of pixel coordinates in different detection boxes, or the area coincidence proportion of different detection boxes.

[0052] Specifically, the corresponding area range of the second detection box is matched with the corresponding area range of the first detection box of the package in the target video frame. If the area range coincidence degree is greater than a preset threshold, it is considered that the corresponding package of the target video frame is damaged; if the area range coincidence degree is less than the preset threshold, it is considered that the corresponding package of the target video frame is not damaged, so as to determine the fourth damage detection result of the target video frame. The technical scheme can realize automatic detection of package damage by matching the detection boxes output by different detection models, and improve the efficiency and accuracy of package damage detection.

[0053] In one example, in the fourth damage detection results of the plurality of target video frames corresponding to the package, if the fourth damage detection result of at least one target video frame indicates that the package is damaged, it is determined that the first damage detection result of the package is damaged; if the fourth damage detection results of the plurality of target video frames corresponding to the package all indicate that the package is not damaged (the package is complete), it is determined that the first damage detection result of the package is not damaged or the package is complete.

[0054] S140, determining a second damage detection result of the package according to the plurality of target video frames corresponding to the package and the type information corresponding to the package.

[0055] The second damage detection result is the same as the first damage detection result, and can also include at least one of the damage state (whether damage occurs), the damage degree, the damage area, and the damage picture of the plurality of video frames of the package.

[0056] Specifically, according to the target video frames corresponding to the target package at a plurality of positions and the type information of the target package, a corresponding package damage identification and detection method is adopted based on different types of packages, so as to determine the second damage detection result of the target package. The technical scheme can effectively improve the accuracy of damage identification and detection of different types of packages by implementing a more targeted package damage detection method for different types of packages.

[0057] S150, determining a target damage detection result of the package according to the first damage detection result and the second damage detection result of the package.

[0058] The target damage detection result refers to the final damage detection result of the package, and can also include at least one of the damage state (whether damage occurs), the damage degree, the damage area, and the damage picture of the plurality of video frames of the package.

[0059] Specifically, the damage degree corresponding to the first damage detection result of the package is compared with the damage degree corresponding to the second damage detection result, so as to determine the damage detection result corresponding to the highest damage degree as the final damage detection result of the package, i.e., the target damage detection result. Through comparison of multiple damage detection results and determination of the most serious damage detection result as the final target damage detection result, the accuracy of the damage detection result of the package can be effectively guaranteed, and the technical problem of insufficient damage detection and identification ability of a single method for the package in some special scenarios and poor reliability can be solved.

[0060] Further, the damaged package identified can be automatically sorted in combination with an automatic package sorting device, so that the operation personnel can timely process the damaged package, thereby effectively improving the automatic identification and processing efficiency of the damaged package.

[0061] In one embodiment, the second damage detection result of the package is determined according to the plurality of target video frames corresponding to the package and the type information corresponding to the package, including: in response to the type information corresponding to the package being a first type, for each target video frame, determining first color information of the package according to the first detection box of the package in the target video frame and pixel color information in the target video frame, determining a fifth damage detection result of the target video frame according to the similarity between the first color information and a plurality of second color information stored in a color database; wherein the first type includes a package type; and determining the second damage detection result of the package according to the fifth damage detection result of the plurality of target video frames corresponding to the package.

[0062] The package refers to a plurality of small packages sent to the same destination being packaged together for transportation.

[0063] The first color information can be understood as a plurality of color information corresponding to the package range. The color database refers to a database pre-stored with a plurality of package color information. The second color information can be understood as at least one color information contained in different packages. The fifth damage detection result can specifically include at least one of similarity result, damage degree, damage area, etc.

[0064] Specifically, if the type information corresponding to the parcel is a set parcel type, i.e., the first type, for each target video frame, a first detection box of the set parcel is determined from the target video frame. By extracting the pixel color information in the first detection box, the first color information of the set parcel is determined. The obtained first color information and the second color information stored in the color database are matched for similarity calculation, so as to determine whether the target video frame is damaged based on the similarity data, and by obtaining the damaged area, picture and other data in the first detection box, the fifth damage detection result is determined. Finally, the fifth damage detection result of the set parcel of the plurality of target video frames is determined as the second damage detection result of the parcel. Since the set parcel is assembled by a plurality of small parcels, the shape of the set parcel is irregular, but the color of the set parcel is relatively uniform, which is convenient for identification. Therefore, the color of the first type of set parcel is extracted in the technical solution, so as to determine whether the set parcel is damaged through pixel color, which improves the efficiency of parcel damage detection.

[0065] Further, in an embodiment, the first color information of the parcel is determined according to the first detection box of the parcel in the target video frame and the pixel color information in the target video frame, further comprising: dividing an image region in the first detection box of the parcel in the target video frame into a plurality of color blocks, determining the average color value of each color block according to the pixel color information in the target video frame, and determining the first color information of the parcel according to the average color information of the plurality of color blocks.

[0066] Wherein, the color block can be understood as a small image region after division.

[0067] Specifically, for the image region of the first detection box of the set parcel in each target video frame, the image region is divided into a plurality of smaller color block regions according to a preset size. And by extracting the pixel color information corresponding to the plurality of color block regions respectively, the average color value of each color block is obtained, and the average color information of the plurality of color blocks is determined as the first color information of the set parcel. The color block region of the plurality of small regions in the technical solution can obtain more accurate color information of different color blocks, so as to improve the accuracy of subsequent color block color matching and parcel damage detection.

[0068] In another implementation, the parcel video information includes a left-eye parcel image and a right-eye parcel image corresponding to each of the video frames; and the determining the second damage detection result of the parcel according to the target video frames corresponding to the parcel and the type information corresponding to the parcel includes: in response to the type information corresponding to the parcel being a second type, determining a first depth image corresponding to each of the target video frames according to the left-eye parcel image and the right-eye parcel image corresponding to each of the target video frames, respectively; for each of the video frames, determining a second depth image corresponding to the parcel from the first depth image corresponding to the target video frame according to the first detection box of the parcel in the target video frame; and determining the second damage detection result of the parcel according to the second depth images corresponding to the parcel.

[0069] The second type includes a first sub-type and a second sub-type, the first sub-type includes at least one of a carton type, a soft package type, and a file type, and the second sub-type includes a special-shaped part. The left-eye parcel image is an image of the parcel captured by a left camera of a binocular shooting device, and the right-eye parcel image is an image of the parcel captured by a right camera of the binocular shooting device. The first depth image is a depth image containing depth information, which is generated based on two images corresponding to a target video frame captured by the binocular shooting device. The second depth image is a depth image determined from the first depth image based on a range of a detection box of the parcel.

[0070] Specifically, if the type information corresponding to the parcel is the second type, a first depth image corresponding to each of the target video frames is determined according to the left-eye parcel image and the right-eye parcel image corresponding to each of the target video frames. For a target video frame in the plurality of video frames, a first detection box of the parcel in the target video frame is determined to determine a second depth image corresponding to a range of the first detection box of the parcel from the first depth image corresponding to the target video frame. Thus, a second damage detection result of the parcel is determined according to the second depth images corresponding to the target video frames of the parcel. The technical solution can detect a parcel of a second type by using a depth image, and can detect parcels of various shapes more flexibly, thereby improving the accuracy and reliability of the parcel damage detection.

[0071] Further, in another embodiment, the second type includes a first sub-type; the first sub-type includes at least one of a carton type, a soft package type, and a document type; and the determining the second damage detection result of the package according to the plurality of second depth images corresponding to the package includes: for each of the second depth images corresponding to the package, performing plane extraction on the second depth image according to a preset multi-plane extraction algorithm to obtain a plurality of target package planes corresponding to the second depth image; for each of the second depth images, dividing the second depth image into a plurality of first grids, respectively determining an average depth of each of the first grids, respectively determining a grid damage detection result of each of the first grids according to the average depth of each of the first grids in the target package plane and an average depth of the target package plane, and determining a plane damage detection result of the target package plane according to the grid damage detection results of the plurality of first grids in the target package plane; determining an image damage detection result of the second depth image according to the plane grid damage detection results of the plurality of target package planes corresponding to the second depth image, and determining the second damage detection result of the package according to the image damage detection results of the plurality of second depth images. By comparing the depth data of the plurality of divided regions with the depth data of the entire package, the technical solution can realize fine area damage detection, and by comparing the depth data of the plurality of target video frames corresponding to the depth images, the accuracy of the package damage detection is effectively ensured.

[0072] The multi-plane extraction algorithm can include a multi-plane RANSAC algorithm. The target package plane specifically refers to a plane image formed by the relatively flat image region in the depth image of the first sub-type package after de-duplication processing. The first grid refers to a depth image region obtained by dividing the second depth image according to a preset size. For example, the division range of the first grid can be 1 cm x 1 cm. The grid damage detection result is used to indicate whether the grid region has damage.

[0073] Optionally, the grid damage detection result of the plurality of first grids in the target package plane is determined according to the depth difference absolute value between the average depth of each of the first grids in the target package plane and the plane average depth data of the target package plane. For example, if the depth difference absolute value result is greater than a preset difference threshold value, it is considered that the first grid region has damage, and the grid damage detection result is damage.

[0074] In another embodiment, the second type can include a second sub-type. Illustratively, the second sub-type includes a special-shaped package. In this case, the determining the second damage detection result of the package according to the second depth images corresponding to the package includes: for each of the second depth images corresponding to the package, dividing the second depth image into a plurality of second grids, and determining the average depth of each of the second grids; for each of the second grids, determining the grid damage detection result of the second grid according to the depth difference between the average depth of the second grid and the average depth of at least one adjacent second grid according to the depth difference; determining the image damage detection result of the second depth image according to the grid damage detection results of the plurality of second grids in the second depth image; and determining the second damage detection result of the package according to the image damage detection results of the plurality of second depth images. Through the calculation of the depth data between adjacent areas in the special-shaped package, the technical problem that the damage detection of the package is not accurate enough due to the irregular shape of the package can be effectively solved, and the damage detection accuracy of the complex-shaped package can be improved by considering the depth data deviation between adjacent areas.

[0075] wherein the special-shaped package refers to a package with an irregular shape. The second grid refers to a depth image area obtained according to another division standard, and the division range of the second grid is smaller than that of the first grid. Illustratively, the division of the second grid can be to further divide the first grid into 3x3 small grids based on the division of the first grid, thereby obtaining a plurality of second grids of 1 / 3 cm.

[0076] Specifically, a plurality of second depth images corresponding to the second sub-type package are obtained, and the second depth image of the special-shaped package is divided into a plurality of second grids, and the average depth data of each second grid is determined. For each second grid, the depth difference between the average depth of each second grid and the average depth of at least one adjacent second grid is determined, so as to determine whether the second grid is damaged according to the depth difference, thereby obtaining the grid damage detection result of each second grid. Finally, the image damage detection result of the second depth image is determined according to the grid damage detection results of the plurality of second grids in the second depth image, and the second damage detection result of the package is determined according to the image damage detection results of the plurality of second depth images.

[0077] The technical scheme of the embodiment of the present application collects package video information of a plurality of packages located on a conveying belt through a binocular shooting device, wherein the package video information comprises a plurality of video frames; thereby obtaining a plurality of video frame images of the packages for subsequent package damage analysis. A plurality of target video frames corresponding to the same package, a first detection box of the package in the target video frames and type information of the package are determined from the plurality of video frames according to the package video information; by determining the target video frames, the first detection box and the type information of the package, the package is comprehensively detected, so as to ensure the accuracy of the package damage detection. For a single package, a first damage detection result of the package is determined according to a plurality of target video frames corresponding to the package and a damage detection model; the damage detection model is used to analyze a plurality of video frame images, thereby realizing efficient package damage detection. A second damage detection result of the package is determined according to a plurality of target video frames corresponding to the package and the type information corresponding to the package; by adopting different damage detection methods for different types of packages, more accurate damage information of different types of packages can be obtained, thereby ensuring the accuracy and wide applicability of the package damage detection method. Finally, a target damage detection result of the package is determined according to the first damage detection result and the second damage detection result of the package. By comparing the damage detection results determined in different ways, the authenticity and reliability of the final damage identification and detection result can be ensured. Through the above technical means, the ability of package damage identification and detection can be effectively improved, and the reliability of the final package damage detection result can be ensured.

[0078] Embodiment two

[0079] Figure 2 A flowchart of a package damage detection method provided by the second embodiment of the present application, the scheme in the present embodiment is based on the scheme in the above-mentioned embodiment, and is a refinement of the scheme of determining a plurality of target video frames corresponding to the same package, a first detection box of the package in the target video frames and type information of the package from the plurality of video frames according to the package video information. The specific implementation can be referred to the description of the present embodiment. Among them, the same or similar technical features as the foregoing embodiments will not be described here. As shown in the following table, the method can specifically include: Figure 2

[0080] S210, collecting package video information of a plurality of packages located on a conveying belt through a binocular shooting device, wherein the package video information comprises a plurality of video frames.

[0081] S220, performing package detection on a plurality of video frames based on a package detection model to obtain a first detection box and type information of each package in each video frame. ​

[0082] The parcel detection model can be used to identify parcels in video images, which can be trained by sample images and their corresponding parcel label data. Further, the parcel label data can include, but is not limited to, a parcel detection box and a parcel type. For example, the parcel label data can also include parcel description data, etc.

[0083] Specifically, by inputting multiple video frames into the parcel detection model, multiple parcels in the video frames are sequentially identified and detected to determine the first detection box of each parcel in each video frame and the type information corresponding to the parcel. Using the parcel detection model can efficiently determine each parcel in the video frame, so as to facilitate subsequent detection of parcel damage.

[0084] S230, track the detected multiple parcels based on a preset tracking algorithm to obtain parcel identifiers of the multiple parcels, and determine the first detection box of the target parcel in the multiple video frames based on the parcel identifiers.

[0085] The tracking algorithm can generally include at least one of SORT algorithm, ByteTrack algorithm, etc. The parcel identifier is used to identify a target parcel to determine the same parcel in multiple video frames.

[0086] Specifically, the detected multiple parcels are tracked and detected using the preset tracking algorithm to generate parcel identifiers corresponding to the multiple parcels, and the first detection box of the target parcel in the multiple video frames can be determined based on the parcel identifiers to determine the position of the target parcel in different position video frames. The present technical solution can effectively guarantee the tracking and identification of the same parcel, thereby improving the accuracy of the damage identification and detection of the same parcel.

[0087] In one embodiment, a plurality of target video frames corresponding to each of the parcels in the multiple video frames are determined from the multiple video frames according to the first detection box of each of the parcels in the multiple video frames, including at least one of the following: determining the distance between the first detection box of the parcel in at least part of the video frames and the center point of the video frame, and determining the target video frame corresponding to the parcel from the multiple video frames according to the distance corresponding to the multiple video frames; determining the target video frame corresponding to the parcel from the multiple video frames according to the capture time of the multiple video frames and the height of the first detection box of the parcel in the video frames.

[0088] The center point of the video frame can be understood as the center point position of the video image.

[0089] In one embodiment, a minimum value of the distances corresponding to the plurality of video frames can be determined, and the video frame corresponding to the minimum value is determined as the target video frame corresponding to the package. This has the advantage that the video frame closest to the binocular camera, or in other words, the video frame at the center of the field of view of the binocular camera, can be determined, and the display size of this video frame is the largest, thereby ensuring that more detailed information of the package can be captured. In another embodiment, a preset value of the distances corresponding to the plurality of video frames can be determined, and the video frame corresponding to the preset value is determined as the target video frame corresponding to the package. The preset value can be determined according to the field of view of the binocular camera. With this technical solution, the image of the package appearing at a preset position in the video range, i.e., the target video frame, can be quickly and conveniently obtained in a simple and effective manner, the determination efficiency of the target video frame is improved, and more comprehensive data basis is provided for subsequent accurate detection of damage of the package.

[0090] In yet another embodiment, the capture times of the plurality of video frames can be sorted according to early and late (including sorting from early to late and / or from late to early), the maximum height of the first detection box of the package in the plurality of sorted video frames is determined, and a target height threshold is determined according to the maximum height; the first video frame in which the height of the first detection box of the package in the plurality of video frames is greater than the target height threshold is determined as the target video frame corresponding to the package. The target height threshold can be obtained by multiplying the maximum height by a preset multiple. In the embodiment of the application, the preset multiple can take a value between 0 and 1, and the specific value can be set according to actual requirements, which is not specifically limited herein, for example, it can be 0.8 times, etc.

[0091] In one example, when determining the plurality of target video frames corresponding to the target package from the plurality of video frames, three video frames in the plurality of video frames can be collected as the target video frames, including but not limited to a video frame in which the package completely enters the field of view of the binocular shooting device, a video frame in which the package is located at the center of the field of view of the binocular shooting device, and a video frame in which the package is about to leave the field of view of the binocular shooting device. First, the video frame in which the detection box of the target package in the plurality of video frames is closest to the center point of the video frame can be determined as the video frame at the center of the field of view. The height of the first detection box of the package in the video frame, i.e., the maximum height, is determined. Second, the plurality of video frames can be sorted in chronological order from early to late, and the first video frame in which the height of the first detection box is greater than or equal to 0.8 times the height of the first detection box in the first video frame can be determined as the video frame in which the package completely enters the field of view of the binocular shooting device. Similarly, the plurality of video frames can be sorted in chronological order from late to early, and the first video frame in which the height of the first detection box is greater than or equal to 0.8 times the maximum height in the first video frame can be determined as the video frame in which the package is about to leave the field of view of the binocular shooting device. Through the above method, the video frame images of the target package at multiple positions can be obtained, so that the multiple parts of the package can be comprehensively damaged detected based on more comprehensive package images, and the reliability of the package damage detection is ensured.

[0092] S240, determining a plurality of target video frames corresponding to each of the packages from the plurality of video frames according to the first detection box of each of the packages in the plurality of video frames.

[0093] Specifically, a plurality of target video frames corresponding to each of the packages can be determined from the plurality of video frames according to the first detection box of each of the packages, so that the target video frames corresponding to each of the packages can be conveniently determined, and the efficiency of subsequent package damage detection can be improved.

[0094] S250, determining a first damage detection result of a single package according to the plurality of target video frames corresponding to the package and a damage detection model.

[0095] S260, determining a second damage detection result of the package according to the plurality of target video frames corresponding to the package and the type information corresponding to the package.

[0096] S270, determining a target damage detection result of the package according to the first damage detection result and the second damage detection result of the package.

[0097] In the technical solution of this invention, firstly, a package detection model is used to detect packages in multiple video frames to obtain a first detection box and type information for each package in each video frame. This allows for efficient and accurate determination of image information of multiple packages in different video frames, facilitating efficient data processing. Based on a preset tracking algorithm, the detected packages are tracked to obtain package identifiers. The first detection box of the target package in the multiple video frames is determined based on the package identifiers. This accurately determines the position of the same package at different locations, enabling accurate and effective damage detection for packages belonging to the same group. Finally, multiple target video frames corresponding to the package are determined from the multiple video frames based on the first detection boxes of each package in the multiple video frames. This filters out invalid video frames, effectively reducing the amount of data required for package damage detection, and achieving more efficient and accurate package damage detection through package images from different locations and from all directions. Therefore, this technical solution, through the above-mentioned technical means, can effectively ensure the accuracy and reliability of target package detection.

[0098] Example 3

[0099] Embodiment 3 of the present invention provides a flowchart of a package damage detection method. To better illustrate the technical solution provided by this embodiment, the following steps are used to illustrate the method. The flowchart of this embodiment is as follows: Figure 3 As shown, specific implementation methods can be found in the description of this embodiment. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here.

[0100] Specifically, the overall implementation process is as follows: Figure 3 As shown.

[0101] 1. Installation of binocular cameras

[0102] Install the binocular camera directly above the belt conveyor or conveyor belt so that the camera is facing the package transport equipment. The installation height depends on the size of the package. It is necessary to ensure that the largest package can be fully in the camera's field of view, with sufficient space around it. The entire package should be visible in the frame for several consecutive frames. Generally, the binocular camera is installed 1.5 meters directly above the package transport equipment.

[0103] 2. Package video capture and depth map generation

[0104] On the operating parcel transport equipment, the parcel passes through the binocular cameras in sequence. The binocular cameras obtain real-time video, left grayscale image, and right grayscale image of the parcel. The disparity map of the left and right cameras is obtained by using a binocular stereo matching algorithm to obtain the depth map of the parcel.

[0105] 3. Package detection

[0106] The trained package target detection model is used to detect the package in the real-time video picture, to obtain the package detection frame and the package category in the video picture.

[0107] 4. Package tracking

[0108] The target tracking algorithm is used to track the package detected by the package target detection model, to generate a unique ID of the package, and to obtain pictures of the package at different positions on the running belt machine.

[0109] 5. Obtain single package picture and depth map

[0110] When the target tracking algorithm is used for detection, multiple frames of images are obtained, in which the package just appears in the field of view, the package is located below the camera field of view, and the package is about to leave the field of view, and the corresponding pictures, package detection frames, package types, and depth maps of the multiple frames of images are obtained.

[0111] Specifically, the multiple frames of images obtained by the target are as follows:

[0112] 1) When the package is located below the binocular camera field of view:

[0113] In the package tracking result, the detection frame closest to the picture center point is found. Let the picture width be w, the height be h, and the picture center point be (w / 2, h / 2). The center point coordinates of the target detection frame are (x, y), the width is wb, and the height is hb. The distance between the detection frame and the picture center point is |h / 2-y|. All detection frames are traversed in turn to find the detection frame with the smallest distance, thereby determining the video picture in which the package is located below the field of view.

[0114] When the package just appears in the binocular camera field of view:

[0115] In the package tracking result, each detection frame is traversed in chronological order, and the first detection frame with a height greater than or equal to 0.8 times the height of the detection frame closest to the picture center point is found.

[0116] When the package is about to leave the binocular camera field of view:

[0117] In the package tracking result, each detection frame is traversed in reverse chronological order, and the first detection frame with a height greater than or equal to 0.8 times the height of the detection frame closest to the picture center point is found.

[0118] 6. Damage package identification based on target detection

[0119] Using the trained damaged package target detection model, according to whether there is a damaged package in the three video frame pictures obtained in step 5, if there is a damaged package in the picture, and the overlap degree of the damaged package detection box and the package detection box is more than 80%, the package is damaged, and the damage degree is determined by obtaining the damaged area and other data, to return the package damage result and evidence picture.

[0120] 7. Damaged package identification based on depth map detection:

[0121] 1) Package target detection result is a package of carton, soft package, and file:

[0122] a. Package depth map preprocessing: according to the depth map and the package detection box, the package depth map is obtained. The package depth map is preprocessed to remove outliers and perform Gaussian filtering to smooth noise.

[0123] b. Package plane extraction and deduplication: for the package depth map in the three states in step 6, use the multi-plane RANSAC algorithm to extract the plane in the package depth map. Remove the smaller plane when the comprehensive similarity of two planes is greater than 90%. The comprehensive similarity is the weighted sum of the normal vector similarity and the position similarity.

[0124] c. Determine whether the deduplicated package plane is damaged: divide the package depth map into 1cm x 1cm grids, and adjust the grid size according to business needs. Calculate the average depth of each grid, and calculate the absolute value of the depth difference between each grid and the package plane. If it is greater than 1cm, the package plane is damaged, otherwise it is intact. Considering that the edges of the package are mostly curved, the threshold of the peripheral grid can be appropriately increased. Add the area of the damaged grid to determine the damage degree, and return the package damage result and evidence picture.

[0125] 2) Package target detection result is a package of special-shaped parts:

[0126] The shape of the special-shaped part is irregular as a whole, such as a circle, or composed of multiple shapes. According to the depth map and the package detection box, the package depth map is obtained. Divide the package depth map into 1cm x 1cm grids, and calculate the average depth of each grid. Calculate the absolute value of the depth difference between each grid and the adjacent grid. When it is greater than 1cm, divide the two grids into 3x3 small grids, and calculate the absolute value of the depth difference between the two grids and the adjacent small grids. When it is greater than 0.5cm, the package is damaged, otherwise it is intact. Considering that the edges of the package are mostly curved, the threshold of the peripheral grid of the package detection box can be appropriately increased. Add the area of the damaged grid to determine the damage degree, and return the package damage result and evidence picture.

[0127] 3) Packages whose target detection result is a package collection:

[0128] The package may have an irregular shape but a uniform color. Damage can be determined by identifying the color. Images of the packages were collected at different times and under different lighting conditions. The colors of the packages were extracted to build a color database. The RGB image within the package detection frame was divided into 1cm x 1cm color blocks. The average color of each area was calculated, and the Euclidean distance similarity between the average color and each color in the package color database was calculated. When all similarities were less than 80%, damage was considered. The sum of the damaged grid areas gave the damaged area of ​​the package in the image. The damage of the three images from step 5 was evaluated sequentially. The largest damaged area among the three images was the damaged area of ​​the package, thus determining the degree of damage and returning the package damage result and evidence images.

[0129] 4) Package damage result returned:

[0130] Specifically, a package is considered slightly damaged if the damaged area is less than or equal to 10 square centimeters, and severely damaged if the damaged area is greater than 10 square centimeters. The system returns the package damage details and area. Package damage is categorized as intact, slightly damaged, or severely damaged. If the package is intact, the damaged area is 0. Damage levels can be customized based on business needs. The system returns the package damage details, damaged area, and damage photos.

[0131] 8. Results fusion:

[0132] The package damage image detection results from step 6 are fused and compared with the package damage detection results corresponding to different package types from step 7. The result with the highest damage level is taken as the final package damage result. If a package is damaged, the business system displays a package damage alarm and promptly notifies relevant production personnel for handling. Furthermore, it can be combined with an automatic sorting device to achieve automatic sorting of damaged packages.

[0133] This technical solution, by employing the above method, enables comprehensive inspection of multiple packages passing through the package conveyor belt in the express processing center using a binocular camera device incorporating intelligent package detection algorithms. This allows for the timely detection and handling of damaged packages, thereby improving customer satisfaction.

[0134] Example 4

[0135] Figure 4 This is a schematic diagram of a system load detection device provided in Embodiment 4 of the present invention. Figure Four As shown, the device includes: a package video acquisition module 401, a package information determination module 402, a first damage detection module 403, a second damage detection module 404, and a damage detection determination module 405.

[0136] The package video acquisition module 401 is configured to acquire package video information of a plurality of packages located on a conveyor belt by using a binocular shooting device, wherein the package video information comprises a plurality of video frames; the package information determination module 402 is configured to determine, according to the package video information, a plurality of target video frames corresponding to a same package, a first bounding box of the package in the target video frames and type information of the package from the plurality of video frames; the first damage detection module 403 is configured to determine, for a single package, a first damage detection result of the package according to a plurality of target video frames corresponding to the package and a damage detection model; the second damage detection module 404 is configured to determine a second damage detection result of the package according to the plurality of target video frames corresponding to the package and the type information of the package; and the damage detection determination module 405 is configured to determine a target damage detection result of the package according to the first damage detection result and the second damage detection result of the package.

[0137] The technical scheme of the embodiment of the present application, the parcel video acquisition module 401 acquires parcel video information of a plurality of parcels located on a conveyor belt through a binocular shooting device, wherein the parcel video information includes a plurality of video frames; thereby obtaining a plurality of video frame images of the parcels for subsequent parcel damage analysis. The parcel information determination module 402 can determine a plurality of target video frames corresponding to the same parcel, a first detection box of the parcel in the target video frame, and type information of the parcel from the plurality of video frames according to the parcel video information; by determining the target video frame, the first detection box, and the type information of the parcel, the parcel is comprehensively detected to ensure the accuracy of the parcel damage detection. The first damage detection module 403 determines the first damage detection result of the parcel according to the plurality of target video frames corresponding to the parcel and a damage detection model for a single parcel; the damage detection model is used to analyze the plurality of video frame images, thereby realizing efficient parcel damage detection. The second damage detection module 404 can determine the second damage detection result of the parcel according to the plurality of target video frames corresponding to the parcel and the type information corresponding to the parcel; by adopting different damage detection methods for different types of parcels, more accurate damage information of different types of parcels can be obtained, thereby ensuring the accuracy and wide applicability of the parcel damage detection method. Finally, the damage detection determination module 405 determines the target damage detection result of the parcel according to the first damage detection result and the second damage detection result of the parcel. By comparing the damage detection results determined in different ways, the authenticity and reliability of the final damage identification detection result can be ensured. The technical scheme can effectively improve the parcel damage identification and detection capability, ensure the reliability of the final parcel damage detection result, and improve the user's satisfaction by timely processing the damaged parcels.

[0138] On the basis of the above-mentioned optional technical solutions, the first damage detection module 403 can further include a parcel detection unit, a parcel tracking unit, and a target video frame determination unit. The parcel detection unit is configured to perform parcel detection on the plurality of video frames based on a parcel detection model to obtain a first detection box and type information of each parcel in each video frame. The parcel tracking unit is configured to track the detected plurality of parcels based on a preset tracking algorithm to obtain parcel identifiers of the plurality of parcels, and determine the first detection box of the target parcel in the plurality of video frames based on the parcel identifiers. The target video frame determination unit is configured to determine a plurality of target video frames corresponding to each parcel from the plurality of video frames according to the first detection box of each parcel in the plurality of video frames.

[0139] On the basis of each of the optional technical solutions above, optionally, the target video frame determination unit can further include a first target video frame determination subunit and a second target video frame determination subunit. The first target video frame determination subunit is configured to determine the distance between the first detection box of the package in at least some of the video frames and the center point of the video frame, respectively, and determine the target video frame corresponding to the package from the plurality of video frames according to the distances corresponding to the plurality of video frames. The second target video frame determination subunit is configured to determine the target video frame corresponding to the package from the plurality of video frames according to the capture time of the plurality of video frames and the height of the first detection box of the package in the video frame.

[0140] On the basis of each of the optional technical solutions above, optionally, the first damage detection module 403 can further include a third damage detection determination unit, a fourth damage detection determination unit, and a first damage detection determination unit. The third damage detection determination unit is configured to provide the target video frame corresponding to the package to a damage detection model for a single target video frame to obtain a third damage detection result corresponding to the target video frame. The fourth damage detection determination unit is configured to determine a fourth damage detection result of the target video frame according to the second detection box and the first detection box of the package in the target video frame in response to the third damage detection result including the second detection box. The second detection box is used to indicate the image region where the damaged package is detected. The first damage detection determination unit is configured to determine the first damage detection result of the package according to the fourth damage detection results of the plurality of target video frames corresponding to the package.

[0141] On the basis of each of the optional technical solutions above, optionally, the fourth damage detection determination unit can further include a coincidence degree determination subunit. The coincidence degree determination subunit is configured to determine the coincidence degree of the second detection box and the first detection box of the package in the target video frame, and determine the fourth damage detection result of the target video frame according to the coincidence degree.

[0142] On the basis of each of the optional technical solutions described above, the second damage detection module 404 can also include a fifth damage detection determination unit and a second damage detection determination unit. The fifth damage detection determination unit is configured to, in response to the type information corresponding to the package being a first type, for each target video frame, determine first color information of the package according to the first detection box of the package in the target video frame and pixel color information in the target video frame, and determine a fifth damage detection result of the target video frame according to a similarity between the first color information and second color information stored in a color database. The first type includes a bale type. The second damage detection determination unit is configured to determine a second damage detection result of the package according to the fifth damage detection results of the plurality of target video frames corresponding to the package.

[0143] On the basis of each of the optional technical solutions described above, the second damage detection module 404 can also include a first depth image determination unit, a second depth image determination unit, and a second damage detection determination unit. The package video information includes a left-eye package image and a right-eye package image corresponding to each video frame. The package video information includes a left-eye package image and a right-eye package image corresponding to each video frame. The first depth image determination unit is configured to, in response to the type information corresponding to the package being a second type, determine a first depth image corresponding to each target video frame according to the left-eye package image and the right-eye package image corresponding to each target video frame, respectively. The second depth image determination unit is configured to, for each video frame, determine a second depth image corresponding to the package from the first depth image corresponding to the target video frame according to the first detection box of the package in the target video frame. The second damage detection determination unit is further configured to determine a second damage detection result of the package according to the second depth images corresponding to the package.

[0144] On the basis of each of the optional technical solutions above, optionally, the second damage detection determination unit can further include: a parcel plane extraction unit, a parcel plane damage detection unit, and a second damage detection determination subunit. The second type includes a first sub-type; the first sub-type includes at least one of a carton type, a soft package type, and a document type; the parcel plane extraction unit is configured to, for each of the second depth images corresponding to the parcel, perform plane extraction on the second depth image according to a preset multi-plane extraction algorithm to obtain a plurality of target parcel planes corresponding to the second depth image; the parcel plane damage detection unit is configured to, for each of the second depth images, divide the second depth image into a plurality of first grids, determine an average depth of each of the first grids, determine a grid damage detection result of each of the first grids according to the average depth of each of the first grids in the target parcel plane and an average depth of the target parcel plane, and determine a plane damage detection result of the target parcel plane according to the grid damage detection results of the plurality of first grids in the target parcel plane; and the second damage detection determination subunit is configured to determine an image damage detection result of the second depth image according to the plane grid damage detection results of the plurality of target parcel planes corresponding to the second depth image, and determine a second damage detection result of the parcel according to the image damage detection results of the plurality of second depth images.

[0145] On the basis of each of the optional technical solutions above, optionally, the second damage detection determination unit can further include: a second grid depth determination unit, an adjacent grid depth difference calculation unit, a second depth image damage detection unit, and a second damage detection determination subunit. The second type includes a second sub-type; the second sub-type includes a special-shaped part; the second grid depth determination unit is configured to, for each of the second depth images corresponding to the parcel, divide the second depth image into a plurality of second grids, and determine an average depth of each of the second grids; the adjacent grid depth difference calculation unit is configured to, for each of the second grids, determine a grid damage detection result of the second grid according to a depth difference between the average depth of the second grid and the average depth of an adjacent second grid of the second grid; the second depth image damage detection unit is configured to determine an image damage detection result of the second depth image according to the grid damage detection results of the plurality of second grids in the second depth image; and the second damage detection determination subunit is further configured to determine a second damage detection result of the parcel according to the image damage detection results of the plurality of second depth images.

[0146] The package damage detection device provided in this embodiment of the invention can execute the package damage detection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the package damage detection method. Technical details not described in detail in this embodiment can be found in any of the package damage detection methods described in this invention.

[0147] Example 5

[0148] Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0149] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0150] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0151] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The processor 11 performs various methods and processes described above, such as a package damage detection method.

[0152] In some embodiments, a package damage detection method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of a package damage detection method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform a package damage detection method by any other suitable means, such as by means of firmware.

[0153] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0154] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a machine or a remote machine or a server.

[0155] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0157] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0158] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0159] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with embodiments of the application. For example, embodiments of the application include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program comprising program code for executing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-described functions defined in the methods of the embodiments of the application are performed.

[0160] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present application. For example, the steps recited in the present application can be performed in parallel, in series, or in a different order, without departing from the desired results of the technical solutions of the present application, and this is not limited herein.

[0161] The above detailed description does not constitute a limitation on the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting package damage, characterized in that, include: The video information of multiple packages located on the conveyor belt is acquired by a binocular camera device, wherein the video information of the packages includes multiple video frames; Based on the package video information, multiple target video frames corresponding to the same package, a first detection box of the package in the target video frame, and the type information of the package are determined from the multiple video frames; For a single package, a first damage detection result for the package is determined based on multiple target video frames corresponding to the package and a damage detection model; The second damage detection result of the package is determined based on the multiple target video frames corresponding to the package and the type information corresponding to the package; The target damage detection result of the package is determined based on the first damage detection result and the second damage detection result of the package.

2. The method for detecting package damage according to claim 1, characterized in that, The step of determining multiple target video frames corresponding to the same package, a first detection box of the package in the target video frames, and the type information of the package from the multiple video frames based on the package video information includes: Package detection is performed on multiple video frames based on a package detection model to obtain the first detection box and type information of each package in each video frame. The multiple detected packages are tracked based on a preset tracking algorithm to obtain package identifiers for the multiple packages, and the first detection box of the target package in the multiple video frames is determined based on the package identifiers. Based on the first detection box that is wrapped in each of the multiple video frames, a plurality of target video frames corresponding to the package are determined from the plurality of video frames.

3. The method for detecting package damage according to claim 2, characterized in that, The step of determining multiple target video frames corresponding to the package from the multiple video frames based on the first detection box of each of the multiple video frames includes at least one of the following: The distance between the first detection box of the package and the center point of the video frame is determined in at least a portion of the video frames respectively, and the target video frame corresponding to the package is determined from the multiple video frames based on the distances corresponding to the multiple video frames; The target video frame corresponding to the package is determined from the multiple video frames based on the acquisition time of the multiple video frames and the height of the first detection frame of the package in the video frame.

4. The method for detecting package damage according to claim 1, characterized in that, The step of determining the first damage detection result of the package based on the multiple target video frames corresponding to the package and the damage detection model includes: For a single target video frame, the target video frame corresponding to the package is provided to the damage detection model to obtain a third damage detection result corresponding to the target video frame; In response to the third damage detection result including the second detection box, a fourth damage detection result of the target video frame is determined based on the second detection box and the first detection boxes of multiple packages in the target video frame; wherein, the second detection box is used to indicate the image area where the detected damaged package is located; The first damage detection result of the package is determined based on the fourth damage detection result of the multiple target video frames corresponding to the package.

5. The method for detecting package damage according to claim 4, characterized in that, The step of determining the fourth damage detection result of the target video frame based on the second detection frame and the first detection frames of multiple packages in the target video frame includes: Determine the overlap between the second detection frame and the first detection frame wrapped in the target video frame, and determine the fourth damage detection result of the target video frame based on the overlap.

6. The method for detecting package damage according to claim 1, characterized in that, The step of determining the second damage detection result of the package based on the multiple target video frames corresponding to the package and the type information corresponding to the package includes: In response to the package's type information being a first type, for each target video frame, the first color information of the package is determined based on the first detection box of the package in the target video frame and the pixel color information in the target video frame. The fifth damage detection result of the target video frame is determined based on the similarity between the first color information and multiple second color information stored in the color database. The first type includes package type. The second damage detection result of the package is determined based on the fifth damage detection result of the multiple target video frames corresponding to the package.

7. The method for detecting package damage according to claim 1, characterized in that, The package video information includes a left-view package image and a right-view package image corresponding to each video frame; determining the second damage detection result of the package based on the multiple target video frames corresponding to the package and the type information corresponding to the package includes: In response to the type information corresponding to the package being a second type, a first depth image corresponding to each target video frame is determined based on the left-eye package image and the right-eye package image corresponding to each target video frame; For each video frame, a second depth image corresponding to the package is determined from the first depth image corresponding to the target video frame based on the first detection box wrapped in the target video frame; The second damage detection result of the package is determined based on multiple second depth images corresponding to the package.

8. The method for detecting package damage according to claim 7, characterized in that, The second type includes a first subtype; the first subtype includes at least one of cardboard box type, soft packaging type, and document type; determining the second damage detection result of the package based on multiple second depth images corresponding to the package includes: For each second depth image corresponding to the package, a preset multi-plane extraction algorithm is used to extract planes from the second depth image to obtain multiple target package planes corresponding to the second depth image; For each second depth map, the second depth map is divided into multiple first grids, the average depth of each first grid is determined, the grid damage detection result of each first grid is determined based on the average depth of each first grid in the target wrapping plane and the average depth of the target wrapping plane, and the plane damage detection result of the target wrapping plane is determined based on the grid damage detection results of multiple first grids in the target wrapping plane; The image damage detection result of the second depth image is determined based on the planar mesh damage detection results of the multiple target package planes corresponding to the second depth image, and the second damage detection result of the package is determined based on the image damage detection results of multiple second depth images.

9. The method for detecting package damage according to claim 7, characterized in that, The second type includes a second subtype; the second subtype includes irregularly shaped parts; determining the second damage detection result of the package based on multiple second depth images corresponding to the package includes: For each second depth image corresponding to the package, the second depth image is divided into multiple second grids, and the average depth of each second grid is determined. For each second grid, the grid damage detection result is determined based on the depth difference between the average depth of the second grid and the average depth of its adjacent second grids. The image damage detection result of the second depth image is determined based on the mesh damage detection results of multiple second grids in the second depth image; The second damage detection result of the package is determined based on the image damage detection results of multiple second depth images.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the package damage detection method as described in any one of claims 1-9.

Images