Unmanned aerial vehicle cargo monitoring method and device based on image recognition and electronic equipment

Abstract

The present application relates to warehouse cargo monitoring technical field, especially in kind based on image recognition's unmanned aerial vehicle cargo monitoring method, device and electronic equipment. The method comprises: constructing warehouse three-dimensional point cloud map and demarcating reference observation point; control unmanned aerial vehicle to execute routine cruise and carry out preliminary state screening to cargo, to trigger depth inspection; after triggering depth inspection, control unmanned aerial vehicle to execute multi-angle patrol and carry out depth inspection to cargo;Based on the results of preliminary state screening and depth inspection to cargo, construct comprehensive evaluation model to determine cargo state. The present application guides unmanned aerial vehicle to collect the image of target cargo from different preset visual angle, and extracts a group of geometric and texture characteristic parameters that can specifically represent the physical integrity of packaging box from it, judges whether its state is abnormal through a layered decision logic, and outputs abnormal type and quantitative index.

Description

Technical Field

[0001] This invention relates to the field of warehouse cargo monitoring technology, and in particular to a method, device and electronic equipment for unmanned aerial vehicle (UAV) cargo monitoring based on image recognition. Background Technology

[0002] With the rapid development of automation and intelligence in the logistics and warehousing industry, efficient and accurate status monitoring and management of massive inventories has become crucial for improving warehouse operational efficiency and reducing the risk of cargo damage. Drones, due to their flexible maneuverability and wide field of view, have shown great potential in warehouse inspection. Meanwhile, advancements in computer vision and 3D perception technologies have made it possible to automatically analyze cargo status through images. Therefore, developing a system that can autonomously, comprehensively, and intelligently monitor cargo status by combining drone platforms with advanced image recognition technology aligns with the development trend of modern smart logistics and has significant practical application value.

[0003] Existing warehouse cargo monitoring methods suffer from the following drawbacks: First, relying on fixed cameras or manual inspections results in large blind spots, poor flexibility, high labor costs, and low efficiency. Second, existing automated monitoring solutions often focus on image analysis from a single angle, making it difficult to comprehensively assess the three-dimensional condition of goods. This limited detection dimension makes them prone to missed detections. Third, existing methods typically only assess single types of defects, lacking a comprehensive quantitative assessment and fusion decision-making mechanism that considers multiple characteristics such as cargo appearance flatness, surface cleanliness, structural integrity, and label clarity. This leads to insufficient accuracy and robustness in condition determination, failing to provide refined anomaly classification information for warehouse management. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, and electronic device for monitoring cargo from unmanned aerial vehicles (UAVs) based on image recognition, so as to solve at least one of the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for monitoring cargo on unmanned aerial vehicles (UAVs) based on image recognition, comprising:

[0007] Construct a 3D point cloud map of the warehouse and calibrate the baseline observation points;

[0008] Control the drone to perform routine patrols and conduct preliminary condition screening of the cargo to trigger in-depth inspections;

[0009] After triggering the in-depth inspection, control the drone to perform multi-angle patrols and conduct in-depth inspections of the cargo;

[0010] Based on the results of preliminary screening and in-depth inspection of the goods, a comprehensive evaluation model is constructed to determine the condition of the goods.

[0011] Furthermore, the warehouse is scanned using a lidar mounted on a drone to obtain high-precision point cloud data. All shelves in the warehouse are treated as cubes, and each shelf has length, width, height and spatial position to construct a set of cubes. Each cube in the set of cubes corresponds to a standard storage location. The high-precision point cloud data containing the set of cubes, aisles and column structures is output as a 3D point cloud map of the warehouse.

[0012] For each standard storage location in the warehouse's 3D point cloud map, the 3D coordinates of three virtual points around it are calculated, including the 3D coordinates of the front observation point, the side observation point, and the top observation point. These three coordinate positions are then used as the reference observation points.

[0013] For the front observation point, take the three-dimensional point cloud coordinates of the center line of the aisle directly in front of the standard storage location and the distance from the front of the shelf equal to the safety distance.

[0014] For the side observation point, take the standard storage location side passage, with the observation axis parallel to the shelf surface and pointing to the side of the storage location, and the three-dimensional point cloud coordinates at a distance from the side of the shelf equal to the safety distance.

[0015] For the top observation point, take the three-dimensional point cloud coordinates directly above the standard cargo location, at a distance equal to the safety distance from the top of the cargo location.

[0016] Furthermore, the front view image is grayscaled and Gaussian filtered for noise reduction. An edge detection algorithm is used to obtain edge pixels in the front view image, and a contour search algorithm is used to find the quadrilateral contour representing the outer contour of the cargo composed of edge pixels. The found quadrilateral contour is corrected. Using the known true rectangular shape of the standard cargo location in the 3D point cloud map and the camera pose, the quadrilateral contour is transformed by perspective and corrected into a standard rectangle.

[0017] The pixels on the four sides of the standard rectangle are classified into four sets: top, bottom, left, and right. The coordinates of the pixels in each set are fitted with a least squares line to obtain four fitted lines. For each pixel on the standard rectangle, the vertical distance from it to the fitted line corresponding to its side is calculated, and the standard deviation of the vertical distance is used as the flatness deviation of the packaging surface. When the flatness deviation of the packaging surface is greater than the deviation threshold, a depth check is triggered.

[0018] Furthermore, the front view image is converted from RGB to Lab color space. In Lab space, the L component represents lightness, and the a and b components represent color. A standard color model is constructed based on the preset front view image: the (L,a,b) values ​​of all pixels in the pure area of ​​the preset front view image are extracted. By calculating the mean and covariance matrix of (L,a,b) respectively, a Gaussian distribution model of the color of the intact packaging surface is established to obtain the standard color model.

[0019] For the front view image to be inspected, within its corresponding standard rectangular area, the Lab value vector of its pixels is set to x=[L,a,b], and the Mahalanobis distance between x and the standard color model is calculated pixel by pixel. The area formed by pixels whose Mahalanobis distance is greater than the color distance threshold is marked as an abnormal area. The ratio of the number of pixels in the abnormal area to the number of pixels in the standard rectangular area is used as the outer surface stain coverage rate. When the outer surface stain coverage rate is greater than the coverage rate threshold, a depth check is triggered.

[0020] Furthermore, based on the location of the standard cargo location in the 3D point cloud map, the pose of the UAV at the side observation point, and the internal parameters of the camera, the projection line segment of the vertical edge of the cargo near the camera is accurately located in the side view image. Then, with this projection line segment as the center, it is extended 20 pixels to both sides, upward to the top of the cargo, and downward to the bottom of the cargo, to extract a rectangular feature area.

[0021] A corner detection algorithm is used within the feature region to detect corners by calculating the grayscale changes as the feature region moves in various directions. The point with the highest response function R value is marked as a corner, and its sharpness is evaluated to obtain sharpness parameters.

[0022] The feature region is divided into multiple 5×5 local regions, and the average of the maximum R values ​​in each local region is calculated as the division mean. The ratio of the R value of the corner point to the division mean is used as the sharpness parameter.

[0023] The position offset of the corner point is evaluated to obtain the offset distance parameter: the vertical distance from the corner point to the projected line segment is calculated as the offset distance parameter;

[0024] The integrity of corner points is evaluated based on sharpness and offset distance parameters to determine the corner integrity score.

[0025] Furthermore, the marker area on the front view, side view, or top view image is converted into a grayscale image. The Brenner gradient is used to calculate the squared difference of grayscale between each pixel and the point two pixels away. The average of the squared differences of grayscale between all valid pixels in the marker area is calculated as the marker sharpness, and the marker blur index is determined based on the marker sharpness.

[0026] Furthermore, the packaging surface flatness deviation, outer surface stain coverage, corner integrity score, and label fuzziness index are mapped to the [0,1] region respectively:

[0027] For the flatness deviation of the packaging surface, set the mapped flatness deviation of the packaging surface to GA, GA=min(flatness deviation of packaging surface / deviation threshold,1);

[0028] For the external surface stain coverage rate, set the mapped external surface stain coverage rate to GB, GB=min(surface stain coverage rate / coverage threshold,1);

[0029] For corner integrity scoring, the mapped corner integrity score is set to GC, where GC = 1 - Sc;

[0030] For the identifier fuzziness index, set the mapped identifier fuzziness index to GD, where GD=F;

[0031] The flatness deviation of the mapped packaging surface, the stain coverage of the outer surface, the corner integrity score, and the label fuzziness index are weighted and fused to determine the cargo anomaly index.

[0032] Furthermore, when the cargo anomaly index is greater than or equal to 0.5, the cargo status is determined to be abnormal, and GA, GB, GC, and GD are traversed to find the one with the largest value to mark its anomaly type; otherwise, the cargo status is determined to be intact.

[0033] On the other hand, the present invention also provides an image recognition-based drone cargo monitoring device, comprising:

[0034] The map building unit is used to construct a 3D point cloud map of the warehouse and to pinpoint the baseline observation points.

[0035] The preliminary screening unit is used to control the drone to perform routine patrols and conduct preliminary condition screening of the cargo in order to trigger a deeper inspection;

[0036] The depth inspection unit is used to control the drone to perform multi-angle patrols and conduct in-depth inspections of the cargo after a depth inspection is triggered.

[0037] The status determination unit is used to construct a comprehensive evaluation model based on the results of preliminary status screening and in-depth inspection of the goods in order to determine the status of the goods.

[0038] On the other hand, the present invention also provides an electronic device, the electronic device comprising:

[0039] One or more processors;

[0040] Storage device for storing one or more programs;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement the image recognition-based drone cargo monitoring method as described above.

[0042] The beneficial effects of this invention are as follows: This invention achieves full automation of the entire process from environmental modeling and cruise route planning to image acquisition, reducing the cost and safety risks of manual inspection. Through a two-stage strategy of preliminary screening + in-depth inspection, it accurately focuses on abnormal targets while ensuring monitoring coverage, achieving the optimal balance between monitoring efficiency and depth. By constructing a comprehensive evaluation model, it scientifically integrates multi-dimensional information such as appearance, structure, and markings, making the cargo status judgment results more comprehensive, objective, and reliable. It can provide strong technical support for cargo damage detection, inventory counting, and quality control in warehouse management, effectively improving the intelligence level and operational efficiency of warehouse management. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the image recognition-based drone cargo monitoring method in this embodiment.

[0045] Figure 2 This is a flowchart of the preliminary status screening method for goods in this embodiment.

[0046] Figure 3 This is a schematic diagram of the image recognition-based drone cargo monitoring device in this embodiment.

[0047] Figure 4 This is a schematic diagram of the electronic device in this embodiment. Detailed Implementation

[0048] The image recognition-based drone cargo monitoring method, apparatus, and electronic device disclosed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated; they can be combined to achieve better technical effects. In the accompanying drawings of the following embodiments, the same reference numerals in each drawing represent the same features or components, which can be applied to different embodiments. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0049] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings are merely for illustrative purposes and to aid those skilled in the art in understanding and reading the invention. They are not intended to limit the conditions under which the invention can be implemented. Any modifications to the structure, changes in proportions, or adjustments to size, provided they do not affect the effectiveness or purpose of the invention, should fall within the scope of the technical content disclosed in the invention. The scope of the preferred embodiments of the present invention includes other implementations, wherein functions may be performed not in the order stated or discussed, including substantially simultaneously or in reverse order, depending on the functions involved. This should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0050] Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0051] In the description of the embodiments of this application, " / " means "or", and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" means: A and B exist alone, B exists alone, and A and B exist simultaneously. In the description of the embodiments of this application, "multiple" refers to two or more embodiments.

[0052] Please see Figure 1 As shown, this is an image recognition-based drone cargo monitoring method according to this embodiment, including:

[0053] Step S1: Construct a 3D point cloud map of the warehouse and calibrate the benchmark observation points.

[0054] Specifically, in step S1 of this embodiment, a lidar mounted on a drone is used to scan the warehouse to obtain high-precision point cloud data. All shelves in the warehouse are treated as cubes, each with length, width, height dimensions, and spatial location, to construct a set of cubes. Each cube in the set corresponds to a standard storage location. The high-precision point cloud data, including the cube set, aisles, and column structures, is output as a 3D point cloud map of the warehouse. This 3D point cloud map can also be generated by the drone performing multiple traversal flights using a visual SLAM algorithm, calculating the drone's own trajectory based on image sequences captured by its camera, and then generating the 3D point cloud map.

[0055] Specifically, in step S1 of this embodiment, for each standard storage location in the warehouse 3D point cloud map, the 3D coordinates of three virtual points around it are calculated, including the 3D coordinates of the front observation point, the side observation point, and the top observation point, and these three coordinate positions are used as reference observation points:

[0056] For the front observation point, take the three-dimensional point cloud coordinates of the center line of the aisle directly in front of the standard storage location and the distance from the front of the shelf equal to the safety distance.

[0057] For the side observation point, take the standard storage location side passage, with the observation axis parallel to the shelf surface and pointing to the side of the storage location, and the three-dimensional point cloud coordinates at a distance from the side of the shelf equal to the safety distance.

[0058] For the top observation point, the three-dimensional point cloud coordinates are taken directly above the standard cargo location, at a distance equal to the safety distance from the top of the cargo location. The safety distance is a preset parameter, the value of which is calculated based on the drone's hovering accuracy, camera focal length, and the required field of view. For example, based on the camera's FOV angle "θ" and the cargo location width "Wb", the minimum distance is calculated using the formula Ds≥(Wb / 2) / tan(θ / 2), and then a margin of, such as 0.2 meters, is added as the final safety distance.

[0059] Specifically, in step S1 of this embodiment, a high-precision 3D point cloud map of the warehouse is constructed using LiDAR or visual SLAM technology, and the shelves are abstracted as a set of cubes, providing an accurate digital spatial reference for cargo monitoring. By intelligently calculating the 3D coordinates of three reference observation points—front, side, and top—for each standard storage location, the standardization, automation, and repeatability of subsequent drone patrols and image acquisition are ensured. This not only avoids the tediousness and errors of manually setting observation points, but also automatically plans safe, compliant, and optimal observation positions based on the actual geometry of the shelves, laying the foundation for the acquisition of high-quality image data.

[0060] Please continue reading. Figure 1 As shown, the image recognition-based drone cargo monitoring method further includes:

[0061] Step S2: Control the drone to perform a routine patrol and conduct a preliminary condition screening of the cargo to trigger a deeper inspection.

[0062] Please see Figure 2 As shown, this is a preliminary condition screening method for goods, including:

[0063] Step S21: Control the drone to perform a regular cruise to take frontal view images.

[0064] Specifically, in step S21 of this embodiment, during the routine patrol of the drone, the drone flies to the front observation point of the standard cargo location and adjusts its attitude so that the camera optical axis is aligned with the center of the standard cargo location in order to take a frontal view image.

[0065] Please continue reading. Figure 2 As shown, the preliminary condition screening method for the goods further includes:

[0066] Step S22: Determine the flatness deviation of the packaging surface based on the front view image.

[0067] Specifically, in step S22 of this embodiment, the front view image is converted to grayscale and subjected to Gaussian filtering for noise reduction. An edge detection algorithm is used to obtain edge pixels in the front view image, and a contour search algorithm is used to find a quadrilateral contour representing the outer contour of the cargo, composed of edge pixels. The found quadrilateral contour is then corrected. Using the known true rectangular shape of the standard cargo location in the 3D point cloud map and the camera pose, a perspective transformation is performed on the quadrilateral contour to correct it into a standard rectangle. In this embodiment, the edge detection algorithm uses Canny edge detection, and the contour search algorithm uses the Suzuki85 algorithm. The purpose of contour correction in this embodiment is that, due to perspective projection, the quadrilateral contour may be a trapezoidal contour, which needs to be corrected. When the found quadrilateral contour is a standard rectangle, no correction is required.

[0068] Specifically, in step S22 of this embodiment, the pixels on the four sides of the standard rectangle are classified into four sets: top, bottom, left, and right. Least squares line fitting is performed on the pixel coordinates of each set to obtain four fitted lines. For each pixel on the standard rectangle, the vertical distance from it to the fitted line corresponding to its side is calculated, and the standard deviation of the vertical distance is used as the packaging surface flatness deviation. When the packaging surface flatness deviation exceeds a deviation threshold, a depth check is triggered. The deviation threshold is obtained by collecting a large number of known images of the front of goods in good condition, slightly dented, or severely dented, calculating their packaging surface flatness deviations, and observing their distribution. The 95th percentile of the distribution of the flatness deviation of packaging surfaces of goods in good condition is used as the deviation threshold.

[0069] Please continue reading. Figure 2 As shown, the preliminary condition screening method for the goods further includes:

[0070] Step S23: Determine the stain coverage of the outer surface based on the front view image.

[0071] Specifically, in step S23 of this embodiment, the front view image is converted from RGB to Lab color space. In Lab space, the L component represents lightness, and the a and b components represent color. A standard color model is constructed based on the preset front view image: the (L,a,b) values ​​of all pixels in the pure area of ​​the preset front view image are extracted. By calculating the mean and covariance matrix of (L,a,b) respectively, a Gaussian distribution model of the color of the intact packaging surface is established to obtain the standard color model.

[0072] Specifically, in step S23 of this embodiment, for the front view image to be detected, within its corresponding standard rectangular area, the Lab value vector of its pixels is set to x=[L,a,b], and the Mahalanobis distance between x and the standard color model is calculated pixel by pixel. Areas consisting of pixels whose Mahalanobis distance is greater than the color distance threshold are marked as abnormal areas. The ratio of the number of pixels in the abnormal areas to the number of pixels in the standard rectangular area is used as the outer surface stain coverage rate. When the outer surface stain coverage rate is greater than the coverage rate threshold, a depth check is triggered. The color distance threshold is determined by the false acceptance rate (FAR) of the standard color model on the validation set. For example, the Mahalanobis distance value when the FAR is 1% is set as the color distance threshold. The coverage rate threshold is set according to management requirements, stipulating that a stain area exceeding a certain percentage, such as 2%, on the packaging surface is considered abnormal.

[0073] Specifically, in step S2 of this embodiment, the drone automatically captures a frontal view image from a preset observation point, and computer vision algorithms are used to analyze the flatness and color cleanliness of the packaging surface, achieving rapid and automated initial screening of the goods' appearance. Correcting the packaging surface contour to a standard rectangle and calculating flatness deviations effectively detects deformations such as dents and bulges in the box. Simultaneously, a standard color model established using the Lab color space and Mahalanobis distance can sensitively identify surface stains, water stains, and other contaminants. This dual screening mechanism efficiently identifies potential anomalies during routine patrols, triggering in-depth inspections only for suspected problematic goods, greatly optimizing the allocation of monitoring resources and improving overall monitoring efficiency.

[0074] Please continue reading. Figure 1 As shown, the image recognition-based drone cargo monitoring method further includes:

[0075] Step S3: After triggering the deep inspection, control the drone to perform multi-angle patrol and deep inspection of the cargo.

[0076] Specifically, in step S3 of this embodiment, the drone is controlled to fly to the side observation point and top observation point of the standard cargo location in sequence to take side view and top view images. Based on the position of the standard cargo location in the three-dimensional point cloud map, the pose of the drone at the side observation point and the internal parameters of the camera, the projection line segment of the vertical edge of the cargo on the side closest to the camera is accurately located in the side view image. With the projection line segment as the center, it extends 20 pixels to both sides, upward to the top of the cargo and downward to the bottom of the cargo, to capture a rectangular feature area.

[0077] A corner detection algorithm is used within the feature region to detect corners by calculating the grayscale changes as the feature region moves in various directions. The point with the highest response function R value is marked as a corner, and its sharpness is evaluated to obtain sharpness parameters.

[0078] The feature region is divided into multiple 5×5 local regions, and the average of the maximum R values ​​in each local region is calculated as the division mean. The ratio of the R value of the corner point to the division mean is used as the sharpness parameter.

[0079] The position offset of the corner point is evaluated to obtain the offset distance parameter: the vertical distance from the corner point to the projected line segment is calculated as the offset distance parameter;

[0080] The integrity of corner points is evaluated based on the sharpness parameter and the offset distance parameter to determine the corner integrity score. The expression for the corner integrity score is: Sc = f1(Rc) × f2(Do), where Sc represents the corner integrity score, Rc represents the sharpness parameter, Do represents the offset distance parameter, f1(Rc) is a mapping function from 0 to 1. When Rc is greater than 3, f1(RC) = 1; when Rc is less than or equal to 3 and greater than or equal to 1.5, the value of f1(RC) is linearly mapped to [0.5, 1]; when Rc is less than 1.5, f1(RC) = 0.3. f2(Do) is a mapping function from 0 to 1. When Do is less than 2, f2(Do) = 1; when Do is greater than or equal to 2 and less than or equal to 10, the value of f2(Do) is linearly mapped to [1, 0.2]; when Do is greater than 10, f2(Do) = 0.2. In this embodiment, the Harris corner detection algorithm is used to detect corner points.

[0081] Specifically, in step S3 of this embodiment, when using the Harris corner detection algorithm to detect corners, if no obvious corner is detected, the corner integrity score is 0. If the sharpness parameter is less than 1, it is determined that no obvious corner is detected.

[0082] Specifically, in step S3 of this embodiment, for the top observation point, the above steps can be used to accurately locate the projection line segment of the horizontal edge of the cargo in the top view image based on the position of the standard cargo location in the three-dimensional point cloud map, the pose of the UAV at the top observation point, and the internal parameters of the camera, and to evaluate the integrity of the corner points and determine the corner point integrity score.

[0083] Specifically, in step S3 of this embodiment, the marking area on the front view, side view, or top view image is converted into a grayscale image. The Brenner gradient is used to calculate the squared difference of grayscale between each pixel and the pixel two pixels away. The average of the squared differences of grayscale between all valid pixels in the marking area is calculated as the marking sharpness. Based on the marking sharpness, a marking blur index is determined. The expression for the marking blur index is: F = 1 - min(CL / Cb, 1), where F represents the marking blur index, CL represents the marking sharpness, and Cb represents the baseline sharpness. The baseline sharpness is obtained by selecting the most recent image of a well-preserved item from the historical inspection records of the standard cargo location and calculating its marking sharpness using the same method.

[0084] Specifically, in step S3 of this embodiment, after the alarm is triggered, the drone is automatically controlled to fly to the side and top observation points for multi-angle inspection, realizing in-depth quantitative analysis of the structural details and label clarity of the goods. By combining a 3D point cloud map to accurately locate the projection of the edges of the goods in the image and focusing on feature areas for corner detection and evaluation, it is possible to accurately determine whether there is structural damage such as breakage or bumps on the outer packaging of the goods. At the same time, by using the Brenner gradient to evaluate the label clarity, problems such as label wear and blurring can be effectively identified. This multi-angle, multi-feature in-depth inspection makes up for the limitations of single-angle screening, making the assessment of the condition of the goods more comprehensive and accurate.

[0085] Please continue reading. Figure 1 As shown, the image recognition-based drone cargo monitoring method further includes:

[0086] Step S4: Based on the results of the preliminary status screening and in-depth inspection of the goods, a comprehensive evaluation model is constructed to determine the status of the goods.

[0087] Specifically, in step S4 of this embodiment, the packaging surface flatness deviation, outer surface stain coverage, corner integrity score, and label fuzziness index are mapped to the [0,1] range respectively:

[0088] For the flatness deviation of the packaging surface, set the mapped flatness deviation of the packaging surface to GA, GA=min(flatness deviation of packaging surface / deviation threshold,1);

[0089] For the external surface stain coverage rate, set the mapped external surface stain coverage rate to GB, GB=min(surface stain coverage rate / coverage threshold,1);

[0090] For corner integrity scoring, the mapped corner integrity score is set to GC, where GC = 1 - Sc;

[0091] For the identifier fuzziness index, set the mapped identifier fuzziness index to GD, where GD=F.

[0092] Specifically, in step S4 of this embodiment, the flatness deviation of the mapped packaging surface, the stain coverage rate of the outer surface, the corner integrity score, and the label fuzziness index are weighted and fused to determine the cargo anomaly index. The expression for the cargo anomaly index is: E=w1×GA+w2×GB+w3×GC+W4×GD, where E represents the cargo anomaly index, w1 represents the dent weight, w2 represents the stain weight, w3 represents the corner damage weight, and w4 represents the label fuzziness weight. In this embodiment, w1=0.3, w2=0.2, w3=0.4, and w4=0.1, and the sum of each weight should be equal to 1.

[0093] Specifically, in step S4 of this embodiment, when the cargo anomaly index is greater than or equal to 0.5, the cargo condition is determined to be abnormal, and the values ​​of GA, GB, GC, and GD are iterated to find the one with the largest value to mark its anomaly type; otherwise, the cargo condition is determined to be intact. When marking the anomaly type, if the value of GA or GC is the largest, it is marked as structural damage; if the value of GB is the largest, it is marked as surface contamination; and if the value of GD is the largest, it is marked as identification soiling. Structural damage can be further divided into minor dents and severe dents. If the value of GA or GC is within [0.3, 0.5], it is marked as minor dent; if the value of GA or GC is within (0.5, 1], it is marked as verification dent.

[0094] Specifically, in step S4 of this embodiment, multiple heterogeneous feature indicators obtained from preliminary screening and in-depth inspection are normalized, mapped, and weighted and fused to construct a comprehensive evaluation model. This model can integrate abnormal information from different dimensions into an intuitive cargo abnormality index and set scientific thresholds for automated status determination and abnormality type classification. This decision-making mechanism based on multi-source data fusion avoids the risk of misjudgment based on a single feature, significantly improves the robustness and reliability of status determination, and provides clear and quantifiable decision-making basis for warehouse management.

[0095] Please see Figure 3 As shown, this is an image recognition-based drone cargo monitoring device according to this embodiment, comprising:

[0096] The map building unit is used to construct a 3D point cloud map of the warehouse and to pinpoint the baseline observation points.

[0097] The preliminary screening unit is used to control the drone to perform routine patrols and conduct preliminary condition screening of the cargo in order to trigger a deeper inspection;

[0098] The depth inspection unit is used to control the drone to perform multi-angle patrols and conduct in-depth inspections of the cargo after a depth inspection is triggered.

[0099] The status determination unit is used to construct a comprehensive evaluation model based on the results of preliminary status screening and in-depth inspection of the goods in order to determine the status of the goods.

[0100] Please see Figure 4 As shown, it is a structural schematic diagram of an electronic device in this embodiment. The electronic device 60 in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0101] like Figure 4 As shown, the electronic device 60 may include a processing unit 61, which can perform various appropriate actions and processes to implement the methods of the embodiments described herein, based on a program stored in ROM 62 or a program loaded from storage device 68 into RAM 63. RAM 63 also stores various programs and data required for the operation of the electronic device 60. The processing unit 61, ROM 62, and RAM 63 are interconnected via bus 64. I / O interface 65 is also connected to bus 64. Typically, the following devices can be connected to I / O interface 65: input devices 66 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 67 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 68 including, for example, magnetic tapes, hard disks, etc.; and communication devices 69. Communication device 69 allows the electronic device 60 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 60 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0102] Specifically, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the methods as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 69, or installed from a storage device 68, or installed from a ROM 62. When the computer program is executed by the processing device 61, it performs the functions defined in the methods of the embodiments of the present invention.

[0103] Specifically, the computer-readable medium described in this embodiment may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0104] Specifically, the computer-readable medium described in this embodiment carries one or more programs. When the electronic device executes one or more of these programs, the electronic device causes the following: to construct a three-dimensional point cloud map of the warehouse and calibrate benchmark observation points; to control the drone to perform routine patrols and conduct preliminary status screening of the goods to trigger in-depth inspections; after triggering in-depth inspections, to control the drone to perform multi-angle patrols and conduct in-depth inspections of the goods; and to construct a comprehensive evaluation model based on the results of the preliminary status screening and in-depth inspections of the goods to determine the status of the goods.

[0105] In the above description, the disclosure of this invention is not intended to limit itself to these aspects. Rather, within the scope of the objectives of this disclosure, components can be selectively and operationally combined in any number. Furthermore, terms such as “comprising,” “encompassing,” and “having” should be interpreted by default as inclusive or open-ended, rather than exclusive or closed, unless explicitly defined as such. All technical, scientific, or other terms are to be understood by those skilled in the art, unless defined as such. Public terms found in dictionaries should not be interpreted in the context of the relevant technical documents in an overly idealistic or impractical manner, unless explicitly defined as such in this disclosure. Any modifications or alterations made by those skilled in the art based on the foregoing disclosure are within the scope of the claims.

Claims

1. A method for monitoring cargo using unmanned aerial vehicles (UAVs) based on image recognition, characterized in that, include: Construct a 3D point cloud map of the warehouse and calibrate the baseline observation points; Control the drone to perform routine patrols and conduct preliminary condition screening of the cargo to trigger in-depth inspections; After triggering the in-depth inspection, control the drone to perform multi-angle patrols and conduct in-depth inspections of the cargo; Based on the results of preliminary screening and in-depth inspection of the goods, a comprehensive evaluation model is constructed to determine the condition of the goods.

2. The image recognition-based UAV cargo monitoring method according to claim 1, characterized in that, The warehouse is scanned using a drone-mounted LiDAR to obtain high-precision point cloud data. All shelves in the warehouse are treated as cubes, and each shelf has length, width, height and spatial position to construct a set of cubes. Each cube in the set of cubes corresponds to a standard storage location. The high-precision point cloud data containing the set of cubes, aisles and column structure is output as a 3D point cloud map of the warehouse. For each standard storage location in the warehouse's 3D point cloud map, the 3D coordinates of three virtual points around it are calculated, including the 3D coordinates of the front observation point, the side observation point, and the top observation point. These three coordinate positions are then used as the reference observation points. For the front observation point, take the three-dimensional point cloud coordinates of the center line of the aisle directly in front of the standard storage location and the distance from the front of the shelf equal to the safety distance. For the side observation point, take the standard storage location side passage, with the observation axis parallel to the shelf surface and pointing to the side of the storage location, and the three-dimensional point cloud coordinates at a distance from the side of the shelf equal to the safety distance. For the top observation point, take the three-dimensional point cloud coordinates directly above the standard cargo location, at a distance equal to the safety distance from the top of the cargo location.

3. The image recognition-based UAV cargo monitoring method according to claim 2, characterized in that, The front view image is converted to grayscale and Gaussian filtered for noise reduction. An edge detection algorithm is used to obtain the edge pixels in the front view image. A contour search algorithm is used to find the quadrilateral contour representing the outer contour of the cargo composed of edge pixels. The quadrilateral contour is corrected. Using the known true rectangular shape of the standard cargo location in the 3D point cloud map and the camera pose, the quadrilateral contour is transformed by perspective and corrected into a standard rectangle. The pixels on the four sides of the standard rectangle are classified into four sets: top, bottom, left, and right. The coordinates of the pixels in each set are fitted with a least squares line to obtain four fitted lines. For each pixel on the standard rectangle, the vertical distance from it to the fitted line corresponding to its side is calculated, and the standard deviation of the vertical distance is used as the flatness deviation of the packaging surface. When the flatness deviation of the packaging surface is greater than the deviation threshold, a depth check is triggered.

4. The image recognition-based unmanned aerial vehicle (UAV) cargo monitoring method according to claim 3, characterized in that, The front view image is converted from RGB to Lab color space. In Lab space, the L component represents lightness and the a and b components represent color. A standard color model is constructed based on the preset front view image: the (L,a,b) values ​​of all pixels in the pure area of ​​the preset front view image are extracted. By calculating the mean and covariance matrix of (L,a,b) respectively, a Gaussian distribution model of the color of the intact packaging surface is established to obtain the standard color model. For the front view image to be inspected, within its corresponding standard rectangular area, the Lab value vector of its pixels is set to x=[L,a,b], and the Mahalanobis distance between x and the standard color model is calculated pixel by pixel. The area formed by pixels whose Mahalanobis distance is greater than the color distance threshold is marked as an abnormal area. The ratio of the number of pixels in the abnormal area to the number of pixels in the standard rectangular area is used as the outer surface stain coverage rate. When the outer surface stain coverage rate is greater than the coverage rate threshold, a depth check is triggered.

5. The image recognition-based UAV cargo monitoring method according to claim 4, characterized in that, Based on the location of the standard cargo location in the 3D point cloud map, the pose of the UAV at the side observation point, and the internal parameters of the camera, the projection line segment of the vertical edge of the cargo near the camera is accurately located in the side view image. Then, with this projection line segment as the center, it is extended 20 pixels to both sides, upward to the top of the cargo, and downward to the bottom of the cargo, to extract a rectangular feature area. A corner detection algorithm is used within the feature region to detect corners by calculating the grayscale changes as the feature region moves in various directions. The point with the highest response function R value is marked as a corner, and its sharpness is evaluated to obtain sharpness parameters. The feature region is divided into multiple 5×5 local regions, and the average of the maximum R values ​​in each local region is calculated as the division mean. The ratio of the R value of the corner point to the division mean is used as the sharpness parameter. The position offset of the corner point is evaluated to obtain the offset distance parameter: the vertical distance from the corner point to the projected line segment is calculated as the offset distance parameter; The integrity of corner points is evaluated based on sharpness and offset distance parameters to determine the corner integrity score.

6. The image recognition-based UAV cargo monitoring method according to claim 5, characterized in that, The marker area on the front view, side view, or top view image is converted into a grayscale image. The Brenner gradient is used to calculate the squared difference of grayscale between each pixel and the point two pixels away. The average of the squared differences of grayscale between all valid pixels in the marker area is calculated as the marker sharpness. The marker blur index is determined based on the marker sharpness.

7. The image recognition-based UAV cargo monitoring method according to claim 6, characterized in that, The packaging surface flatness deviation, outer surface stain coverage, corner integrity score, and label fuzziness index are respectively mapped to the [0,1] region: For the flatness deviation of the packaging surface, set the mapped flatness deviation of the packaging surface to GA, GA=min(flatness deviation of packaging surface / deviation threshold,1); For the external surface stain coverage rate, set the mapped external surface stain coverage rate to GB, GB=min(surface stain coverage rate / coverage threshold,1); For corner integrity scoring, the mapped corner integrity score is set to GC, where GC = 1 - Sc; For the identifier fuzziness index, set the mapped identifier fuzziness index to GD, where GD=F; The flatness deviation of the mapped packaging surface, the stain coverage of the outer surface, the corner integrity score, and the label fuzziness index are weighted and fused to determine the cargo anomaly index.

8. The image recognition-based unmanned aerial vehicle (UAV) cargo monitoring method according to claim 7, characterized in that, When the cargo anomaly index is greater than or equal to 0.5, the cargo status is determined to be abnormal. Then, GA, GB, GC, and GD are traversed to find the one with the largest value to mark its anomaly type. Otherwise, the cargo status is determined to be intact.

9. An image recognition-based drone cargo monitoring device, applied to the image recognition-based drone cargo monitoring method as described in any one of claims 1-8, characterized in that, include: The map building unit is used to construct a 3D point cloud map of the warehouse and to pinpoint the baseline observation points. The preliminary screening unit is used to control the drone to perform routine patrols and conduct preliminary condition screening of the cargo in order to trigger a deeper inspection; The depth inspection unit is used to control the drone to perform multi-angle patrols and conduct in-depth inspections of the cargo after a depth inspection is triggered. The status determination unit is used to construct a comprehensive evaluation model based on the results of preliminary status screening and in-depth inspection of the goods in order to determine the status of the goods.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image recognition-based unmanned aerial vehicle cargo monitoring method as described in any one of claims 1-8.

Images