Warehouse goods anomaly detection method and device based on image recognition technology

Abstract

The invention relates to the technical field of warehouse cargo management, in particular to a warehouse cargo anomaly detection method and device based on an image recognition technology, and the method comprises the steps: carrying out the image collection of a goods shelf in a target warehouse according to a preset sampling period based on an unmanned plane, so as to obtain a goods shelf storage image, carrying out the preprocessing of the goods shelf storage image, and obtaining a goods shelf anomaly detection result; a standard goods shelf storage image is obtained; and carrying out edge detection on the standard goods shelf storage image to obtain a goods shelf storage contour, and carrying out linear detection on the goods shelf storage contour to obtain a standard goods shelf storage contour tolerance. Through comprehensive detection based on the stockout quantity and the tidiness coefficient, the management condition of the warehouse can be comprehensively evaluated, potential problems can be timely found, the warehousing operation is optimized, the dependence on human resources is reduced, the cost brought by manual inspection is reduced, the safety is improved, and the manual operation risk is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of warehouse cargo management, and in particular to a method and device for detecting abnormality of warehouse cargo based on image recognition technology. Background Art

[0002] Image recognition technology is a technology that uses computer algorithms and artificial intelligence to analyze and understand image content. It can identify objects, scenes, text and other information in images and convert them into processable data.

[0003] Traditional methods rely on manpower for regular inspections, which is time-consuming and inefficient, and cannot detect problems in a timely manner. Due to the subjectivity and limitations of manual inspections, it is difficult to accurately judge the status of goods in many cases. This human factor leads to false alarms and omissions, resulting in inaccurate inventory information, which in turn affects subsequent replenishment and allocation decisions; and traditional methods cannot achieve real-time monitoring of warehouse status. Many problems can only be discovered during regular inspections. This lag makes it difficult to solve potential problems in a timely manner, affecting the overall efficiency of warehouse management; and in traditional methods, the cleanliness of shelves often depends on personal subjective judgment and lacks objective standards, which not only affects the efficiency of goods placement, but also increases the difficulty of finding items, reducing overall work efficiency. If the goods on the shelves are not placed neatly, it may lead to the inability to accurately find the required goods when picking up the goods, which in turn causes misplacing or omissions, and ultimately causes goods abnormalities. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a warehouse cargo abnormality detection method and device based on image recognition technology.

[0005] The technical solution adopted to solve the above technical problems is: a warehouse cargo anomaly detection method based on image recognition technology, comprising: Based on the drone, images of shelves in the target warehouse are collected according to a preset sampling period to obtain shelf storage images, and the shelf storage images are preprocessed to obtain standard shelf storage images; Performing edge detection on the standard shelf storage image to obtain a shelf storage contour, performing straight line detection on the shelf storage contour to obtain a standard shelf storage contour, and matching the standard shelf storage contour with a preset template contour to obtain a matching degree between the two; The matching degree is compared with a preset matching degree threshold. If the matching degree is less than the preset matching degree threshold, image difference is performed between the standard shelf storage contour and the template contour to obtain a differential contour. Otherwise, it indicates that the shelf is fully stocked with goods. Out-of-stock statistics are performed on the shelf based on the differential contour to obtain the out-of-stock quantity of the shelf. Cargo target detection is performed on the standard shelf storage image based on the trained target detection model to obtain a cargo target detection result, a cover coefficient of the cargo is calculated based on the cargo target detection result, and a neatness coefficient of the shelf is evaluated based on the cover coefficient of the cargo, wherein the cover coefficient refers to the degree to which the cargo is blocked by other objects on the shelf, expressed as a ratio, and is used to evaluate the visibility and neatness of the shelf; The target warehouse is comprehensively inspected based on the out-of-stock quantity and neatness coefficient of all the shelves in the target warehouse to obtain the abnormality detection coefficient of the target warehouse.

[0006] Preferably, the shelf storage image is preprocessed to obtain a standard shelf storage image, including: The shelf storage image is processed by a gamma unsharp masking algorithm to obtain a first enhanced image, and a weight calculation is performed on the first enhanced image to obtain a first weight. The calculation formula of the first weight is as follows: ; in, represents the first weight, represents the index of the first enhanced image, represents the average value of the pixel values ​​of the first enhanced image, represents the variance of the pixel values ​​of the first enhanced image; The shelf storage image is processed by a multi-scale Retinex and color restoration algorithm to obtain a second enhanced image, and a second weight of the second enhanced image is calculated. The calculation formula of the second weight is as follows: ; in, represents the second weight, represents the index of the second enhanced image, represents the average value of the pixel values ​​of the second enhanced image, Represents the variance of the pixel values ​​of the second enhanced image.

[0007] Preferably, the preprocessing operation is performed on the shelf storage image to obtain a standard shelf storage image, and further includes: Based on the first weight, the first enhanced image is multiplied pixel by pixel through a Gaussian pyramid and a Laplacian pyramid to obtain a first output image. The expression of the first output image is as follows: ; in, represents the first output image, represents the Gaussian pyramid layer, represents the first layer, represents the first weight after normalization; The second enhanced image is multiplied pixel-wise by a Gaussian pyramid and a Laplacian pyramid based on the second weight to obtain a second output image. The expression of the second output image is as follows: ; in, represents the second output image, represents the normalized second weight; The first output image and the second output image are reconstructed to obtain a standard shelf storage image, and the expression of the standard shelf storage image is as follows: ; in, Represents a standard shelf storage image.

[0008] Preferably, the target detection model includes a backbone network, a feature aggregation module and a detection module, wherein the backbone network is used to extract the overall feature map of the standard shelf storage image, and the overall feature map is evenly split into two feature maps along the channel dimension. and , the two feature maps and Input to the feature aggregation module, the feature aggregation module includes a global attention unit, a local attention unit and a fusion unit. The global attention unit is used to extract the long-range relationship of each pixel in space, and the local attention unit is used to extract the local features of the region of interest in the spatial dimension of a given feature map. The fusion unit includes two 1×1 convolutional layers and a heavy parameter block. The outputs of the global attention unit and the local attention unit are fused by element-by-element addition, and finally the features are expanded into a one-dimensional sequence. The detection module is used to detect the feature aggregation module to obtain the cargo target detection result, wherein the expression of the global attention unit is as follows: ; in, represents the global attention operator, represents matrix multiplication, represents the Softmax activation function, represents a 1×1 point-wise convolutional layer, represents a multilayer perceptron, represents the global spatial attention, Represents a transpose operation; The expression of the local attention unit is as follows: ; in, represents the local attention operator, represents element-wise matrix multiplication, represents the Sigmoid activation function, Represents three stacked 1×1 convolutional layers and a 3×3 depthwise separable convolutional layer.

[0009] Preferably, performing linear detection on the shelf storage contour to obtain a standard shelf storage contour includes: Calculate the pixel gradient field of the shelf storage contour, take the pixel point with the largest gradient among the unmarked pixels as the starting point, and traverse the unmarked pixels along the gradient direction of the starting point; Add the pixel points whose gradient direction angle is less than the preset angle threshold to the straight line support area, mark them, and recalculate the direction of the straight line support area to determine whether there are unmarked pixel points. If so, repeat the above operation. Otherwise, calculate the line point density of the straight line support area. If the line point density of the straight line support area is less than the preset density threshold, it is determined whether the angle threshold has been adjusted. If it has been adjusted, the radius of the straight line support area is reduced according to a preset ratio. Otherwise, the standard deviation of the gradient direction angle between pixel points within the preset range of the starting point of the straight line support area is used as the new angle threshold.

[0010] Preferably, performing linear detection on the shelf storage contour to obtain a standard shelf storage contour further includes: If the line point density of the linear support area is not less than a preset density threshold, the linear support area is regarded as a valid linear support area, and the valid linear support area is divided into a plurality of small rectangles; Calculate the distances from all line points in the sub-rectangle to the width of the rectangle, and project to the width of the sub-rectangle, and count the number of line points in each equally divided small rectangle to establish a statistical histogram; Determine whether the histogram has a peak value. If a peak value appears, reduce the number of line points in the small rectangle where the branch is located and shorten the width of the sub-rectangle. Otherwise, determine whether the number of directed lines in the effective straight line support area is less than a preset straight line number threshold; If it is less than the preset straight line quantity threshold, the valid straight line support area is a qualified straight line support area. If it is not less than the preset straight line quantity threshold, the valid straight line support area is an unqualified straight line support area. The valid straight line support area is removed to obtain the standard shelf storage outline.

[0011] Preferably, matching the standard shelf storage profile with a preset template profile to obtain a matching degree between the two includes: Extracting boundary features between the standard shelf storage outline and the preset template outline, wherein the boundary features include corner points, edges and curvatures, and constructing a feature point set based on the boundary features; Calculate the distance between the feature point set of the standard shelf storage contour and the feature point set of the template contour based on the Euclidean distance; The distance is compared with a preset distance threshold to obtain the number of feature points that match the feature point set of the standard shelf storage contour and the feature point set of the template contour, and the number of feature points is the matching degree.

[0012] Preferably, performing out-of-stock statistics on the shelf based on the differential profile to obtain the out-of-stock quantity of the shelf includes: Performing image segmentation on the differential contour based on a global threshold method to obtain a contour image, and counting the pixel area of ​​the contour image based on a pixel counting method; The pixel area is matched with a preset goods pixel table to obtain the out-of-stock quantity corresponding to the pixel area.

[0013] Preferably, the calculation formula of the masking coefficient is as follows: ; in, Indicates the target exist The masking factor of the frame, Indicates the target A rectangular detection box, express Target in frame The position information of the rectangular detection box in the image coordinate system, Indicates the target The center coordinates of Indicates the target The aspect ratio, Indicates the target of high, Indicates that the target The area size obtained by the coordinate information is Representation and Target The total number of adjacent targets.

[0014] The technical solution adopted to solve the above technical problems is: a warehouse cargo anomaly detection device based on image recognition technology, which is applicable to the warehouse cargo anomaly detection method based on image recognition technology, comprising: An image processing unit, the image processing unit is used to collect images of shelves in a target warehouse based on the drone according to a preset sampling period to obtain shelf storage images, and perform preprocessing operations on the shelf storage images to obtain standard shelf storage images; A contour matching unit, the contour matching unit is used to perform edge detection on the standard shelf storage image to obtain a shelf storage contour, perform straight line detection on the shelf storage contour to obtain a standard shelf storage contour, and match the standard shelf storage contour with a preset template contour to obtain a matching degree between the two; an out-of-stock statistics unit, the out-of-stock statistics unit being used to compare the matching degree with a preset matching degree threshold, and if the matching degree is less than the preset matching degree threshold, performing image difference between the standard shelf storage contour and the template contour to obtain a differential contour, otherwise, it indicates that the shelf is fully stocked, and out-of-stock statistics are performed on the shelf based on the differential contour to obtain the out-of-stock quantity of the shelf; A neatness detection unit, the neatness detection unit is used to perform cargo target detection on the standard shelf storage image based on a trained target detection model to obtain a cargo target detection result, calculate a masking coefficient of the cargo based on the cargo target detection result, and evaluate a neatness coefficient of the shelf based on the masking coefficient of the cargo, wherein the masking coefficient refers to the degree to which the cargo is blocked by other objects on the shelf, expressed as a ratio, and is used to evaluate the visibility and neatness of the shelf; An anomaly detection unit is used to perform a comprehensive detection on the target warehouse based on the out-of-stock quantity and neatness coefficient of all the shelves in the target warehouse to obtain an anomaly detection coefficient of the target warehouse.

[0015] The beneficial effects of the present invention are as follows: (1) The present invention uses drones to automatically collect images, which reduces the workload of manual inspections and improves detection efficiency. Through a preset sampling period, the warehouse status can be monitored in a timely manner, and the situation of missing or improperly placed goods can be quickly discovered. (2) The present invention combines edge detection, line detection and template matching to accurately identify the status of items on the shelf, reduce the possibility of false alarms and missed alarms, and accurately count the number of out-of-stock items through analysis of differential contours. The neatness of the shelves can be evaluated through the masking coefficient, providing data support for warehouse management. (3) Through comprehensive detection based on the number of out-of-stock items and the neatness coefficient, the present invention can comprehensively evaluate the management status of the warehouse, discover potential problems in a timely manner, optimize warehouse operations, reduce dependence on human resources, reduce the cost of manual inspections, and at the same time improve safety and reduce the risk of human operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A schematic flow chart of the steps of an overall method in an embodiment of the present invention; Figure 2 A schematic diagram of the device architecture of an overall device in an embodiment of the present invention.

[0017] Figure numerals: 1. Image processing unit; 2. Contour matching unit; 3. Out-of-stock statistics unit; 4. Neatness detection unit; 5. Abnormality detection unit. DETAILED DESCRIPTION

[0018] Embodiment 1, as Figure 1 As shown, the present invention proposes a warehouse cargo anomaly detection method based on image recognition technology, comprising: S1. Based on the drone, images of shelves in the target warehouse are collected according to a preset sampling period to obtain shelf storage images, and the shelf storage images are preprocessed to obtain standard shelf storage images; S2, performing edge detection on the standard shelf storage image to obtain the shelf storage contour, performing straight line detection on the shelf storage contour to obtain the standard shelf storage contour, and matching the standard shelf storage contour with the preset template contour to obtain the matching degree between the two; S3, comparing the matching degree with a preset matching degree threshold. If the matching degree is less than the preset matching degree threshold, performing image difference between the standard shelf storage contour and the template contour to obtain a differential contour. Otherwise, it means that the shelf is fully stocked with goods. The out-of-stock quantity of the shelf is counted based on the differential contour to obtain the out-of-stock quantity of the shelf. S4, performing cargo target detection on the standard shelf storage image based on the trained target detection model to obtain cargo target detection results, calculating the cargo cover coefficient based on the cargo target detection results, and evaluating the neatness coefficient of the shelf based on the cargo cover coefficient, wherein the neatness coefficient is proportional to the cover coefficient; S5. Perform a comprehensive inspection on the target warehouse based on the out-of-stock quantity and neatness coefficient of all shelves in the target warehouse to obtain the abnormality detection coefficient of the target warehouse.

[0019] In the present invention, drone refers to an unmanned aircraft, which is usually used for shooting, monitoring and data collection; sampling period refers to the time interval for drone image collection to ensure that data is obtained within a predetermined time; edge detection refers to an image processing technology used to identify edges in images (places where brightness changes significantly), and commonly used algorithms include Canny edge detection; contour refers to the boundary line of an object in an image, indicating the shape and position of the object; template image compares a standard shape (template) with a target image to determine the similarity between the two to detect a specific shape or pattern; image difference is a commonly used image processing technology used to detect changes between images, and its basic principle is to find the difference between two or more images by comparing them so as to identify the movement and change of objects; masking coefficient refers to the degree to which goods are blocked by other objects on the shelf, usually expressed as a ratio, and is used to evaluate the visibility and neatness of the shelf; the abnormality detection coefficient judges the overall operation status of the warehouse and identifies whether there are abnormalities by comprehensively analyzing the out-of-stock quantity and neatness coefficient of the shelf.

[0020] Embodiment 2, a warehouse cargo anomaly detection method based on image recognition technology proposed by the present invention, compared with embodiment 1, this embodiment further includes: performing a preprocessing operation on the shelf storage image to obtain a standard shelf storage image, including: A1. Process the shelf storage image using a gamma unsharp mask algorithm to obtain a first enhanced image. Perform weight calculation on the first enhanced image to obtain a first weight. The calculation formula of the first weight is as follows: ; in, represents the first weight, represents the index of the first enhanced image, represents the average value of the pixel values ​​of the first enhanced image, represents the variance of the pixel values ​​of the first enhanced image; A2. Perform multi-scale Retinex and color restoration algorithm processing on the shelf storage image to obtain a second enhanced image, and calculate the second weight of the second enhanced image. The calculation formula of the second weight is as follows: ; in, represents the second weight, represents the index of the second enhanced image, represents the average value of the pixel values ​​of the second enhanced image, Represents the variance of the pixel values ​​of the second enhanced image.

[0021] In this embodiment, gamma unsharpening is an image enhancement technology that reduces the sharpening effect by adjusting the gamma value of the image, smoothes the details in the image, and enhances the visual effect; mask algorithm refers to the use of mask technology to select and process specific areas of an image, which is usually used to enhance or repair the image; multi-scale Retinex refers to an image processing method that analyzes images at different scales to enhance brightness and color while reducing the impact of lighting changes; color restoration refers to the process of restoring the color of an image to a more realistic or expressive state, usually in combination with other image enhancement technologies.

[0022] In an optional embodiment, preprocessing the shelf storage image to obtain a standard shelf storage image further includes: A3. Performing a pixel-level multiplication operation on the first enhanced image through a Gaussian pyramid and a Laplacian pyramid based on the first weight to obtain a first output image. The expression of the first output image is as follows: ; in, represents the first output image, represents the Gaussian pyramid layer, represents the first layer, represents the first weight after normalization; A4. Perform a pixel-level multiplication operation on the second enhanced image through a Gaussian pyramid and a Laplacian pyramid based on the second weight to obtain a second output image. The expression of the second output image is as follows: ; in, represents the second output image, represents the normalized second weight; A5. Reconstruct the first output image and the second output image to obtain a standard shelf storage image. The expression of the standard shelf storage image is as follows: ; in, Represents a standard shelf storage image.

[0023] It should be noted that Gaussian pyramid refers to an image pyramid structure, which generates a multi-scale image representation by blurring the image layer by layer; Laplacian pyramid refers to extracting the edge and detail information of the image by differentiating each layer of the Gaussian pyramid, and is usually used for image reconstruction and enhancement; pixel-level multiplication refers to the operation of multiplying each pixel of the image one by one, and is usually used to combine two images or an image with weights.

[0024] In an optional embodiment, the target detection model includes a backbone network, a feature aggregation module and a detection module. The backbone network is used to extract the overall feature map of the standard shelf storage image, and the overall feature map is evenly split into two feature maps along the channel dimension. and , the two feature maps and Input to the feature aggregation module, which includes a global attention unit, a local attention unit and a fusion unit. The global attention unit is used to extract the long-range relationship of each pixel in space. The local attention unit is used to extract the local features of the region of interest in the spatial dimension of a given feature map. The fusion unit includes two 1×1 convolutional layers and a heavy parameter block. The outputs of the global attention unit and the local attention unit are fused by element-by-element addition. Finally, the features are expanded into a one-dimensional sequence. The detection module is used to detect the feature aggregation module to obtain the cargo target detection result. The expression of the global attention unit is as follows: ; in, represents the global attention operator, represents matrix multiplication, represents the Softmax activation function, represents a 1×1 point-wise convolutional layer, represents a multilayer perceptron, represents the global spatial attention, Represents a transpose operation; The expression of the local attention unit is as follows: ; in, represents the local attention operator, represents element-wise matrix multiplication, represents the Sigmoid activation function, Represents three stacked 1×1 convolutional layers and a 3×3 depthwise separable convolutional layer.

[0025] It should be noted that the re-parameter block is a network structure used to improve the efficiency and performance of the model during the training and inference stages. It reduces the amount of computation by rearranging parameters or simplifying calculations; feature expansion is the process of converting a high-dimensional feature map into a one-dimensional sequence, which is usually performed before entering the detection module so that the extracted features can be input into the fully connected layer or other subsequent processing modules; the detection module is responsible for the final target detection of the results of feature aggregation, which usually includes classification and bounding box regression operations to identify targets and their locations in the image.

[0026] In an optional embodiment, a straight line detection is performed on the shelf storage contour to obtain a standard shelf storage contour, including: B1. Calculate the pixel gradient field of the shelf storage contour, take the pixel point with the largest gradient among the unmarked pixels as the starting point, and traverse the unmarked pixels along the gradient direction of the starting point; B2. Add the pixels whose gradient direction angle is less than the preset angle threshold to the straight line support area, mark them, and recalculate the direction of the straight line support area to determine whether there are unmarked pixels. If so, repeat the above operation. Otherwise, calculate the line point density of the straight line support area. B3. If the line point density of the straight line support area is less than the preset density threshold, determine whether the angle threshold has been adjusted. If it has been adjusted, reduce the radius of the straight line support area according to the preset ratio. Otherwise, use the standard deviation of the gradient direction angle between the pixel points within the preset range of the starting point of the straight line support area as the new angle threshold.

[0027] It should be noted that the pixel gradient field is a representation in image processing, which is used to describe the degree and direction of pixel intensity change in an image. The gradient is obtained by calculating the brightness change of the neighborhood around each pixel, usually using the Sobel operator or other edge detection algorithms. High gradient values ​​indicate that there are significant edges or contours in the image. The starting point is the first point selected from the unmarked pixels, usually the pixel with the largest gradient value. It is the starting point for subsequent traversal and processing. The gradient direction refers to the direction of the pixel gradient, usually expressed as an angle or unit vector, which points to the direction with the largest brightness change in the image. The linear support area refers to a group of pixels formed around a certain initial point under certain conditions. These pixels form a linear structure or support to some extent. This area can be used to describe the boundary of the target object. The line point density refers to the ratio of the number of valid pixels in the linear support area to the area of ​​the area. It is used to evaluate the "compactness" of the linear support area, that is, whether the area forms a continuous line.

[0028] In an optional embodiment, performing straight line detection on the shelf storage contour to obtain a standard shelf storage contour further includes: B4. If the line point density of the straight line support area is not less than the preset density threshold, the straight line support area is regarded as a valid straight line support area, and the valid straight line support area is divided into a plurality of small rectangles; B5. Calculate the distance from all line points in the sub-rectangle to the width of the rectangle, and project to the wide side of the sub-rectangle, and count the number of line points in each equally divided small rectangle to establish a statistical histogram; B6. Determine whether the histogram has a peak. If so, reduce the number of line points in the small rectangle where the branch is located and shorten the width of the sub-rectangle. Otherwise, determine whether the number of directed lines in the effective straight line support area is less than a preset straight line number threshold. B7. If it is less than the preset straight line quantity threshold, the effective straight line support area is a qualified straight line support area. If it is not less than the preset straight line quantity threshold, the effective straight line support area is an unqualified straight line support area. The effective straight line support area is removed to obtain the standard shelf storage outline.

[0029] It should be noted that projection is to calculate the distance from the line point to the wide side of the rectangle and map it to the side. In this way, the distribution of the line points in the rectangle can be quantified; the statistical histogram is a graph used to represent the distribution of line points in each small rectangle. The wide side of each small rectangle and the number of line points form a bar graph, showing the distribution of line points in different positions.

[0030] In an optional embodiment, matching the standard shelf storage profile with the preset template profile to obtain a matching degree between the two includes: C1. Extract boundary features of the standard shelf storage outline and the preset template outline, where the boundary features include corner points, edges and curvatures, and construct a feature point set based on the boundary features; C2, calculating the distance between the feature point set of the standard shelf storage contour and the feature point set of the template contour based on the Euclidean distance; C3. Compare the distance with a preset distance threshold to obtain the number of feature points that match the feature point set of the standard shelf storage contour and the feature point set of the template contour. The number of feature points is the matching degree.

[0031] In an optional embodiment, out-of-stock statistics are performed on the shelf based on the differential profile to obtain the out-of-stock quantity of the shelf, including: D1. Perform image segmentation on the differential contour based on the global threshold method to obtain a contour image, and count the pixel area of ​​the contour image based on the pixel counting method; D2. Match the pixel area with the preset goods pixel table to obtain the out-of-stock quantity corresponding to the pixel area.

[0032] It should be noted that the global threshold method is an image segmentation technology. By setting a threshold, the pixels in the image are divided into two categories: foreground (the part of interest) and background. All pixel values ​​greater than the threshold are classified as foreground, and all pixel values ​​less than or equal to the threshold are classified as background. This method is suitable for images with obvious contrast; the pixel counting method is to estimate the area of ​​a specific area in the image by counting the number of pixels in the area. This method is often used to analyze the size of the target object in the image or the space it occupies.

[0033] In an optional embodiment, the calculation formula of the masking coefficient is as follows: ; in, Indicates the target exist The masking factor of the frame, Indicates the target A rectangular detection box, express Target in frame The position information of the rectangular detection box in the image coordinate system, Indicates the target The center coordinates of Indicates the target The aspect ratio, Indicates the target of high, Indicates that the target The area size obtained by the coordinate information is Representation and Target The total number of adjacent targets.

[0034] Embodiment three, as Figure 2 As shown, the present invention proposes a warehouse cargo anomaly detection device based on image recognition technology, which is applicable to the warehouse cargo anomaly detection method based on image recognition technology, including: The image processing unit 1 is used to collect images of shelves in a target warehouse based on the drone according to a preset sampling period to obtain shelf storage images, and perform preprocessing operations on the shelf storage images to obtain standard shelf storage images; Contour matching unit 2, the contour matching unit 2 is used to perform edge detection on the standard shelf storage image to obtain the shelf storage contour, perform straight line detection on the shelf storage contour to obtain the standard shelf storage contour, and match the standard shelf storage contour with the preset template contour to obtain the matching degree between the two; The out-of-stock statistics unit 3 is used to compare the matching degree with a preset matching degree threshold. If the matching degree is less than the preset matching degree threshold, the standard shelf storage contour and the template contour are image-differentiated to obtain a differential contour. Otherwise, it means that the shelf is fully stocked. The out-of-stock statistics of the shelf are performed based on the differential contour to obtain the out-of-stock quantity of the shelf. The neatness detection unit 4 is used to perform cargo target detection on the standard shelf storage image based on the trained target detection model to obtain cargo target detection results, calculate the cargo cover coefficient based on the cargo target detection results, and evaluate the neatness coefficient of the shelf based on the cargo cover coefficient, wherein the neatness coefficient is proportional to the cover coefficient; The abnormality detection unit 5 is used to perform a comprehensive detection on the target warehouse based on the out-of-stock quantity and neatness coefficient of all shelves in the target warehouse to obtain the abnormality detection coefficient of the target warehouse.

[0035] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A warehouse cargo anomaly detection method based on image recognition technology, characterized in that: include: Based on the drone, images of shelves in the target warehouse are collected according to a preset sampling period to obtain shelf storage images, and the shelf storage images are preprocessed to obtain standard shelf storage images; Performing edge detection on the standard shelf storage image to obtain a shelf storage contour, performing straight line detection on the shelf storage contour to obtain a standard shelf storage contour, and matching the standard shelf storage contour with a preset template contour to obtain a matching degree between the two; The matching degree is compared with a preset matching degree threshold. If the matching degree is less than the preset matching degree threshold, image difference is performed between the standard shelf storage contour and the template contour to obtain a differential contour. Otherwise, it indicates that the shelf is fully stocked with goods. Out-of-stock statistics are performed on the shelf based on the differential contour to obtain the out-of-stock quantity of the shelf. Cargo target detection is performed on the standard shelf storage image based on the trained target detection model to obtain a cargo target detection result, a cover coefficient of the cargo is calculated based on the cargo target detection result, and a neatness coefficient of the shelf is evaluated based on the cover coefficient of the cargo, wherein the cover coefficient refers to the degree to which the cargo is blocked by other objects on the shelf, expressed as a ratio, and is used to evaluate the visibility and neatness of the shelf; The target warehouse is comprehensively inspected based on the out-of-stock quantity and neatness coefficient of all the shelves in the target warehouse to obtain the abnormality detection coefficient of the target warehouse.

2. The method for detecting abnormality of warehouse goods based on image recognition technology according to claim 1, characterized in that: The shelf storage image is preprocessed to obtain a standard shelf storage image, including: The shelf storage image is processed by a gamma unsharp masking algorithm to obtain a first enhanced image, and a weight calculation is performed on the first enhanced image to obtain a first weight. The calculation formula of the first weight is as follows: ； in, represents the first weight, represents the index of the first enhanced image, represents the average value of the pixel values ​​of the first enhanced image, represents the variance of the pixel values ​​of the first enhanced image; The shelf storage image is processed by a multi-scale Retinex and color restoration algorithm to obtain a second enhanced image, and a second weight of the second enhanced image is calculated. The calculation formula of the second weight is as follows: ； in, represents the second weight, represents the index of the second enhanced image, represents the average value of the pixel values ​​of the second enhanced image, Represents the variance of the pixel values ​​of the second enhanced image.

3. The method for detecting abnormality of warehouse goods based on image recognition technology according to claim 2 is characterized in that: The method further comprises: performing a preprocessing operation on the shelf storage image to obtain a standard shelf storage image. Based on the first weight, the first enhanced image is multiplied pixel by pixel through a Gaussian pyramid and a Laplacian pyramid to obtain a first output image. The expression of the first output image is as follows: ； in, represents the first output image, represents the Gaussian pyramid layer, represents the first layer, represents the first weight after normalization; The second enhanced image is multiplied pixel-wise by a Gaussian pyramid and a Laplacian pyramid based on the second weight to obtain a second output image. The expression of the second output image is as follows: ； in, represents the second output image, represents the normalized second weight; The first output image and the second output image are reconstructed to obtain a standard shelf storage image, and the expression of the standard shelf storage image is as follows: ； in, Represents a standard shelf storage image.

4. The method for detecting abnormality of warehouse goods based on image recognition technology according to claim 1, characterized in that: The target detection model includes a backbone network, a feature aggregation module and a detection module. The backbone network is used to extract the overall feature map of the standard shelf storage image and evenly split the overall feature map into two feature maps along the channel dimension. and , the two feature maps and Input to the feature aggregation module, the feature aggregation module includes a global attention unit, a local attention unit and a fusion unit. The global attention unit is used to extract the long-range relationship of each pixel in space, and the local attention unit is used to extract the local features of the region of interest in the spatial dimension of a given feature map. The fusion unit includes two 1×1 convolutional layers and a heavy parameter block. The outputs of the global attention unit and the local attention unit are fused by element-by-element addition, and finally the features are expanded into a one-dimensional sequence. The detection module is used to detect the feature aggregation module to obtain the cargo target detection result, wherein the expression of the global attention unit is as follows: ； in, represents the global attention operator, represents matrix multiplication, represents the Softmax activation function, represents a 1×1 point-wise convolutional layer, represents a multilayer perceptron, represents the global spatial attention, Represents a transpose operation; The expression of the local attention unit is as follows: ； in, represents the local attention operator, represents element-wise matrix multiplication, represents the Sigmoid activation function, Represents three stacked 1×1 convolutional layers and a 3×3 depthwise separable convolutional layer.

5. The method for detecting abnormality of warehouse goods based on image recognition technology according to claim 1 is characterized in that: The shelf storage contour is subjected to linear detection to obtain a standard shelf storage contour, including: Calculate the pixel gradient field of the shelf storage contour, take the pixel point with the largest gradient among the unmarked pixels as the starting point, and traverse the unmarked pixels along the gradient direction of the starting point; Add the pixel points whose gradient direction angle is less than the preset angle threshold to the straight line support area, mark them, and recalculate the direction of the straight line support area to determine whether there are unmarked pixel points. If so, repeat the above operation. Otherwise, calculate the line point density of the straight line support area. If the line point density of the straight line support area is less than the preset density threshold, it is determined whether the angle threshold has been adjusted. If it has been adjusted, the radius of the straight line support area is reduced according to a preset ratio. Otherwise, the standard deviation of the gradient direction angle between pixel points within the preset range of the starting point of the straight line support area is used as the new angle threshold.

6. The method for detecting abnormality of warehouse goods based on image recognition technology according to claim 5 is characterized in that: The method further comprises: performing a straight line detection on the shelf storage contour to obtain a standard shelf storage contour; If the line point density of the linear support area is not less than a preset density threshold, the linear support area is regarded as a valid linear support area, and the valid linear support area is divided into a plurality of small rectangles; Calculate the distances from all line points in the sub-rectangle to the width of the rectangle, and project to the width of the sub-rectangle, and count the number of line points in each equally divided small rectangle to establish a statistical histogram; Determine whether the histogram has a peak value. If a peak value appears, reduce the number of line points in the small rectangle where the branch is located and shorten the width of the sub-rectangle. Otherwise, determine whether the number of directed lines in the effective straight line support area is less than a preset straight line number threshold; If it is less than the preset straight line quantity threshold, the valid straight line support area is a qualified straight line support area. If it is not less than the preset straight line quantity threshold, the valid straight line support area is an unqualified straight line support area. The valid straight line support area is removed to obtain the standard shelf storage outline.

7. The method for detecting abnormality of warehouse goods based on image recognition technology according to claim 1, characterized in that: Matching the standard shelf storage profile with the preset template profile to obtain a matching degree between the two, including: Extracting boundary features between the standard shelf storage outline and the preset template outline, wherein the boundary features include corner points, edges and curvatures, and constructing a feature point set based on the boundary features; Calculate the distance between the feature point set of the standard shelf storage contour and the feature point set of the template contour based on the Euclidean distance; The distance is compared with a preset distance threshold to obtain the number of feature points that match the feature point set of the standard shelf storage contour and the feature point set of the template contour, and the number of feature points is the matching degree.

8. The method for detecting abnormality of warehouse goods based on image recognition technology according to claim 1, characterized in that: Performing out-of-stock statistics on the shelf based on the differential profile to obtain the out-of-stock quantity of the shelf includes: Performing image segmentation on the differential contour based on a global threshold method to obtain a contour image, and counting the pixel area of ​​the contour image based on a pixel counting method; The pixel area is matched with a preset goods pixel table to obtain the out-of-stock quantity corresponding to the pixel area.

9. The method for detecting abnormality of warehouse goods based on image recognition technology according to claim 1, characterized in that: The calculation formula of the masking coefficient is as follows: ； in, Indicates the target exist The masking factor of the frame, Indicates the target A rectangular detection box, express Target in frame The position information of the rectangular detection box in the image coordinate system, Indicates the target The center coordinates of Indicates the target The aspect ratio, Indicates the target of high, Indicates that the target The area size obtained by the coordinate information is Representation and Target The total number of adjacent targets.

10. A warehouse cargo anomaly detection device based on image recognition technology, which is applicable to a warehouse cargo anomaly detection method based on image recognition technology according to any one of claims 1 to 9, characterized in that: include: An image processing unit (1), the image processing unit (1) being used to collect images of shelves in a target warehouse based on a drone according to a preset sampling period to obtain shelf storage images, and to perform a preprocessing operation on the shelf storage images to obtain standard shelf storage images; A contour matching unit (2), the contour matching unit (2) being used to perform edge detection on the standard shelf storage image to obtain a shelf storage contour, perform straight line detection on the shelf storage contour to obtain a standard shelf storage contour, and match the standard shelf storage contour with a preset template contour to obtain a matching degree between the two; The out-of-stock statistics unit (3) is used to compare the matching degree with a preset matching degree threshold value. If the matching degree is less than the preset matching degree threshold value, the image difference between the standard shelf storage contour and the template contour is performed to obtain a differential contour. Otherwise, it means that the shelf is fully stocked with goods. The out-of-stock statistics of the shelf are performed based on the differential contour to obtain the out-of-stock quantity of the shelf. A neatness detection unit (4), the neatness detection unit (4) is used to perform cargo target detection on the standard shelf storage image based on a trained target detection model to obtain cargo target detection results, calculate the masking coefficient of the cargo based on the cargo target detection results, and evaluate the neatness coefficient of the shelf based on the masking coefficient of the cargo, wherein the masking coefficient refers to the degree to which the cargo is blocked by other objects on the shelf, expressed as a ratio, and is used to evaluate the visibility and neatness of the shelf; An abnormality detection unit (5), the abnormality detection unit (5) is used to perform a comprehensive detection on the target warehouse based on the out-of-stock quantity and neatness coefficient of all the shelves in the target warehouse, so as to obtain an abnormality detection coefficient of the target warehouse.

Images