Parcel deviation identification method, device, and equipment and storage medium

Abstract

This invention relates to the field of artificial intelligence and discloses a method, apparatus, device, and storage medium for package deviation identification. The method includes: acquiring original monitoring image samples and labeling the cross-belt regions and packages in the original monitoring image samples; training a model based on the labeled original monitoring image samples to obtain a cross-belt identification model and a package positioning model; acquiring a monitoring image to be identified and inputting the monitoring image to be identified into the cross-belt identification model and the package positioning model respectively to obtain the cross-belt mask image and package positioning information in the monitoring image to be identified; and identifying deviated packages in the monitoring image based on the package positioning information and the cross-belt mask image. This method uses computer vision technology to identify package deviation, enabling real-time tracking of monitored packages, reducing the likelihood of missed detections, improving the identification rate, and providing timely alerts for deviated packages to notify relevant personnel, thereby improving logistics and transportation efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, apparatus, device, and storage medium for identifying package deviation. Background Technology

[0002] Nowadays, people's lives are increasingly inseparable from express delivery, and logistics transportation is constantly being improved and optimized, empowering smart cities. During the sorting and distribution process, when packages pass through the curved tracks of cross-belts, items are prone to moving or being thrown off due to gravity and inertia, preventing smooth delivery. These items may fall off the rails or deviate from the track, blocking the transport of other items and hindering their smooth delivery. Traditional handling methods require manual inspection, which is labor-intensive and prone to omissions. Summary of the Invention

[0003] The main objective of this invention is to solve the technical problem of omissions in the existing process of handling packages that deviate from the conveyor belt.

[0004] The first aspect of this invention provides a method for package deviation identification, comprising: acquiring original surveillance image samples and labeling the cross-band regions and packages in the original surveillance image samples; training a model based on the labeled original surveillance image samples to obtain a cross-band identification model and a package positioning model; acquiring a surveillance image to be identified and inputting the surveillance image to be identified into the cross-band identification model and the package positioning model to obtain a cross-band mask image and package positioning information of the package in the surveillance image to be identified; and identifying the deviated package in the surveillance image to be identified based on the package positioning information and the cross-band mask image.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the step of training the model based on the labeled original surveillance image samples to obtain a cross-strip recognition model and a package location model includes: dividing the labeled original surveillance image samples into a training set and a test set; training the cross-strip recognition model based on the training set and a preset instance segmentation model neural network; training the package location model based on the training set and a preset target detection network; and verifying the detection effects of the cross-strip recognition model and the package location model through the test set.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the step of inputting the standard driving data into a preset scenario complexity evaluation model and constructing a test set based on the scenario complexity output by the scenario complexity evaluation model includes: processing the standard driving data of the target area through a sliding window according to a preset time span to obtain multiple time-based scenarios; inputting the time-based scenarios into the preset scenario complexity evaluation model to obtain the scenario complexity corresponding to the multiple time-based scenarios; and constructing a test set based on the scenario complexity corresponding to each time-based scenario.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the cross-strip recognition model includes a feature extraction network, a target region coarse extraction network, and a cross-strip segmentation network; the step of acquiring the monitoring image to be identified and inputting the monitoring image to be identified into the cross-strip recognition model and the package positioning model respectively to obtain the cross-strip mask image and the package positioning information of the package in the monitoring image to be identified includes: acquiring the monitoring image to be identified; inputting the monitoring image to be identified into the cross-strip recognition model, and obtaining the feature image corresponding to the monitoring image to be identified through the feature extraction network in the cross-strip recognition model; inputting the feature image into the target region coarse extraction network in the cross-strip recognition model to obtain the corresponding target region feature image; inputting the target region feature image into the cross-strip segmentation network to obtain the cross-strip mask image; and inputting the monitoring image to be identified into the package positioning model to obtain the package positioning information of the package.

[0008] Optionally, in the fourth implementation of the first aspect of the present invention, the target region coarse extraction network is constructed based on a Feature Pyramid Network (FPN), a Region Extraction Network (RPN), and a Region of Interest (RoI) Align network. The step of inputting the feature image into the target region coarse extraction network of the cross-band recognition model to obtain the corresponding target region feature image includes: inputting the feature image into the FPN of the target region coarse extraction network, and segmenting the feature image into images of different sizes through the FPN; inputting the images of different sizes into the RPN, calculating the target pre-selected region of the cross-band through the RPN, performing multi-scale feature fusion based on the images of different sizes and the target pre-selected region to obtain a feature tensor; and inputting the feature tensor into the RoI Align, adjusting the target pre-selected region through the RoI Align to obtain the corresponding target region feature image.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the target detection network is CenterNet; the step of training the package localization model based on the training set and the preset target detection network includes: inputting the training set into CenterNet to obtain the predicted labels and embedding vectors of the training set; calculating a second loss value of the predicted labels and the package labels in the annotated original surveillance image samples; calculating a third loss value of the embedding vectors, and using the second loss value and the third loss value as the total loss value of the target detection network; and using gradient descent to iteratively train CenterNet with the total loss value to obtain the package localization model.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, identifying the off-center package in the monitoring image to be identified based on the package location information and the cross-belt mask image includes: obtaining pixel information of the cross-belt mask image in the monitoring image to be identified to obtain a first pixel set corresponding to the cross-belt mask image; obtaining corresponding pixel information in the monitoring image to be identified according to the package location information corresponding to each package to obtain a second pixel set corresponding to each package; determining whether the second pixel set corresponding to each package is a proper subset of the first pixel set; and identifying the package corresponding to the second pixel set that is a proper subset of the first pixel set as an off-center package.

[0011] A second aspect of the present invention provides a package deviation identification device, comprising: a sample acquisition module for acquiring original monitoring image samples and labeling the cross-band regions and packages in the original monitoring image samples; a model training module for training a model based on the labeled original monitoring image samples to obtain a cross-band identification model and a package positioning model; a model input module for acquiring a monitoring image to be identified and inputting the monitoring image to be identified into the cross-band identification model and the package positioning model to obtain a cross-band mask image and package positioning information of the package in the monitoring image to be identified; and an identification module for identifying deviated packages in the monitoring image to be identified based on the package positioning information and the cross-band mask image.

[0012] Optionally, in a first implementation of the second aspect of the present invention, the sample acquisition module is specifically used for: acquiring the original surveillance video stream; performing frame-by-frame processing on the original surveillance video stream to obtain original surveillance image samples; and using preset annotation software to annotate the cross-band regions and wrap-ups in the original surveillance image samples.

[0013] Optionally, in a second implementation of the second aspect of the present invention, the model training module specifically includes: a sample partitioning unit, used to partition the labeled original surveillance image samples into a training set and a test set; a first training unit, used to train a crossband recognition model based on the training set and a preset instance segmentation model neural network; a second training unit, used to train a package location model based on the training set and a preset target detection network; and a testing unit, used to verify the detection effects of the crossband recognition model and the package location model through the test set respectively.

[0014] Optionally, in a third implementation of the second aspect of the present invention, the crossband recognition model includes a feature extraction network, a target region coarse extraction network, and a crossband segmentation network; the model input module specifically includes: a feature extraction unit, used to acquire a monitoring image to be identified, and to obtain a feature image corresponding to the monitoring image to be identified by passing the feature extraction network in the crossband recognition model; a region coarse extraction unit, used to input the feature image into the target region coarse extraction network in the crossband recognition model to obtain a corresponding target region feature image; a mask unit, used to input the target region feature image into the crossband segmentation network to obtain a crossband mask image; and a package positioning unit, used to input the monitoring image to be identified into the package positioning model to obtain package positioning information.

[0015] Optionally, in a fourth implementation of the second aspect of the present invention, the target region coarse extraction network is constructed based on a Feature Pyramid Network (FPN), a Region Extraction Network (RPN), and a Region of Interest (RoI) Align network. Specifically, the region coarse extraction unit is used to: input the feature map into the FPN of the target region coarse extraction network, and segment the feature image into images of different sizes through the FPN; input the images of different sizes into the RPN, and calculate the target pre-selected region of the cross-band through the RPN; perform multi-scale feature fusion based on the images of different sizes and the target pre-selected region to obtain a feature tensor; input the feature tensor into the RoI Align, and adjust the target pre-selected region through the RoI Align to obtain the corresponding target region feature image.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the target detection network is CenterNet, and the second training unit is specifically used for: inputting the training set into CenterNet to obtain the predicted label and embedding vector of the training set; calculating the second loss value of the predicted label and the package label in the annotated original surveillance image sample; calculating the third loss value of the embedding vector, and using the second loss value and the third loss value as the total loss value of the target detection network; using the gradient descent method, iteratively training CenterNet with the total loss value to obtain the package localization model.

[0017] Optionally, in a sixth implementation of the second aspect of the present invention, the identification module is specifically used to: obtain pixel information of the cross-band mask image in the monitoring image to be identified, and obtain a first pixel set corresponding to the cross-band mask image; obtain pixel information corresponding to each package in the monitoring image to be identified based on the package positioning information corresponding to each package, and obtain a second pixel set corresponding to each package; determine whether the second pixel set corresponding to each package is a proper subset of the first pixel set; and identify the package corresponding to the second pixel set that is a proper subset of the first pixel set as an off-target package.

[0018] A third aspect of the present invention provides a package deviation identification device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the package deviation identification device to perform the steps of the package deviation identification method described above.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described package deviation identification method.

[0020] In the technical solution of this invention, original surveillance image samples are collected, and the cross-strip areas and packages in the original surveillance image samples are labeled; based on the labeled original surveillance image samples, a model is trained to obtain a cross-strip recognition model and a package positioning model; a surveillance image to be identified is acquired, and the surveillance image to be identified is input into the cross-strip recognition model and the package positioning model respectively to obtain the cross-strip mask image and the package positioning information in the surveillance image to be identified; based on the package positioning information and the cross-strip mask image, the deviated packages in the surveillance image to be identified are identified. This method uses computer vision technology to identify package deviation, can track packages in real time, is less likely to miss detection, improves the recognition rate, and can promptly alert relevant personnel to handle deviated packages, thereby improving logistics and transportation efficiency. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the first embodiment of the package deviation recognition method in this invention;

[0022] Figure 2 This is a schematic diagram of a second embodiment of the package deviation recognition method in this invention;

[0023] Figure 3 This is a schematic diagram of a third embodiment of the package deviation recognition method in this invention;

[0024] Figure 4 This is a schematic diagram of one embodiment of the package deviation recognition device in this invention;

[0025] Figure 5 This is a schematic diagram of another embodiment of the package deviation recognition device in this invention;

[0026] Figure 6 This is a schematic diagram of one embodiment of the package deviation recognition device in this invention. Detailed Implementation

[0027] This application provides a method, apparatus, device, and storage medium for identifying package deviation, addressing the technical problem of omissions in existing processes for handling packages that deviate from the conveyor belt.

[0028] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the package deviation identification method in this invention includes:

[0030] 101. Collect raw surveillance image samples and label the cross-band areas and packages in the raw surveillance image samples;

[0031] It is understood that the executing entity of this invention can be a package deviation identification device, a terminal, or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example.

[0032] In practical applications, to better ensure transportation timeliness and package safety, existing logistics transportation systems install cameras in office areas such as distribution centers and outlets for monitoring. In this embodiment, the monitored video stream is processed to obtain original monitoring image samples, which are then labeled. In this embodiment, the labeling of the original monitoring image samples is mainly done manually using preset labeling software, such as Labelme. Labelme can label images in the form of polygons, rectangles, circles, polylines, line segments, and points, marking the intersecting areas and packages in the original monitoring image samples.

[0033] 102. Based on the labeled original surveillance image samples, the model was trained to obtain the cross-belt recognition model and the package localization model, respectively.

[0034] In this embodiment, a crossband recognition model is trained based on the MaskRcnn network architecture and a package localization model is trained based on the CenterNet network architecture, respectively.

[0035] 103. Obtain the monitoring image to be identified. Input the monitoring image to be identified into the cross-belt recognition model and the package positioning model respectively to obtain the cross-belt mask image and the package positioning information of the package in the monitoring image to be identified.

[0036] In this embodiment, Mask R-CNN is an instance segmentation algorithm architecture that can be modified with different branches to perform various tasks, including object classification, object detection, semantic segmentation, instance segmentation, and human pose recognition. The basic framework of Mask R-CNN is based on Faster R-CNN, with a fully connected segmentation subnetwork added after the basic feature network. When an image of arbitrary size is input into this network, the system performs two tasks through a deep convolutional network. The first task is the Faster R-CNN's RPN network, which primarily identifies candidate regions. The second task is object detection. The Mask R-CNN network extracts candidate regions through a region recommendation network. This region recommendation network is a lightweight neural network that performs convolutional operations by scanning the feature map through a sliding window. It combines different sizes and aspect ratios to generate overlapping regions, i.e., anchors, and provides default preset position information for each anchor for subsequent candidate region extraction. The region recommendation network outputs two types of information for each anchor. The first is a prediction of the anchor's foreground or background category. The foreground category represents the probability that one or more target classes are present in the anchor, while the background category refers to other objects besides the target to be detected, which will be filtered out later. The second information is a fine-tuning of the preset bounding box. When the center of the target does not completely coincide with the center of the foreground anchor, i.e., there is an offset, the percentage change in the output position information (x, y, w, h) is used to accurately adjust the anchor position and make the fit to the target position more accurate. There is overlap among the foreground anchors. The non-maximum suppression method is used to filter out anchors with low foreground scores and retain the anchors with the highest scores. The final region of interest is the cross-band region in this embodiment. The cross-band recognition model trained by the network can output a mask image of the cross-band in the monitoring image.

[0037] In practical applications, CenterNet is an anchor-free object detection network. By transforming the object detection problem into a standard keypoint estimation problem, the image is fed into a fully convolutional network to obtain a heatmap. The peak points of the heatmap are the center points. The position of the peak points of each feature map predicts the width and height information of the object, thus predicting the position of the object in the image. In this embodiment, the object is the package.

[0038] 104. Based on package location information and cross-belt mask images, identify deviated packages in the monitoring images to be identified.

[0039] In this embodiment, the package location information and the cross-belt mask image output by the cross-belt recognition model and the package location model are used to identify whether the package location information and the cross-belt mask image have an intersection in the monitoring image to be identified. If they do, the package corresponding to the package location information is determined to be an off-target package in the monitoring image to be identified. If not, it means that the package corresponding to the package location information has not deviated in the monitoring image to be identified.

[0040] In this embodiment, original surveillance image samples are collected, and the cross-strip areas and packages in the original surveillance image samples are labeled. Based on the labeled original surveillance image samples, a model is trained to obtain a cross-strip recognition model and a package location model. A surveillance image to be identified is acquired and input into the cross-strip recognition model and the package location model, respectively, to obtain a cross-strip mask image and package location information in the surveillance image to be identified. Based on the package location information and the cross-strip mask image, deviated packages in the surveillance image to be identified are identified. This method uses computer vision technology to identify package deviation, enabling real-time tracking of monitored packages, reducing the likelihood of missed detections, improving the recognition rate, and providing timely alerts for deviated packages to notify relevant personnel for handling, thereby improving logistics and transportation efficiency.

[0041] Please see Figure 2 The second embodiment of the package deviation identification method in this invention includes:

[0042] 201. Obtain the original surveillance video stream;

[0043] 202. Perform frame segmentation on the original surveillance video stream to obtain original surveillance image samples;

[0044] In this embodiment, original surveillance image samples are obtained from the original surveillance video stream by video frame extraction. Video frame extraction involves extracting several frames from the video stream at regular intervals through video encoding and decoding, simulating taking a photo at regular intervals. In this embodiment, a fixed number of frames are extracted per second, for example, two frames. That is, if the original surveillance video stream is 30 seconds long, then 60 original surveillance image samples are extracted.

[0045] 203. Use the preset annotation software to annotate the intersecting areas and packages in the original surveillance image samples;

[0046] In this embodiment, the annotation software 1abelme is used to annotate the original monitoring image sample with labels for the cross-band areas and the wrapping areas. The label for the cross-band area is the outline of the cross-band area, and the label for the wrapping area is the outline of the wrapping area. This yields JSON files of the cross-band areas and their corresponding labels, as well as JSON files of the wrapping areas and their corresponding labels in the original monitoring image sample.

[0047] 204. Based on the labeled original surveillance image samples, the model was trained to obtain the cross-belt recognition model and the package localization model, respectively.

[0048] 205. Obtain the monitoring image to be identified. Input the monitoring image to be identified into the cross-belt recognition model and the package positioning model respectively to obtain the cross-belt mask image and the package positioning information of the package in the monitoring image to be identified.

[0049] 206. Obtain the pixel information of the crossband mask image in the monitoring image to be identified, and obtain the first pixel set corresponding to the crossband mask image;

[0050] 207. Based on the package location information of each package, obtain the corresponding pixel information in the monitoring image to be identified, and obtain the second pixel set corresponding to each package;

[0051] 208. Determine whether the set of second pixels corresponding to each package is a proper subset of the set of first pixels;

[0052] 209. Identify the package corresponding to the second pixel set, which is a proper subset of the first pixel set, as an off-package.

[0053] In practical applications, various methods can be used to identify deviated packages in a monitoring image based on package location information and cross-band mask images. For example, by determining the position of the cross-band mask image in the monitoring image, the cross-band region in the monitoring image can be identified. Package location information can then be used to determine if the corresponding package is within the cross-band region. If it is, the package corresponding to the package location information is identified as a deviated package in the monitoring image; otherwise, it indicates that the package corresponding to the package location information has not deviated in the monitoring image. In this embodiment, the main method is to determine whether the pixels of the cross-band mask image in the monitoring image overlap with the pixels corresponding to the package location information, thereby determining whether the corresponding package is deviated. Pixel overlap indicates whether the second pixel set corresponding to each package is a proper subset of the first pixel set. If so, the package corresponding to the second pixel set that is a proper subset of the first pixel set is identified as a deviated package.

[0054] This embodiment, building upon the previous embodiment, details the process of constructing a complete test set based on standardized driving data from the target area. Following preset time intervals, the standardized driving data of the target area is processed using a sliding window to obtain multiple first sub-scenes. Time information corresponding to each first sub-scene is acquired, and based on this time information, a simulated destination is determined for each first sub-scene. The simulated destination is then set as a simulation factor for the corresponding first sub-scene. All first sub-scenes and their corresponding simulation factors are aggregated to obtain the complete test set. This method constructs a complete driving set using standardized driving data and utilizes a scenario complexity evaluation model to construct a refined test set, thereby improving the complexity and comprehensiveness of the simulated scenarios in the target area and ultimately enhancing the accuracy of package deviation recognition.

[0055] Please see Figure 3 The third embodiment of the package deviation identification method in this invention includes:

[0056] 301. Collect raw surveillance image samples and label the cross-band areas and packages in the raw surveillance image samples;

[0057] 302. Divide the labeled original surveillance image samples into training set and test set;

[0058] 303. Based on the training set and a preset instance segmentation model neural network, a crossband recognition model is trained.

[0059] A crossband recognition model was trained based on the MaskRcnn network architecture, and a package localization model was trained based on the CenterNet network architecture.

[0060] In this embodiment, the instance segmentation model neural network includes a feature extraction network, a target region coarse extraction network, and a crossband segmentation network. The crossband recognition model is trained through the following process: the training set is input into the instance segmentation model neural network, and the feature extraction network in the instance segmentation model neural network obtains the feature image corresponding to the training set; the feature image is input into the target region coarse extraction network in the instance segmentation model neural network to obtain the corresponding target region feature image; the target region feature image is input into the crossband segmentation network to obtain the region segmentation result; the region segmentation result and the training set are input into a preset loss function to calculate the first loss value; it is determined whether the first loss value is less than a preset loss threshold; if not, the model parameters of the instance segmentation model neural network are updated based on the first loss value, and the process returns to the step of inputting the training set into the instance segmentation model neural network until the first loss value is less than the preset loss threshold; if yes, the crossband recognition model is generated based on the model parameters of the instance segmentation model neural network. In this embodiment, the target region coarse extraction network is constructed based on the Feature Pyramid Network (FPN), the Region Extraction Network (RPN), and the Region of Interest (RoI) Align. The process of inputting the feature image into the target region coarse extraction network of the instance segmentation model neural network to obtain the corresponding target region feature image includes: inputting the feature image into the FPN of the target region coarse extraction network, where the FPN segments the feature image into images of different sizes; inputting the images of different sizes into the RPN, where the RPN calculates the target pre-selected regions of the crossbands; performing multi-scale feature fusion based on the images of different sizes and the target pre-selected regions to obtain a feature tensor; and inputting the feature tensor into the RoI Align, where the RoI Align adjusts the target pre-selected regions to obtain the corresponding target region feature image.

[0061] 304. Based on the training set and a pre-set target detection network, a package localization model is trained.

[0062] In this embodiment, the target detection network is CenterNet, and the package localization model is trained through the following steps: inputting the training set into CenterNet to obtain the predicted labels and embedding vectors of the training set; calculating the second loss value of the predicted labels and the package labels in the original monitoring image samples after annotation; calculating the third loss value of the embedding vectors, and using the second loss value and the third loss value as the total loss value of the target detection network; using the gradient descent method, iteratively training CenterNet with the total loss value to obtain the package localization model.

[0063] 305. Verify the detection performance of the cross-belt recognition model and the package localization model using test sets respectively;

[0064] 306. Obtain the monitoring image to be identified, and in the cross-band recognition model of the monitoring image to be identified, obtain the feature image corresponding to the monitoring image to be identified through the feature extraction network in the cross-band recognition model.

[0065] 307. Input the feature map into the FPN of the target region coarse extraction network, and use the FPN to segment the feature image into images of different sizes;

[0066] 308. Input images of different sizes into the RPN, calculate the target pre-selection region of the cross band through the RPN, and perform multi-scale feature fusion based on images of different sizes and target pre-selection regions to obtain the feature tensor;

[0067] In this embodiment, the number of anchors in the RPN can be reduced. Anchors represent K windows of different sizes and aspect ratios on the feature image during point observation. These K windows are sorted in descending order of size, and windows at the bottom of the list are deleted according to a preset ratio. In this embodiment, the preset ratio is set to 50%, meaning the last K / 2 anchors are deleted. Since the target to be segmented is mainly intersecting bands, and the target is relatively large in the environment, reducing the number of anchors in the RPN effectively reduces the number of computational steps, thereby greatly improving computational efficiency.

[0068] 309. Input the feature tensor into RoI Align, and adjust the target pre-selected region through RoI Align to obtain the corresponding target region feature image;

[0069] 310. Input the feature image of the target region into the cross-band segmentation network to obtain the cross-band mask image;

[0070] 311. Input the monitoring image to be identified into the package location model to obtain the package location information.

[0071] This embodiment, based on the previous embodiment, describes in detail the training process of the contour recognition model. The contour recognition model can recognize hair contours based on the pixel values ​​of different regions of a face image, with high recognition accuracy, making subsequent calculation of hair detail information more accurate.

[0072] The package deviation recognition method in the embodiments of the present invention has been described above. The package deviation recognition device in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 4 One embodiment of the package deviation identification device in this invention includes:

[0073] The sample acquisition module 401 is used to acquire raw surveillance image samples and to label the cross-band areas and packages in the raw surveillance image samples.

[0074] The model training module 402 is used to train the model based on the labeled original surveillance image samples to obtain the cross-belt recognition model and the package localization model, respectively.

[0075] The model input module 403 is used to acquire the monitoring image to be identified, and input the monitoring image to be identified into the cross-belt identification model and the package positioning model respectively, to obtain the cross-belt mask image and the package positioning information of the package in the monitoring image to be identified;

[0076] The identification module 404 is used to identify deviated packages in the monitoring image to be identified based on the package location information and the cross-belt mask image.

[0077] In this embodiment of the invention, the package deviation identification device operates the aforementioned package deviation identification method. The device acquires original monitoring image samples and labels the cross-belt areas and packages within these samples. Based on the labeled original monitoring image samples, a model is trained to obtain a cross-belt identification model and a package positioning model. A monitoring image to be identified is acquired and input into both the cross-belt identification model and the package positioning model to obtain a cross-belt mask image and package positioning information. Based on the package positioning information and the cross-belt mask image, the deviated package in the monitoring image is identified. This method uses computer vision technology to identify package deviation, enabling real-time tracking of monitored packages, reducing the likelihood of missed detections, improving the identification rate, and providing timely alerts for deviated packages to notify relevant personnel for handling, thereby improving logistics and transportation efficiency.

[0078] Please see Figure 5 A second embodiment of the package deviation identification device in this invention includes:

[0079] The sample acquisition module 401 is used to acquire raw surveillance image samples and to label the cross-band areas and packages in the raw surveillance image samples.

[0080] The model training module 402 is used to train the model based on the labeled original surveillance image samples to obtain the cross-belt recognition model and the package localization model, respectively.

[0081] The model input module 403 is used to acquire the monitoring image to be identified, and input the monitoring image to be identified into the cross-belt identification model and the package positioning model respectively, to obtain the cross-belt mask image and the package positioning information of the package in the monitoring image to be identified;

[0082] The identification module 404 is used to identify deviated packages in the monitoring image to be identified based on the package location information and the cross-belt mask image.

[0083] In this embodiment, the sample acquisition module is specifically used to: acquire the original surveillance video stream; perform frame-by-frame processing on the original surveillance video stream to obtain original surveillance image samples; and use preset annotation software to annotate the cross-band regions and wrap-ups in the original surveillance image samples.

[0084] In this embodiment, the model training module 402 specifically includes: a sample partitioning unit 4021, used to partition the labeled original surveillance image samples into a training set and a test set; a first training unit 4022, used to train a cross-strip recognition model based on the training set and a preset instance segmentation model neural network; a second training unit 4023, used to train a package location model based on the training set and a preset target detection network; and a testing unit 4024, used to verify the detection effects of the cross-strip recognition model and the package location model through the test set, respectively.

[0085] In this embodiment, the crossband recognition model includes a feature extraction network, a target region coarse extraction network, and a crossband segmentation network. The model input module 403 specifically includes: a feature extraction unit 4031, used to acquire the monitoring image to be identified, and to obtain a feature image corresponding to the monitoring image to be identified by using the feature extraction network in the crossband recognition model; a region coarse extraction unit 4032, used to input the feature image into the target region coarse extraction network in the crossband recognition model to obtain a corresponding target region feature image; a mask unit 4033, used to input the target region feature image into the crossband segmentation network to obtain a crossband mask image; and a package positioning unit 4034, used to input the monitoring image to be identified into the package positioning model to obtain package positioning information.

[0086] In this embodiment, the target region coarse extraction network is constructed based on the Feature Pyramid Network (FPN), the Region Extraction Network (RPN), and the Region of Interest (RoI) Align. The region coarse extraction unit 4032 is specifically used for: inputting the feature map into the FPN of the target region coarse extraction network, and segmenting the feature image into images of different sizes through the FPN; inputting the images of different sizes into the RPN, calculating the target pre-selected region of the cross band through the RPN, performing multi-scale feature fusion based on the images of different sizes and the target pre-selected region to obtain a feature tensor; inputting the feature tensor into RoIAlign, and adjusting the target pre-selected region through RoIAlign to obtain the corresponding target region feature image.

[0087] In this embodiment, the target detection network is CenterNet, and the second training unit 4023 is specifically used for: inputting the training set into CenterNet to obtain the predicted labels and embedding vectors of the training set; calculating the second loss value of the predicted labels and the package labels in the annotated original surveillance image samples; calculating the third loss value of the embedding vectors, and using the second loss value and the third loss value as the total loss value of the target detection network; using the gradient descent method, iteratively training CenterNet with the total loss value to obtain the package localization model.

[0088] In this embodiment, the identification module 404 is specifically used to: obtain pixel information of the cross-belt mask image in the monitoring image to be identified, and obtain a first pixel set corresponding to the cross-belt mask image; obtain the corresponding pixel information in the monitoring image to be identified according to the package positioning information corresponding to each package, and obtain a second pixel set corresponding to each package; determine whether the second pixel set corresponding to each package is a proper subset of the first pixel set; and identify the package corresponding to the second pixel set that is a proper subset of the first pixel set as an off-set package.

[0089] This embodiment, based on the previous embodiment, describes in detail the specific functions of each module and the unit composition of some modules. It uses computer vision technology to identify package deviation, enabling real-time tracking of monitored packages, reducing the likelihood of missed detections, improving the recognition rate, and promptly alerting relevant personnel to deviated packages, thereby improving logistics and transportation efficiency.

[0090] above Figure 4 and Figure 5 The package deviation identification device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The package deviation identification device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0091] Figure 6This is a schematic diagram of a package deviation identification device 600 provided in an embodiment of the present invention. The package deviation identification device 600 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 610 (e.g., one or more processors) and a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) for storing application programs 633 or data 632. The memory 620 and storage media 630 can be temporary or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the package deviation identification device 600. Furthermore, the processor 610 may be configured to communicate with the storage media 630 and execute the series of instruction operations in the storage media 630 on the package deviation identification device 600 to implement the steps of the above-described package deviation identification method.

[0092] The package deviation identification device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 6 The illustrated package deviation identification device structure does not constitute a limitation on the package deviation identification device provided in this application. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0093] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the package deviation identification method.

[0094] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying package deviation, characterized in that, The package deviation identification method includes: Collect raw surveillance image samples and label the cross-band areas and packages in the raw surveillance image samples; Models were trained based on labeled original surveillance image samples to obtain cross-belt recognition model and package localization model, respectively. The monitoring image to be identified is obtained, and the monitoring image to be identified is input into the cross-belt identification model and the package positioning model respectively to obtain the cross-belt mask image and the package positioning information of the package in the monitoring image to be identified. Based on the package location information and the cross-band mask image, identify the off-center packages in the monitoring image to be identified; The cross-strip recognition model includes a feature extraction network, a target region coarse extraction network, and a cross-strip segmentation network. The process of acquiring the monitoring image to be identified and inputting it into the cross-strip recognition model and the package location model to obtain the cross-strip mask image and package location information from the monitoring image includes: acquiring the monitoring image to be identified; inputting the monitoring image into the cross-strip recognition model; obtaining a feature image corresponding to the monitoring image through the feature extraction network in the cross-strip recognition model; inputting the feature image into the target region coarse extraction network in the cross-strip recognition model to obtain a corresponding target region feature image; inputting the target region feature image into the cross-strip segmentation network to obtain a cross-strip mask image; and inputting the monitoring image to be identified into the package location model to obtain the package location information.

2. The package deviation identification method according to claim 1, characterized in that, The process of acquiring raw surveillance image samples and annotating the intersecting areas and packages in the raw surveillance image samples includes: Obtain the raw surveillance video stream; The original surveillance video stream is processed by frame segmentation to obtain original surveillance image samples; The cross-band areas and packages in the original surveillance image samples are labeled using preset labeling software.

3. The package deviation identification method according to claim 1, characterized in that, The model training based on the labeled original surveillance image samples yields the following models: a cross-belt recognition model and a package localization model. The labeled original surveillance image samples are divided into training set and test set; Based on the training set and the preset instance segmentation model neural network, a crossband recognition model is trained. Based on the training set and the preset target detection network, a package localization model is trained. The detection performance of the cross-belt recognition model and the package location model was verified using the test set, respectively.

4. The package deviation identification method according to claim 3, characterized in that, The target region coarse extraction network is constructed based on the feature pyramid network FPN, the region extraction network RPN, and the region of interest matching network RoI Align. The step of inputting the feature image into the target region coarse extraction network in the crossband recognition model to obtain the corresponding target region feature image includes: The feature map is input into the FPN of the target region coarse extraction network, and the FPN segments the feature image into images of different sizes. Images of different sizes are input into the RPN, and the RPN calculates the target pre-selected region of the cross band. Multi-scale feature fusion is performed based on images of different sizes and target pre-selected regions to obtain the feature tensor. The feature tensor is input into RoI Align, and the target pre-selected region is adjusted by RoI Align to obtain the corresponding target region feature image.

5. The package deviation identification method according to claim 3, characterized in that, The target detection network is CenterNet; The package localization model trained based on the training set and the preset target detection network includes: The training set is input into CenterNet to obtain the predicted labels and embedding vectors of the training set; Calculate the second loss value of the package label in the predicted label and the labeled original surveillance image sample; Calculate the third loss value of the embedding vector, and use the second loss value and the third loss value as the total loss value of the target detection network; The package localization model is obtained by iteratively training the CenterNet using the gradient descent method and the total loss value.

6. The package deviation identification method according to any one of claims 1-5, characterized in that, The step of identifying deviated packages in the monitoring image to be identified based on the package location information and the cross-band mask image includes: Obtain the pixel information of the crossband mask image in the monitoring image to be identified, and obtain the first pixel set corresponding to the crossband mask image; Based on the package location information corresponding to each package, obtain the corresponding pixel information in the monitoring image to be identified, and obtain the second pixel set corresponding to each package; Determine whether the set of second pixels corresponding to each package is a proper subset of the set of first pixels; The package corresponding to the second pixel set, which is a proper subset of the first pixel set, is identified as an off-package.

7. A package deviation detection device, characterized in that, The package deviation detection device includes: The sample acquisition module is used to acquire raw surveillance image samples and to label the cross-band areas and packages in the raw surveillance image samples. The model training module is used to train models based on labeled original surveillance image samples, resulting in cross-band recognition model and package localization model, respectively. The model input module is used to acquire the monitoring image to be identified, and input the monitoring image to be identified into the cross-belt identification model and the package positioning model respectively, to obtain the cross-belt mask image and the package positioning information of the package in the monitoring image to be identified; The identification module is used to identify deviated packages in the monitoring image to be identified based on the package location information and the cross-belt mask image; The cross-band recognition model includes a feature extraction network, a target region coarse extraction network, and a cross-band segmentation network. The model input module specifically includes: a feature extraction unit, used to acquire the monitoring image to be recognized, and to obtain a feature image corresponding to the monitoring image to be recognized through the feature extraction network in the cross-band recognition model; a region coarse extraction unit, used to input the feature image into the target region coarse extraction network in the cross-band recognition model to obtain a corresponding target region feature image; a mask unit, used to input the target region feature image into the cross-band segmentation network to obtain a cross-band mask image; and a package location unit, used to input the monitoring image to be recognized into the package location model to obtain package location information.

8. A package deviation detection device, characterized in that, The package deviation identification device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the package deviation identification device to perform the steps of the package deviation identification method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the package deviation identification method as described in any one of claims 1-6.

Images