Methods, devices, equipment and media for detecting the delivery status of goods

Abstract

This invention relates to the field of logistics technology and discloses a method for detecting the delivery status of items: acquiring a shipment image, identifying the item in the shipment image using a target detection model to obtain the shipment item; acquiring a color image sequence and a depth image sequence corresponding to the shipment item, determining the movement trajectory of the shipment item based on the color image sequence and depth image sequence; identifying the status of the movement trajectory using a trajectory analysis model to obtain the trajectory status; acquiring weight change data, and determining the current status of the shipment item based on the weight change data and trajectory status. This invention achieves the fusion of multiple sensing modes through shipment images, color image sequences, and depth image sequences, thereby avoiding the limitations of a single sensing mode and improving the accuracy of trajectory recognition. By using weight change data and trajectory status, cross-validation of multi-dimensional information is achieved, improving the accuracy of item status recognition and thus preventing missed shipments.

Description

Technical Field

[0001] This invention relates to the field of logistics technology, and in particular to a method, apparatus, equipment and medium for detecting the delivery status of goods. Background Technology

[0002] Existing locker systems typically include identity verification, electronic waybill management, and process control logic based on locker status sensors. Successful parcel delivery is determined solely by the physical signal of the locker door closing. However, this method has two drawbacks: firstly, its limited sensing capabilities result in a lack of item tracking; secondly, it fails to verify changes in the actual state of the locker interior. Therefore, a new detection method is urgently needed to address these issues. Summary of the Invention

[0003] This invention provides a method, apparatus, device, and medium for detecting the delivery status of items, in order to solve the problem in the prior art that only a single physical signal is relied upon to determine whether the delivery was successful.

[0004] A method for detecting the delivery status of an item includes: Acquire a shipping image, and use a target detection model to identify the item in the shipping image to obtain the shipping item; Obtain the color image sequence and depth image sequence corresponding to the mailed item, and determine the movement trajectory of the mailed item based on the color image sequence and the depth image sequence; The trajectory state is obtained by identifying the state of the movement trajectory using a trajectory analysis model; Obtain weight change data, and determine the current status of the item to be shipped based on the weight change data and the trajectory status.

[0005] A delivery status detection device, comprising: The item recognition module is used to acquire the image of the package being shipped, and to identify the item in the image by using a target detection model; The trajectory determination module is used to acquire a color image sequence and a depth image sequence corresponding to the mailed item, and determine the movement trajectory of the mailed item based on the color image sequence and the depth image sequence. The state recognition module is used to identify the state of the movement trajectory through the trajectory analysis model to obtain the trajectory state; The status determination module is used to acquire weight change data and determine the current status of the item to be sent based on the weight change data and the trajectory status.

[0006] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor being used to perform the above-described item delivery status detection method.

[0007] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting the delivery status of an item.

[0008] The aforementioned method, apparatus, equipment, and medium for detecting the delivery status of items, in this invention, identify items in a package delivery image using a target detection model, thereby determining the item being delivered. By using color and depth image sequences, the movement trajectory of the item in three-dimensional space is determined, achieving the fusion of multiple sensing modes and avoiding the limitations of a single sensing mode, thus improving the accuracy of trajectory recognition. Through a trajectory analysis model, the status of the movement trajectory is identified, thereby determining whether the item has been placed in the designated slot. By using weight change data and trajectory status, cross-validation of multi-dimensional information is achieved, thereby improving the accuracy of item status recognition and preventing missed deliveries. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of a method for detecting the delivery status of items according to an embodiment of the present invention; Figure 2 This is a flowchart of step S10 of the item delivery status detection method in one embodiment of the present invention; Figure 3 This is a flowchart of step S20 of the item delivery status detection method in one embodiment of the present invention; Figure 4 This is a flowchart of step S30 of the item delivery status detection method in one embodiment of the present invention; Figure 5 This is a schematic block diagram of an item delivery status detection device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the movement trajectory of an item delivery in one embodiment of the present invention; Figure 7 This is a schematic diagram of the moving speed scenario for item delivery in one embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] The terms “first,” “second,” “third,” “fourth,” etc., in this application (if present) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0013] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0014] In one embodiment, such as Figure 1 As shown, a method for detecting the delivery status of an item is provided, including the following steps: S10: Acquire a shipment image, and use a target detection model to identify the item in the shipment image to obtain the shipment item.

[0015] In essence, a package drop image refers to an image captured by a camera of a user dropping off a package. This can be a single image or multiple consecutive images, captured by a camera inside the locker compartment. An object detection model is a model trained on a large amount of sample data, used to identify the shape and material of express packages, as well as images of handheld packages under different lighting and occlusion conditions within the locker environment. The item being dropped off refers to the object the user is dropping off; this can be one item or multiple items.

[0016] Specifically, after initiating the parcel delivery process and assigning a corresponding compartment, the wide-angle camera, depth sensor, and dynamic weighing unit within that compartment are simultaneously activated. Then, a parcel delivery image is acquired and input into a target detection model. The model, trained during development, learns to recognize package features, thereby identifying objects in the image that match package characteristics and determining them as the parcel being delivered.

[0017] S20: Obtain the color image sequence and depth image sequence corresponding to the item being sent, and determine the movement trajectory of the item being sent based on the color image sequence and the depth image sequence.

[0018] Understandably, a color image sequence refers to RGB images of the item being shipped at different times. A depth image sequence refers to depth maps of the item being shipped and the compartment plane at different times. A movement trajectory refers to the movement path of the item being shipped from when the locker door opens to when it closes.

[0019] Specifically, the process involves acquiring the color image sequence and depth image sequence corresponding to the item being shipped. Then, based on the two-dimensional coordinates of the item in the color image at each moment in the color image sequence, and combined with the depth image at the same moment in the depth image sequence, the two-dimensional coordinates of the item are converted into three-dimensional coordinates. Finally, based on all the three-dimensional coordinates in chronological order, the movement trajectory of the item is generated.

[0020] S30: The trajectory is identified by the trajectory analysis model to obtain the trajectory status.

[0021] Understandably, the trajectory status is used to characterize whether a movement trajectory is a normal delivery trajectory. Specifically, a normal delivery trajectory is one where the movement trajectory moves towards the grid opening and enters the grid opening plane. An abnormal delivery trajectory is one where the movement trajectory moves towards the grid opening, enters the grid opening plane, and then returns to a position outside the grid opening plane; or, if the movement trajectory pauses before the grid opening plane or fails to enter the grid opening plane, it is also considered an abnormal delivery trajectory.

[0022] Specifically, a trajectory analysis model is acquired, and the movement trajectory is input into the model. The trajectory analysis model identifies the movement trajectory to determine whether it is normal. Specifically, by analyzing parameters such as trajectory curvature, smoothness, and speed, a trajectory is considered normal if the trajectory curvature and smoothness indicate the item is moving towards the compartment and the speed exhibits a single-peak pattern. Conversely, an abnormal delivery trajectory is identified if the trajectory curvature and smoothness indicate the item is moving towards or away from the compartment and the speed exhibits a multi-peak pattern.

[0023] S40: Obtain weight change data, and determine the current status of the item to be sent based on the weight change data and the trajectory status.

[0024] Understandably, weight change data refers to the weight data measured by the weight sensor inside the compartment. Specifically, when the user has not placed the item in the compartment, the weight change data is zero. The current status indicates whether the item has been successfully delivered.

[0025] Specifically, the system acquires weight change data uploaded by the weight sensor inside the parcel compartment. Then, based on the weight change data and the trajectory status, it determines the current status of the parcel. That is, if the weight change data indicates a weight change and the trajectory status is a normal delivery trajectory, the current status is determined to be a successful delivery; otherwise, it is considered an abnormal delivery trajectory, and the corresponding exception handling operation is executed.

[0026] In the item delivery status detection method of this invention, a target detection model is used to identify the items in the package delivery image, thereby determining the item being delivered. Color image sequences and depth image sequences are used to determine the movement trajectory of the item in three-dimensional space, achieving the fusion of multiple sensing modes and avoiding the limitations of a single sensing mode, thus improving the accuracy of trajectory recognition. A trajectory analysis model is used to identify the movement trajectory status, thereby determining whether the item has been placed in the designated slot. Cross-validation of multi-dimensional information is achieved through weight change data and trajectory status, thereby improving the accuracy of item status recognition and preventing missed deliveries.

[0027] In one embodiment, such as Figure 2 As shown, in step S10, the process of identifying the item in the mailing image using a target detection model to obtain the mailing item includes: S101, the mail image is subjected to feature extraction at different scales through the backbone network in the target detection model to obtain the extracted features.

[0028] S102, the neck network in the target detection model is used to fuse all the extracted features in different dimensions to obtain fused features.

[0029] S103, the head network in the target detection model is used to predict all the fused features to obtain an object that conforms to the package feature.

[0030] Understandably, the target detection model in this embodiment is a lightweight model, such as YOLOv5s, YOLOX-s, YOLO11x, etc. Feature extraction refers to features extracted from different dimensions of the mail image. Feature fusion refers to the fusion of features from different dimensions. Package features refer to features that match the mail package.

[0031] Specifically, the image of the mailed item is enhanced by first randomly selecting the coordinates (xc, yc) of a reference point for stitching the mailed item image. Multiple images are then randomly selected, and each image is resized and scaled according to the reference point. These images are then placed at preset positions on a larger image of a specified size, such as top left, top right, bottom left, and bottom right. Next, based on the size transformation method of each image, the mapping relationship is mapped to image labels, and the larger image is stitched together according to the specified horizontal and vertical coordinates to obtain the enhanced image.

[0032] Then, the backbone network in the object detection model extracts features from the enhanced image at different scales. Specifically, the enhanced image is convolved using 2D convolution, normalization, and the SiLU activation function in the convolution module to reduce dimensionality and enhance nonlinearity, thus obtaining the first convolutional features. These first convolutional features are then extracted using the C2f module. This involves reducing the number of channels, extracting features through the first convolution, concatenating the input and output using residual connections, and finally restoring the number of channels to obtain the first extracted features. Next, the convolution module and the C2f module extract the first extracted features at different scales. After the last convolution module and the C2f module, the SPPF module performs multiple consecutive max pooling operations and residual connections on the extracted features from the previous dimension, resulting in extracted features at different scales.

[0033] Next, the neck network in the object detection model performs feature fusion across different dimensions on all extracted features. Specifically, the first extracted feature of the last dimension is upsampled and fused with the first extracted feature of the previous dimension to obtain the first feature. This first feature is then passed through a C2f layer and upsampled before being fused with the first extracted feature of the previous dimension to obtain the first-dimensional feature. The first-dimensional feature is then passed through a convolutional layer and fused with the first feature to obtain the second feature. This second feature is then passed through a C2f layer to obtain the second-dimensional feature. Finally, the second-dimensional feature is passed through a convolutional layer and fused with the first extracted feature of the last dimension to obtain the third feature. This third feature is then passed through a C2f layer to obtain the third-dimensional feature. These three features are then defined as the fused features.

[0034] Finally, the head network in the object detection model performs prediction processing on all the fused features. Specifically, the detect layer predicts the category of each feature dimension, and threshold filtering eliminates low-confidence predictions. Non-maximum suppression (NMS) eliminates redundant categories, thus obtaining objects that match the wrapping features. The working principle of NMS is as follows: for each target category, the object with the highest confidence is selected as a reference, and only the most likely object is retained.

[0035] S104. After marking objects that meet the characteristics of a parcel, the marked objects are identified as items to be sent.

[0036] Specifically, when an object matching the characteristics of a package is detected, a tracking tag is assigned to the object, i.e., the object is marked for tracking throughout the delivery process. Then, the object is identified as the item to be sent.

[0037] In this embodiment, a backbone network is used to extract features at different scales. A neck network is used to fuse features from different dimensions. A head network is used to identify objects in the parcel image that match the characteristics of a package, thereby enabling the labeling of identified objects and the determination of the parcel item, thus improving the accuracy of parcel item identification.

[0038] In one embodiment, such as Figure 3 As shown, in step S20, the process of determining the movement trajectory of the mailed item based on the color image sequence and the depth image sequence includes: S201, In the color image sequence, the location of the mailed item is identified to obtain the two-dimensional item coordinates at different times.

[0039] S202, perform coordinate transformation on all the coordinates of the two-dimensional objects according to the depth image sequence to obtain the coordinates of the three-dimensional objects corresponding to each of the two-dimensional objects.

[0040] S203, sort all the coordinates of the three-dimensional items according to the order of all time points to obtain the movement trajectory of the sent item.

[0041] In essence, two-dimensional item coordinates refer to the coordinates of the item being shipped in a plane. Three-dimensional item coordinates refer to the coordinates of the item being shipped in the three-dimensional space of the storage compartment.

[0042] Specifically, after obtaining the color image sequence and depth image sequence, the location of the mailed item in the color images at different times is determined by tracking markers, and the pixel coordinates of the center point of the mailed item in the color images at different times are determined, thus obtaining the two-dimensional coordinates of the mailed item at different times. This two-dimensional coordinates of the mailed item at different times can be determined by jointly analyzing images captured by multiple cameras.

[0043] Then, coordinate transformation is performed on all two-dimensional item coordinates based on the depth image sequence, that is, the depth image at the same moment as the color image is determined, and the depth value of the item being sent in the depth image is determined. Based on the two-dimensional item coordinates and depth value at the same moment, the three-dimensional pixel coordinates are determined. Then, the three-dimensional pixel coordinates are converted into three-dimensional world coordinates relative to the grid plane according to the preset camera parameters, thus obtaining the three-dimensional item coordinates of the item being sent at different moments.

[0044] In one embodiment, the pixel coordinates (u, v) of the center point of the two-dimensional bounding box are paired with the depth value d in the depth map acquired by the depth sensor at the same time to form a three-dimensional observation point (u, v, d). Here, (u, v) represents the pixel position of the target in the image, and d represents the actual distance between the point and the camera. Then, using pre-calibrated camera parameters (such as focal length, optical center, distortion coefficients, etc.), perspective transformation is used to convert the point (u, v, d) in the pixel coordinate system into three-dimensional world coordinates (x, y, z) relative to the opening of the express delivery locker.

[0045] Next, all three-dimensional object coordinates are sorted according to the order of all times, that is, the chronological order of all three-dimensional object coordinates at different times is sorted, and a smooth curve is used to connect all three-dimensional object coordinates to obtain the movement trajectory of the sent item. In one embodiment, the three-dimensional world coordinates of the sent item are sorted by time to form a motion trajectory sequence: {P1(x, y, z), P2(x, y, z), ..., P}, which accurately describes the movement process of the item in space.

[0046] In this embodiment, the pixel coordinates of the item being shipped in two-dimensional space are determined using a color image sequence. The pixel coordinates of the item being shipped in three-dimensional space are determined using a depth image sequence, and the transformation of these pixel coordinates is also achieved. This results in the determination of the three-dimensional item coordinates, which in turn enables the generation of the movement trajectory, improving the accuracy of trajectory recognition.

[0047] In one embodiment, such as Figure 4 As shown, in step S30, the process of identifying the state of the movement trajectory using a trajectory analysis model to obtain the trajectory state includes: S301, the movement trajectory is feature extracted through the first branch of the trajectory analysis model to obtain the motion features of the object.

[0048] S302, the movement trajectory is feature extracted through the second branch of the trajectory analysis model to obtain trajectory morphology features.

[0049] S303, the movement trajectory is feature extracted through the third branch of the trajectory analysis model to obtain behavioral event features.

[0050] S304, input the object motion features, the trajectory morphology features and the behavioral event features into the time-series branch of the trajectory analysis model, and obtain the trajectory state output by the time-series branch.

[0051] Understandably, the first branch refers to the branch used to extract motion features of the object. The second branch refers to the branch used to extract trajectory morphology features. The third branch refers to the branch used to extract behavioral event features. The temporal branch refers to the branch used to determine whether the trajectory state is normal, for example, in a one-dimensional CNN network or GRU network.

[0052] Specifically, the first branch of the trajectory analysis model extracts features from the movement trajectory, specifically the velocity in the Z-direction (the direction towards the grid), thus obtaining the item's motion characteristics. For example, it extracts the trend of velocity change in the Z-direction. Similarly, the second branch of the trajectory analysis model extracts features from the movement trajectory, specifically the trajectory curvature, smoothness, and distance from the grid plane, thus obtaining the trajectory morphology features. For example, it extracts the trend of motion change relative to the grid plane. Next, the third branch of the trajectory analysis model extracts features from the movement trajectory, specifically analyzing the final position of the movement trajectory relative to the grid plane, thus obtaining behavioral event features. For example, it analyzes the movement trajectory to extract features of the final location of the shipped item.

[0053] Finally, the movement characteristics, trajectory morphology characteristics, and behavioral event characteristics of the item are input into the temporal branch of the trajectory analysis model. The temporal branch determines the specific content corresponding to each feature. Then, based on the specific content corresponding to each feature, it is determined whether the trajectory is a normal delivery trajectory or an abnormal delivery trajectory. For example, a normal delivery trajectory is characterized by continuous movement towards the grid and crossing the grid plane; an abnormal delivery trajectory is characterized by pausing in front of the grid, entering the grid plane and turning back, or failing to reach the grid plane and retracting.

[0054] In this embodiment, by using different branches in the trajectory analysis model, the characteristics of object movement, trajectory morphology, and behavioral events are extracted, thereby enabling the identification of trajectory status and determining whether the delivery trajectory is normal, thus improving the accuracy of trajectory identification.

[0055] In one embodiment, such as Figure 6 and Figure 7 As shown, in step S304, the step of inputting the object motion features, the trajectory morphology features, and the behavior event features into the time-series branch to obtain the trajectory state output by the time-series branch includes: When the trajectory morphology is smooth and inward, the item movement is characterized by a single-peak velocity, and the behavioral event is entering the grid plane, the trajectory state is determined to be a normal delivery trajectory.

[0056] When the trajectory morphology is characterized by a smooth inward trajectory followed by a reversal, the item movement is characterized by a multi-peaked speed, and the behavioral event is characterized by not entering the grid plane, the trajectory state is determined to be an abnormal delivery trajectory.

[0057] Understandably, a normal delivery trajectory refers to a continuous movement of the trajectory towards and across the grid's plane, meaning the user places the item into the grid. An abnormal delivery trajectory refers to the trajectory stopping before the grid's plane, turning back, or retracting before reaching the grid's plane, meaning the user did not place the item into the grid, or the weight in the grid did not change. A single-peak speed pattern means the speed's change trend has one peak. A multi-peak speed pattern means the speed's change trend has multiple peaks.

[0058] Specifically, the movement characteristics, trajectory morphology characteristics, and behavioral event characteristics of the item are input into the temporal branch. The temporal branch then determines these characteristics. When the trajectory morphology is smooth and inward-facing, the item's movement characteristic is a single-peaked velocity, and the behavioral event characteristic is entering the grid plane, the movement trajectory is determined to be the normal delivery path, and its trajectory state is defined as a normal delivery trajectory. For example, in... Figure 6 The middle left image and Figure 7 The velocity change curve shown in Figure 1 shows a smooth inward trajectory, a single-peak velocity, and entry into the grid plane. Similarly, by determining the movement characteristics, trajectory morphology characteristics, and behavioral event characteristics of the item through temporal branching, when the trajectory morphology is a smooth inward trajectory followed by a reversal, the item movement characteristics are a multi-peak velocity, and the behavioral event characteristic is not entering the grid plane, the movement trajectory is determined to be the path of abnormal delivery, and the trajectory state is determined to be an abnormal delivery trajectory. For example, in Figure 6 The middle right side of the image and Figure 7 The velocity change curve shown in Figure 2 has a smooth trajectory that curves inward and then turns back. The velocity exhibits a multi-peak shape (i.e., a pause occurs when exiting) and does not enter the grid plane (including entering and exiting the grid plane).

[0059] In this embodiment, the content is specifically represented by the object's motion characteristics, trajectory morphology characteristics, and behavioral event characteristics, and the cross-validation of multiple features is achieved, thereby determining the trajectory state and improving the accuracy of trajectory state recognition.

[0060] In one embodiment, in step S40, determining the current status of the mailed item based on the weight change data and the trajectory status includes: When the trajectory status represents a normal delivery trajectory and the weight change data represents a change in weight, the current status is determined to be a successful delivery status.

[0061] When the trajectory state represents a normal delivery trajectory and the weight change data represents no weight change, the current state is determined to be the first delivery abnormal state.

[0062] When the trajectory status represents an abnormal delivery trajectory, and / or the weight change data represents no weight change, the current state is determined to be a second abnormal delivery state.

[0063] Understandably, a successful delivery status refers to a state where both the trajectory status and weight change are normal. A first abnormal delivery status refers to a state where the trajectory status is normal, but the weight change is abnormal. A second abnormal delivery status refers to a state where at least one of the trajectory status and / or weight change is abnormal.

[0064] Specifically, when the trajectory status of the item indicates a normal delivery trajectory and the weight change data indicates a weight change, the item is determined to have been successfully delivered, and its current status is set as a successful delivery status. Weight change data can be acquired after the locker door closes, achieving triple cross-validation of trajectory status, weight change, and closing signal, thus improving the accuracy of item status identification. In one embodiment, when the trajectory status of the item indicates a normal delivery trajectory and the weight change data indicates no weight change, the item is determined to have not been successfully delivered, and its current status is set as a first delivery anomaly status. In another embodiment, when the trajectory status of the item indicates an abnormal delivery trajectory and / or the weight change data indicates no weight change—that is, when at least one of the trajectory status and weight change data is abnormal—the item is determined to have not been successfully delivered, and its current status is set as a second delivery anomaly status. Here, the current status can be determined as the second delivery anomaly status when the trajectory status indicates an abnormal delivery trajectory, eliminating the need to further determine the weight change data.

[0065] In this embodiment, the delivery success status, the first delivery failure status, and the second delivery failure status are determined by using trajectory status and weight change data. This enables cross-validation of multi-dimensional information, improves the accuracy of item status recognition, and thus avoids missing items during delivery.

[0066] In one embodiment, after step S40, after determining the current state of the mailed item based on the weight change data and the trajectory status, the method further includes: When the current status is "delivery successful", the logistics status of the sent item will be changed to "collection successful".

[0067] When the current state is the first delivery abnormal state, an alarm prompt corresponding to the first delivery abnormal state is executed, the waybill is locked, and the delivery video is saved.

[0068] When the current state is the second delivery abnormality state, execute the real-time alarm corresponding to the second delivery abnormality state, and change the logistics status of the sent item to pickup failure.

[0069] In essence, logistics status refers to the categorized and described status information of goods during transportation, based on their different stages and states. This information is used for real-time monitoring and tracking of the goods' location and transportation status. Successful pickup signifies the completion of package handover to the user. Alarm notification refers to sending information to designated personnel for processing. Real-time alarm refers to continuous alerts via sound, light, and information notifications. Delivery video refers to video footage of the user placing the package into the designated compartment. Locked waybill means the waybill information is locked and cannot be modified. A waybill is a standardized document in the logistics / express delivery industry that records crucial information about the delivery of goods.

[0070] Specifically, when the current status of the item being sent is "delivery successful," meaning the user places the item in the slot, the item's logistics status is changed to "collection successful." In one embodiment, when the current status is "first delivery abnormality," an alarm corresponding to the first delivery abnormality is triggered, the waybill corresponding to the item is locked, and the delivery video corresponding to the item is saved. The delivery video can be viewed by a pre-defined personnel corresponding to the alarm to determine if the item is in the slot. If the item is present, the waybill is released, and the item's logistics status is changed to "collection successful." If the item is not present, the item's logistics status is changed to "collection failed." In another embodiment, when the current status is "second delivery abnormality," a real-time alarm corresponding to the second delivery abnormality is triggered, i.e., a continuous alarm is triggered, and after the alarm ends, the item's logistics status is changed to "collection failed." The system can immediately send the waybill information and delivery video to the designated personnel for reporting, allowing the designated personnel to stop the real-time alarm.

[0071] In this embodiment, different current states are used to determine different delivery states, thereby determining the corresponding actions to be performed for different delivery states. This ensures that users actually send packages and avoids missing items from being sent.

[0072] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0073] In one embodiment, an item delivery status detection device is provided, which corresponds one-to-one with the item delivery status detection method described in the above embodiments. For example... Figure 5As shown, the item delivery status detection device includes an item recognition module 10, a trajectory determination module 20, a status recognition module 30, and a status determination module 40. Detailed descriptions of each functional module are as follows: The item recognition module 10 is used to acquire a mailing image and perform item recognition on the mailing image using a target detection model to obtain the mailing item. The trajectory determination module 20 is used to acquire a color image sequence and a depth image sequence corresponding to the mailed item, and determine the movement trajectory of the mailed item based on the color image sequence and the depth image sequence. The state recognition module 30 is used to perform state recognition on the movement trajectory through the trajectory analysis model to obtain the trajectory state; The status determination module 40 is used to acquire weight change data and determine the current status of the item to be sent based on the weight change data and the trajectory status.

[0074] In one embodiment, the item recognition module 10 includes: The feature extraction unit is used to extract features from the mail image at different scales through the backbone network in the target detection model to obtain extracted features. The feature fusion unit is used to perform feature fusion of all the extracted features in different dimensions through the neck network in the target detection model to obtain fused features; The object prediction unit is used to predict all the fused features through the head network in the target detection model to obtain objects that conform to the wrapping features; The dispatch item unit is used to identify the marked object as a dispatch item after marking the object that meets the characteristics of a package.

[0075] In one embodiment, the trajectory determination module 20 includes: A location recognition unit is used to identify the location of the mailed item in the color image sequence and obtain two-dimensional item coordinates at different times; A coordinate transformation unit is used to perform coordinate transformation on all the coordinates of the two-dimensional objects according to the depth image sequence to obtain the coordinates of the three-dimensional objects corresponding to each of the two-dimensional object coordinates. The trajectory generation unit is used to sort all the coordinates of the three-dimensional items according to the order of all time points to obtain the movement trajectory of the item being sent.

[0076] In one embodiment, the state recognition module 30 includes: The object motion feature unit is used to extract features from the movement trajectory through the first branch in the trajectory analysis model to obtain the object motion features; The trajectory morphology feature unit is used to extract features from the movement trajectory through the second branch in the trajectory analysis model to obtain trajectory morphology features; The behavioral event feature unit is used to extract features from the movement trajectory through the third branch in the trajectory analysis model to obtain behavioral event features. The trajectory state unit is used to input the object motion characteristics, trajectory morphology characteristics and behavioral event characteristics into the temporal branch of the trajectory analysis model, and obtain the trajectory state output by the temporal branch.

[0077] In one embodiment, the trajectory state unit includes: The normal delivery trajectory subunit is used to determine the trajectory state as a normal delivery trajectory when the trajectory morphology feature is a smooth inward trajectory, the item movement feature is a single-peak velocity, and the behavioral event feature is entering the grid plane. An abnormal delivery trajectory subunit is used to determine that the trajectory state is an abnormal delivery trajectory when the trajectory morphology features are smooth inward and turning back, the item movement features are multi-peak speed, and the behavioral event features are not entering the grid plane.

[0078] In one embodiment, the state determination module 40 includes: A delivery success status unit is used to determine the current status as a delivery success status when the trajectory status represents a normal delivery trajectory and the weight change data represents a change in weight. The first abnormal trajectory unit is used to determine the current state as the first abnormal delivery state when the trajectory state represents a normal delivery trajectory and the weight change data represents no change in weight. The second abnormal trajectory unit is used to determine the current state as the second abnormal delivery state when the trajectory state represents an abnormal delivery trajectory and / or the weight change data represents no weight change.

[0079] In one embodiment, the device further includes: The pickup success unit is used to change the logistics status of the sent item to pickup success when the current status is delivery success status; An abnormal alarm unit is used to execute an alarm prompt corresponding to the first delivery abnormal state when the current state is the first delivery abnormal state, and to lock the waybill and save the delivery video. The anomaly reporting unit is used to execute a real-time alarm corresponding to the second delivery anomaly state when the current state is the second delivery anomaly state, and to change the logistics status of the sent item to pickup failure.

[0080] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor being used to perform the above-described item delivery status detection method.

[0081] Specific limitations regarding the computer equipment, processor, and their various units and modules can be found in the above-described limitations of the item delivery status detection method, and will not be repeated here. Each module in the aforementioned processor can be implemented entirely or partially through software, hardware, or a combination thereof. Understandably, the processor includes a processor, memory, network interface, and database connected via a device bus. Each module of the processor can be embedded in hardware or independent of the processor, or stored in memory as software, so that the processor can call and execute the operations corresponding to each module. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores operating devices, computer programs, and a database. The internal memory provides an environment for the operation of the operating devices and computer programs in the non-volatile storage media. The database stores the data used in the item delivery status detection method in the above embodiments. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an item delivery status detection method.

[0082] In one embodiment, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described item delivery status detection method.

[0083] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0085] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting the delivery status of an item, characterized in that, include: Acquire a shipping image, and use a target detection model to identify the item in the shipping image to obtain the shipping item; Obtain the color image sequence and depth image sequence corresponding to the mailed item, and determine the movement trajectory of the mailed item based on the color image sequence and the depth image sequence; The trajectory state is obtained by identifying the state of the movement trajectory using a trajectory analysis model; Obtain weight change data, and determine the current status of the item to be shipped based on the weight change data and the trajectory status.

2. The method for detecting the delivery status of items as described in claim 1, characterized in that, The step of identifying the item in the mailing image using a target detection model to obtain the mailing item includes: The target detection model uses the backbone network in the target detection model to extract features from the mail image at different scales, thereby obtaining the extracted features. The neck network in the target detection model is used to fuse all the extracted features in different dimensions to obtain fused features. The head network in the target detection model is used to predict all the fused features to obtain objects that conform to the package features; After marking objects that match the characteristics of a package, the marked objects are identified as items to be shipped.

3. The method for detecting the delivery status of items as described in claim 1, characterized in that, Determining the movement trajectory of the mailed item based on the color image sequence and the depth image sequence includes: In the color image sequence, the location of the mailed item is identified to obtain the two-dimensional item coordinates at different times; Based on the depth image sequence, coordinate transformation is performed on all the coordinates of the two-dimensional objects to obtain the coordinates of the three-dimensional objects corresponding to each of the two-dimensional objects. The coordinates of all the three-dimensional items are sorted according to the order of all time points to obtain the movement trajectory of the sent item.

4. The method for detecting the delivery status of items as described in claim 1, characterized in that, The step of identifying the state of the movement trajectory through a trajectory analysis model to obtain the trajectory state includes: The movement characteristics of the object are obtained by extracting features from the movement trajectory through the first branch of the trajectory analysis model. The second branch of the trajectory analysis model is used to extract features from the movement trajectory to obtain trajectory morphology features; The movement trajectory is feature-extracted through the third branch of the trajectory analysis model to obtain behavioral event features. The object motion features, trajectory morphology features, and behavioral event features are input into the temporal branch of the trajectory analysis model to obtain the trajectory state output by the temporal branch.

5. The method for detecting the delivery status of items as described in claim 4, characterized in that, The step of inputting the object motion features, the trajectory morphology features, and the behavioral event features into the time-series branch to obtain the trajectory state output by the time-series branch includes: When the trajectory morphology is smooth and inward, the item movement is characterized by a single-peak velocity, and the behavioral event is entering the grid plane, the trajectory state is determined to be a normal delivery trajectory. When the trajectory morphology is characterized by a smooth inward trajectory followed by a reversal, the item movement is characterized by a multi-peaked speed, and the behavioral event is characterized by not entering the grid plane, the trajectory state is determined to be an abnormal delivery trajectory.

6. The method for detecting the delivery status of items as described in claim 1, characterized in that, Determining the current status of the mailed item based on the weight change data and the trajectory status includes: When the trajectory status represents a normal delivery trajectory and the weight change data represents a change in weight, the current status is determined to be a successful delivery status. When the trajectory state represents a normal delivery trajectory and the weight change data represents no change in weight, the current state is determined to be the first delivery abnormal state. When the trajectory status represents an abnormal delivery trajectory, and / or the weight change data represents no weight change, the current state is determined to be a second abnormal delivery state.

7. The method for detecting the delivery status of items as described in claim 6, characterized in that, After determining the current status of the mailed item based on the weight change data and the trajectory status, the method further includes: When the current status is "delivery successful", the logistics status of the sent item will be changed to "collection successful". When the current state is the first delivery abnormal state, an alarm prompt corresponding to the first delivery abnormal state is executed, the waybill is locked, and the delivery video is saved; When the current state is the second delivery abnormality state, execute the real-time alarm corresponding to the second delivery abnormality state, and change the logistics status of the sent item to pickup failure.

8. A device for detecting the delivery status of an item, characterized in that, include: The item recognition module is used to acquire the image of the package being shipped, and to identify the item in the image by using a target detection model; The trajectory determination module is used to acquire a color image sequence and a depth image sequence corresponding to the mailed item, and determine the movement trajectory of the mailed item based on the color image sequence and the depth image sequence. The state recognition module is used to identify the state of the movement trajectory through the trajectory analysis model to obtain the trajectory state; The status determination module is used to acquire weight change data and determine the current status of the item to be sent based on the weight change data and the trajectory status.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor being used to perform the article delivery status detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the article delivery status detection method as described in any one of claims 1 to 7.

Images