A cargo warehousing automatic checking method based on mobile terminal image recognition
By using mobile image recognition and 3D spatial association technology, the problems of strong dependence on fixed equipment and separation of recognition and verification in existing technologies have been solved, enabling flexible and accurate goods entry verification and adapting to the needs of diverse warehousing scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- YUNNAN JIUJI INFORMATION TECH CO LTD
- Filing Date
- 2026-02-26
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, goods entry verification relies excessively on fixed identification equipment and preset identification stations, resulting in high system deployment costs and limited operating scenarios; the identification and verification processes are separated in space and time, leading to delayed error detection; and in complex scenarios such as batch entry of multiple goods, overlapping of multiple tasks, and code obstruction, there is a lack of effective spatial resolution and dynamic binding mechanisms.
By using mobile image recognition, continuous image frames of the code attached to the surface of the goods are acquired in real time. Combined with visual inertial odometry, the camera pose is obtained. The three-dimensional spatial position is calculated using a multi-view geometric method to generate a three-dimensional anchor point sequence. The three-dimensional spatial proximity and motion consistency measures are calculated to establish the spatial relationship between the goods and the task. Physical binding pairs are generated through a bipartite graph optimal matching algorithm, and the binding strategy is dynamically adjusted to adapt to different operating modes.
It enables real-time and accurate goods entry verification in diverse warehousing scenarios, reduces equipment costs, improves the flexibility and accuracy of identification, reduces the lag in error detection, and adapts to the needs of complex operating scenarios.
Smart Images
Figure CN122155598A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of logistics and warehousing technology, and more specifically, to an automatic verification method for goods entering the warehouse based on mobile terminal image recognition. Background Technology
[0002] With the development of the logistics and warehousing industry, the accuracy and efficiency of goods receiving verification have become important factors affecting the quality of warehousing operations. Traditional receiving verification methods mostly rely on manual barcode verification or fixed automated identification equipment, which have problems such as complex operation procedures, high equipment deployment costs, and insufficient flexibility.
[0003] In the prior art, Chinese patent CN118505114A discloses a warehousing goods receiving system and method based on radio frequency identification and image processing. This solution sets up a fixed identification area, uses a tag reader and a status reader to identify the real-time status of the RFID tag and the goods respectively, and matches the storage strategy through a server. This solution relies on a preset physical identification area and fixed acquisition equipment, which is not suitable for mobile operation scenarios without fixed workstations. Moreover, identification and verification are separated in space, and the goods have already left the operation site when an error is detected.
[0004] Chinese patent CN111798181A discloses a method and system for automatic customer order warehousing based on image recognition. This solution compares order information with goods information through a control center and controls a retrieval robot to retrieve the goods. An image recognition verification module then compares the goods information with the order information. This solution relies on automated equipment such as conveyor belts and robotic arms and is suitable for large-scale automated warehouses. However, for flexible operation scenarios such as small and medium-sized warehouses, temporary storage, and returns processing, it suffers from problems such as high equipment costs, long deployment cycles, and difficulty in rapid adaptation.
[0005] In summary, existing technical solutions generally suffer from problems such as strong dependence on fixed facilities, separation of identification and verification processes, and inability to complete real-time binding during the movement of goods, making it difficult to meet the flexible and ever-changing needs of warehousing operations. Summary of the Invention
[0006] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an automatic goods entry verification method based on mobile terminal image recognition, aiming to solve the following problems in the prior art: Firstly, the over-reliance on fixed identification equipment and preset identification stations for goods entry verification results in high system deployment costs and limited operational scenarios. Secondly, the identification and verification processes are separated in space and time, resulting in delayed error detection and invalid logistics. Third, when faced with complex scenarios such as multiple goods entering the warehouse in batches, multiple tasks overlapping, and codes being obscured, existing methods lack effective spatial resolution and dynamic binding mechanisms. To achieve the above objectives, the present invention provides the following technical solution: an automatic verification method for goods entering the warehouse based on mobile terminal image recognition, comprising the following steps: S1. In response to the warehouse entry verification command, the mobile terminal continuously collects continuous image frames containing the code attached to the surface of the goods, providing raw image data for subsequent identification and positioning. S2. In the continuous image frames, the cargo identification code and task association code are identified in real time, and the two-dimensional image coordinates of each code in each image frame are extracted. To obtain the code content information and its position in the image plane; S3. Synchronously acquire the camera pose information of the mobile terminal at each frame image acquisition time. The camera pose is determined by a rotation matrix. Translation vector This indicates that it provides a camera position and attitude reference for mapping two-dimensional image coordinates to three-dimensional space; S4. For the same cargo identification code or the same task association code appearing in multiple consecutive frames of images, determine the two-dimensional image coordinates of the code in different frames. and the camera pose of the corresponding frame The three-dimensional spatial position of the code in the world coordinate system was calculated using a multi-view geometric method. It is updated iteratively over time to generate confidence-based information. The three-dimensional anchor point sequence elevates the observation of the code from a two-dimensional pixel plane to a three-dimensional physical space; S5. Generate the spatiotemporal trajectory of each code based on the three-dimensional anchor point sequence, and calculate the spatiotemporal trajectories of any two codes within a preset time window. Three-dimensional spatial proximity Measurement of Motion Consistency To quantify the spatial proximity and motion synchronization between codes; S6. Based on the three-dimensional spatial proximity Measurement of Motion Consistency The cargo identification code that meets the association conditions and the task association code are constructed as a physical binding pair to establish a spatial association relationship between cargo and task; S7. The physical binding pair is sent to the server, which retrieves the goods file and task file according to the binding relationship, compares the data, and receives the verification result returned by the server to complete the automatic verification of the goods entry information.
[0007] Furthermore, in step S3, the camera pose information is output in real time as a six-degree-of-freedom camera pose by fusing image feature point tracking data and inertial measurement unit data through a mobile terminal. This provides the camera position and pose corresponding to each frame of image for subsequent 3D reconstruction; The multi-view geometric method in step S4 includes: establishing a spatial ray equation based on the pinhole camera model; solving for the initial values of the three-dimensional coordinates of the observed coordinates and corresponding camera poses of the same code in at least two frames of images; and applying the multi-view geometric method to the continuous time window. The three-dimensional coordinates and camera pose within the image are jointly optimized using bundle adjustment to minimize reprojection error, resulting in the three-dimensional anchor point sequence.
[0008] Furthermore, in step S5, motion consistency measurement Calculated by extracting two codes within a time window. The three-dimensional position sequence at the common moment is used to generate a displacement vector sequence. , Calculate the Pearson correlation coefficient between the two sequences. or dynamic time-warped distance The similarity between the displacement direction and amplitude is used to determine whether two codes have synchronous motion characteristics; if ,or Then determine The two codes have motion consistency.
[0009] Furthermore, in step S5, the three-dimensional spatial proximity... The three-dimensional Euclidean distance between two codes at the same moment or within a time window The average three-dimensional Euclidean distance within the code is used to characterize the proximity of two codes in physical space; The association condition in step S6 is: two codes and ,in That is, a connection can only be established if both spatial proximity and motion synchronization are simultaneously satisfied. When the same cargo identification code and multiple task association codes simultaneously meet the association conditions, the weighted sum of three-dimensional spatial proximity and motion consistency metrics is used as the matching cost. The bipartite graph optimal matching algorithm is used to determine the unique physical binding pair, and the optimal assignment is achieved under multiple candidate conditions.
[0010] Furthermore, after step S4, a three-dimensional spatial clustering and entity segmentation step is included: performing three-dimensional Euclidean distance clustering based on the three-dimensional spatial positions of all codes at the current time, and clustering codes with Euclidean distances less than a certain value. The codes are grouped into the same cluster, and the spatially adjacent code sets are initially filtered. Calculate the three-dimensional normal vector of the plane containing each code. , to obtain the spatial orientation information of the code surface; Within the same cluster, the angle between the three-dimensional normal vectors Multiple codes are merged into the same cargo entity, meaning that codes with the same orientation and spatial proximity are considered to belong to the same physical cargo surface, and this cargo entity is used as a whole to participate in the construction of subsequent physical binding pairs, thus avoiding multiple codes on the same cargo being bound independently.
[0011] Furthermore, before step S6, a dynamic binding strategy adaptive step is included: real-time analysis of the angular velocity magnitude of the mobile terminal. and camera pose change rate It identifies the operator's current operating mode to adapt to the parameter requirements under different work behaviors; When the hovering stabilization mode is detected, the third preset threshold is lowered. Extend the preset time window and increase the first weighting coefficient. Decrease the second weighting coefficient In fine-alignment scenarios, the strictness of spatial proximity criteria is improved and the distance weight is strengthened; When identified as a scanning movement mode, 3D anchor point generation and association determination are performed simultaneously on all codes appearing within a single frame image, and the size of the image is reduced. Increase Strengthen motion consistency criteria and achieve batch processing in fast-moving scenarios; When the view is identified as zoomed-out panoramic mode, the generation of physical binding pairs is paused, while the display of the existing 3D anchor point sequence and physical binding pairs is maintained to avoid misbinding at long distances.
[0012] Furthermore, step S6 also includes a spatial conflict detection and error avoidance step: detecting whether there are two different task association codes with a three-dimensional spatial distance. And each document satisfies the association condition with different cargo identification codes or cargo entities, that is, it is determined whether there are multiple documents that are simultaneously associated with different goods and the documents are close to each other; If a conflict exists, it is determined to be a task conflict. The generation of the physical binding pairs involved in the conflict is suspended and a conflict prompt message is output, prompting the operator to separate the overlapping documents. After the conflict is resolved, the association determination is re-executed to avoid incorrect binding caused by incorrect document pasting or overlap.
[0013] Furthermore, the construction of physical binding pairs in step S6 includes two stages: transient binding and steady-state binding. When the cargo identification code and the task association code first meet the association conditions and become a matching result, a transient physical binding pair is generated and the moment of first satisfaction is recorded. , as a candidate binding state; Within the preset confirmation time window Continuous internal monitoring of two codes and If always satisfied and ( , If the transient physical binding pair is converted into a steady-state physical binding pair and assigned a unique binding identifier, then the physical binding pair will be converted into a steady-state physical binding pair. Otherwise, the transient physical binding pair is released, and the erroneous matching caused by transient noise is filtered out through the stability verification in the time dimension.
[0014] Furthermore, the confidence information of the three-dimensional anchor point sequence in step S4 Based on the number of observations Reprojection error Synthesized and generated to characterize the reliability of 3D position estimation; In steps S5 and S6, the confidence level is... The code corresponding to the 3D anchor point does not participate in spatiotemporal trajectory generation, 3D spatial proximity calculation, motion consistency measurement, or association condition determination. It is only considered after the number of observations in subsequent frames increases, the reprojection error decreases, and the confidence level improves. Then incorporate it into the calculation to ensure that the three-dimensional position involved in the association criterion has sufficient credibility.
[0015] Furthermore, the specific steps for the server to perform data comparison in step S7 include: The server receives the physical binding pair information and parses out the cargo identification code and task association code to obtain the binding relationship; The cargo file is obtained by querying the cargo database based on the cargo identification code. The cargo file contains at least the attributes of the goods receivable. The task file is obtained by querying the task database according to the task association code. The task file contains at least the attributes of the goods actually received. Compare the attributes of the goods to be received with the attributes of the goods actually received, generate a verification result, and complete the information verification of a single item of goods. When a task file contains multiple goods details, the server marks the current goods code as matched based on the established physical binding pairs, and continues to receive binding pairs of the same task code with other goods codes until all details are matched, thus realizing item-by-item verification and progress tracking in batch warehousing scenarios. The server will compare the verification result with the physical binding identifier. The system is associated with storage and pushed to mobile terminals in real time, allowing operators to receive the verification results on-site.
[0016] The technical effects and advantages of this invention are as follows: This invention acquires consecutive image frames containing cargo identification codes and task association codes via a mobile terminal, and simultaneously obtains camera pose information output by a visual inertial odometry system, providing a pose reference for subsequent 3D reconstruction. Based on a multi-view geometry method, the 2D observations of the same code in multiple image frames are triangulated to the world coordinate system, generating a 3D anchor point sequence with confidence. This process elevates the code from 2D pixel space to 3D physical space, eliminating the ambiguity of perspective projection and helping to improve the reliability of single-frame recognition under conditions such as occlusion and tilt. The confidence mechanism, through a comprehensive evaluation of the number of observations and reprojection errors, filters valid 3D positions, helping to reduce the risk of mismatches.
[0017] This invention generates the spatiotemporal trajectory of each code based on a three-dimensional anchor point sequence. By calculating the three-dimensional spatial proximity and motion consistency metric of any two codes within a preset time window, a dual-dimensional association criterion of space and motion is established. The three-dimensional spatial proximity characterizes the physical distance between codes, while the motion consistency metric quantifies whether two codes exhibit synchronous motion characteristics through the correlation coefficient of displacement vector sequences or dynamic time warping distance. This dual-dimensional criterion can effectively distinguish between accidental proximity and genuine consignment of goods, helping to improve the binding accuracy in scenarios such as dense stacking of multiple goods and cross-operations.
[0018] This invention constructs a bipartite graph of cargo identification codes and task association codes that meet the association conditions. Using a weighted sum of spatial proximity and motion consistency metrics as the matching cost, the Hungarian algorithm is employed to solve for the minimum weighted matching, generating physical binding pairs. This method can automatically optimize matching when multiple task codes and multiple cargo codes coexist, helping to correct errors from manual pasting. Furthermore, the physical binding pair construction process includes two stages: transient binding and steady-state binding. By continuously monitoring the stability of the association conditions within a time window, misbinding caused by transient noise is avoided, thus improving the reliability of the binding results.
[0019] This invention merges multiple codes with an Euclidean distance and a normal vector angle less than a preset threshold into a single cargo entity through three-dimensional spatial clustering and entity segmentation steps. This process aggregates multiple codes that physically belong to the same cargo into a single binding unit, avoiding duplicate binding or erroneous splitting of the same cargo due to multiple codes being affixed, thus improving the integrity of entity recognition. The entity segmentation results are updated in real time and synchronized to the subsequent association determination process, achieving unification of code-level observation and entity-level binding.
[0020] This invention employs a dynamic binding strategy with adaptive steps to analyze the mobile terminal's angular velocity, camera pose change rate, and focal length parameters in real time. It identifies operator operation modes such as hovering stability, scanning movement, and zooming out for panoramic views, and dynamically adjusts the spatial proximity threshold, time window width, and matching cost weighting coefficient accordingly. This adaptive mechanism helps adapt to different operating habits and work scenarios, improving operational smoothness while ensuring binding accuracy. The linear parameter transition strategy avoids binding state jitter during mode switching, contributing to a more continuous user experience. Attached Figure Description
[0021] Figure 1 This is an overall flowchart of the method of the present invention.
[0022] Figure 2 This is a flowchart of the three-dimensional reconstruction and trajectory calculation process of the present invention.
[0023] Figure 3 This is the adaptive branch diagram for the dynamic binding strategy of this invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] Example 1 As attached Figures 1 to 3 The following is a method for automatic verification of goods entering the warehouse based on mobile image recognition, and its specific implementation details are as follows: S1. In response to the warehouse entry verification command, the mobile terminal continuously collects consecutive image frames containing the surface code of the goods.
[0026] In one preferred implementation, the operator holds a smartphone, opens a dedicated inbound verification application, and clicks the "Start Inbound" button in the center of the screen. The application generates an inbound verification command. The smartphone uses its rear main camera, which, in another preferred implementation, is set to a resolution of 1920×1080 pixels, a frame rate of 30 frames per second, and autofocus mode enabled. The camera continuously captures image frames, each a color image in RGB format.
[0027] Cargo surface affix codes include two types of graphic codes: Goods identification code: As a preferred implementation, a Quick Response Code (QR code) is used. The code content is prefixed with "SKU:" followed by an 18-digit alphanumeric combination, containing unique identification information for the goods. This code is affixed by the supplier to a prominent position on the outer packaging of the goods during the production process.
[0028] Task Association Code: As a preferred implementation, a Portable Data File 417 (PDF417) or Quick Response Code is used. The code content is prefixed with "TASK:" followed by a 20-digit alphanumeric combination, containing the inbound task document identification information. This code is printed by the warehouse management system when the task is generated and affixed to the outer packaging of the goods by the operator upon receipt.
[0029] The application continuously captures image frames, and each frame is sent to the subsequent processing pipeline.
[0030] S2. In the continuous image frames, identify the cargo identification code and task association code in real time, and extract the two-dimensional image coordinates of each code in each frame image.
[0031] Each image frame is first converted to grayscale, and then detected using a multi-format barcode recognition engine. The engine takes a grayscale image frame as input and outputs the detected code content string, code type, and code location information within the image. As a preferred implementation, a Fast Response Code and Portable Data File 417 code decoder from an open-source computer vision library are used.
[0032] The location information is represented by the pixel coordinates of the four vertices of the smallest bounding rectangle of the code region. In this embodiment, the average value of the four vertex coordinates is taken as the two-dimensional image coordinates of the code, denoted as . ,in The horizontal pixel coordinates are... These are the vertical pixel coordinates, with the origin located at the top left corner of the image.
[0033] Code type determination rules: If the code content begins with "SKU:", it is determined to be a goods identification code; if the code content begins with "TASK:", it is determined to be a task association code; all other code content is ignored. After each frame recognition is completed, all recognized codes and their two-dimensional image coordinates, code content, code type, and frame timestamp are encapsulated as observation data and transmitted to S3 and S4.
[0034] S3. Synchronously acquire the camera pose information of the mobile terminal at the moment of each frame image acquisition.
[0035] Smartphones integrate a visual inertial odometry (VIO) module, which outputs camera pose in real time through the following methods: Feature point extraction and tracking: The input is the grayscale image of the current frame, and the output is the set of image feature points and their two-dimensional coordinates. As a preferred implementation, a directional fast feature point extraction and rotation brief descriptor (ORB) algorithm is used to extract 500 feature points per frame, and the motion of the feature points between consecutive frames is tracked using a sparse optical flow algorithm.
[0036] Inertial Measurement Unit (IMU) Data Acquisition: Input is raw sensor data from the six-axis inertial measurement unit built into the smartphone; output is three-axis angular velocity. and triaxial acceleration The output frequency is 200Hz. Angular velocity is measured in radians per second, and acceleration is measured in meters per second. .
[0037] Tightly coupled nonlinear optimization: The input consists of feature point tracking data and inertial measurement unit pre-integrated data, and the output is the six-DOF camera pose for each frame. As a preferred implementation, a sliding window optimization framework is used, with the window size set to 15 frames.
[0038] The optimization variables include the camera pose, 3D coordinates of feature points, and inertial measurement unit (IMU) bias for each frame within the window. The optimization objective is to minimize the weighted sum of visual reprojection error and IMU pre-integration error, and the optimization algorithm employs the Gauss-Newton method.
[0039] The camera pose is represented by a transformation matrix, and the rotation component is denoted as... The translation component is ,Right now ,in for Rotation matrix, for Translation vector, defined as the camera position in the world coordinate system. The world coordinate system origin is set to the camera position at the instant the visual inertial odometry module starts. The axis is perpendicular to the ground and points upwards.
[0040] The camera pose of each frame is bound to the timestamp of that frame and output synchronously to S4.
[0041] S4. For the same cargo identification code or the same task association code appearing in multiple consecutive frames of images, based on the two-dimensional image coordinates of the code in different frames and the camera pose of the corresponding frames, the three-dimensional spatial position of the code in the world coordinate system is calculated using a multi-view geometric method, and iteratively updated over time to generate a three-dimensional anchor point sequence with confidence information.
[0042] The purpose of this step is to elevate the two-dimensional code points in the image to three-dimensional physical space, so that subsequent spatial relationship calculations can be freed from the ambiguity of perspective projection.
[0043] The application maintains a global hash table of code objects, using the code content string as the key. Each code object contains the following fields: Code type: Goods identification code or task association code.
[0044] Observation record list: Stores the observation data for each frame in chronological order, including two-dimensional image coordinates. Camera pose Frame timestamp.
[0045] 3D position: The currently estimated world coordinates, denoted as .
[0046] Confidence level: denoted as The value range is [0,1].
[0047] Update the flags.
[0048] Failure timer: Records the duration for which the confidence level is below a threshold.
[0049] When S2 outputs new observation data, it first checks if a code object with the same code content already exists in the hash table. If it does not exist, a new code object is created and the observation is added; if it exists, the observation record is appended.
[0050] As a preferred implementation, 3D reconstruction is triggered when the number of observation records for a certain code object reaches 3 frames. This frame count balances reconstruction timeliness and accuracy. A multi-view geometric method is employed, with the following specific steps: (1) Establish the equation for the space ray.
[0051] For the Frame observation, camera position Camera rotation matrix Set the camera intrinsic parameter matrix: , in , Focal length , These are the coordinates of the optical center. As a preferred implementation, , , The unit is pixels.
[0052] coordinates of the two-dimensional image Convert to normalized image coordinates: .
[0053] The direction vector of the ray in the world coordinate system .
[0054] The three-dimensional spatial location of the code satisfy: ,in For depth factors.
[0055] (2) Solve for the initial value of the three-dimensional position.
[0056] For at least 3 frames of observation, the simultaneous equations eliminate... , get about A system of linear equations. The input is the ray equations for each frame of observations, and the output is... The least squares solution.
[0057] As a preferred implementation, the singular value decomposition method is used to solve the overdetermined system of equations, with the objective of minimizing the sum of squared reprojection errors for each frame.
[0058] (3) Sliding window bundle adjustment optimization.
[0059] Set the width of the sliding window Frame. Select all observation records and corresponding camera poses for this code within the 20 frames prior to the current time.
[0060] Construct an optimization problem: the optimization variable is and the camera pose of each frame within the window The attitude is represented by Euler angles.
[0061] The input is all observations within the window. and the corresponding initial camera pose values, the output is the optimized value. and the optimized camera pose. The optimization objective is to minimize the reprojection error of all observations: , in The projection function is expressed as follows: .
[0062] The Levenberg-Marquardt algorithm is used for iterative solution, with an upper limit of 30 iterations. The convergence condition is that the change in parameters is less than 1 / 30. (Unit: meters or radians), the optimized version is updated. .
[0063] (4) Confidence calculation.
[0064] Confidence The calculation formula is: , in This represents the total number of observations for the current code object. As a preferred implementation method, a preset baseline number of observations is used. Take 10; This represents the average reprojection error of all observations since the most recent bundle adjustment, expressed in pixels. If... Approaching 0, then The contribution increases; to avoid division by zero errors, when At that time, take .
[0065] After each optimization, update the code object and and at that moment and The anchor points are stored in chronological order to form a three-dimensional sequence, which is maintained using a linked list structure. Each anchor point contains a timestamp. ,Location Confidence level .
[0066] Confidence screening and failure handling: Setting the tenth preset threshold , used to determine the validity of the code.
[0067] Confidence Updated over time: with the number of observations Increase and reprojection error Decrease Gradually increase.
[0068] when In this case, the system automatically re-includes the code in the association determination process in the next frame. As a preferred implementation, if the confidence level of a certain code object is lower than [a certain value] for 30 consecutive seconds... The system automatically clears all observation records for the code object and marks it as pending deletion to avoid the accumulation of invalid data.
[0069] S5. Generate the spatiotemporal trajectory of each code based on the three-dimensional anchor point sequence, and calculate the three-dimensional spatial proximity and motion consistency metric of any two codes within a preset time window.
[0070] To determine whether two codes belong to the same physical transport unit, it is necessary to consider not only their instantaneous spatial distance but also whether they move synchronously in time. Motion consistency measures are used to quantify this synchronicity.
[0071] The spatiotemporal trajectory of each code is formed by connecting its three-dimensional anchor point sequence in chronological order. A preset time window is set. In a preferred implementation, the system uses a sliding window with a step size of 1.0 second.
[0072] For any two codes and During the time window Within, perform the following calculations: (1) Three-dimensional spatial proximity, denoted as .
[0073] If two codes share a common 3D anchor point within the window, then the average of the 3D Euclidean distances at all common moments is taken. Let the set of common moments be... ,for ,distance ,but .
[0074] If there is no common moment, then take the end moment of the window. The Euclidean distance between the three-dimensional positions of the two codes. The position of time is obtained through linear interpolation: if the code is in If there is no anchor point at any given time, take the two nearest anchor points. , Linear interpolation with time weighting, the interpolation formula is: , in , This represents the three-dimensional position vector at the corresponding moment. If the code has only one historical anchor point, then the position of that anchor point is directly taken as... Time and location.
[0075] (2) Motion consistency measure, denoted as , is a binary variable, where 1 indicates that it has motion consistency and 0 indicates that it does not.
[0076] Extract the 3D position sequence of the two codes at all common moments within the window, requiring perfect timestamp alignment. For each moment... (Except for the initial window time), calculate the displacement vector: , .
[0077] Two displacement vector sequences are obtained. and The lengths are all .
[0078] Calculate the Pearson correlation coefficient between the two sequences. Calculate the correlation coefficients for each of the three components of the displacement vector and then take the average: , in , for The mean of the sequence, for The mean of the sequence; , correspond , Quantity.
[0079] Set the first preset threshold As a preferred implementation method, if ,but ; If the sequence length If the variance of a certain component is zero, making it impossible to calculate the correlation coefficient, then the dynamic time warping (DTW) distance is used as an alternative.
[0080] The input consists of two displacement vector sequences, and the output is the cumulative distance. .
[0081] Set a second preset threshold Rice, as a preferred implementation method. If... ,but ;otherwise .when hour, It is equal to the Euclidean distance between the two displacement vectors.
[0082] In calculation and At that time, for any code, if the confidence level of the latest anchor point at the current moment... If the code is invalid, it is considered an invalid position at the current time and does not participate in any distance calculation or motion consistency measurement. Its corresponding anchor point sequence is ignored within the window.
[0083] 3D spatial proximity Measurement of Motion Consistency Output to S6.
[0084] S6. Based on the three-dimensional spatial proximity and motion consistency metric, construct a physical binding pair between the cargo identification code and the task association code that meet the association conditions.
[0085] Set the third preset threshold Rice, as a preferred implementation method; fourth preset threshold .
[0086] For any cargo identification code Associated code with any task If the following conditions are met simultaneously: and , Then determine and The association conditions are met.
[0087] When the same cargo identification code and multiple task association codes simultaneously meet the association conditions, a unique binding is performed.
[0088] Define the matching cost function: , in The first weighting coefficient, These are the second weighting coefficients, all of which are preset constants. As a preferred implementation method, in the conventional mode... , .
[0089] Construct a bipartite graph by associating all the cargo identification codes and task association codes that meet the association conditions in the current scenario, with the left node set... right-side node set The edge weight is The Hungarian algorithm is used to solve the minimum weight matching problem. The input is a cost matrix and the output is a set of matching results. Each cargo identifier code matches at most one task association code, and each task association code matches at most one cargo identifier code.
[0090] when and When the number of nodes is unequal, the algorithm adds virtual nodes and sets the weight of the virtual edges to a large constant, keeping the excess nodes in an unmatched state. This matching result is the physical binding pair.
[0091] When constructing the bipartite graph, all confidence levels The code is directly removed from the candidate set and does not participate in association determination and matching.
[0092] The construction of physical binding pairs is divided into two stages: transient binding and steady-state binding. (1) Transient binding phase.
[0093] when and When the association condition is met for the first time and becomes a Hungarian match result, the system generates a transient physical binding pair and records the moment of the first meeting. Set the seventh preset threshold. Meters, eighth preset threshold As a preferred implementation method, a preset confirmation time window is set. Seconds, as a preferred implementation method.
[0094] exist During this period, continuous monitoring and 3D spatial proximity and motion consistency measurement If at each moment within this window... All less than and All equal to Then, the transient physical binding pair is converted into a steady-state physical binding pair, and a unique binding identifier is assigned, denoted as . As a preferred implementation, the binding identifier uses a universally unique identifier (UUID).
[0095] If at any time within the window or Then the transient physical binding pair will be released. and Restore to the unbound state. If during monitoring... or The confidence level suddenly dropped to The transient physical binding pair will then be immediately released.
[0096] (2) Spatial conflict detection and error avoidance.
[0097] Spatial conflict detection is performed synchronously during the construction of physical binding pairs. A ninth preset threshold is set. Rice, as a preferred implementation method.
[0098] Detection condition: Two different task association codes exist. , Its three-dimensional spatial distance ,and With a certain cargo identification code The association conditions are met, and at the same time With a certain cargo identification code The association conditions are met, and and For different cargo codes.
[0099] If the above conditions are detected, a task conflict is determined. The system immediately suspends the generation of all physical binding pairs involved in the conflict and displays a red warning box in the center of the mobile terminal screen with the message "Task Conflict: Multiple overlapping documents, please separate the documents." After the operator separates the overlapping documents, the system automatically resolves the conflict and re-executes the association determination and matching in S6.
[0100] Multi-detail task association processing: When the task file corresponding to the task association code contains multiple goods details, the server needs to perform one-to-many association during the comparison stage.
[0101] The mobile terminal generates only a physical binding pair between a task code and a single item code. After receiving this binding pair, the server retrieves the complete list of details based on the task code and iterates through the list of unmatched item codes for the current pallet. It compares the receivable item attributes in the details with the item files of the unmatched item codes one by one. If a match is found, a one-to-many association is established and recorded. This process is described in detail in S7.
[0102] S7. The physical binding pair is sent to the server, which retrieves the cargo file and task file based on the binding relationship, compares the data, and receives the verification result returned by the server.
[0103] The mobile terminal sends steady-state physical binding pair information to the enterprise warehouse management server via a wireless network in JavaScript Object Notation (JSON) format, and uses Hypertext Transfer Security Protocol (HTTPS) for encrypted transmission. Each message contains the following fields (the following is an example of one implementation): { "bid":"550e8400-e29b-41d4-a716-446655440000", "goods_code": "SKU: 690123456789012345", "task_code": "TASK: R20241105001", "bind_time": 1699171200000 } Server-side data comparison steps: E1. Parse the physical binding pair information and extract the goods_code and task_code.
[0104] E2. Query the goods database based on the goods_code to obtain the goods file. The goods file must at least contain the attributes of the goods receivable. As a preferred implementation, the attributes of the goods receivable are SKU codes.
[0105] E3. Query the task database based on the task_code to obtain the task file. The task file must at least contain the actual goods received attributes. As a preferred implementation, the actual goods received attributes are the task's SKU code. If the task file contains multiple details, extract the detail list.
[0106] E4. Compare the attributes of the goods received with those of the goods actually received. Comparison rule: If the two strings are exactly the same, the verification result is "consistent"; otherwise, it is "inconsistent", and the difference information is added.
[0107] E5. If the task file contains multiple details and the current item code and task code have already established a physical binding pair, the server will further perform batch association: mark the current item code as matched and remove it from the list of unmatched item codes. Simultaneously, matching entries in the task details are marked as completed. The server continues to receive binding pairs of the same task code with other item codes until all details are matched or a timeout occurs.
[0108] E6. The server associates and stores the single verification result and batch association status with the physical binding pair identifier in the database, and pushes them to the mobile terminal in real time via the full-duplex communication protocol (WebSocket).
[0109] The mobile terminal receives the verification result returned by the server.
[0110] S8, 3D spatial annotation and augmented reality feedback steps.
[0111] The mobile terminal generates visual labels in a real-time preview based on the established steady-state physical binding pairs and the 3D anchor point positions of their corresponding cargo entities. This step is achieved using augmented reality (AR) technology.
[0112] The specific implementation is as follows: For each steady-state physical binding pair and its corresponding cargo entity, obtain the entity's current three-dimensional spatial position, denoted as . If an entity is composed of multiple codes, Take the arithmetic mean of the three-dimensional positions of each code.
[0113] Get the camera pose of the current frame and camera intrinsic parameter matrix .
[0114] Will Projecting onto the current frame image plane: First, Transform to camera coordinate system: Then project it onto the pixel coordinate system: , , in , , They are respectively of , , Quantity.
[0115] exist A visual label is rendered at the designated location. As a preferred implementation, the label uses a semi-transparent black rectangular background, measures 160×80 pixels, and has a corner radius of 8 pixels. Label content: Top left corner: Circular icon, 16 pixels in diameter, green indicates "consistent", red indicates "inconsistent".
[0116] Middle: Goods name (obtained from the goods file, the first 8 Chinese characters are extracted), font size 14 pixels.
[0117] Bottom right corner: Last 4 digits of the SKU code, font size 12 pixels.
[0118] Depth testing is enabled during rendering to ensure that the label is not obscured by the goods themselves; at the same time, a low-pass filter is used to smooth the label position and reduce jitter.
[0119] The visual label updates its rendering position in real time as the mobile terminal's viewing angle changes, always maintaining its position attached to the image of the corresponding cargo entity.
[0120] S9. Steps for 3D spatial clustering and entity segmentation.
[0121] The purpose of this step is to merge multiple codes that physically belong to the same cargo entity into a single unit, preventing multiple codes affixed to the same cargo from being bound as independent entities, thereby improving matching accuracy.
[0122] This step is executed in parallel with S5 and S6, and as a preferred implementation, it is triggered once every 1 second.
[0123] (1) Three-dimensional Euclidean distance clustering.
[0124] The input is the 3D spatial location of all code objects at the current moment (using the latest anchor point), and the output is the cluster label for each code. A density-based spatial clustering algorithm (DBSCAN) is used, with a fifth preset threshold set. Using meters as the neighborhood radius is a preferred implementation method; the minimum number of neighborhood points is set to 1. Points with Euclidean distances less than... The codes belong to the same cluster.
[0125] (2) Calculation of code plane normal vector.
[0126] For each code, retrieve the two-dimensional image coordinates in its latest frame and the corresponding three-dimensional anchor point position. Using the code as the center, select the four corner points of its circumscribed rectangle in the image, and use the corresponding frame camera pose to backproject the four corner points into three-dimensional space to obtain four three-dimensional points.
[0127] The input consists of the coordinates of four 3D points, and the output is the normal vector of the fitted plane. The least squares method is used to fit the plane, yielding its equation. normal vector Normalized to a unit vector.
[0128] (3) Entity merger.
[0129] Within the same cluster, calculate the angle between the normal vectors of any two codes. : Take the absolute value and output the degree.
[0130] Set the sixth preset threshold This is a preferred implementation method. If... If the two codes belong to the same physical cargo surface, they are considered to be merged into the same cargo entity.
[0131] After the merger, the three-dimensional spatial position of the cargo entity Take the arithmetic mean of the three-dimensional positions of all contained codes, and the entity's normal vector. Take the average (normalized) of the normal vectors of each code. This entity participates as an independent unit in the association determination and binding of S5 and S6.
[0132] The entity identifier is generated by concatenating the strings containing each code and taking the MD5 hash value. After the segmentation is completed, the system updates the "Entity ID" field of each code object. Subsequent steps are calculated on an entity-by-entity basis.
[0133] S10, Adaptive steps for dynamic binding strategy.
[0134] This step automatically adjusts the binding parameters by analyzing the operator's movement behavior in real time, so that the system can maintain a smooth verification experience under different operating habits.
[0135] This step runs continuously, executing once per frame.
[0136] Real-time data collection: angular velocity magnitude The input is data from the inertial measurement unit gyroscope.
[0137] Camera pose change rate ,in The frame interval time. , is the camera position vector for consecutive frames.
[0138] Camera focal length : Obtain the current zoom level through the camera application programming interface and convert it to an equivalent 35mm focal length.
[0139] Recognition mode and parameter adjustment: (1) Hovering Stability Mode.
[0140] Set the first rate threshold rad / s, first duration threshold s, as a preferred implementation method.
[0141] when and m / s, and the duration of this state When the time comes, it is identified as hovering stable mode.
[0142] Adjust parameters: Third preset threshold The height was reduced from 0.3 meters to 0.15 meters.
[0143] Preset time window The time limit has been increased from 3.0 seconds to 5.0 seconds.
[0144] First weighting coefficient The second weighting coefficient was adjusted from 0.6 to 0.7. Adjusted from 0.4 to 0.3.
[0145] If the above conditions are no longer met and persist for 0.5 seconds, the system will linearly transition the parameters to normal values within 1.0 second to avoid binding state jitter caused by sudden parameter changes. The linear interpolation formula is: , in The parameter value at the end of the mode. This is a normal value. Second.
[0146] (2) Scanning movement mode.
[0147] Set a second rate threshold rad / s, as a preferred implementation method.
[0148] when and When the speed is m / s, it is identified as a scanning movement mode.
[0149] Adjust parameters: For all codes appearing within a single frame, instead of waiting for accumulation across multiple frames, steps S4 to S6 are executed directly based on the existing 3D anchor points in the current and historical frames. That is, for newly entered codes, if a historical 3D anchor point already exists, it immediately participates in the association determination; otherwise, a code object is created first, and reconstruction is performed in subsequent frames.
[0150] First weighting coefficient Adjusted to 0.4 Adjusted to 0.6.
[0151] When the above conditions are no longer met and this continues for 0.5 seconds, the system will linearly transition the parameters to normal values within 1.0 second.
[0152] (3) Zoom out panoramic mode.
[0153] Set the first focal length threshold mm (equivalent to 35mm focal length), as a preferred embodiment, is the dividing line between typical wide-angle and telephoto.
[0154] When the camera focal length parameter At that time, it was identified as a zoomed-out panoramic mode.
[0155] Adjust strategy: Suspend the execution of S6, meaning no new physical binding pairs will be generated.
[0156] Maintain updates to existing 3D anchor point sequences and render visualization tags for existing physical binding pairs.
[0157] when At that time, the system immediately resumes execution of S6.
[0158] All preset thresholds and weighting coefficients given in this embodiment are preferred implementations and are not the only limitations. Those skilled in the art can calibrate them according to the actual deployment environment.
[0159] The method described in this embodiment enables real-time spatial binding and warehousing verification of goods and tasks by simply having an operator move a handheld mobile terminal around the goods, without relying on fixed identification equipment or preset identification stations. The specific parameters, algorithms, and thresholds described above are preferred embodiments and not the only limitations of this invention. Those skilled in the art can adjust them according to actual scenario requirements, and any modifications, equivalent substitutions, or improvements made should be included within the scope of protection of this invention.
Claims
1. A goods warehousing automatic checking method based on mobile terminal image recognition, characterized in that, Includes the following steps: S1. The mobile terminal responds to the warehouse entry verification command and continuously collects continuous image frames containing the surface code of the goods. S2. In the continuous image frames, identify the cargo identification code and task association code in real time, and extract the two-dimensional image coordinates of each code in each frame image. S3. Synchronously acquire the camera pose information of the mobile terminal at the moment of each frame image acquisition; S4. For the same cargo identification code or the same task association code appearing in multiple consecutive frames of images, based on the two-dimensional image coordinates of the code in different frames and the camera pose of the corresponding frames, the three-dimensional spatial position of the code in the world coordinate system is calculated using a multi-view geometric method, and iteratively updated over time to generate a three-dimensional anchor point sequence with confidence information. S5. Generate the spatiotemporal trajectory of each code based on the three-dimensional anchor point sequence, and calculate the three-dimensional spatial proximity and motion consistency metric of any two codes within a preset time window. S6. Based on the three-dimensional spatial proximity and motion consistency measurement, construct the cargo identification code and task association code that meet the association conditions into a physical binding pair; S7. The physical binding pair is sent to the server, which retrieves the cargo file and task file based on the binding relationship, compares the data, and receives the verification result returned by the server.
2. The automatic goods entry verification method based on mobile terminal image recognition according to claim 1, characterized in that, In step S3, the camera pose information is output in real time as a six-degree-of-freedom camera pose by fusing image feature point tracking data and inertial measurement unit data through a mobile terminal. The multi-view geometry method in step S4 includes: establishing a spatial ray equation based on the pinhole camera model, solving for the initial values of three-dimensional coordinates for the observation coordinates and corresponding camera poses of the same code in at least two frames of images, and jointly optimizing the three-dimensional coordinates and camera poses within a continuous time window using bundle adjustment to obtain the three-dimensional anchor point sequence.
3. The automatic goods entry verification method based on mobile terminal image recognition according to claim 1, characterized in that, In step S5, the motion consistency metric is calculated in the following manner: Extract the three-dimensional position sequence of the two codes at the common moment within the time window, generate the displacement vector sequence, and calculate the Pearson correlation coefficient or dynamic time warping distance between the two sequences. If the Pearson correlation coefficient is greater than the first preset threshold, or the dynamic time warping distance is less than the second preset threshold, then the two codes are determined to have motion consistency.
4. The automatic goods entry verification method based on mobile terminal image recognition according to claim 1, characterized in that, In step S5, the three-dimensional spatial proximity is the three-dimensional spatial Euclidean distance between the two codes at the same time or the average three-dimensional spatial Euclidean distance within the time window. The association conditions in step S6 are: the three-dimensional spatial proximity of the two codes is less than the third preset threshold, and the motion consistency metric is equal to the fourth preset threshold. When the same cargo identification code and multiple task association codes simultaneously meet the association conditions, the weighted sum of three-dimensional spatial proximity and motion consistency metric is used as the matching cost. The weighted sum is composed of the product of the first weighting coefficient and the three-dimensional spatial proximity plus the product of the second weighting coefficient and the motion consistency metric. The bipartite graph optimal matching algorithm is used to determine the unique physical binding pair.
5. The automatic goods entry verification method based on mobile terminal image recognition according to claim 4, characterized in that, Step S4 is followed by a three-dimensional spatial clustering and entity segmentation step: Based on the three-dimensional spatial position of all codes at the current time, three-dimensional Euclidean distance clustering is performed, and codes with Euclidean distance less than the fifth preset threshold are grouped into the same cluster; Calculate the three-dimensional normal vector of the plane containing each code; Within the same cluster, multiple codes with an angle between their three-dimensional normal vectors less than the sixth preset threshold are merged into the same cargo entity, and this cargo entity is used as a whole to participate in the construction of subsequent physical binding pairs.
6. The automatic goods entry verification method based on mobile terminal image recognition according to claim 5, characterized in that, The step S6 is preceded by a dynamic binding strategy adaptation step: Real-time analysis of mobile terminal angular velocity data and camera pose change rate to identify the operator's current operating mode; When the hovering stabilization mode is identified, the third preset threshold is reduced, the preset time window is extended, the first weighting coefficient is increased, and the second weighting coefficient is decreased. When the scanning movement mode is identified, three-dimensional anchor point generation and association determination are performed simultaneously on all codes appearing in a single frame image, and the first weighting coefficient is reduced and the second weighting coefficient is increased. When the view is identified as zoomed-out panoramic mode, the generation of physical binding pairs is paused, while the display of the existing 3D anchor point sequence and physical binding pairs is maintained.
7. The automatic goods entry verification method based on mobile terminal image recognition according to claim 6, characterized in that, Step S6 also includes a spatial conflict detection and error avoidance step: Detect whether there are two different task association codes whose three-dimensional spatial distance is less than the ninth preset threshold, and which respectively satisfy the association conditions with different cargo identification codes or cargo entities; If a conflict exists, it is determined to be a task conflict. The generation of the physical binding pairs involved in the conflict is suspended and a conflict prompt message is output. The association determination is re-executed after the conflict is resolved.
8. The automatic goods entry verification method based on mobile terminal image recognition according to claim 7, characterized in that, The construction of physical binding pairs in step S6 includes two stages: transient binding and steady-state binding. When the cargo identification code and the task association code meet the association conditions for the first time and become a matching result, a transient physical binding pair is generated and the moment of first satisfaction is recorded. Within a preset confirmation time window, continuously monitor the three-dimensional spatial proximity and motion consistency of the two codes. If the association condition is always met, convert the transient physical binding pair into a steady-state physical binding pair and assign a unique binding identifier; otherwise, release the transient physical binding pair.
9. The automatic verification method for goods entering the warehouse based on mobile terminal image recognition according to claim 8, characterized in that, In step S4, the confidence information of the three-dimensional anchor point sequence is generated based on the number of observations and reprojection error. In steps S5 and S6, the codes corresponding to three-dimensional anchor points with confidence levels below the tenth preset threshold are not included in the spatiotemporal trajectory generation, three-dimensional spatial proximity calculation, motion consistency measurement, and association condition determination. They will be included in the calculation when the confidence level in subsequent frames increases to above the tenth preset threshold.
10. The automatic verification method for goods entering the warehouse based on mobile terminal image recognition according to any one of claims 1 to 9, characterized in that, The specific steps for data comparison performed by the server in step S7 include: The server receives the physical binding pair information and parses out the cargo identification code and task association code; it then queries the cargo database based on the cargo identification code to obtain the cargo file, which at least contains the attributes of the goods receivable. The task file is obtained by querying the task database according to the task association code. The task file contains at least the attributes of the goods actually received. Compare the attributes of the goods to be received with those of the goods actually received, and generate a verification result. When a task file contains multiple goods details, the server marks the current goods code as matched based on the established physical binding pairs, and continues to receive binding pairs of the same task code with other goods codes until all details are matched. The server associates the verification result with the physical binding identifier, stores it, and pushes it to the mobile terminal in real time.