Intelligent door-type channel machine and commodity identification method

Abstract

The present invention relates to the technical field of commodity identification, and in particular to an intelligent gate-type channel machine and a commodity identification method. The present invention proposes the following scheme: a commodity list is generated by reading commodity radio frequency tags through a radio frequency scanner, a commodity channel image is captured by multiple cameras, the contour, depth map and trajectory information of the commodity are extracted, three-dimensional reconstruction is performed, and commodity splitting and matching are performed; a time sequence sorting model is constructed based on channel diameter reduction information and trajectory prediction, and the time sequence of commodity arrival at a sorting port is accurately predicted; finally, the commodity list is optimized according to the arrival sequence of the commodities, and the sorting equipment is controlled to perform corresponding sorting operations; the present invention improves the efficiency and accuracy of commodity identification and sorting by accurately capturing the movement trajectory and time sequence characteristics of the commodities.

Description

Technical Field

[0001] The present invention relates to the technical field of commodity identification, and in particular to an intelligent door-type channel machine and a commodity identification method. Background Art

[0002] With the rapid development of e-commerce and automated logistics industries, the efficiency and accuracy of commodity sorting operations have become key issues in logistics and warehousing systems. In the traditional commodity identification and sorting process, although radio frequency identification (RFID) technology and computer vision technology are used to identify and track commodities, due to the overlap, occlusion, and channel deformation of commodities during transportation and sorting, accurate tracking and real-time identification are faced with challenges. Especially in complex environments such as intelligent portal channel machines, how to effectively distinguish commodities passing through the same channel and predict the order in which they arrive at the sorting port is still a technical problem to be solved.

[0003] For example, the Chinese patent application with the authorization announcement number CN109145816B provides a commodity identification method and system, which includes: a neural network module obtains commodity features and transmits the commodity features to a channel domain attention module, wherein the commodity features include commodity-related features and commodity-irrelevant features; and the channel domain attention module distinguishes the commodity-related features from the commodity-irrelevant features, and transmits at least the commodity-related features to the next neural network module. This technical solution can improve the accuracy of commodity identification.

[0004] The above technical solution has the problem raised by this background technology: it cannot effectively handle the position changes, occlusion and channel deformation of goods in dynamic channels, resulting in insufficient accuracy and real-time performance in complex environments. To solve the above problems, this application designs an intelligent gate-type channel machine and a product recognition method. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and provide an intelligent gate channel machine and a commodity identification method. A commodity list is generated by reading the commodity radio frequency tags through a radio frequency scanner, and the commodity channel image is captured by multiple cameras to extract the contour, depth map and trajectory information of the commodity, perform three-dimensional reconstruction and perform commodity splitting and matching. Based on the channel diameter reduction information and trajectory prediction, a time sequence sorting model is constructed to accurately predict the time sequence of the arrival of the commodities at the sorting port. Finally, the commodity list is optimized according to the arrival order of the commodities, and the sorting equipment is controlled to perform the corresponding sorting operations. By accurately capturing the movement trajectory and time sequence characteristics of the commodities, the efficiency and accuracy of commodity identification and sorting are improved.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A commodity identification method is applied to an intelligent gate-type channel machine, wherein the intelligent gate-type channel machine comprises a commodity channel with a continuously decreasing diameter and gates spanning both sides of the commodity channel, wherein the gates are provided with a radio frequency scanner and a plurality of cameras arranged along the extension direction of the commodity channel, and two adjacent cameras are separated by a set distance, wherein the method comprises:

[0008] Read the radio frequency tags of commodities passing through the gate by the radio frequency scanner to generate a commodity list;

[0009] The camera is used to capture images of a batch of commodities passing through the commodity channel to obtain multiple commodity placement images;

[0010] Determine the order information of the arrival of the commodities at the sorting port according to the commodity list and each commodity placement image; wherein determining the order information of the arrival of the commodities at the sorting port comprises: processing the commodity placement image, identifying the commodities in combination with the commodity list, predicting the commodity trajectory according to the channel diameter reduction information and the trajectory information, outputting the time sequence of the arrival of the commodities at the sorting port through a time sequence sorting model, and adjusting the commodity list according to the time sequence;

[0011] The batch of commodities is sorted according to the sequence information.

[0012] The information of determining the order in which the commodities arrive at the sorting port includes:

[0013] Processing the product placement image to extract contour information, depth map and trajectory information;

[0014] Reconstructing the product placement image in three dimensions according to the contour information and the depth map, splitting the batch of products into individual products and marking serial numbers, matching the contour information of the individual products with the product list one by one, and marking the serial numbers corresponding to the products in the product list;

[0015] Calculating channel diameter reduction information according to the camera position, predicting the trajectory of the commodity according to the channel diameter reduction information and the trajectory information, and calculating trajectory prediction information for each commodity;

[0016] Constructing a time sequence sorting model, taking the trajectory prediction information as an input parameter of the time sequence sorting model, modeling the trajectory prediction information at a time point through the time sequence sorting model, and outputting the time sequence of the commodities arriving at the sorting port;

[0017] The commodity list is adjusted according to the time sequence to obtain the order information of the commodities arriving at the sorting port.

[0018] The processing of the commodity placement image includes:

[0019] Binarize the product placement image, crop the product area of ​​the binary image by Hough transform, perform background subtraction on the product area, and calculate the contour information of the product by morphological opening operation;

[0020] Calculate the statistical features of the product placement image, calculate the depth estimation value, air scattering depth and spatial geometric depth of the product placement image according to the statistical features, build a deep fusion network, and calculate the depth map through the deep fusion network;

[0021] The corner features and grayscale centroid of each product placement image are extracted, and the product feature points of the image are calculated according to the angle between the corner features and the grayscale centroid. The image frames at different time points are matched with the feature points according to the product feature points to calculate the trajectory information.

[0022] 3D reconstruction of product placement images, including:

[0023] According to the contour information, the object boundary points are extracted in each image frame, the object boundary points are associated with the depth value information to generate point cloud coordinates, the point cloud coordinates corresponding to the multiple depth maps are registered, and the three-dimensional point cloud coordinates of the product are calculated;

[0024] Set a loss threshold, estimate the resolution based on the depth value information of multiple frames of product placement images, and calculate the depth loss value. If the depth loss value is greater than or equal to the loss threshold, it means that there is a product occlusion at the depth value position. Perform continuity constraint projection on the current position, iteratively correct the 3D point cloud coordinates of the current position according to the geometric constraints corresponding to the current position, and calculate the 3D point cloud coordinates of the occluded product based on the correction result;

[0025] The commodity display image is 3D reconstructed according to the 3D point cloud coordinates, and the 3D model of the commodity is texture mapped according to the contour information.

[0026] The calculating of the channel diameter reduction information according to the camera position includes:

[0027] According to multiple commodity channel images, the edge points of the channel are extracted, and the edge points are projected into a unified world coordinate system according to the position of the camera. The edge points are fitted by a high-order curve fitting algorithm to generate a geometric model of the channel.

[0028] Performing synchronous sampling along the channel direction of the geometric model, calculating the cross-sectional area, edge curvature and shrinkage rate of the channel section according to the sampling results, and calculating the channel geometry change parameters according to the cross-sectional area, edge curvature and shrinkage rate;

[0029] According to the external parameter matrix and internal parameter matrix of the camera, the three-dimensional model of the product at each position is projected onto the geometric model, and the geometric model is segmented along the channel direction according to the number of cameras, and the three-dimensional distribution law of the products in the segmented interval is analyzed, and the distribution change characteristics of the products are calculated according to the three-dimensional distribution law;

[0030] The channel diameter shrinkage information is calculated according to the channel geometry change parameter and the commodity distribution change characteristic. The calculation formula of the channel diameter shrinkage information is:

[0031] ;

[0032] in, Indicates channel reduction information. Indicates the spatial position of the first camera, represents the spatial position of the last camera, z represents the spatial position of the channel in the direction of camera arrangement, The weight function representing the influence of channel geometry change on the shrinkage information, represents the geometric characteristics of the channel at position z, Represents the weight function of the impact of commodity distribution on shrinkage information, represents the distribution law of goods at position z, represents the rate of change of the camera's viewing angle at position z, Indicates the spatial resolution in the channel direction, which is used to calculate the step size of discrete changes.

[0033] The calculating of the trajectory prediction information of each commodity includes:

[0034] Using the center line of the geometric model as a reference line and the center of mass of the product in each product placement image as a reference point, the offset distance is calculated and a deviation curve is drawn according to the channel direction;

[0035] According to the channel diameter reduction information, the speed and direction in the trajectory information are corrected;

[0036] Starting from the last trajectory point, the deviation curve is continuously drawn according to the correction results of speed and direction to generate the trajectory point at the next moment until the trajectory point reaches the channel focus and generates trajectory prediction information.

[0037] The constructing of the time series sorting model comprises:

[0038] The input layer is used to organize the trajectory prediction information into time series data and normalize it;

[0039] The temporal encoding layer is used to extract the temporal features of the trajectory and construct the dependency relationship between time steps by processing the time series data;

[0040] The time modeling layer is used to integrate the dependencies of commodities in time steps and predict the time when commodities arrive at the sorting port.

[0041] The obtaining of the order information of the commodities arriving at the sorting port includes:

[0042] According to the time sequence of the goods arriving at the sorting port output by the time sequence sorting model, the serial numbers of the goods are matched with the time sequence;

[0043] Rearrange the commodities in the commodity list in chronological order to generate a commodity list arranged in the order of arrival at the sorting port;

[0044] Based on the rearranged product list, output optimized product sequence information.

[0045] The step of sorting the batch of commodities comprises:

[0046] Match the serial numbers of the products with the order information of the products arriving at the sorting port one by one;

[0047] According to the time sequence information of the goods arriving at the sorting port, control the execution actions of the sorting equipment, including the opening, closing and physical displacement operations of the sorter;

[0048] According to the arrival time of the goods and the designated sorting destination, the corresponding goods are sorted to their corresponding sorting exits;

[0049] An intelligent door-type channel machine, comprising:

[0050] A commodity channel, which is a channel structure with a continuously shrinking diameter and is used to guide batches of commodities to pass along a predetermined path;

[0051] The gate spans across both sides of the commodity channel and includes:

[0052] RFID scanner, used to read the RFID tag information of commodities and generate a commodity list;

[0053] The drive unit is used to control the opening and closing of the gate to ensure that the goods enter the commodity channel at the set time interval;

[0054] The multi-camera system is arranged along the extension direction of the commodity channel, with a set distance between adjacent cameras, including:

[0055] The camera is used to capture the image of the product placement and extract the product's contour information, center of mass position and trajectory information;

[0056] Depth sensors, integrated into some cameras, are used to obtain depth map data of product placement;

[0057] The processing module is integrated in the intelligent gate channel machine and communicates with the external computing system to perform a commodity identification method, including:

[0058] Preprocess and 3D reconstruct product images;

[0059] Predict product trajectories and sort arrival times;

[0060] Control the sorting equipment to sort the goods based on the order in which the goods arrive at the sorting port;

[0061] The sorting device is arranged at the end of the commodity channel and is used to sort the commodities to the corresponding exit according to the time sequence of the commodities arriving at the sorting exit and the designated sorting destination. The sorting device includes:

[0062] Sorting slide rails are used to guide the goods to slide in a specified direction;

[0063] The sorting execution device completes the sorting of goods through a robotic arm or a flip device.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] 1. The present invention combines radio frequency identification technology and image processing technology to accurately obtain the real-time location information of goods in the dynamic channel, and optimizes the arrival order of goods through trajectory prediction and time sequence sorting model, thereby improving the accuracy of commodity identification and sorting efficiency;

[0066] 2. In a complex channel environment, the present invention can effectively deal with problems such as commodity obstruction and channel deformation, and realize efficient and intelligent automatic sorting of commodities. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0068] Figure 1 This is a flow chart of a commodity identification method according to Embodiment 1 of the present invention;

[0069] Figure 2 This is a schematic diagram of the structure of the commodity channel of Example 1 of the present invention;

[0070] Figure 3 This is a schematic diagram of eliminating reflection points in Embodiment 1 of the present invention;

[0071] Figure 4 This is a deep fusion network structure diagram of Example 1 of the present invention;

[0072] Figure 5 This is a structural diagram of the timing sorting model of Example 1 of the present invention.

[0073] Reference numerals: 101, commodity channel; 102, gate; 103, radio frequency scanner; 104, first camera; 105, second camera; 106, third camera. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0075] Example 1

[0076] See also Figure 1 and Figure 2 The present invention provides an embodiment: a commodity identification method, applied to an intelligent gate-type channel machine, the intelligent gate-type channel machine comprises a commodity channel with a continuously shrinking diameter and gates across both sides of the commodity channel, the gates are equipped with a radio frequency scanner and a plurality of cameras arranged along the extension direction of the commodity channel, and two adjacent cameras are separated by a set distance, and the specific steps of the method are as follows:

[0077] S1: Use an RFID scanner to read the RFID tags of goods passing through the gate and generate a list of goods;

[0078] In this step, the RFID scanner uses radio frequency to identify the radio frequency tag (RFID tag) of the product, thereby obtaining the unique identifier (ID) of each product. Before the products passing through the gate enter the product channel, they will trigger the reading process of the RFID scanner and generate a product list. Each item in the list includes the product ID, tag information and other additional information (including product category, size, weight, etc.).

[0079] S2: Use a camera to capture images of batches of goods passing through the product channel to obtain a sequence of product placement images;

[0080] In this step, multiple cameras are used to capture a sequence of images of goods flowing in the product channel. The cameras take full-scale images of the goods at a set distance and angle. Each image provides two-dimensional image data of the goods, including information such as the shape, position, size, and relative position of the goods. In order to ensure efficient tracking, the camera configuration should ensure that every key moment of the goods in the product channel can be captured, especially when multiple goods pass in parallel, to ensure that the image data of any product is not missed.

[0081] S3: Process the product placement image sequence, split the batch of products into individual products and mark the serial numbers;

[0082] In this step, each product is separated from the image through image segmentation and recognition algorithms. The processing includes removing the background, eliminating ghost images, and segmenting overlapping products. Each separated product will be marked with a unique product serial number and matched with the ID information in the product list. This process is a key step to ensure that the products can be accurately tracked during trajectory prediction and sorting operations.

[0083] S4: Calculate the channel diameter reduction information according to the camera position, predict the trajectory of the goods, and calculate the trajectory prediction information of each product;

[0084] In this step, the channel shrinkage information is calculated using the relative positions of multiple cameras and the geometric model of the channel. The channel shrinkage information reflects the impact of spatial constraints on goods in the channel, which affects the speed and trajectory of the goods. By combining with the initial trajectory information of each product, the future position of the product is predicted. The prediction process takes into account the channel shrinkage effect, the dynamic characteristics of the goods, and the flow density in the channel.

[0085] S5: constructing a time sequence sorting model, taking the trajectory prediction information as an input parameter of the time sequence sorting model, modeling the trajectory prediction information at a time point through the time sequence sorting model, and outputting the time sequence of the commodities arriving at the sorting port;

[0086] S6: adjusting the commodity list according to the time sequence, obtaining the sequence information of the commodities arriving at the sorting port, and sorting the batch commodities according to the sequence information;

[0087] In this step, the arrival time sequence of the goods output by the time-series sorting model is used to sort the original goods list. The goods in the goods list are rearranged according to the predicted arrival time to ensure that the goods that arrive first are sorted first. After sorting, the sorting order information of the goods will be passed to the sorting equipment for it to perform the corresponding sorting operations, ensuring that the goods arrive at the sorting port in the planned order and are accurately delivered to the designated destination through appropriate sorting actions.

[0088] like Figure 2 As shown, in a specific implementation, the intelligent portal channel machine includes a commodity channel 101 with a gradually decreasing diameter, in which a plurality of groups of first cameras 104, second cameras 105 and third cameras 106 arranged along the extension direction are arranged, and a radio frequency scanner 103 is configured at the gate 102. Figure 2 The example device shown in the figure is equipped with three cameras to realize dynamic identification and trajectory prediction of goods, but in actual use, the number of cameras can be flexibly adjusted according to the channel length and identification requirements. Figure 2 The multiple oval patterns represent different products. The quantity of products is Figure 2The figure is only for illustration. In actual application scenarios, more products can be simultaneously and accurately identified.

[0089] The design of commodity channel 101 adopts a gradually decreasing diameter. Figure 2 The direction of the middle arrow is the direction of the reduction, which is intended to optimize the flow path of the goods, so that the goods are arranged in an orderly manner along a fixed direction, and reduce the interference of random placement of goods on subsequent identification and sorting. The radio frequency scanner 103 is located at the gate 102. By scanning the radio frequency tag (RFID) of the goods, it quickly generates a list of goods and identifies the initial information of the goods, including key data such as the name of the goods and the batch number. This information is then combined with the image and depth information collected by the camera to provide multi-modal input.

[0090] The camera array arranged along the commodity channel 101 is used to capture the dynamic images of the commodities in the commodity channel 101 in real time, and obtain the images and position data of the commodities from multiple perspectives. These data are processed by the deep learning model to extract the contour information, depth map and motion trajectory characteristics of the commodities. Combined with the dynamic and static algorithms, the motion trajectory of the commodities can be accurately predicted, and the time sequence of the commodities arriving at the sorting port can be calculated accordingly.

[0091] In this embodiment, the dynamic and static algorithm is mainly reflected in combining the data of the RFID tag with the image data, obtaining static information through the RFID tag, and accurately determining the dynamic position and state of the product through the product image captured by the camera.

[0092] That is, RFID provides static information (such as product ID) for each product, while image data provides dynamic information (such as the movement trajectory of the product). The dynamic and static algorithms can use the combination of the two to perform 3D reconstruction and trajectory prediction.

[0093] Furthermore, although radio frequency identification technology can accurately read the RFID tag information of each product, thereby identifying the basic information of the product, such as product ID, type, specifications, etc., in actual applications, RFID technology cannot accurately identify the specific location of the product in the smart portal channel, especially when the products are stacked, partially blocked, or multiple products are staggered. It is impossible to directly use RFID information to match the products with their actual locations in the product channel, thereby affecting the accuracy of subsequent tasks such as sorting and scheduling. Due to the diversity of products, complex stacking methods and occlusion problems, the recognition accuracy is often low, and the accuracy of product tracking and sorting is poor.

[0094] In order to overcome this problem, image processing and deep learning technologies are combined to further optimize the RFID technology. Specifically, although RFID can provide the identity information of the product, the image data can provide the actual spatial position of the product in the channel. First, the product area is extracted through binary image and Hough transform, and the spatial position of the product is accurately located with the help of depth map and 3D reconstruction technology. The specific steps of S3 are as follows:

[0095] S3.1: Binarize the product placement image and display it, crop the product area of ​​the binary image by Hough transform, perform background subtraction on the product area, and calculate the contour information of the product by morphological opening operation;

[0096] The binarization image processing uses the threshold method to set a threshold, mark the area with pixel values ​​greater than the threshold as the foreground (product area), and the area less than the threshold as the background, and perform binarization on the product placement image to distinguish the products from the background. The binarized image has only two colors (black and white), making the product area more prominent and suppressing the cluttered and irrelevant parts of the background.

[0097] Hough transform can effectively identify areas in an image that conform to specific geometric shapes (such as straight lines, circles, etc.). In this step, it is mainly used to cut out the contour area of ​​the product, and through shape detection of the product area, it can accurately extract products with obvious geometric features (such as rectangles, circles, etc.). For products with complex shapes, the extraction effect can be optimized through multiple iterations and parameter adjustments.

[0098] In an image of a product display taken by a camera, the products may partially overlap or have a cluttered background. Through binarization, Hough transform, and background subtraction, the outline of each product can be effectively distinguished and irrelevant background information can be removed to ensure that the outline of each product is clearer.

[0099] S3.2: Calculate the statistical features of the product placement image, calculate the depth estimation value, air scattering depth and spatial geometric depth of the product placement image according to the statistical features, construct a deep fusion network, and calculate the depth map through the deep fusion network;

[0100] By analyzing the statistical characteristics of product placement images, including the morphological characteristics of the products (such as aspect ratio, angle, etc.) and the brightness distribution of the images, we can further infer the spatial distribution information of the products, especially in scenarios where products are stacked or blocked. We can infer the depth position of each product in space (i.e., the positioning in the Z-axis direction).

[0101] The deep fusion network extracts deep features through a multi-layer convolutional neural network (CNN) and uses temporal and spatial information to synthesize and optimize depth information, thereby improving the accuracy of the depth map, especially in complex scenes (such as partial occlusion of goods, lighting changes, etc.), which can reduce errors.

[0102] S3.3: Reconstruct the product placement image in three dimensions according to the contour information and the depth map, split the batch of products into individual products and mark the serial numbers, match the contour information of the individual products with the product list one by one, and mark the serial numbers corresponding to the products in the product list;

[0103] By calculating the coordinates of the goods in three-dimensional space, we can overcome the shortcomings of relying solely on two-dimensional images, obtain the specific position of the goods in space, and complete the product splitting by analyzing the spatial coordinates of each product, ensuring that each product occupies a unique position in the reconstructed three-dimensional space, avoiding overlap and confusion between products. The serial number of the product corresponds one-to-one with the information in the product list to ensure data consistency, effectively handle problems such as product stacking and occlusion, and accurately identify and split each product in a complex product arrangement. By accurately calibrating each product, the efficiency and accuracy of subsequent operations (such as trajectory prediction, sorting, etc.) can be ensured.

[0104] In practical applications, such as warehouse management or automated sorting systems, goods may be randomly placed and have different shapes and sizes. In this case, 3D reconstruction can help the system accurately identify each product through spatial coordinates and separate the stacked goods into separate items based on spatial position information, thus avoiding the misjudgment that may be caused by a single 2D image processing method.

[0105] S3.1 The specific steps are as follows:

[0106] S3.1.1: Traverse each pixel in the product placement image, calculate the sum of the grayscale weighted differences between the pixel neighborhood and the pixel, compare the sum of the grayscale weighted differences with the binarization threshold, and if it is less than the binarization threshold, set the pixel to zero to obtain a pixel binarization image;

[0107] Grayscale weighted difference refers to the weighted average of the grayscale differences between each pixel and its surrounding pixels. The weight is usually determined by the distance or similarity of the pixels in the neighborhood. The grayscale weighted difference is compared with the preset binarization threshold. If the sum of the grayscale weighted differences is less than the binarization threshold, the point is considered to belong to the background and is set to zero. Otherwise, it is retained as a foreground pixel to form a binary image.

[0108] Edge information in an image can be effectively captured through the gray - scale differences of local pixels. Since the gray - scale changes are usually significant in the edge region, calculating the weighted difference can highlight the object contours in the image. By comparing the gray - scale weighted difference with the binarization threshold, noise in the image can be filtered out, and only key information is retained, thereby improving the accuracy of subsequent image - processing steps. In the commodity channel, if multiple commodities are placed side by side and the colors of the commodities and the background are relatively similar, simple direct binarization may not effectively segment the commodities from the background. By calculating the gray - scale weighted difference of each pixel with its neighborhood, the edges of the commodities can be highlighted, avoiding interference from background noise and improving the accurate extraction of the commodity region.

[0109] S3.1.2: Perform peak statistics on the pixel binarized image through the Hough transform, determine the straight - line intersection coordinates based on the peaks, obtain the commodity display area, project all white pixels in the commodity display area, and extract the commodity contour area;

[0110] The Hough transform maps the pixel points in the image to the Hough space. In this space, the straight - line features of the pixel points are represented as a set of peaks. By statistically analyzing these peaks, the intersection positions of the straight lines in the image can be determined, and this intersection point is the boundary of the commodity display area. Using this intersection - point information, the commodity contour can be accurately extracted, and further projection operations can be carried out to extract the specific contour area of the commodity;

[0111] The Hough transform can effectively identify the straight - line features in the commodity image. Especially when there are obvious straight - line structures on the edges or surfaces of the commodities, it can efficiently extract the contours.

[0112] S3.1.3: Calculate the pixel - ratio features of all pixel points in the commodity contour area, perform background subtraction on the image according to the pixel - ratio features, and calculate the contour information of the commodity through the opening operation of morphology. The formula for the contour information is:

[0113] ;

[0114] Among them, represents the contour feature of the commodity contour area, i represents a single Gaussian distribution of the commodity contour area in the Gaussian mixture model, K represents the total number of Gaussian distributions of the commodity contour area in the Gaussian mixture model, represents the weight value of the i - th Gaussian distribution, represents the probability density function of the Gaussian distribution, X represents the commodity contour area, represents the mean value of the Gaussian distribution of the commodity contour area, represents the variance of the Gaussian distribution of the commodity contour area, represents the erosion operation, represents the expansion operation, T represents the mean of the Gaussian distribution of the product contour area in the mixed Gaussian model;

[0115] Pixel ratio features are usually encoded through information such as brightness, contrast, and texture of the pixel to describe the appearance characteristics of the object. Through these features, the product contour area can be effectively distinguished from the background area. The background subtraction operation is used to further remove unnecessary background information to make the object contour more prominent. Then, the morphological opening operation is used to further optimize the extraction of the product contour, remove small noise areas, and correct the integrity of the product contour.

[0116] The specific steps of S3.2 are as follows:

[0117] S3.2.1: Convert the color space of the product placement image into grayscale, detect the dark area of ​​the product placement image, determine the position of the reflection area, calculate the texture of the reflection area, compensate the product placement image according to the calculation result of the reflection area texture, and eliminate the reflection points in the product placement image;

[0118] In actual industrial scenarios, the reflected light on the surface of the product may cause highlights or unreal reflection points to appear in the reflective area of ​​the image, which will interfere with the subsequent depth estimation and object recognition. To solve this problem, the product placement image is first converted into a grayscale image to simplify subsequent processing. Then, the dark area is detected by image analysis technology to identify the reflective area in the product image. Next, the texture information of the reflective area is used to compensate these areas. This compensation step helps to eliminate the reflection points in the image, thereby ensuring the accuracy of depth estimation and providing cleaner and interference-free image data for subsequent image processing steps. The core advantage of reflective area compensation is that it can remove the interference caused by the reflection of the product surface, especially in scenes with highly reflective materials or large changes in lighting conditions. By reducing the interference of reflection, the quality of the image can be effectively improved, providing more realistic image data for subsequent depth calculation. In this way, the real appearance and spatial position of the product can be accurately obtained, especially in highly reflective environments, compensating for reflection points can significantly improve the recognition rate and processing accuracy.

[0119] See also Figure 3 , assuming that reflection point 1 and reflection point 2 are reflection points and have the same texture direction and a reflection angle of 225°, the search direction is maintained until a non-reflection position appears, and the pixel information of the pixel at the non-reflection position is provided to the reflection area to eliminate the reflection point.

[0120] S3.2.2: Calculate the luminous intensity and texture intensity of the product placement image, find the dark area of ​​the product placement image after reflected light compensation, calculate the gradient value of the entire product placement image, and calculate the depth estimation value of the product placement image;

[0121] The purpose of calculating the luminous intensity and texture intensity of an image is to more accurately describe the surface of the product and its lighting conditions. The luminous intensity reflects the reflective characteristics of the product surface, while the texture intensity reflects the details of the object surface. The image after reflection compensation usually exposes some dark areas, especially areas with uneven lighting. By analyzing the gradient values ​​of the product placement image, areas with larger gradients usually indicate the edges of objects or parts with more significant depth changes. This can reveal the depth differences between different areas of the product surface, thereby further calculating the depth estimation value of the product.

[0122] S3.2.3: Calculate the brightness, saturation and chromaticity characteristics of the product placement image, and calculate the air scattering depth based on the pixel values ​​of the product placement image;

[0123] Air scattering refers to the scattering effect of light when it passes through air, which can cause blur or distortion in distant areas of the image. By calculating the brightness, saturation, and chromaticity features in the product placement image, combined with the surface lighting conditions of the object, the depth information of air scattering can be effectively estimated. This method further refines the depth estimation by capturing changes in lighting and color in the image, making the depth information more accurate, especially at long distances or in uneven lighting conditions.

[0124] S3.2.4: Extract continuous curves in the product placement image, perform straight line fitting on the curves, obtain straight lines existing in the product placement image, obtain significant vanishing lines in the product placement image, use the intersection of the significant vanishing line and the straight line of the product placement image as the vanishing point of the product placement image, and calculate the spatial geometric depth based on the vanishing point of the product placement image;

[0125] Straight lines and vanishing lines in product placement images are important clues for identifying the spatial structure of objects. By extracting continuous curves in product placement images and performing straight line fitting on the curves, the system can identify the straight line features in the image. The intersection of these straight line features and the vanishing lines can be used as the vanishing points of the product placement images, and then the spatial geometric depth can be calculated. This process infers the spatial geometric information of the product through the vanishing points, thereby achieving more accurate depth reconstruction.

[0126] S3.2.5: The depth map is calculated through a deep fusion network. The input of the network is the depth estimation value of the product placement image, air scattering depth, spatial geometric depth and other features. The depth features are fused through four input branches (corresponding to different depth information). Each input branch uses a convolutional neural network (CNN) for feature extraction, and extracts depth features through operations such as convolution and pooling. Then, the 512 features extracted by each branch are cascaded and input into the fully connected layer for feature fusion, and finally a depth map with fused depth information is obtained.

[0127] See also Figure 4, the deep fusion network structure diagram of the embodiment of the present invention, whose input is the depth estimation value of the product placement image , air scattering depth , spatial geometric depth and product placement images , the product placement image size is set to 16×16, and the deep feature fusion network has four input branches, among which, , , and The input branch uses a CNN network. Each branch contains three convolutional layers and a maximum pooling layer. The number of filters is 64, 128, 256, and 512 respectively. The size and step size of the convolution kernel are 3×3 and 1 respectively, and ReLU is used as the nonlinear activation function. The pooling factor and step size of the pooling layer are 2×2 and 2 respectively. Among them, , , The weights of the three input branches of the CNN network are shared. These branches generate fixed-dimensional feature vectors for different dimensions of depth information such as depth estimation, air scattering depth, and spatial geometric depth. Subsequently, these feature vectors are cascaded to form a comprehensive feature vector, which is used as the input of the fully connected layer (FC layer). In the FC layer, the first fully connected layer projects the comprehensive feature vector into a high-dimensional feature space with lower dimensions. The main function of this layer is to compress the comprehensive features of the input and extract global depth feature information at the same time. The ReLU activation function is used to further enhance the representation ability of nonlinear features, thereby ensuring that the features can adapt to the depth changes in complex scenes. Then, the second fully connected layer further processes the output of the first layer and maps the 512-dimensional feature vector to a 3-dimensional fusion weight space. These weights are used to quantify the contribution of each input branch in generating the final depth map, thereby achieving effective fusion and optimization of depth estimation, air scattering depth, and spatial geometric depth. By learning the weights of different branches, the FC layer can dynamically adjust the impact of each input branch on the generation of the depth map, solving the errors caused by factors such as occlusion and illumination changes. Finally, the comprehensive features optimized by the FC layer are sent to the pooling layer to generate an optimized depth map.

[0128] By combining image processing technology and depth information, accurate positioning and three-dimensional reconstruction of goods in space can be achieved, thus overcoming the limitation that RFID tags can only provide product information but not precise location, and ensuring the accuracy of positioning and sorting of goods in the channel. Specifically, the specific steps of S3.3 are as follows:

[0129] S3.3.1: Extract the boundary points of the object in each image frame according to the contour information, associate the boundary points of the object with the depth value information, generate point cloud coordinates, align the point cloud coordinates corresponding to the multiple depth maps, and calculate the three-dimensional point cloud coordinates of the product;

[0130] Based on the contour information extracted from the product placement image, the boundary points of the object are extracted in each frame of the image through edge detection technology. These boundary points are the geometric features of the product and can be regarded as the key nodes of the product's external contour. Through the depth map (that is, the depth value of each pixel), these boundary points are associated with the corresponding depth value information to generate the three-dimensional spatial coordinates of each boundary point, that is, the point cloud coordinates. The point cloud is a set of coordinate points that represent the shape of a three-dimensional object. These point clouds can provide the specific position and shape of the product in space.

[0131] Then the point cloud coordinates extracted from multiple images are registered. The purpose of registration is to combine the point cloud data extracted from images taken at different time points, and fuse the 3D data from different perspectives into a unified 3D point cloud model through a certain mathematical model. By optimizing the matching point cloud, the errors under different camera perspectives are eliminated, and the consistency of the 3D coordinates of the product is achieved.

[0132] Specifically, the contour information provides a two-dimensional bounding box for each product, while the depth information provides the third dimension of the bounding box for the product. Combining these contour information and depth data, a corresponding depth value can be assigned to each boundary point of each product, thereby constructing a three-dimensional point cloud coordinate.

[0133] S3.3.2: Set a loss threshold, estimate the resolution based on the depth value information of multiple frames of product placement images, and calculate the depth loss value. If the depth loss value is greater than or equal to the loss threshold, it means that there is a product occlusion at the depth value position. Perform continuity constraint projection on the current position, iteratively correct the 3D point cloud coordinates of the current position according to the geometric constraints corresponding to the current position, and calculate the 3D point cloud coordinates of the occluded product based on the correction result;

[0134] In the process of processing image and depth data, depth information loss is often encountered due to product stacking, perspective problems or other reasons, that is, the depth values ​​of some products cannot be accurately captured, and may be missing or distorted. According to the depth value information of multiple frames of product placement images, the resolution estimate of each depth point is calculated, that is, the accuracy of the depth data is evaluated. If the depth loss value of certain areas exceeds the preset threshold (for example, the depth loss value is greater than the set threshold), it indicates that there may be an occlusion problem in the area, resulting in the inability to accurately obtain the depth information of the product. In order to deal with this situation, the system will perform continuity constraint projection on these areas, that is, infer the depth values ​​of these areas based on the currently known information, and correct them according to geometric constraints (such as the boundaries and physical shapes of the products).

[0135] S3.3.3: Reconstruct the product placement image in three dimensions according to the three-dimensional point cloud coordinates, and perform texture mapping on the three-dimensional model of the product through contour information.

[0136] 3D reconstruction makes the product no longer a simple 2D image or point cloud data, but a 3D model with actual shape and structure. Texture mapping gives the product model a more realistic appearance, improving the visibility and recognition of the product in the channel. In actual product identification, this process greatly enhances the accuracy and visualization of the product model, especially in the subsequent sorting, tracking and transportation process, accurate 3D models are crucial to achieve efficient sorting.

[0137] The specific steps of S4 are as follows:

[0138] S4.1: extract the corner features and grayscale centroid of each product placement image, calculate the product feature points of the image according to the angle between the corner features and the grayscale centroid, perform feature point matching on image frames at different time points according to the product feature points, and calculate trajectory information;

[0139] First, each frame of the product placement image is processed to extract the corner features and grayscale centroid, and then the product feature points of the image are calculated through the angle between the corner features and the grayscale centroid. The core purpose of this operation is to extract the unique identification of the product from the image for subsequent matching. The corner feature is a point with a large amount of information in the image, which can better reflect the edge and contour information of the product, while the grayscale centroid is the brightness center of the product image, reflecting the overall distribution of the product in the image. By combining the two, the morphological information of the product can be captured more comprehensively.

[0140] By matching feature points, the system can find the location of the same product in image frames at different time points and calculate the trajectory information of the product. For example, when a product moves from one end of a channel to the other, the feature points of each frame of the image will change with the movement of the product. By tracking these changes, the system can accurately calculate the movement trajectory of the product.

[0141] Specifically, there are multiple products in the channel, and some of them block each other. If we only rely on traditional visual image processing technology, the products may not be recognized or be recognized incorrectly. By combining corner features and grayscale centroids, the system can accurately identify and track products, ensuring accurate positioning of products even in the case of occlusion or interlacing.

[0142] S4.2: Extract channel edge points through product placement images, fit and generate geometric models, calculate cross-sectional parameters, and calculate channel shrinkage information based on product distribution;

[0143] The channel edge points are extracted from the product placement image, and the geometric model of the channel is generated by the geometric fitting method, and the cross-sectional parameters of the channel are further calculated. The purpose of this step is to analyze the geometric shape of the channel and the distribution characteristics of the products at different positions, especially the shrinking effect of the channel. In the intelligent door channel, the channel is usually gradually narrowed, and this shrinking phenomenon will affect the movement trajectory and speed of the product. Therefore, accurately calculating the geometric shape of the channel is crucial to accurately predict the trajectory of the product.

[0144] By extracting the edge points of the channel, the boundary shape of the channel can be obtained. The edge points are generally composed of the inner wall of the channel. These points can be obtained from the commodity channel image through image processing algorithms (such as edge detection, contour extraction, etc.). These points are fitted using a high-order curve fitting algorithm to generate a geometric model of the channel. This geometric model can accurately describe the shape change of the channel, especially when the channel gradually narrows, and provides a detailed description of the channel shrinkage information.

[0145] According to the geometric model, the cross-sectional parameters of the channel, such as cross-sectional area, curvature, etc., can be calculated. The cross-sectional area reflects the spatial size of the channel at different positions, while the curvature describes the degree of change in the channel shape. Through these parameters, the impact of the channel on the goods when the goods move in the channel can be accurately analyzed.

[0146] Combined with the distribution of goods, the geometric changes in the channel will have different effects on the movement of goods. For example, in the narrow part of the channel, the goods may encounter greater resistance and their speed will decrease. By considering the distribution of goods and the geometric characteristics of the channel, the system can more accurately estimate the dynamic information such as the speed and direction of the goods.

[0147] Specifically, by modeling the geometry of the channel, we can clearly understand and predict the impact of the narrowing of the channel on the movement of goods, especially at the narrowing of the channel, where the movement of goods will be subject to more physical constraints. In this way, we can make a more accurate prediction of the movement trajectory of the goods. By extracting the edge information of the channel and calculating the cross-sectional parameters, we can understand the geometric changes of the channel, so that we can fully consider the physical characteristics of the channel when predicting the trajectory of the goods, and avoid errors caused by relying solely on the trajectory data of the goods. Assume that there are multiple goods moving in the channel and the channel will narrow at certain positions. In the part where the channel shrinks, the movement speed of the goods may slow down. If this is not taken into account, it may lead to inaccurate predictions of the time when the goods arrive at the sorting port. By calculating the degree of contraction and cross-sectional changes of the channel through the geometric model, the movement speed of the goods can be corrected to ensure that the prediction results are more realistic.

[0148] S4.3: Predicting the trajectory of the commodity according to the channel diameter reduction information and the trajectory information, and calculating the trajectory prediction information of each commodity.

[0149] The channel diameter reduction information calculated in the previous steps is combined with the product trajectory information to predict the future trajectory of the product. At this time, it not only relies on the movement trajectory of the product in the channel, but also takes the geometric changes of the channel (such as diameter reduction, curvature change, etc.) as correction factors to further refine the trajectory prediction. The purpose of this step is to accurately predict the future trajectory and arrival time of the product by combining the geometric characteristics of the channel and the historical trajectory of the product.

[0150] Specifically, in the application scenario of the channel machine, since the speed of the goods does not change relatively, that is, the movement of the goods is mainly determined by the combined effect of gravity and the shape of the channel, in most cases, the trajectory prediction of the goods can be relatively stable and rely on the previous state of motion. However, the geometric shape of the channel (such as diameter reduction) and the interaction between goods (such as collision, extrusion, etc.) will affect the movement trajectory of the goods. By combining the channel diameter reduction information, the trajectory prediction of the goods can be updated in real time. Although the speed of the goods remains basically unchanged, the movement path, speed and direction of the goods may change as the channel gradually narrows. Especially in the diameter reduction section of the channel, the goods will be subject to more physical constraints (such as collision or mutual pushing between goods due to smaller space). Therefore, the channel diameter reduction information calculated based on the geometric model can effectively predict the dynamic changes of goods at specific locations. For example, the channel diameter reduction information can provide data on the change of the channel cross section, helping the prediction model understand the behavior pattern of goods at different channel positions. For example, the narrowing of the channel in some places may cause the speed of the goods to slow down or change, thereby affecting the predicted trajectory of the goods. In order to adapt to this change, the system can adjust the travel path of the goods in real time through the prediction model to ensure the accuracy of the trajectory prediction. Furthermore, in an environment where multiple goods are moving at the same time, the mutual interference between goods may cause physical squeezing or displacement, thereby changing their original trajectory. In this case, the commodity distribution change parameters of the channel shrinkage information provide an effective reference, through which a certain degree of estimation and correction can be made to the interference between goods.

[0151] S4.2 The specific steps are as follows:

[0152] S4.2.1: Extract the edge points of the channel based on multiple product channel images, project the edge points to a unified world coordinate system based on the position of the camera, fit the edge points using a high-order curve fitting algorithm, and generate a geometric model of the channel;

[0153] In the intelligent gate-type channel machine, the geometric shape of the commodity channel may change with different environments, such as the curvature and narrowing of the inner wall of the channel. Through multiple commodity channel images, the edge points of the channel are extracted. These edge points are usually composed of clearer feature parts in the image taken by the camera, such as the walls on both sides of the channel, gates, etc.

[0154] By projecting the edge points into a unified world coordinate system, the perspective differences in the image can be eliminated, making the images taken by different cameras consistent. At this point, these edge points can be fitted with the help of a high-order curve fitting algorithm to obtain an accurate geometric model of the commodity channel. This geometric model can reflect the actual shape of the channel (such as a straight line, curved or contracted shape) and provide a basis for subsequent trajectory prediction.

[0155] Assume that in some intelligent gate-type channels, the channel gradually shrinks from wide to narrow. If a simple straight line model is used to represent the channel boundary, the predicted trajectory will not reflect the impact of the narrowing of the channel, resulting in the movement of the goods in the channel being misestimated. By fitting the channel edge and generating a geometric model, the problem that traditional methods cannot handle complex channel shapes can be effectively solved. High-order curve fitting can provide a more accurate representation of the channel shape, especially when the channel is curved or narrow, which can greatly improve the accuracy of geometric modeling and avoid the errors caused by simple linear models. For example, the channel may be narrowed. If fine fitting is not used, the model will not be able to accurately reflect this change, thereby affecting the accuracy of the product trajectory prediction.

[0156] S4.2.2: Perform synchronous sampling along the channel direction of the geometric model, calculate the cross-sectional area, edge curvature and shrinkage rate of the channel section according to the sampling results, and calculate the channel geometric change parameter according to the cross-sectional area, edge curvature and shrinkage rate;

[0157] The cross-sectional area represents the area of ​​the channel intercepted at a certain position, and the width and height of the channel determine this area. In the commodity channel, the change of the cross-sectional area directly affects the efficiency of the commodity passing through, especially when the channel shrinks or bends. The edge curvature represents the degree of curvature of the curve at the edge of the channel. The greater the curvature, the more severe the curvature of the channel edge. A higher curvature value usually means that the channel has a larger turn or complex geometric changes at this point, which may affect the movement trajectory of the commodity. The shrinkage rate refers to the rate of change of the width of the channel at different positions.

[0158] Specifically, if the channel gradually narrows from the entrance to the sorting port, the speed and trajectory of the goods in the channel will be significantly affected. The calculation of the shrinkage rate can help accurately predict the movement of the goods. Through these sampling results, the geometric change parameters of the channel can be accurately calculated. These parameters are crucial for subsequent trajectory prediction. They reflect the changing rules of the geometric shape of the channel at different locations of the goods, which helps to correct the movement trajectory of the goods at different locations. For example, when there are sharp turns or narrow openings in the channel, the geometric changes of the channel will cause the speed and movement direction of the goods to change. Only through careful geometric analysis can the movement of goods be accurately predicted and the sorting process optimized.

[0159] S4.2.3: Based on the external parameter matrix and internal parameter matrix of the camera, project the three-dimensional model of the product at each position onto the geometric model, segment the geometric model along the channel direction according to the number of cameras, analyze the three-dimensional distribution law of the products in the segmented interval, and calculate the distribution change characteristics of the products according to the three-dimensional distribution law;

[0160] Specifically, based on the camera’s internal and external parameter matrices (i.e., the camera’s internal parameters and external positions), the 3D model of the product at each position is projected onto the geometric model. Through comprehensive analysis of multiple camera perspectives, it is possible to ensure that the 3D distribution of the product in the channel is fully captured.

[0161] Next, the channel is segmented according to the number of cameras along the direction of the geometric model, and the distribution patterns of the goods in the segmented intervals are analyzed. For example, goods may be concentrated in certain locations, or may be stagnant or piled up in certain curved or narrow areas. By analyzing these distribution patterns, the distribution change characteristics of the goods can be obtained, such as the tendency of goods to gather in a certain section of the channel, or the change in speed of goods when passing through the channel.

[0162] Through this three-dimensional distribution law analysis, the spatial position and distribution characteristics of the goods in the channel can be more accurately considered when predicting the trajectory of the goods. For example, when multiple goods gather in a certain area of ​​the channel, their movement trajectory may be affected by the goods in front and behind, resulting in speed changes or trajectory deviations. By analyzing these distribution laws, necessary adjustments can be made during trajectory prediction to avoid incorrect trajectory judgments.

[0163] S4.2.4: Calculate the channel diameter reduction information according to the channel geometry change parameter and the commodity distribution change characteristics. The calculation formula of the channel diameter reduction information is:

[0164] ;

[0165] in, Indicates channel reduction information. Indicates the spatial position of the first camera, represents the spatial position of the last camera, z represents the spatial position of the channel in the direction of camera arrangement, The weight function representing the influence of channel geometry change on the shrinkage information, represents the geometric characteristics of the channel at position z, Represents the weight function of the impact of commodity distribution on shrinkage information, represents the distribution law of goods at position z, represents the rate of change of the camera's viewing angle at position z, Indicates the spatial resolution in the channel direction, which is used to calculate the step size of discrete changes.

[0166] Channel shrinkage information characterizes the geometric shrinkage effect that goods may encounter when moving in the channel and the impact of the distribution of goods on the movement trajectory. The calculation results can provide the specific movement of goods in the channel, provide an important basis for the final trajectory prediction of the goods, accurately simulate the physical movement characteristics of the goods in the channel, and adjust the trajectory prediction of the goods based on this information. The acquisition of shrinkage information can effectively avoid trajectory errors caused by channel shrinkage, bending and other problems, and ensure the accuracy and efficiency of the sorting process.

[0167] Specifically, if a section of the channel shrinks due to equipment failure, the movement of the goods passing through this section will be significantly affected. If the system does not accurately calculate the shrinkage information of this section, it may predict that the goods can pass smoothly, causing the goods to miss the sorting port. By calculating the shrinkage information, the predicted trajectory can be adjusted to ensure that the goods can reach the correct sorting port in time and avoid errors caused by channel shrinkage.

[0168] The specific steps of S4.3 are as follows:

[0169] S4.3.1: Using the center line of the geometric model as the baseline and the center of mass of the product in each product placement image as the reference point, calculate the offset distance and draw the deviation curve according to the channel direction;

[0170] The centroid is the center of gravity of all pixels of the product in the image. In the product channel, the movement path of the product will be affected by the channel geometry and external forces. Therefore, the movement trajectory of the product in the channel will usually be offset to a certain extent.

[0171] By calculating the offset distance between the center of mass of the product and the center line of the geometric model and drawing the deviation curve along the channel direction, the deviation between the current position of the product and the ideal path (the center line of the geometric model) can be obtained. This deviation is an important basis for predicting the future position of the product, because the position change of the product in the channel is not completely along a straight line or a standard path, but is constrained by the channel shape.

[0172] S4.3.2: Correcting the speed and direction in the trajectory information according to the channel reduction information;

[0173] When goods pass through the channel, as the channel gradually shrinks, the speed and direction of the goods will change. Through real-time calculation and analysis of channel shrinkage information, the geometric characteristics of the channel at different positions can be obtained, thereby predicting the movement patterns of goods in these areas. Channel shrinkage has a significant restraining effect on the movement of goods. For example, when goods enter the shrinkage area, they may slow down or deviate from the original trajectory due to space limitations.

[0174] The revision process includes:

[0175] When the commodity enters the reduced-diameter area, the reduction of the channel space will cause the flow rate of the commodity to slow down. The space in the reduced-diameter area is analyzed through a real-time geometric model, and then the speed of the commodity is adjusted to ensure that the movement of the commodity in the reduced-diameter area is not overly affected.

[0176] Due to the change in the shape of the channel, the commodity may have a slight direction deviation. The movement direction of the commodity will be corrected in real time according to the change law of the channel shape to ensure that the commodity is predicted according to the actual situation of the channel shape.

[0177] Specifically, for example, when the commodity enters a gradually narrowing channel section, the flow rate of the commodity will gradually slow down due to the narrowing of the channel. If the speed is not corrected in time, it may be predicted that the commodity will continue to flow at the original speed, resulting in a deviation in the prediction time.

[0178] S4.3.3: Starting from the last trajectory point, continue to draw the deviation curve according to the correction results of the speed and direction to generate the next moment's trajectory point until the trajectory point reaches the end of the channel, generating trajectory prediction information.

[0179] Inside the channel, since the movement of the commodity is affected by the geometric changes of the channel, the calculation of the trajectory point needs to be carried out step by step until the commodity reaches the end position of the channel, such as the sorting port or the exit. The key lies in dynamically tracking the position change of the commodity and continuing to predict the movement path of the commodity based on the previously calculated deviation curve. When generating each new trajectory point, update the trajectory using the current correction parameters (speed, direction) to generate the position of the commodity at the next moment until the commodity reaches the destination.

[0180] Specifically, on the channel of the intelligent portal channel machine, the trajectory of the commodity is usually relatively single, that is, the commodity basically moves along the forward direction of the channel, and there are no complex turns or large deviations in the channel. Therefore, the core factors of trajectory prediction mainly focus on the geometric transformation of the channel and the distribution change of the commodity in the channel.

[0181] Please refer to Figure 5 , the structure diagram of the time series sorting model in the embodiment of the present invention. The time series sorting model includes:

[0182] An input layer, which is used to organize the trajectory prediction information into time series data and perform normalization processing;

[0183] In the input layer, the model takes the trajectory prediction information of multiple commodities as input and organizes it into time series data. The trajectory prediction information usually includes dynamic information such as the speed, position, and acceleration of the commodity at different time nodes. Since the dimensions and numerical ranges of these information may be inconsistent, the input data usually needs to be normalized.

[0184] The temporal encoding layer processes the time series data, extracts the temporal features of the trajectory, and constructs the dependency between time steps;

[0185] LSTM processes the input information (product location, speed, acceleration and other features) of each time step step by step, and captures the dependencies between different positions of products on the time axis through its internal state (memory unit). For example, the speed of a product at a certain position in a channel depends not only on the speed at that moment, but also on the speed state at the previous moments. LSTM can process this information and store long-term information in its memory unit, thereby understanding the dynamic changes of the trajectory.

[0186] LSTM can pass information to the next time step through the hidden layer state, so that the subsequent time step can use the state of the previous moment to update its own state. Therefore, the LSTM network can not only extract features from the input information at the current moment, but also use the contextual information of the previous moment to enhance the prediction of the current moment.

[0187] For example, when goods pass through the channel, their speed and acceleration may change in multiple time steps. LSTM will determine whether the goods will speed up or slow down based on the historical speed and acceleration, and adjust the predicted trajectory accordingly. The construction of this dependency is very important because the time sequence of goods arriving at the sorting port is closely related to the movement state of the goods.

[0188] The time modeling layer is used to integrate the dependencies of commodities in time steps and predict the time when commodities arrive at the sorting port.

[0189] The task of the time modeling layer is to integrate the dependencies of products at different time steps based on the features obtained from the time coding layer and make the final time prediction. Modeling is performed through the fully connected layer combined with the self-attention mechanism. In this process, the self-attention mechanism can help the model assign different weights to different time points according to their importance, thereby achieving more accurate modeling of important time steps.

[0190] The specific steps of S6 are as follows:

[0191] S6.1: According to the time sequence of the arrival of the goods at the sorting port output by the time sequence sorting model, the serial number of the goods is matched with the time sequence;

[0192] S6.2: Rearrange the commodities in the commodity list in chronological order to generate a commodity list arranged in the order of arrival at the sorting port;

[0193] S6.3: Outputting optimized product sequence information according to the rearranged product list;

[0194] S6.4: Match the serial numbers of the commodities with the order information of the commodities arriving at the sorting port one by one;

[0195] S6.5: Control the execution of the sorting equipment according to the time sequence information of the goods arriving at the sorting port, including the opening, closing and physical displacement operation of the sorter;

[0196] S6.6: Sort the corresponding goods to their corresponding sorting exits according to the arrival time of the goods and the designated sorting destination.

[0197] Example 2

[0198] An intelligent door-type channel machine, comprising:

[0199] A commodity channel, which is a channel structure with a continuously shrinking diameter and is used to guide batches of commodities to pass along a predetermined path;

[0200] The gate spans across both sides of the commodity channel and includes:

[0201] RFID scanner, used to read the RFID tag information of commodities and generate a commodity list;

[0202] The drive unit is used to control the opening and closing of the gate to ensure that the goods enter the commodity channel at the set time interval;

[0203] The multi-camera system is arranged along the extension direction of the commodity channel, with a set distance between adjacent cameras, including:

[0204] High-resolution cameras are used to capture images of product placement and extract product contour information, center of mass position, and trajectory information;

[0205] Depth sensors, integrated into some cameras, are used to obtain depth map data of product placement;

[0206] The processing module, based on the intelligent portal machine, communicates with the external computing system, including:

[0207] Preprocess and 3D reconstruct product images;

[0208] Predict product trajectories and sort arrival times;

[0209] Control the sorting equipment to sort the goods based on the order in which the goods arrive at the sorting port;

[0210] The sorting device is arranged at the end of the commodity channel and is used to sort the commodities to the corresponding exit according to the time sequence of the commodities arriving at the sorting exit and the designated sorting destination. The sorting device includes:

[0211] Sorting slide rails are used to guide the goods to slide in a specified direction;

[0212] The sorting execution device completes the sorting of goods through a robotic arm or a flip device.

[0213] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A commodity identification method, applied to an intelligent gate-type channel machine, the intelligent gate-type channel machine comprises a commodity channel with a continuously shrinking diameter and gates across both sides of the commodity channel, the gates are equipped with a radio frequency scanner and a plurality of cameras arranged along the extension direction of the commodity channel, and two adjacent cameras are separated by a set distance, characterized in that: The method comprises: Read the radio frequency tags of commodities passing through the gate by the radio frequency scanner to generate a commodity list; The camera is used to capture images of a batch of commodities passing through the commodity channel to obtain multiple commodity placement images; According to the commodity list and each commodity placement image, determining the order information of the commodities arriving at the sorting port, wherein determining the order information of the commodities arriving at the sorting port comprises: processing the commodity placement image, identifying the commodities in combination with the commodity list, predicting the commodity trajectory according to the channel diameter reduction information and the trajectory information, obtaining trajectory prediction information, outputting the time sequence of the commodities arriving at the sorting port through a time sequence sorting model, and adjusting the commodity list according to the time sequence; The time series sorting model includes: The input layer is used to organize the trajectory prediction information into time series data and normalize it; The temporal encoding layer is used to extract the temporal features of the trajectory and construct the dependency relationship between time steps by processing the time series data; The time modeling layer is used to integrate the dependencies of commodities in time steps and predict the time when commodities arrive at the sorting port; The batch of commodities is sorted according to the sequence information.

2. A commodity identification method according to claim 1, characterized in that: The information of determining the order in which the commodities arrive at the sorting port includes: Processing the product placement image to extract contour information, depth map and trajectory information; Reconstructing the product placement image in three dimensions according to the contour information and the depth map, splitting the batch of products into individual products and marking serial numbers, matching the contour information of the individual products with the product list one by one, and marking the serial numbers corresponding to the products in the product list; Calculating channel diameter reduction information according to the camera position, predicting the trajectory of the commodity according to the channel diameter reduction information and the trajectory information, and calculating trajectory prediction information for each commodity; Constructing a time sequence sorting model, taking the trajectory prediction information as an input parameter of the time sequence sorting model, modeling the trajectory prediction information at a time point through the time sequence sorting model, and outputting the time sequence of the commodities arriving at the sorting port; The commodity list is adjusted according to the time sequence to obtain the order information of the commodities arriving at the sorting port.

3. A commodity identification method according to claim 2, characterized in that: The processing of the commodity placement image includes: Binarize the product placement image, crop the product area of ​​the binary image by Hough transform, perform background subtraction on the product area, and calculate the contour information of the product by morphological opening operation; Calculate the statistical features of the product placement image, calculate the depth estimation value, air scattering depth and spatial geometric depth of the product placement image according to the statistical features, build a deep fusion network, and calculate the depth map through the deep fusion network; The corner features and grayscale centroid of each product placement image are extracted, and the product feature points of the image are calculated according to the angle between the corner features and the grayscale centroid. The image frames at different time points are matched with the feature points according to the product feature points to calculate the trajectory information.

4. A commodity identification method according to claim 3, characterized in that: The three-dimensional reconstruction of the commodity placement image includes: According to the contour information, the object boundary points are extracted in each image frame, the object boundary points are associated with the depth value information to generate point cloud coordinates, the point cloud coordinates corresponding to the multiple depth maps are registered, and the three-dimensional point cloud coordinates of the product are calculated; Set a loss threshold, estimate the resolution based on the depth value information of multiple frames of product placement images, and calculate the depth loss value. If the depth loss value is greater than or equal to the loss threshold, it means that there is a product occlusion at the depth value position. Perform continuity constraint projection on the current position, iteratively correct the 3D point cloud coordinates of the current position according to the geometric constraints corresponding to the current position, and calculate the 3D point cloud coordinates of the occluded product based on the correction result; The commodity placement image is three-dimensionally reconstructed according to the three-dimensional point cloud coordinates, and the three-dimensional model of the commodity is textured using contour information.

5. A commodity identification method according to claim 4, characterized in that: The calculating of the channel diameter reduction information according to the camera position includes: According to multiple commodity channel images, the edge points of the channel are extracted, and the edge points are projected into a unified world coordinate system according to the position of the camera. The edge points are fitted by a high-order curve fitting algorithm to generate a geometric model of the channel. Performing synchronous sampling along the channel direction of the geometric model, calculating the cross-sectional area, edge curvature and shrinkage rate of the channel section according to the sampling results, and calculating the channel geometry change parameters according to the cross-sectional area, edge curvature and shrinkage rate; According to the external parameter matrix and internal parameter matrix of the camera, the three-dimensional model of the product at each position is projected onto the geometric model, and the geometric model is segmented along the channel direction according to the number of cameras, and the three-dimensional distribution law of the products in the segmented interval is analyzed, and the distribution change characteristics of the products are calculated according to the three-dimensional distribution law; The channel diameter shrinkage information is calculated according to the channel geometry change parameter and the commodity distribution change characteristic. The calculation formula of the channel diameter shrinkage information is: ； in, Indicates channel reduction information. Indicates the spatial position of the first camera, represents the spatial position of the last camera, z represents the spatial position of the channel in the direction of camera arrangement, The weight function representing the influence of channel geometry change on the shrinkage information, represents the geometric characteristics of the channel at position z, Represents the weight function of the impact of commodity distribution on shrinkage information, represents the distribution law of goods at position z, represents the rate of change of the camera's viewing angle at position z, Indicates the spatial resolution in the channel direction, which is used to calculate the step size of discrete changes.

6. A commodity identification method according to claim 5, characterized in that: The calculating of the trajectory prediction information of each commodity includes: Using the center line of the geometric model as a reference line and the center of mass of the product in each product placement image as a reference point, the offset distance is calculated and a deviation curve is drawn according to the channel direction; According to the channel diameter reduction information, the speed and direction in the trajectory information are corrected; Starting from the last trajectory point, the deviation curve is continuously drawn according to the correction results of speed and direction to generate the trajectory point at the next moment until the trajectory point reaches the end of the channel and generates trajectory prediction information.

7. A commodity identification method according to claim 2, characterized in that 。 8. A commodity identification method according to claim 2, characterized in that: The obtaining of the order information of the commodities arriving at the sorting port includes: According to the time sequence of the goods arriving at the sorting port output by the time sequence sorting model, the serial numbers of the goods are matched with the time sequence; Rearrange the commodities in the commodity list in chronological order to generate a commodity list arranged in the order of arrival at the sorting port; Based on the rearranged product list, output optimized product sequence information.

9. A commodity identification method according to claim 1, characterized in that: The step of sorting the batch of commodities comprises: Match the serial numbers of the products with the order information of the products arriving at the sorting port one by one; According to the time sequence information of the goods arriving at the sorting port, control the execution actions of the sorting equipment, including the opening, closing and physical displacement operations of the sorter; According to the arrival time of the goods and the designated sorting destination, the corresponding goods are sorted to their corresponding sorting outlets.

10. An intelligent door-type channel machine, characterized in that: The intelligent portal channel machine comprises: A commodity channel, which is a channel structure with a continuously shrinking diameter and is used to guide batches of commodities to pass along a predetermined path; The gate spans across both sides of the commodity channel and includes: RFID scanner, used to read the RFID tag information of commodities and generate a commodity list; The drive unit is used to control the opening and closing of the gate to ensure that the goods enter the commodity channel at the set time interval; The multi-camera system is arranged along the extension direction of the commodity channel, with a set distance between adjacent cameras, including: The camera is used to capture the image of the product placement and extract the product's contour information, center of mass position and trajectory information; Depth sensors, integrated into some cameras, are used to obtain depth map data of product placement; A processing module, integrated in an intelligent gate-type channel machine, communicates with an external computing system, and is used to execute a commodity identification method according to any one of claims 1 to 9, comprising: Preprocess and 3D reconstruct product images; Predict product trajectories and sort arrival times; Control the sorting equipment to sort the goods based on the order in which the goods arrive at the sorting port; The sorting device is arranged at the end of the commodity channel and is used to sort the commodities to the corresponding exit according to the time sequence of the commodities arriving at the sorting exit and the designated sorting destination. The sorting device includes: Sorting slide rails are used to guide the goods to slide in a specified direction; The sorting execution device completes the sorting of goods through a robotic arm or a flip device.

Images