Method and apparatus for identifying a single item of merchandise

Abstract

The application discloses a single-piece commodity identification method and device, and relates to the technical field of automatic production. A specific embodiment of the method comprises the following steps: identifying image data of a commodity effective area in commodity transmission image data according to a preset area size, calling an identification model to identify commodity data in the image data, obtaining first commodity mask data, simultaneously performing edge detection on the image data to obtain commodity contour data and then generating second commodity mask data, combining the obtained mask data to remove redundancy, obtaining target mask data, and determining a final single-piece commodity area by identifying the number of connected domains in the target mask data. The embodiment implements a multi-modal fusion intelligent identification method, and improves the accuracy of single-piece commodity identification.

Description

Technical Field

[0001] This invention relates to the field of automated production technology, and in particular to a method and apparatus for identifying individual items. Background Technology

[0002] In modern industrial automated production scenarios, goods typically undergo production, inspection, and packaging processes via complete automated production lines. During the conveyor belt transport of goods on these lines, situations frequently arise where multiple items are very close together or even stacked, affecting the accuracy of subsequent single-item-based operations. Therefore, it is necessary to inspect the goods on the conveyor belt to identify standard single items, while rejecting multiple or stacked items and re-entering them into the conveyor belt. Traditional inspection methods primarily rely on direct image data recognition and template matching to determine whether an item is a single piece, but this approach suffers from low accuracy in complex environments. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a method and apparatus for identifying single items, which can perform multimodal fusion of product image data transmitted by the product to detect the product and improve the accuracy of identifying single items.

[0004] To achieve the above objectives, according to one aspect of the present invention, a method for identifying a single item is provided, comprising: Image data of the effective area of ​​the product in the transmitted image data is identified according to the preset area size; The recognition model is invoked to identify the product data in the image data, thereby obtaining the first product mask data; Edge detection is performed on the image data to obtain product outline data, and second product mask data is generated based on the product outline data; Based on the first product mask data and the second product mask data, duplicates are removed to obtain target mask data. Then, based on the number of connected components in the target mask data and the preset single product determination rules, the single product region in the effective product region is identified.

[0005] Optionally, the recognition model is a lightweight visual segmentation model, and the step of calling the recognition model to identify the product data in the image data to obtain the first product mask data includes: Generate multiple types of prompt data based on the image data and preset text prompts; The lightweight visual segmentation model is invoked to identify product data in the image data based on the multi-type cue data, thereby obtaining the first product mask data.

[0006] Optionally, the step of performing edge detection on the image data to obtain product outline data includes: The image data is converted into grayscale image data, and edge detection is performed based on the grayscale image data to obtain edge image data; Contour recognition is performed based on the edge image data to obtain the product contour data.

[0007] Optionally, before deduplicating the first product mask data and the second product mask data to obtain the target mask data, the method further includes: dividing the image data into grids to obtain multiple grid points in the image data; performing edge prediction on the multiple grid points to obtain mask data corresponding to each grid point; and deduplicating the mask data corresponding to each grid point to obtain third product mask data. The step of deduplicating the first product mask data and the second product mask data to obtain the target mask data includes: deduplicating the first product mask data, the second product mask data and the third product mask data to obtain the target mask data.

[0008] Optionally, the step of deduplicating the mask data corresponding to each grid point to obtain the third product mask data includes: Calculate the intersection-union ratio (CIRR) of the mask data corresponding to any two grid points based on the mask data corresponding to each grid point, and determine the overlapping mask data based on the CIRR; The third product mask data is obtained by deduplicating the overlapping mask data.

[0009] Optionally, the step of deduplicating the first product mask data, the second product mask data, and the third product mask data to obtain the target mask data includes: Based on the first confidence score corresponding to the first product mask data, the second confidence score corresponding to the second product mask data, and the third confidence score corresponding to the third product mask data, the first product mask data, the second product mask data, and the third product mask data are sorted in descending order of confidence score to obtain a mask data sequence. Calculate the intersection-union ratio (CIU) of any two mask data in the mask data sequence, and determine the overlapping mask data in the mask data sequence based on the calculation result; The first mask data that appears first in the overlapping mask data sequence is determined, and the mask data other than the first mask data in the overlapping mask data sequence is taken as the data to be deduplicated. The mask data in the mask data sequence is deduplicated to obtain the target mask data.

[0010] Optionally, the preset single-item product determination rule is that the number of connected domains in the effective area of ​​the product is 1 and the product area corresponding to the connected domain in the effective area of ​​the product does not exceed the preset product area threshold.

[0011] Optionally, before identifying the image data of the effective area of ​​the product in the transmitted image data according to the preset area size, the method further includes: The product transmission image data is obtained from the image buffer, wherein the image data in the image buffer is image frame data captured from the product transmission video data at a preset frame rate.

[0012] Optionally, the method further includes: The minimum bounding rectangle of the item in the single item area is determined, and the adjustment angle of the item is determined based on the minimum bounding rectangle, so as to adjust the conveying direction of the item based on the adjustment angle.

[0013] According to another aspect of the present invention, a device for identifying a single item is provided, comprising: The first recognition module is used to recognize the image data of the effective area of ​​the product in the product transmitted image data according to the preset area size; The second recognition module is used to call the recognition model to recognize the product data in the image data and obtain the first product mask data; The detection module is used to perform edge detection on the image data to obtain product outline data, and generate second product mask data based on the product outline data; The third identification module is used to deduplicate the first product mask data and the second product mask data to obtain target mask data, and to identify the single product area in the effective product area based on the number of connected components in the target mask data and the preset single product determination rules.

[0014] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the single-item identification method provided in the embodiments of the present invention.

[0015] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the single-item identification method provided in the embodiments of the present invention.

[0016] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the single-item identification method provided in the embodiments of the present invention.

[0017] One embodiment of the above invention has the following advantages or beneficial effects: it can combine model recognition with edge detection methods to achieve a multimodal fusion recognition strategy, thereby improving the accuracy of product recognition results. Furthermore, the multimodal fusion recognition strategy can solve the problem of low recognition accuracy when using a single recognition method, while balancing recognition efficiency and accuracy, resulting in better recognition performance in automated production scenarios. In addition, the recognition model-based method can improve recognition accuracy in complex scenarios, thereby improving the overall accuracy of the recognition process.

[0018] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0019] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main steps of a method for identifying a single item according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the main modules of a single-item identification device according to an embodiment of the present invention; Figure 3 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 4 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0020] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0022] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0023] In modern automated production scenarios, to ensure the quality of automated production operations and avoid omissions in product handling, product detection is typically performed during product transfer. This involves identifying non-standard single-item scenarios such as multiple items being too close together or stacked. Ideally, produced goods are transferred at certain intervals. However, due to potential equipment malfunctions on the production line, these non-standard single-item scenarios can occur, affecting the smooth progress of subsequent packaging processes. Therefore, product detection identifies non-standard single-item scenarios and controls other equipment to remove these items and reposition them on the transfer equipment, eliminating their negative impact and improving the product qualification rate of the production process. Existing product detection methods mainly fall into two categories: traditional image processing algorithms, which rely on edge detection and template matching for product detection and counting; and deep learning detection methods, which use convolutional neural networks for object detection and segmentation models. However, these methods often have the following drawbacks in practical applications: traditional algorithms suffer from poor detection accuracy and robustness in complex backgrounds, overlapping products, and scenarios with changing poses. Deep learning models struggle to meet high frame rate requirements in industrial scenarios, exhibiting weak zero-shot adaptation capabilities and difficulty in learning with few samples. Existing solutions cannot simultaneously achieve high-precision segmentation, real-time processing, and rapid adaptation to new products, resulting in high deployment and maintenance costs.

[0024] Taking into full account the shortcomings of the above methods, this invention provides a method for identifying single items, which can realize intelligent detection based on multimodal fusion. While ensuring accurate identification of single items, it further improves the identification efficiency and ensures the smooth operation of production processes in industrial automation scenarios.

[0025] Figure 1 This is a schematic diagram illustrating the main steps of a method for identifying a single item according to an embodiment of the present invention. Figure 1 As shown, the method for identifying a single item mainly includes steps S101 to S104.

[0026] Step S101: Identify the image data of the effective area of ​​the product in the product transmission image data according to the preset area size.

[0027] To improve the accuracy of subsequent product outline recognition, the acquired product transport image data needs to be initially segmented to determine the effective area representing the product within the image data. This product transport image data refers to the image data captured by the acquisition device, including the product transport device and the product transport area on it. When identifying the effective product area, the effective area can be defined within the product transport image data according to a preset area size, thus obtaining the image data of that effective area. The preset area size can be set by professionals based on experience.

[0028] In one embodiment, the preset area size can also be calculated based on the product conveying parameters. Specifically, the width of the preset area can be the width of the product conveying device (e.g., a belt), while the length needs to be determined based on the conveying speed of the product conveying device and the packaging speed of the products. For example, if the belt speed is 200 mm / s and the packaging speed is 30 pieces / min, then the effective area length is 200 mm / s. 60 / 30=400mm.

[0029] In one embodiment, when defining the effective product area, the product conveying equipment can be pre-modeled to identify the product conveying equipment area and the product area in the product conveying image data, and the effective product area can be accurately defined based on the identification results. The modeling can be performed using either a static background method or a dynamic background method. Static background method: Before the product conveying equipment starts, an image without product conveying is collected as baseline data. In subsequent data collection, newly collected image data is compared with the current baseline data to identify the effective product area. Dynamic background method: Due to factors such as changes in lighting, static backgrounds may become ineffective. A Gaussian Mixture Model (GMM) can be selected to build a statistical model for each pixel to record the color distribution of that pixel over a certain period. When the difference between the collected image data and the model value is significant, it is determined to be a pixel representing a product, thus defining the effective product area.

[0030] In one embodiment, to ensure the smooth progress of subsequent recognition processes, the obtained image data of the effective product area can be further subjected to perspective transformation and various standardization operations. Specifically, for the obtained image data of the effective product area, the corner points of the effective product area are first identified, then the transformation matrix is ​​calculated. Based on the calculation results, the original image data is corrected to a bird's-eye view, and the lens distortion of the acquisition device is further corrected. It should be noted that the defined effective product area is a rectangular area, but due to factors such as lens distortion, this area is usually represented as a trapezoid or rhombus in the image data, and needs to be restored to a rectangle through the above perspective transformation operation. Specifically, the four vertices of the target area (original vertices) are first marked, the image is cropped, and then the largest length and width are found to obtain the vertices of the target rectangle (target vertices). The transformation matrix M is calculated using the original vertices and the target vertices. Finally, the target area is completely transformed according to matrix M to obtain the transformed image data.

[0031] After completing the above perspective transformation operation, it is also necessary to perform format standardization, scale the transformed image data to a fixed size such as 480x360, convert it to RG4 color space, and use Letter4ox (scale proportionally and fill the edges to maintain the aspect ratio of the image without distortion) to maintain the aspect ratio.

[0032] Step S102: Call the recognition model to recognize the product data in the image data to obtain the first product mask data.

[0033] After obtaining the image data of the valid product area, product recognition is required. First, a recognition model is used to segment the image data, identifying product data and non-product data, and generating a first product mask based on the recognition results. This first product mask represents the product data identified by the model recognition method. Specifically, the recognition model encodes the image data and decodes it based on the encoding result. Based on the decoding result, product edges are identified to obtain the first product mask data. The obtained first product mask data can be a binary image, where 0 represents the background area and 1 represents the product area. During the recognition model's recognition process, prompt data can be generated based on the defined valid product area, enabling the recognition model to quickly and accurately generate the first product mask data based on the prompt data and image data.

[0034] Step S103: Perform edge detection on the image data to obtain product outline data, and generate second product mask data based on the product outline data.

[0035] In addition to directly calling the model to identify product data, edge detection algorithms are also needed for identification. This process can be executed asynchronously with the model recognition process. Specifically, edge detection algorithms (such as the Canny algorithm) are used to detect product outline data in the image data. Then, based on the obtained product outline data, product data and non-product data in the image data are identified, and a second product mask data is further generated. Besides the Canny algorithm, edge detection algorithms such as the Sobel algorithm can also be used.

[0036] Step S104: Based on the first product mask data and the second product mask data, deduplication is performed to obtain the target mask data, and the single product area in the valid product area is identified based on the number of connected components in the target mask data and the preset single product judgment rules.

[0037] After obtaining the product mask data identified through different methods, the first and second product mask data can be fused to remove duplicate mask data. Specifically, the overlapping area between the first and second product mask data is identified, and duplicate first and second product mask data are determined based on the overlapping area. Duplicate mask data refers to mask data that satisfies the condition that the overlapping area is greater than a preset threshold. For duplicate first and second product mask data, one of the mask data can be randomly selected for deletion. The deduplicated target mask data is then obtained. The number of connected components in the target mask data is counted; this number represents the number of products in the identified valid product area. Based on this number of connected components and a preset single-item product determination rule, it is determined whether the current valid product area is a single-item product area. The preset single-item product determination rule includes a connected component count greater than 1 or a connected component area greater than a preset area threshold. The preset area threshold can be set empirically. This preset area threshold is related to the product's placement posture on the product conveying device and the acquisition angle of the acquisition device.

[0038] In one embodiment, for the identified single-item product area, the products in that area are determined to be single items that meet the production / packaging conditions. Products in non-single-item product areas do not meet the production / packaging conditions and need to be rejected before being placed back into the product conveying equipment for conveying and packaging. The rejection operation can be performed by a robotic arm.

[0039] The single-item identification method according to embodiments of the present invention combines model recognition with edge detection methods to achieve a multimodal fusion identification strategy, thereby improving the accuracy of the product identification results. Furthermore, the multimodal fusion identification strategy can solve the problem of low identification accuracy when using a single identification method, while balancing identification efficiency and accuracy, resulting in better identification performance in automated production scenarios. In addition, the model-based identification method can improve identification accuracy in complex scenarios, thereby improving the overall accuracy of the identification process.

[0040] In one embodiment, the recognition model is a lightweight visual segmentation model. The recognition model is called to identify product data in image data to obtain first product mask data, including: generating multi-type prompt data based on image data and preset text prompts; and calling the lightweight visual segmentation model to identify product data in image data based on multi-type prompt data to obtain first product mask data.

[0041] To improve the recognition efficiency of the identification model, a lightweight visual segmentation model can be selected for product data recognition. For example, the SAM2 model (an image and video segmentation model) can be used for recognition. While improving recognition efficiency, this model supports zero-shot recognition, requiring no pre-training and achieving accurate recognition results even during cold starts. Specifically, when using the SAM2 model, multiple types of prompt data can be constructed based on image data and preset text prompts, including point, box, and text prompt data. Based on this multi-type prompt data, the SAM2 model is called to identify product data in the image data, thereby generating the first product mask data.

[0042] When performing recognition, the SAM2 model first performs preliminary mask recognition based on image data (box-type cue data) and text-type cue data of the effective product area. Then, it generates point-type cue data based on the preliminary mask recognition results, and performs recognition based on the point, box, and text-type cue data. The box-type cue data is used to indicate to the model that the product data is within the effective product area. The point-type cue data includes points within and outside the product area, indicating to the model the range of the product data.

[0043] In one embodiment, the prototype SAM2 model is too large to be directly deployed to the edge. Therefore, a lightweight SAM2 model can be selected for deployment.

[0044] In one embodiment, in addition to the SAM2 model, instance segmentation models such as SAM, Mask R-CNN (a deep learning model) lightweight version, etc., can also be selected to achieve the same technical effect of the present invention.

[0045] The single-item identification method according to embodiments of the present invention can select a lightweight identification model to accurately identify product data, achieve the purpose of efficient identification, and at the same time, ensure the real-time requirements of the identification process.

[0046] In one embodiment, edge detection is performed on image data to obtain product outline data, including: converting the image data into grayscale image data, and performing edge detection based on the grayscale image data to obtain edge image data; and performing outline recognition based on the edge image data to obtain product outline data.

[0047] When performing edge detection, the image data first needs to be converted to grayscale to obtain grayscale image data. Based on the grayscale image data, the Canny algorithm is used for edge detection to obtain edge image data, which is the preliminary product outline data. Based on this edge image data, further prediction of points within the outline can be performed to refine the preliminary product outline data, resulting in the final, accurate product outline data.

[0048] Specifically, the edge detection process can include the following steps: grayscale conversion → Canny edge detection → morphological operations → contour finding → contour in-point prediction → mask generation. Morphological operations: Create a 3x3 kernel: `kernel=np.ones((3, 3), np.uint8)`. `cv2.dilate(edges, kernel, iterations=2)` dilates the edges, connecting broken edges. `cv2.erode(edges, kernel, iterations=1)` performs erosion to remove noise. Contour finding: `cv2.findContours(edges,cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)` finds all contours in the image and returns a list and hierarchy of contours. Contour filtering and processing: Iterate through each detected contour and calculate the contour area: `cv2.contourArea(contour)`. If the contour area is between the minimum and maximum values ​​specified in the configuration file, process the contour. Use `cv2.fillPoly` to create a filled polygon mask to obtain the final product contour data.

[0049] In one embodiment, before deduplicating the first product mask data and the second product mask data to obtain the target mask data, the method further includes: dividing the image data into grids to obtain multiple grid points in the image data; performing edge prediction on the multiple grid points to obtain mask data corresponding to each grid point, and deduplicating the mask data corresponding to each grid point to obtain third product mask data; deduplicating the first product mask data and the second product mask data to obtain the target mask data includes: deduplicating the first product mask data, the second product mask data, and the third product mask data to obtain the target mask data.

[0050] Besides model-based recognition and edge detection methods, grid-based recognition methods can also be used. When deduplicating mask data, the mask data results obtained from these three methods can be combined to further improve the accuracy of the recognition results. First, the image data of the effective product area is divided according to a preset grid size, resulting in multiple grid points distributed throughout the image data. Prediction is performed on each grid point to obtain the product area recognition data corresponding to each grid point. That is, each grid point is treated as a point within the product's range for edge prediction. Overlap detection is then performed on the product area recognition data corresponding to each grid point. For multiple overlapping product area recognition data, only one is retained. After deduplication of all product area recognition data, the final third product mask data is obtained. When deduplicating product mask data obtained from different methods, the first, second, and third product mask data can be combined.

[0051] The single-item identification method according to embodiments of the present invention adds a grid-based identification process to model recognition and edge detection, combining the three to form a multimodal fusion method. This ensures the accuracy of the identification results while improving model recognition efficiency. High identification accuracy can also be achieved in complex identification scenarios.

[0052] In one embodiment, deduplication of the mask data corresponding to each grid point is performed to obtain third product mask data, including: calculating the intersection-union ratio (CIRR) of the mask data corresponding to any two grid points based on the mask data corresponding to each grid point, and determining the overlapping mask data based on the CIRR; and deduplication of the overlapping mask data to obtain third product mask data.

[0053] When deduplicating the mask data corresponding to grid points, the mask data for each grid point can be determined first based on the product area identification data corresponding to each grid point. Then, the intersection-over-union (IoU) ratio of the mask data corresponding to any two grid points can be calculated. This IoU ratio reflects the degree of overlap between different mask data. Based on the IoU ratio, two mask data points with an IoU ratio greater than a preset threshold are identified as duplicate data and deduplicated. After completing the deduplication operation for all overlapping mask data, the final third product mask data can be obtained.

[0054] In one embodiment, in addition to calculating the intersection-union ratio, the degree of overlap of the mask data can also be determined by calculating parameters such as the similarity of different mask data and the area of ​​the overlapping region. Specific parameters can be determined according to actual business needs.

[0055] In one embodiment, deduplication is performed based on the first product mask data, the second product mask data, and the third product mask data to obtain the target mask data, including: Based on the first confidence score corresponding to the first product mask data, the second confidence score corresponding to the second product mask data, and the third confidence score corresponding to the third product mask data, the first, second, and third product mask data are sorted in descending order of confidence score to obtain a mask data sequence. The intersection-union ratio (IUU) of any two mask data in the mask data sequence is calculated, and the overlapping mask data in the mask data sequence is determined based on the calculation result. The first mask data in the overlapping mask data sequence is determined, and the mask data other than the first mask data in the overlapping mask data sequence is used as the data to be deduplicated. The mask data in the mask data sequence is deduplicated to obtain the target mask data.

[0056] When fusing and deduplicating results from different modalities, the first, third, and fourth mask data are sorted as a whole according to their confidence scores from highest to lowest, resulting in an ordered sequence of mask data. Next, the intersection-over-union (IoU) ratio between any two mask data points in the sequence is calculated, and multiple duplicate mask data points (i.e., overlapping mask data) are identified based on a preset IoU threshold. For these duplicate mask data points, only the mask data with the highest order is retained as the first mask data, while the other mask data are treated as deduplicated data to obtain the final target mask data.

[0057] In one embodiment, during the deduplication process described above, when deduplicating at least two duplicate mask data, deduplication can be performed based on the confidence score corresponding to each mask data. Specifically, mask data with lower confidence scores is removed, while the mask data with the highest confidence score is retained. Furthermore, while obtaining the product mask data through model recognition, edge detection, and grid-based recognition methods, a corresponding confidence score is also generated for that product mask data.

[0058] In one embodiment, the preset rule for determining a single item is that the number of connected domains in the effective area of ​​the item is 1 and the area of ​​the item corresponding to the connected domain in the effective area of ​​the item does not exceed a preset item area threshold.

[0059] The pre-defined rules for determining a single item need to identify both multi-item and stacked item scenarios. In multi-item scenarios, the number of connected components in the effective region is greater than 1, while in stacked item scenarios, the area of ​​each item is larger than that of a single item. Therefore, the corresponding rule for determining a single item can be set to ensure that the number of connected components in the effective region of the item is 1, and that the area of ​​the item corresponding to each connected component in the effective region does not exceed a pre-defined item area threshold. This pre-defined item area threshold can be determined based on manual experience or calculated based on the average area of ​​items transmitted historically.

[0060] The product area corresponding to the connected component needs to be converted according to the pixel-to-actual size ratio. The length and width are obtained from the smallest bounding rectangle to get the final product area.

[0061] In one embodiment, before identifying the image data of the effective area of ​​the product in the product transmission image data according to the preset area size, the method further includes: obtaining the product transmission image data from the image buffer, wherein the image data in the image buffer is image frame data captured from the product transmission video data at a preset frame rate.

[0062] To ensure the real-time performance of the recognition process, an image buffer can be set up to store the real-time acquired product delivery data. This can be achieved by capturing frames from an RTSP (Real-Time Streaming Protocol) stream at 30 FPS (frames per second) using the VideoCapture class based on OpenCV (an open-source, cross-platform computer vision library). This product delivery image data can be stored in the image buffer before subsequent recognition processing and retrieved during later processing. To ensure real-time performance, unprocessed image data can be selectively deleted from the image buffer according to certain data discarding rules. For example, images stored for longer than a preset threshold can be deleted, or multiple images with the longest storage time can be deleted periodically at a certain frequency. Specific data discarding rules can be set according to business requirements.

[0063] Since the recognition process may take a long time, exceeding the image acquisition speed, when retrieving product transmission image data from the image buffer later, in order to ensure real-time performance, the latest product transmission image data in the image buffer can be used to avoid continuously processing historical product transmission image data, which would affect the real-time performance of the recognition process.

[0064] In one embodiment, the method further includes: determining the minimum bounding rectangle of the item in the single item area, and determining the adjustment angle of the item based on the minimum bounding rectangle, so as to adjust the conveying direction of the item based on the adjustment angle.

[0065] To improve packaging efficiency, the system can identify whether a product is a single item and its placement angle. For items with incorrect angles, it can identify the smallest bounding rectangle of the product within the single-item area and calculate the adjustment angle based on this rectangle. This corrects the product's placement angle and adjusts its conveyor direction, improving subsequent packaging efficiency. Simultaneously, the conveyor belt's forward direction should be set to 0°, and the reverse direction to 180° to avoid inconsistent directions. Angle commands can be generated based on the adjustment angle to drive the robotic arm to correct the product's orientation, adapting to the subsequent packaging process.

[0066] The single-item identification method provided by this invention can construct a multimodal fusion identification architecture based on different identification methods for product data, fusing the results of model identification and algorithm identification to improve the overall accuracy of the identification process. Simultaneously, by selecting a lightweight model, the efficiency of the identification process is further improved, ensuring real-time requirements. Furthermore, setting a buffer during data acquisition avoids data accumulation, enabling the identification process to handle the latest acquired image data, further guaranteeing the real-time requirements of the identification process.

[0067] Figure 2 The diagram shows the main modules of the single-item identification device provided in an embodiment of the present invention. Figure 2 As shown, the single-item identification device 200 mainly includes a first identification module 201, a second identification module 202, a detection module 203, and a third identification module 204.

[0068] The first recognition module 201 is used to recognize the image data of the effective area of ​​the product in the product transmission image data according to the preset area size; The second recognition module 202 is used to call the recognition model to recognize the commodity data in the image data and obtain the first commodity mask data; The detection module 203 is used to perform edge detection on the image data to obtain the product outline data, and generate second product mask data based on the product outline data; The third identification module 204 is used to deduplicate the first product mask data and the second product mask data to obtain the target mask data, and to identify the single product area in the effective product area based on the number of connected components in the target mask data and the preset single product judgment rules.

[0069] The single-item identification device according to embodiments of the present invention can combine model recognition with edge detection methods to achieve a multimodal fusion identification strategy, thereby improving the accuracy of the product identification results. Furthermore, the multimodal fusion identification strategy can solve the problem of low identification accuracy when using a single identification method, while balancing identification efficiency and accuracy, resulting in better identification performance in automated production scenarios. In addition, the identification method based on the identification model can improve identification accuracy in complex scenarios, thereby improving the overall accuracy of the identification process.

[0070] In one embodiment, the recognition model is a lightweight visual segmentation model, and the second recognition module 202 is used for: Generate multiple types of prompt data based on image data and preset text prompts; A lightweight visual segmentation model is invoked to identify product data in image data based on multi-type cue data, thereby obtaining the first product mask data.

[0071] In one embodiment, the detection module 203 is configured to: The image data is converted into grayscale image data, and edge detection is performed based on the grayscale image data to obtain edge image data; Contour recognition is performed based on edge image data to obtain product contour data.

[0072] In one embodiment, the single-item identification device 200 further includes: a division module 205 (not shown in the figure), used to divide the image data into grids to obtain multiple grid points in the image data; and a prediction module 206 (not shown in the figure), used to perform edge prediction on the multiple grid points to obtain mask data corresponding to each grid point, and to remove duplicates from the mask data corresponding to each grid point to obtain third item mask data. The third identification module 204 is used to deduplicatize the first product mask data, the second product mask data, and the third product mask data to obtain the target mask data.

[0073] In one embodiment, the prediction module 206 is configured to: Calculate the intersection-union ratio (CIRR) of the mask data corresponding to any two grid points based on the mask data corresponding to each grid point, and determine the overlapping mask data based on the CIRR. The overlapping mask data is deduplicated to obtain the third product mask data.

[0074] In one embodiment, the third identification module 204 is used for: Based on the first confidence score corresponding to the first product mask data, the second confidence score corresponding to the second product mask data, and the third confidence score corresponding to the third product mask data, the first product mask data, the second product mask data, and the third product mask data are sorted in descending order of confidence score to obtain the mask data sequence. Calculate the intersection-union ratio (CIU) of any two mask data in the mask data sequence, and determine the overlapping mask data in the mask data sequence based on the calculation result; The first mask data in the overlapping mask data sequence is determined, and the mask data other than the first mask data in the overlapping mask data sequence is taken as the data to be deduplicated. The mask data in the mask data sequence is deduplicated to obtain the target mask data.

[0075] In one embodiment, the preset rule for determining a single item is that the number of connected domains in the effective area of ​​the item is 1 and the area of ​​the item corresponding to the connected domain in the effective area of ​​the item does not exceed a preset item area threshold.

[0076] In one embodiment, the single-item identification device 200 further includes: an acquisition module 207 (not shown in the figure), used for: Product transmission image data is obtained from the image buffer. The image data in the image buffer consists of image frames captured from the product transmission video data at a preset frame rate.

[0077] In one embodiment, the single-item identification device 200 further includes an adjustment module 208 (not shown in the figure), used to determine the minimum bounding rectangle of the item in the single-item area, and to determine the adjustment angle of the item based on the minimum bounding rectangle, so as to adjust the conveying direction of the item based on the adjustment angle.

[0078] The single-item identification device provided in this embodiment of the invention can construct a multimodal fusion identification architecture based on different identification methods for product data, fusing the results of model identification and algorithm identification to improve the overall accuracy of the identification process. Simultaneously, by selecting a lightweight model, the efficiency of the identification process is further improved, ensuring real-time requirements. Furthermore, setting a buffer during data acquisition avoids data accumulation, enabling the identification process to handle the latest acquired image data, further guaranteeing the real-time requirements of the identification process.

[0079] Figure 3 An exemplary system architecture 300 is shown, which can be applied to the single-item identification method or single-item identification device according to embodiments of the present invention.

[0080] like Figure 3As shown, system architecture 300 may include terminal devices 301, 302, and 303, a network 304, and a server 305. Network 304 serves as the medium for providing communication links between terminal devices 301, 302, and 303 and server 305. Network 304 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0081] Users can use terminal devices 301, 302, and 303 to interact with server 305 via network 304 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 301, 302, and 303, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0082] Terminal devices 301, 302, and 303 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0083] Server 305 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 301, 302, and 303 (for example only). The backend management server can analyze and process data such as received single-item identification requests, and feed back the processing results (such as identification results - for example only) to the terminal devices.

[0084] It should be noted that the single-item identification method provided in this embodiment of the invention is generally executed by server 305, and correspondingly, the single-item identification device is generally set in server 305.

[0085] It should be understood that Figure 3 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0086] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer system 400 suitable for implementing terminal devices or servers of the present invention. Figure 4 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0087] like Figure 4As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0088] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. Drive 410 is also connected to I / O interface 405 as needed. Removable media 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 410 as needed so that computer programs read from them can be installed into storage section 408 as needed.

[0089] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit (CPU) 401, it performs the functions defined above in the system of this invention.

[0090] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0092] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a first identification module, a second identification module, a detection module, and a third identification module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the first identification module can also be described as "a module for identifying the effective area of ​​a commodity in the commodity transmission image data according to a preset area size."

[0093] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: Image data of the effective area of ​​the product in the transmitted image data is identified according to the preset area size; The recognition model is invoked to identify the product data in the image data, and the first product mask data is obtained; Edge detection is performed on the image data to obtain product outline data, and second product mask data is generated based on the product outline data; The target mask data is obtained by deduplicating the first and second product mask data, and the single product region in the valid product region is identified based on the number of connected components in the target mask data and the preset single product determination rules.

[0094] According to the technical solution of this invention, model recognition and edge detection methods can be combined to achieve a multimodal fusion recognition strategy, improving the accuracy of product recognition results. Furthermore, the multimodal fusion recognition strategy can solve the problem of low recognition accuracy when using a single recognition method, while balancing recognition efficiency and accuracy, resulting in better recognition performance in automated production scenarios. In addition, the recognition model-based method can improve recognition accuracy in complex scenarios, thereby improving the overall accuracy of the recognition process.

[0095] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for identifying a single item, characterized in that, include: Image data of the effective area of ​​the product in the transmitted image data is identified according to the preset area size; The recognition model is invoked to identify the product data in the image data, thereby obtaining the first product mask data; Edge detection is performed on the image data to obtain product outline data, and second product mask data is generated based on the product outline data; Based on the first product mask data and the second product mask data, duplicates are removed to obtain target mask data. Then, based on the number of connected components in the target mask data and the preset single product determination rules, the single product region in the effective product region is identified.

2. The method according to claim 1, characterized in that, The recognition model is a lightweight visual segmentation model. The step of calling the recognition model to identify product data in the image data and obtain first product mask data includes: Generate multiple types of prompt data based on the image data and preset text prompts; The lightweight visual segmentation model is invoked to identify product data in the image data based on the multi-type cue data, thereby obtaining the first product mask data.

3. The method according to claim 1, characterized in that, The step of performing edge detection on the image data to obtain product outline data includes: The image data is converted into grayscale image data, and edge detection is performed based on the grayscale image data to obtain edge image data; Contour recognition is performed based on the edge image data to obtain the product contour data.

4. The method according to claim 1, characterized in that, Before deduplicating the first product mask data and the second product mask data to obtain the target mask data, the method further includes: dividing the image data into grids to obtain multiple grid points in the image data; performing edge prediction on the multiple grid points to obtain mask data corresponding to each grid point; and deduplicating the mask data corresponding to each grid point to obtain the third product mask data. The step of deduplicating the first product mask data and the second product mask data to obtain the target mask data includes: deduplicating the first product mask data, the second product mask data and the third product mask data to obtain the target mask data.

5. The method according to claim 4, characterized in that, The step of deduplicating the mask data corresponding to each grid point to obtain the third product mask data includes: Calculate the intersection-union ratio (CIRR) of the mask data corresponding to any two grid points based on the mask data corresponding to each grid point, and determine the overlapping mask data based on the CIRR. The third product mask data is obtained by deduplicating the overlapping mask data.

6. The method according to claim 4, characterized in that, The process of deduplicating the first product mask data, the second product mask data, and the third product mask data to obtain the target mask data includes: Based on the first confidence score corresponding to the first product mask data, the second confidence score corresponding to the second product mask data, and the third confidence score corresponding to the third product mask data, the first product mask data, the second product mask data, and the third product mask data are sorted in descending order of confidence score to obtain a mask data sequence. Calculate the intersection-union ratio (CIU) of any two mask data in the mask data sequence, and determine the overlapping mask data in the mask data sequence based on the calculation result; The first mask data that appears first in the overlapping mask data sequence is determined, and the mask data other than the first mask data in the overlapping mask data sequence is taken as the data to be deduplicated. The mask data in the mask data sequence is deduplicated to obtain the target mask data.

7. The method according to claim 1, characterized in that, The preset single-item product determination rule is that the number of connected domains in the effective area of ​​the product is 1 and the product area corresponding to the connected domain in the effective area of ​​the product does not exceed the preset product area threshold.

8. The method according to claim 1, characterized in that, Before identifying the effective area of ​​the product in the transmitted product image data according to the preset area size, the method further includes: The product transmission image data is obtained from the image buffer, wherein the image data in the image buffer is image frame data captured from the product transmission video data at a preset frame rate.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The minimum bounding rectangle of the item in the single item area is determined, and the adjustment angle of the item is determined based on the minimum bounding rectangle, so as to adjust the conveying direction of the item based on the adjustment angle.

10. A device for identifying a single item, characterized in that, include: The first recognition module is used to recognize the image data of the effective area of ​​the product in the product transmitted image data according to the preset area size; The second recognition module is used to call the recognition model to recognize the product data in the image data and obtain the first product mask data; The detection module is used to perform edge detection on the image data to obtain product outline data, and generate second product mask data based on the product outline data; The third identification module is used to deduplicate the first product mask data and the second product mask data to obtain target mask data, and to identify the single product area in the effective product area based on the number of connected components in the target mask data and the preset single product determination rules.

11. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

Images