Method, device and medium for quickly retrieving goods on e-commerce platform through photographing
By using a pre-set image segmentation grid and saliency score filtering, combined with a pre-set image segmentation model, the problems of high model expansion costs and over-segmentation in e-commerce platform product retrieval are solved, achieving efficient and accurate product subject identification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN AIMALL TECHNOLOGY CO LTD
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-05
AI Technical Summary
In e-commerce platform product retrieval, existing technologies suffer from high costs and poor flexibility in expanding dedicated object detection models, while general segmentation models suffer from over-segmentation, resulting in poor retrieval accuracy and user experience.
Multiple sub-images of the product to be identified are generated by a preset image segmentation grid. The saliency value and central region of each pixel are determined. The cue point set is merged, and the preset image segmentation model is used for segmentation. The target product segmentation image is selected by combining saliency coverage and centrality scores.
It enables accurate extraction of the main product from complex product images without the need for training, improving the accuracy and efficiency of retrieval and reducing model training and maintenance costs.
Smart Images
Figure CN122157236A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image recognition technology, specifically to a method, apparatus, device, and medium for quickly retrieving products from an e-commerce platform by taking a photo. Background Technology
[0002] In product retrieval and e-commerce applications, automatically outlining the main product from user-uploaded or web-fetched images is a crucial preprocessing step for visual search, category identification, background blurring, and specification verification. Current mainstream technologies have limitations: Dedicated object detection models: These require collecting and labeling massive amounts of data for specific product categories (such as footwear, apparel, and 3C digital products) to train the detector (e.g., the YOLO series). Their drawback is the high cost of expansion: when faced with a massive, long-tail, and frequently updated e-commerce product database, the model needs constant retraining, making it difficult to cover all categories and lacking flexibility.
[0003] Direct application of general segmentation models: This involves using models like the Segment Anything Model (SAM) and its high-speed variants (such as FastSAM) for zero-shot segmentation. These models, designed for general purposes, suffer from severe oversegmentation problems with product images: they not only segment the main product but also output irrelevant background elements such as product labels, decorative patterns, text in the background, desktop textures, and even body parts of the model as independent objects. This generates a large amount of irrelevant "noise" results, requiring complex and unstable post-processing for filtering, severely interfering with the accuracy of the retrieval system and the user experience. Therefore, current technology lacks a solution that can intelligently and accurately focus on and extract a unique main product from complex product images without requiring training. Summary of the Invention
[0004] The purpose of this disclosure is to provide a method, apparatus, device, and medium for quickly retrieving products from an e-commerce platform by taking a photo.
[0005] To achieve the above objectives, the first aspect of this disclosure provides a method for quickly retrieving products from an e-commerce platform by taking a photo, the method comprising: The image of the product to be identified is segmented based on a preset image segmentation grid to generate multiple sub-images of the product to be identified, wherein each sub-image of the product to be identified has the same image size; Determine the first average saliency value of each pixel in the first product sub-image to be identified, wherein the first product sub-image to be identified is any one of the plurality of product sub-images to be identified; If the first average saliency value is greater than a set threshold, the pixel with the largest pixel saliency value is determined from the first sub-image of the product to be identified as the target candidate pixel corresponding to the first sub-image of the product to be identified, based on the multiple pixel saliency values corresponding to each pixel in the first sub-image of the product to be identified. Determine multiple central region pixels corresponding to each product sub-image to be identified, and merge the multiple central region pixels and at least one target candidate pixel to obtain the prompt pixel set corresponding to the product image to be identified; Based on the set of prompting pixels, the image of the product to be identified is segmented using a preset image segmentation model to generate multiple product segmentation images and multiple confidence scores corresponding to each product segmentation image; Based on the multiple confidence scores and the multiple product segmentation images, a target product segmentation image is determined, and based on the target product segmentation image, the product category corresponding to the product image to be identified is determined.
[0006] Optionally, in some embodiments, determining the target product segmentation image based on the plurality of confidence scores and the plurality of product segmentation images includes: Determine multiple candidate bounding boxes corresponding to each product segmentation image; Based on the multiple candidate boxes and the multiple confidence scores, at least one initial segmentation image that meets the set conditions is selected from the multiple product segmentation images; Determine the comprehensive score of at least one product subject corresponding to each of the at least one initial segmented images; Based on the comprehensive score of at least one product entity, the initial segmentation image with the highest comprehensive score of the product entity is determined as the target product segmentation image.
[0007] Optionally, in some embodiments, determining the comprehensive score of at least one product subject corresponding to each of the at least one initial segmented images includes: The first centrality score of the first initial segmentation image is determined based on the normalized distance between the center of the first candidate box and the center of the first image in the first initial segmentation image, wherein the first initial segmentation image is any one of the at least one initial segmentation images. Based on the inclusion relationship between at least one candidate box corresponding to the at least one initial segmented image and the first candidate box, a first integrity score of the first initial segmented image is determined; Based on the multiple pixel saliency values corresponding to each pixel in the first initial segmentation image, the first saliency region coverage score of the first initial segmentation image is determined. Based on the first centrality score, the first integrity score, and the first saliency region coverage score, the first comprehensive score of the product subject corresponding to the first initial segmentation image is determined.
[0008] Optionally, in some embodiments, determining the first salient region coverage score of the first initial segmentation image based on multiple pixel saliency values corresponding to each pixel point in the first initial segmentation image includes: Based on the multiple pixel saliency values, determine multiple target pixels whose pixel saliency values are greater than a set threshold; Determine the pixel ratio corresponding to the plurality of target pixels; Based on the pixel ratio, the first salient region coverage score of the first initial segmentation image is determined.
[0009] Optionally, in some embodiments, determining the first comprehensive score of the product subject corresponding to the first initial segmented image based on the first centrality score, the first integrity score, and the first saliency region coverage score includes: Obtain the multiple weight values corresponding to each score; The first centrality score, the first integrity score, and the first saliency region coverage score are weighted and summed based on the multiple weight values to obtain the first comprehensive score of the product entity.
[0010] Optionally, in some embodiments, the setting conditions include at least one of the following: Confidence score ; The area ratio of the candidate boxes is in the range of (0.05, 0.9).
[0011] Optionally, in some embodiments, determining the first average saliency value of each pixel in the first sub-image of the product to be identified includes: Determine the pixel saliency value between each pixel in the first sub-image of the product to be identified and other pixels near that pixel; Based on the multiple pixel saliency values corresponding to each pixel point in the first product sub-image to be identified, a first average saliency value corresponding to the first product sub-image to be identified is determined.
[0012] According to a second aspect of this disclosure, an apparatus is provided for quickly retrieving products from an e-commerce platform by taking a photo, the apparatus comprising: The first generation module is used to segment the image of the product to be identified based on a preset image segmentation grid, and generate multiple sub-images of the product to be identified, wherein the image size of each sub-image of the product to be identified is the same. The first determining module is used to determine the first average saliency value of each pixel in the first product sub-image to be identified, wherein the first product sub-image to be identified is any one of the plurality of product sub-images to be identified; The determination module is used to determine the pixel with the largest pixel significance value from the first product sub-image to be identified as the target candidate pixel corresponding to the first product sub-image, based on multiple pixel significance values corresponding to each pixel in the first product sub-image to be identified, when the first average significance value is greater than a set threshold. The second determining module is used to determine multiple central region pixels corresponding to each product sub-image to be identified, and to merge the multiple central region pixels and at least one target candidate pixel to obtain a set of prompt pixels corresponding to the product image to be identified. The second generation module is used to segment the image of the product to be identified according to the set of prompt pixels using a preset image segmentation model, and generate multiple product segmentation images and multiple confidence scores corresponding to each product segmentation image. The execution module is used to determine a target product segmentation image based on the multiple confidence scores and the multiple product segmentation images, and to determine the product category corresponding to the product image to be identified based on the target product segmentation image.
[0013] According to a third aspect of this disclosure, an electronic device is provided, comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of the first aspects of this disclosure.
[0014] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any of the first aspects of this disclosure.
[0015] The above technical solution involves segmenting an image of a product to be identified using a preset image segmentation grid, generating multiple sub-images of the product to be identified. A first average saliency value is determined for each pixel in the first sub-image. Based on the multiple saliency values corresponding to each pixel in the first sub-image, the pixel with the largest saliency value is selected as the target candidate pixel for the first sub-image. Multiple central region pixels are determined for each sub-image, and these central region pixels and at least one target candidate pixel are merged to obtain a set of prompt pixels corresponding to the product to be identified. Based on this set, the product image is segmented using a preset image segmentation model, generating multiple product segmentation images and multiple confidence scores for each segmentation image. A target product segmentation image is determined based on the multiple confidence scores and the multiple product segmentation images, and the product category corresponding to the product image is determined based on the target product segmentation image. Thus, by using quantitative features such as centrality, completeness, and saliency coverage, a relatively accurate product subject can be extracted from the photograph, ensuring the accuracy of product subject identification.
[0016] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description
[0017] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a method for quickly retrieving products from an e-commerce platform by taking a photo, according to an exemplary embodiment.
[0018] Figure 2 This is a block diagram illustrating an apparatus for quickly retrieving products from an e-commerce platform by taking a photo, according to an exemplary embodiment.
[0019] Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation
[0020] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.
[0021] Figure 1 This is a flowchart illustrating a method for quickly retrieving products from an e-commerce platform by taking a photo, according to an exemplary embodiment. Figure 1As shown, the method includes the following steps: Step S101: Perform image segmentation on the image of the product to be identified based on a preset image segmentation grid to generate multiple sub-images of the product to be identified.
[0022] In some embodiments, the image sizes of each product sub-image to be identified are all the same. For example, the preset image segmentation grid is a nine-square grid that is equally divided proportionally. After the product image to be identified is segmented by the preset image segmentation grid, it is divided into 9 product sub-images to be identified.
[0023] Step S102: Determine the first average saliency value of each pixel in the first sub-image of the product to be identified.
[0024] In some embodiments, the first product sub-image to be identified is any one of a plurality of product sub-images to be identified.
[0025] Optionally, in some embodiments, step S102 above includes: Determine the pixel saliency value between each pixel in the first sub-image of the product to be identified and other pixels near that pixel; Based on the multiple pixel saliency values corresponding to each pixel point in the first product sub-image to be identified, the first average saliency value corresponding to the first product sub-image to be identified is determined.
[0026] This module aims to generate a set of guiding point cues for subsequent segmentation models, making it more likely that their initial attention will cover the main product.
[0027] First, the image is processed using the StaticSaliencyFineGrained saliency detection algorithm to obtain the saliency map S. map This algorithm is a classic saliency detection method based on biological visual attention mechanisms. Its core principle is to simulate the human visual system's sensitivity to local contrast. It defines the saliency value of each pixel by calculating the difference in low-level visual features such as color and brightness between each pixel and its surrounding area. Although in product images, not only the main product but also areas with strong contrast to the background, such as trademarks, price tags, background decorations, and mannequin accessories, are also marked as "salient" and cannot be directly used, this algorithm ensures that at least one of the generated cue points is on the main product, reducing missed detections in subsequent models.
[0028] The image is then divided into 9 sub-regions using a 3x3 grid. For each region R k Calculate its internal average significance value V. k If V kIf the value is greater than a set threshold (e.g., 0.6), the region is considered to contain visually significant content, and the pixel with the highest significance value in this region is selected as a candidate point. .all Form set P salient Ensure that the cue points are relevant to the image content.
[0029] Simultaneously, a fixed spatial point set P is generated that is independent of the image content but conforms to the prior knowledge of product photography composition. prior Select 9 encrypted points in the central area and add them to the cue point set (e.g., ...). Figure 2 This step forces the model to focus on common product placement locations, such as the image center, providing basic spatial coverage assurance.
[0030] Finally, the two point sets are merged, and duplicate points in the merged point set whose Euclidean distance is less than a threshold (e.g., less than 20 pixels) are removed, retaining the points with higher significance values, to obtain the final prompt point set P. final This invention employs a hybrid strategy that balances "content salience" and "commodity spatial prior".
[0031] Step S103: If the first average saliency value is greater than the set threshold, the pixel with the largest pixel saliency value is determined from the first product sub-image to be identified as the target candidate pixel corresponding to the first product sub-image based on the multiple pixel saliency values corresponding to each pixel in the first product sub-image to be identified.
[0032] Step S104: Determine multiple central region pixels corresponding to each product sub-image to be identified, and merge the multiple central region pixels and at least one target candidate pixel to obtain the prompt pixel set corresponding to the product image to be identified.
[0033] Step S105: Based on the set of prompted pixels, the image of the product to be identified is segmented using a preset image segmentation model to generate multiple product segmentation images and multiple confidence scores corresponding to each product segmentation image.
[0034] For example, in this embodiment, the set of prompt pixels includes multiple prompt pixels. Each prompt pixel is used to prompt the preset image segmentation model to identify and segment the image information at the location of the pixel. That is, the coordinates of the pixel serve as clues for the model to segment the specific product. The preset image segmentation model (e.g., an instance segmentation model based on a visual Transformer or CNN architecture) will process the image and prompt information simultaneously. The model's task is to identify all different product instances in the image and generate an accurate pixel-level segmentation mask for each instance. For each product instance identified by the model, the process will generate two key outputs: (1) Product segmentation image: a binary mask image or an image after cropping / cutting out the corresponding area of the original image, clearly separating the product from the background and other products. (2) Multiple confidence scores: This is usually not a single score, but may be a set of scores used to evaluate the quality and reliability of this segmentation in multiple dimensions. For example: Identification confidence: the degree of confidence of the model in whether the region is the "target product". Segmentation accuracy score: the degree of fit between the generated mask and the real contour of the product. Classification confidence (if the model includes a classification head): The predicted score for the specific category of the product (e.g., "bottled water," "potato chips"). The final result is a series of independent product images and their corresponding quality score lists, which can be directly used for subsequent retail scenarios such as inventory counting, price recognition, and automated checkout.
[0035] The training process of the aforementioned preset image segmentation model includes: I. Input Sample; Each training sample is a triple: 1. Original Product Images: Scene images containing one or more products, such as photos of product shelves or product displays. Images are the direct objects of model learning.
[0036] 2. Hints: These are crucial for guiding the model on "which object to segment." During training, various hints are simulated during inference, including: Cueing: Randomly click one or more points (foreground points) on the target product, or click in the background area (background points). This is the form of the "cue pixel set" during training.
[0037] Tip: Use a rectangle to roughly outline the target product.
[0038] Mask hint: Provide a coarse or incomplete mask to allow the model to be optimized.
[0039] Text prompt: Describe it in natural language, such as "a red Coke can".
[0040] During training, these cue types and locations are randomly sampled to enhance the model's robustness.
[0041] 3. True Labels: These are the "standard answers" that the model needs to learn, including: Instance segmentation mask: A pixel-level binary mask image corresponding to each product instance (foreground is 1, background is 0). If an image has N products, there will be N such masks.
[0042] Optional - Category label: The category for each product instance (e.g., "Coca-Cola 330ml can").
[0043] II. Output; Given an input (image, cue), the model outputs a prediction, which, during training, is compared to the "true label" to calculate the loss. The output typically includes: Multiple prediction masks: The model may output multiple (e.g., 3) candidate masks for a single cue, each mask corresponding to a different possible interpretation of the object.
[0044] The confidence score for each mask: Each candidate mask is predicted with a quality score (IoU score), which represents the likelihood that the model considers the mask to be correct.
[0045] Training process and loss function; The core of training is minimizing the loss function, that is, reducing the gap between the model's predictions and the "standard answer". Commonly used losses include: Mask loss: Calculates the difference between the predicted mask and the ground truth mask. Commonly used are Dice Loss or a combination of Focal Loss and Dice Loss, which penalize pixel-level classification (foreground / background) errors.
[0046] IoU regression loss: penalizes the difference between the predicted confidence score (predicted IoU) and the true IoU between the predicted mask and the true mask. This ensures that the confidence of the model's predictions is calibrated and reliable.
[0047] After iterative training on a large amount of triplet data (image, cue, ground truth mask), some parameters of the model were tuned to their optimal values. At this point, the model learned generalization ability: when encountering new images and cues it had never seen before, it could still output the correct segmentation mask and a reasonable confidence score.
[0048] Optionally, in some embodiments, the above step of determining the target product segmentation image based on multiple confidence scores and multiple product segmentation images includes: Determine multiple candidate bounding boxes corresponding to each product segmentation image; Based on multiple candidate bounding boxes and multiple confidence scores, at least one initial segmentation image that meets the set conditions is selected from multiple product segmentation images; Determine the comprehensive score of at least one product subject corresponding to at least one initial segmented image; Based on the comprehensive score of at least one product entity, the initial segmentation image with the highest comprehensive score of the product entity is determined as the target product segmentation image.
[0049] Optionally, in some embodiments, the above step of determining at least one comprehensive score for each product subject corresponding to at least one initial segmented image includes: The first centrality score of the first initial segmentation image is determined based on the normalized distance between the center of the first candidate box and the center of the first image in the first initial segmentation image. The first initial segmentation image is any one of at least one initial segmentation image. A first integrity score of the first initial segmentation image is determined based on the inclusion relationship between at least one candidate box corresponding to at least one initial segmentation image and the first candidate box. The first salient region coverage score of the first initial segmentation image is determined based on the multiple pixel saliency values corresponding to each pixel in the first initial segmentation image. Based on the first centrality score, the first integrity score, and the first saliency region coverage score, the first comprehensive score of the first commodity subject corresponding to the first initial segmentation image is determined.
[0050] Optionally, in some embodiments, the above step of determining a first salient region coverage score of the first initial segmentation image based on multiple pixel saliency values corresponding to each pixel point in the first initial segmentation image includes: Based on multiple pixel saliency values, identify multiple target pixels whose pixel saliency values are greater than a set threshold; Determine the pixel ratio corresponding to multiple target pixels; Based on the pixel ratio, the first salient region coverage score of the first initial segmentation image is determined.
[0051] Optionally, in some embodiments, the above step of determining the first comprehensive score of the product subject corresponding to the first initial segmented image based on the first centrality score, the first integrity score, and the first saliency region coverage score includes: Obtain the multiple weight values corresponding to each score; The first commodity entity comprehensive score is obtained by weighting and summing the first centrality score, the first completeness score, and the first saliency region coverage score according to multiple weight values.
[0052] Optionally, in some embodiments, the setting conditions include at least one of the following: Confidence score ; The area ratio of the candidate boxes is in the range of (0.05, 0.9).
[0053] For example, this module utilizes the zero-shot capability of a general segmentation model to generate initial segmentation candidates based on the aforementioned cue points. The image is then compared with the final cue point set P. final The input is fed into the FastSAM model. The model outputs an initial set of segmentation results. B i To divide the bounding box, M i For the segmentation mask, S i This represents the confidence score predicted by the model. At this point, R0 raw There are still product components other than the main product and various background distractions.
[0054] Multi-level post-processing filtering based on product characteristics: This module employs cascading filtering logic to simulate the human thought process of selecting subjects. raw The results that best match the characteristics of the "main product" are selected from these.
[0055] First, use primary geometry and confidence filtering to select the model output confidence S. i Candidate boxes with a value greater than 0.5 and an area ratio within the set range (0.05, 0.9) are selected. This step quickly removes obviously invalid noise (such as extremely small textures) and misclassified backgrounds.
[0056] Then, a comprehensive score is calculated for each candidate box. i This score incorporates multiple prior characteristics of the product entity: (1) Centrality score S center Calculate the normalized distance from the center of the bounding box to the center of the image; the score is negatively correlated with the distance. This reflects the compositional prior that "the subject is usually centered."
[0057] (2) Integrity score S integrity : Check the containment relationship between this box and other high-scoring boxes. If a large box almost completely contains one or more significant small boxes, the small boxes are considered to be part of the large box's product area, and the small box's score is penalized.
[0058] (3) Significant area coverage score S salient : Using the saliency map S generated by module one map Calculate the candidate box mask M i Within a given area, the proportion of pixels whose salience exceeds a salience threshold. The main product should cover most of the salience portion of its area.
[0059] The formula for calculating the overall score is: , where α, β, γ, δ are adjustable weights.
[0060] Finally, the single candidate box with the highest overall score is output as the final extracted main product.
[0061] Step S106: Based on multiple confidence scores and multiple product segmentation images, determine the target product segmentation image, and based on the target product segmentation image, determine the product category corresponding to the product image to be identified.
[0062] For example, this embodiment uses a hybrid cue point generation method: employing a nine-square grid partitioning, regional saliency threshold judgment, and a two-stage point set merging and deduplication strategy, it combines adaptive content point selection based on regional saliency assessment with a fixed spatial point set based on compositional priors to generate a hybrid cue point set for product segmentation. For multi-feature fusion post-processing oriented towards the product subject: utilizing the saliency region coverage score calculated from the preceding saliency map, the association between preceding and following modules is established, and a cascading filtering and scoring mechanism specifically designed to select prominent product subjects from the output of a general segmentation model is developed.
[0063] Existing product detection technologies require the collection and labeling of large-scale datasets for specific product categories, which suffers from poor scalability, high costs, and delays. E-commerce scenarios involve a massive, long-tail range of product categories, with new products constantly emerging, making the training and maintenance of dedicated models for each category time-consuming and labor-intensive. Furthermore, directly using general segmentation models for product extraction results in severe "oversegmentation" and "subject blurring" problems. The model treats the product itself, brand logo, decorative patterns, packaging, background text, and even model limbs as equal objects in the output, generating a large number of irrelevant candidate boxes.
[0064] This embodiment introduces prior knowledge from the product domain to intelligently suppress interference: This invention encodes implicit priors such as "the main body of the product is usually centered," "packaging should not replace the product," and "the main body should occupy a prominent area" into explicit filtering rules through quantitative features such as centrality, completeness, and saliency coverage. This effectively suppresses typical interfering elements such as background text, logos, models, and packaging boxes. An end-to-end, collaborative optimization pipeline is constructed. The "saliency coverage score" in post-processing directly utilizes the saliency map calculated in the prompt generation stage, forming a collaborative closed loop of pre-module guidance and post-module verification, enhancing the consistency and robustness within the system. While maintaining the advantage of zero samples, it achieves specialization effects, avoiding the huge costs of data collection and training for dedicated models.
[0065] The above technical solution involves segmenting an image of a product to be identified using a preset image segmentation grid, generating multiple sub-images of the product to be identified. A first average saliency value is determined for each pixel in the first sub-image. Based on the multiple saliency values corresponding to each pixel in the first sub-image, the pixel with the largest saliency value is selected as the target candidate pixel for the first sub-image. Multiple central region pixels are determined for each sub-image, and these central region pixels and at least one target candidate pixel are merged to obtain a set of prompt pixels corresponding to the product to be identified. Based on this set, the product image is segmented using a preset image segmentation model, generating multiple product segmentation images and multiple confidence scores for each segmentation image. A target product segmentation image is determined based on the multiple confidence scores and the multiple product segmentation images, and the product category corresponding to the product image is determined based on the target product segmentation image. Thus, by using quantitative features such as centrality, completeness, and saliency coverage, a relatively accurate product subject can be extracted from the photograph, ensuring the accuracy of product subject identification.
[0066] Figure 2 This is a schematic diagram illustrating an apparatus for quickly retrieving products from an e-commerce platform by taking a photograph, according to an exemplary embodiment. Figure 2 As shown, the device 100 includes: The first generation module 110 is used to perform image segmentation on the image of the product to be identified based on a preset image segmentation grid, and generate multiple sub-images of the product to be identified, wherein the image size of each sub-image of the product to be identified is the same. The first determining module 120 is used to determine the first average saliency value of each pixel in the first product sub-image to be identified, wherein the first product sub-image to be identified is any one of a plurality of product sub-images to be identified; The determination module 130 is used to determine the pixel with the largest pixel significance value from the first product sub-image to be identified as the target candidate pixel corresponding to the first product sub-image, based on the multiple pixel significance values corresponding to each pixel in the first product sub-image to be identified when the first average significance value is greater than a set threshold. The second determining module 140 is used to determine multiple central region pixels corresponding to each product sub-image to be identified, and to merge the multiple central region pixels and at least one target candidate pixel to obtain a set of prompt pixels corresponding to the product image to be identified. The second generation module 150 is used to segment the image of the product to be identified according to the set of prompt pixels and a preset image segmentation model to generate multiple product segmentation images and multiple confidence scores corresponding to each product segmentation image. The execution module 160 is used to determine the target product segmentation image based on multiple confidence scores and multiple product segmentation images, and to determine the product category corresponding to the product image to be identified based on the target product segmentation image.
[0067] Optionally, in some embodiments, the execution module 160 includes: The first determination submodule is used to determine multiple candidate boxes corresponding to each product segmentation image; The filtering submodule is used to filter at least one initial segmentation image that meets the set conditions from multiple product segmentation images based on multiple candidate boxes and multiple confidence scores. The second determining submodule is used to determine the comprehensive score of at least one commodity subject corresponding to at least one initial segmented image; Based on the comprehensive score of at least one product entity, the initial segmentation image with the highest comprehensive score of the product entity is determined as the target product segmentation image.
[0068] Optionally, the second determining submodule includes: The first determining unit is configured to determine a first centrality score of the first initial segmentation image based on the normalized distance between the center of the first candidate box and the center of the first image in the first initial segmentation image, wherein the first initial segmentation image is any one of at least one initial segmentation image. The second determining unit is used to determine a first integrity score of the first initial segmentation image based on the inclusion relationship between at least one candidate box corresponding to at least one initial segmentation image and the first candidate box; The third determining unit is used to determine the first salient region coverage score of the first initial segmentation image based on the multiple pixel saliency values corresponding to each pixel point in the first initial segmentation image. The fourth determining unit is used to determine the first comprehensive score of the first commodity subject corresponding to the first initial segmentation image based on the first centrality score, the first integrity score, and the first saliency region coverage score.
[0069] Optionally, in some embodiments, the third determining unit is used for: Based on multiple pixel saliency values, identify multiple target pixels whose pixel saliency values are greater than a set threshold; Determine the pixel ratio corresponding to multiple target pixels; Based on the pixel ratio, the first salient region coverage score of the first initial segmentation image is determined.
[0070] Optionally, in some embodiments, the fourth determining unit is used for: Obtain the multiple weight values corresponding to each score; The first commodity entity comprehensive score is obtained by weighting and summing the first centrality score, the first completeness score, and the first saliency region coverage score according to multiple weight values.
[0071] Optionally, in some embodiments, the setting conditions include at least one of the following: Confidence score ; The area ratio of the candidate boxes is in the range of (0.05, 0.9).
[0072] Optionally, the first determining module 120 is used for: Determine the pixel saliency value between each pixel in the first sub-image of the product to be identified and other pixels near that pixel; Based on the multiple pixel saliency values corresponding to each pixel point in the first product sub-image to be identified, the first average saliency value corresponding to the first product sub-image to be identified is determined.
[0073] The above technical solution involves segmenting an image of a product to be identified using a preset image segmentation grid, generating multiple sub-images of the product to be identified. A first average saliency value is determined for each pixel in the first sub-image. Based on the multiple saliency values corresponding to each pixel in the first sub-image, the pixel with the largest saliency value is selected as the target candidate pixel for the first sub-image. Multiple central region pixels are determined for each sub-image, and these central region pixels and at least one target candidate pixel are merged to obtain a set of prompt pixels corresponding to the product to be identified. Based on this set, the product image is segmented using a preset image segmentation model, generating multiple product segmentation images and multiple confidence scores for each segmentation image. A target product segmentation image is determined based on the multiple confidence scores and the multiple product segmentation images, and the product category corresponding to the product image is determined based on the target product segmentation image. Thus, by using quantitative features such as centrality, completeness, and saliency coverage, a relatively accurate product subject can be extracted from the photograph, ensuring the accuracy of product subject identification.
[0074] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0075] Figure 3This is a block diagram illustrating an electronic device according to an exemplary embodiment. Figure 3 As shown, the electronic device 300 may include a processor 301 and a memory 302. The electronic device 300 may also include one or more of a multimedia component 303, an input / output (I / O) interface 304, and a communication component 305.
[0076] The processor 301 controls the overall operation of the electronic device 300 to complete all or part of the steps in the method of quickly retrieving products from an e-commerce platform by taking a picture. The memory 302 stores various types of data to support the operation of the electronic device 300. This data may include, for example, instructions for any application or method operating on the electronic device 300, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 303 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 302 or transmitted via communication component 305. The audio component also includes at least one speaker for outputting audio signals. I / O interface 304 provides an interface between processor 301 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 305 is used for wired or wireless communication between the electronic device 300 and other devices. Wireless communication may include Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination of these. Therefore, the corresponding communication component 305 may include a Wi-Fi module, a Bluetooth module, or an NFC module.
[0077] In an exemplary embodiment, the electronic device 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method of quickly retrieving products from an e-commerce platform by taking a picture.
[0078] In another exemplary embodiment, a computer-readable storage medium is also provided, which includes program instructions that, when executed by a processor, implement the steps of the method for quickly retrieving products from an e-commerce platform by taking a photograph. For example, the computer-readable storage medium may be the memory 302 that includes the program instructions, which may be executed by the processor 301 of the electronic device 300 to complete the method for quickly retrieving products from an e-commerce platform by taking a photograph.
[0079] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a processor, which, when executed by the processor, implements the steps of the method described above for quickly retrieving products from an e-commerce platform by taking a picture.
[0080] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.
[0081] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction.
[0082] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.
Claims
1. A method for quickly retrieving products from an e-commerce platform by taking a photo, characterized in that, The method includes: The image of the product to be identified is segmented based on a preset image segmentation grid to generate multiple sub-images of the product to be identified, wherein each sub-image of the product to be identified has the same image size; Determine the first average saliency value of each pixel in the first product sub-image to be identified, wherein the first product sub-image to be identified is any one of the plurality of product sub-images to be identified; If the first average saliency value is greater than a set threshold, the pixel with the largest pixel saliency value is determined from the first sub-image of the product to be identified as the target candidate pixel corresponding to the first sub-image of the product to be identified based on the multiple pixel saliency values corresponding to each pixel in the first sub-image of the product to be identified. Determine multiple central region pixels corresponding to each product sub-image to be identified, and merge the multiple central region pixels and at least one target candidate pixel to obtain the prompt pixel set corresponding to the product image to be identified; Based on the set of prompting pixels, the image of the product to be identified is segmented using a preset image segmentation model to generate multiple product segmentation images and multiple confidence scores corresponding to each product segmentation image; Based on the multiple confidence scores and the multiple product segmentation images, a target product segmentation image is determined, and based on the target product segmentation image, the product category corresponding to the product image to be identified is determined.
2. The method for quickly retrieving products from an e-commerce platform by taking a photo, as described in claim 1, is characterized in that... The step of determining the target product segmentation image based on the multiple confidence scores and the multiple product segmentation images includes: Determine multiple candidate bounding boxes corresponding to each product segmentation image; Based on the multiple candidate boxes and the multiple confidence scores, at least one initial segmentation image that meets the set conditions is selected from the multiple product segmentation images; Determine the comprehensive score of at least one product subject corresponding to each of the at least one initial segmented images; Based on the comprehensive score of at least one product entity, the initial segmentation image with the highest comprehensive score of the product entity is determined as the target product segmentation image.
3. The method for quickly retrieving products from an e-commerce platform by taking a photo, as described in claim 2, is characterized in that... Determining the comprehensive score of at least one product subject corresponding to each of the at least one initial segmented images includes: The first centrality score of the first initial segmentation image is determined based on the normalized distance between the center of the first candidate box and the center of the first image in the first initial segmentation image, wherein the first initial segmentation image is any one of the at least one initial segmentation images. Based on the inclusion relationship between at least one candidate box corresponding to the at least one initial segmented image and the first candidate box, a first integrity score of the first initial segmented image is determined; Based on the multiple pixel saliency values corresponding to each pixel in the first initial segmentation image, the first saliency region coverage score of the first initial segmentation image is determined. Based on the first centrality score, the first integrity score, and the first saliency region coverage score, the first comprehensive score of the product subject corresponding to the first initial segmentation image is determined.
4. The method for quickly retrieving products from an e-commerce platform by taking a photo, as described in claim 3, is characterized in that... The step of determining the first salient region coverage score of the first initial segmentation image based on the multiple pixel saliency values corresponding to each pixel point in the first initial segmentation image includes: Based on the multiple pixel saliency values, determine multiple target pixels whose pixel saliency values are greater than a set threshold; Determine the pixel ratio corresponding to the plurality of target pixels; Based on the pixel ratio, the first salient region coverage score of the first initial segmentation image is determined.
5. The method for quickly retrieving products from an e-commerce platform by taking a photo, as described in claim 3, is characterized in that... The step of determining the first comprehensive score of the product subject corresponding to the first initial segmented image based on the first centrality score, the first integrity score, and the first saliency region coverage score includes: Obtain the multiple weight values corresponding to each score; The first centrality score, the first integrity score, and the first saliency region coverage score are weighted and summed based on the multiple weight values to obtain the first comprehensive score of the product entity.
6. The method for quickly retrieving products from an e-commerce platform by taking a photo, as described in claim 2, is characterized in that... The setting conditions include at least one of the following: Confidence score ; The area ratio of the candidate boxes is in the range of (0.05, 0.9).
7. The method for quickly retrieving products from an e-commerce platform by taking a photo, as described in any one of claims 1-6, is characterized in that... Determining the first average saliency value of each pixel in the first sub-image of the product to be identified includes: Determine the pixel saliency value between each pixel in the first sub-image of the product to be identified and other pixels near that pixel; Based on the multiple pixel saliency values corresponding to each pixel point in the first product sub-image to be identified, a first average saliency value corresponding to the first product sub-image to be identified is determined.
8. A device for quickly retrieving products from an e-commerce platform by taking a photo, characterized in that, The device includes: The first generation module is used to segment the image of the product to be identified based on a preset image segmentation grid, and generate multiple sub-images of the product to be identified, wherein the image size of each sub-image of the product to be identified is the same. The first determining module is used to determine the first average saliency value of each pixel in the first product sub-image to be identified, wherein the first product sub-image to be identified is any one of the plurality of product sub-images to be identified; The determination module is used to determine the pixel with the largest pixel significance value from the first product sub-image to be identified as the target candidate pixel corresponding to the first product sub-image, based on multiple pixel significance values corresponding to each pixel in the first product sub-image to be identified, when the first average significance value is greater than a set threshold. The second determining module is used to determine multiple central region pixels corresponding to each product sub-image to be identified, and to merge the multiple central region pixels and at least one target candidate pixel to obtain a set of prompt pixels corresponding to the product image to be identified. The second generation module is used to segment the image of the product to be identified according to the set of prompt pixels using a preset image segmentation model, and generate multiple product segmentation images and multiple confidence scores corresponding to each product segmentation image. The execution module is used to determine a target product segmentation image based on the multiple confidence scores and the multiple product segmentation images, and to determine the product category corresponding to the product image to be identified based on the target product segmentation image.
9. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.