System and method for verifying product live shots

Abstract

A product recommendation method, comprising: providing product information having a product description, a main image, and a plurality of backup images; for each backup image: performing saliency segmentation to obtain a segment and a background, defining a segment bounding box enclosing the segment, extracting image features from the segment bounding box, extracting text features from the product description, extracting main image features from the main image, matching the image features of the segment bounding box with the text features and the main image features, determining the backup image as a qualified backup image if the image features of the segment bounding box match the text features and the main image features; when the number of qualified backup images is equal to or greater than a threshold number, recommending the product to a customer.

Description

[0001] REFERENCE TO RELATED APPLICATIONS

[0002] In the description of the present disclosure, some references are cited and discussed, which can include patents, patent applications, and various publications. The citation and / or discussion of such references in the present disclosure is provided merely for the purpose of illustrating the present disclosure and does not constitute an admission that any such reference is "prior art" to the present disclosure described herein. All references cited and discussed in the specification are incorporated herein by reference in their entirety and the disclosure of each reference is hereby incorporated by reference herein, to the same extent as if each reference were individually incorporated by reference herein. TECHNICAL FIELD

[0003] The present disclosure relates generally to e-commerce, and more particularly to systems and methods for automatically verifying product photos to provide recommended products that pass verification criteria. BACKGROUND

[0004] The background description provided herein is for the purpose of generally presenting the context of the disclosure. The work of the inventors, to the extent the inventors' work is not

[0005] In e-commerce platforms, sellers can provide multiple photos for a product to be sold on the e-commerce platform. Product photos are a key factor in driving e-commerce conversion, where the quality, angle, color, and design template of the photos are very important in conveying product information to customers. To ensure high-quality product photos, e-commerce platforms can employ human inspection and verification of each uploaded product photo. This process is slow and inaccurate.

[0006] Therefore, there is a need in the art to address the aforementioned deficiencies and shortcomings. SUMMARY

[0007] In certain aspects, the present disclosure provides methods and apparatuses for automatically extracting useful information from product photos, cross-verifying between textual descriptions, numerical attributes, and other images, and enhancing key factors in e-commerce evaluation metrics. The effects achieved by certain embodiments of the present disclosure include: (1) automatically extracting the color and texture of a product from a given image and verifying the image through the textual description and numerical attributes of the image; (2) automatically extracting the color and texture of a product from a given image and verifying the image through other images in the same stock keeping unit (SKU); (3) automatically extracting salient objects, evaluating object layout, and comparing the image with different design templates; and (4) automatically enhancing image quality, such as image contrast, brightness, and sharpness.

[0008] In certain aspects, the present disclosure relates to a product recommendation method. In certain embodiments, the method is performed by a computing device, the method comprising:

[0009] providing product information, wherein the product information comprises a product description, a main image, and a plurality of backup images;

[0010] extracting a text feature from the product description and a main image feature from the main image;

[0011] for each backup image of the plurality of backup images, extracting an image feature from the backup image, matching the image feature with the text feature and the main image feature, and determining the backup image as a qualified backup image if the image feature matches the text feature and the main image feature; and

[0012] when a number of qualified backup images is equal to or greater than a threshold number, recommending the product with updated product information to a customer. The updated product information comprises the product description, the main image, and the threshold number of recommended backup images.

[0013] In certain embodiments, the extracting the image feature comprises: performing saliency segmentation to obtain a mask of the backup image, the mask comprising a segment and a background, a saliency value of the segment being greater than a saliency value of the background; defining a segment bounding box enclosing the segment; and extracting the image feature from the segment bounding box.

[0014] In certain embodiments, an average saliency value of the threshold number of recommended backup images is greater than an average saliency value of a remaining number of the qualified backup images.

[0015] In certain embodiments, the method further comprises, before extracting the image feature from the segment bounding box: refining the segment in the segment bounding box to obtain a refined segment bounding box; and enhancing image quality of the refined segment bounding box to obtain an enhanced segment bounding box. The image feature is extracted from the enhanced segment bounding box, and the qualified image is obtained based on the enhanced segment bounding box.

[0016] In certain embodiments, the refining the segment in the segment bounding box comprises:

[0017] removing abnormal pixels from the segment;

[0018] normalizing saliency values of pixels of the segment to obtain normalized saliency values in a range of 0 to 1;

[0019] applying an adaptive threshold to the normalized saliency values to define background pixels and foreground pixels;

[0020] setting saliency values of pixels in an outline of the segment to 1; and

[0021] The saliency values around the edges of the contours are smoothly reduced to 0.

[0022] In certain embodiments, outlier pixels are removed from the segment by erosion and dilation. In certain embodiments, an adaptive threshold is applied by comparing the saliency value of a pixel with the average or median saliency value of its neighboring pixels. When the saliency value of the pixel is less than the average or median, the pixel is defined as a background pixel. Otherwise, the pixel is defined as a foreground pixel. In certain embodiments, k pixels k-pixel boxes, where k is an odd number. The pixel is at the center of the pixel box, while the k pixels The remaining pixels in the k-pixel box are neighboring pixels. In certain embodiments, k is 3, 5, 7, 9, or an odd number larger than 9. In one embodiment, k is 3.

[0023] In certain embodiments, the normalized saliency value can also be in the range of 0 to 255.

[0024] In certain embodiments, the enhancing the image quality of the refined segment boundary box includes performing image sharpening and adaptive image histogram equalization on the refined segment boundary box.

[0025] In certain embodiments, the segment of the backup image includes a plurality of segments, the method further comprises, before extracting the image features from the segment boundary box:

[0026] calculating a 0-degree image moment of each segment boundary box in the plurality of segment boundary boxes;

[0027] deleting a small segment boundary box and retaining a large segment boundary box from the plurality of segment boundary boxes, wherein the 0-degree image moment of the small segment boundary box is less than a 0-degree image moment threshold, and the 0-degree image moment of the large segment boundary box is equal to or greater than the 0-degree image moment threshold; and

[0028] when the number of large segment boundary boxes is greater than one: calculating an average saliency value of each large segment boundary box, and selecting the large segment boundary box with the highest average saliency value as the segment boundary box for extracting the image features.

[0029] In certain embodiments, the 0-degree image moment threshold is 10% of the area of the backup image.

[0030] In certain embodiments, the extracting the image features from the segment bounding box includes extracting general image features from the segment bounding box using a convolutional network, extracting a color histogram from the segment bounding box, extracting top-k colors from the color histogram, and combining the general image features and the top-k colors to form the image features.

[0031] In certain embodiments, the top-k colors are extracted using k-means clustering. In certain embodiments, k is 3.

[0032] In certain embodiments, the determining the backup image as a qualified backup image includes matching the image features to the text features using a transformer, and matching the image features to the main image features using cosine similarity.

[0033] In certain embodiments, the threshold number is 2.

[0034] In certain aspects, the disclosure relates to a product recommendation system. In certain embodiments, the system includes a computing device. The computing device includes a processor and a storage device storing computer executable code that, when executed at the processor, is configured to:

[0035] provide product information, wherein the product information includes a product description, a main image, and a plurality of backup images;

[0036] extract text features from the product description, extract main image features from the main image;

[0037] for each backup image of the plurality of backup images, extract image features from the backup image, match the image features to the text features and the main image features, determine the backup image as a qualified backup image if the image features match the text features and the main image features; and

[0038] recommend the product with updated product information to a customer when a number of qualified backup images is equal to or greater than a threshold number. The updated product information includes the product description, the main image, and the threshold number of qualified backup images.

[0039] In certain embodiments, the computer executable code is configured to extract the image features by: performing saliency segmentation to obtain a mask of the backup image, the mask including a segment and a background, a saliency value of the segment being greater than a saliency value of the background; defining a segment bounding box that encloses the segment; and extracting the image features from the segment bounding box.

[0040] In some embodiments, the threshold number of recommended backup images has an average saliency value that is greater than an average saliency value of the remaining number of the qualified backup images.

[0041] In some embodiments, the computer-executable code is further configured to, before extracting the image features from the segment bounding box: refine the segment in the segment bounding box to obtain a refined segment bounding box; and enhance image quality of the refined segment bounding box to obtain an enhanced segment bounding box. The computer-executable code is configured to extract the image features from the enhanced segment bounding box, and the qualified image is obtained based on the enhanced segment bounding box.

[0042] In some embodiments, the computer-executable code is configured to refine the segment in the segment bounding box by:

[0043] removing outlier pixels from the segment;

[0044] normalizing saliency values of pixels of the segment to obtain normalized saliency values in a range of 0 to 1;

[0045] applying an adaptive threshold to the normalized saliency values to define background pixels and foreground pixels;

[0046] setting saliency values of pixels in an outline of the segment to 1; and

[0047] smoothly decreasing saliency values around edges of the outline to 0.

[0048] In some embodiments, the computer-executable code is configured to enhance image quality of the refined segment bounding box by performing image sharpening and adaptive image histogram equalization on the refined segment bounding box.

[0049] In some embodiments, the segment of the backup image includes a plurality of segments, and the computer-executable code is further configured to:

[0050] calculate a 0-degree image moment of each segment bounding box of a plurality of segment bounding boxes;

[0051] delete a small segment bounding box from the plurality of segment bounding boxes and keep a large segment bounding box, wherein the 0-degree image moment of the small segment bounding box is less than a 0-degree image moment threshold, and the 0-degree image moment of the large segment bounding box is equal to or greater than the 0-degree image moment threshold; and

[0052] When the number of large fragment bounding boxes is greater than one: calculate the average saliency value of each large fragment bounding box, and select the large fragment bounding box with the highest average saliency value as the fragment bounding box for extracting the image features.

[0053] In some embodiments, the 0-degree image moment threshold is 10% of the area of ​​the spare image.

[0054] In some embodiments, the computer-executable code is configured to extract image features from the fragment bounding box by: extracting general image features from the fragment bounding box using a convolutional network; extracting a color histogram from the fragment bounding box; extracting the top k colors from the color histogram; and combining the general image features and the top k colors to form the image features.

[0055] In some embodiments, the computer-executable code is configured to use k-means clustering to extract the top k colors. In some embodiments, k is 3.

[0056] In some embodiments, the computer-executable code is configured to determine the alternative image as a qualified alternative image by: matching the image features with the text features using a converter; and matching the image features with the main image features using cosine similarity.

[0057] In some embodiments, the number of thresholds is 2.

[0058] In some aspects, this disclosure relates to a non-transitory computer-readable medium storing computer-executable code. The computer-executable code, when executed at a processor of a computing device, is configured to perform the methods described above.

[0059] These and other aspects of this disclosure will become apparent from the following description of preferred embodiments in conjunction with the accompanying drawings and their description, although variations and modifications therein may affect the novel conception of this disclosure without departing from it. Attached Figure Description

[0060] The accompanying drawings illustrate one or more embodiments of this disclosure and, together with the written description, serve to explain the principles of this disclosure. The same reference numerals are used throughout the drawings to refer to the same or similar elements of the embodiments.

[0061] Figure 1 A product recommendation system according to certain embodiments of the present disclosure is illustrated schematically.

[0062] Figure 2 Image modules according to certain embodiments of the present disclosure are schematically depicted.

[0063] Figure 3 An example of saliency segmentation is schematically depicted in accordance with some embodiments of the present disclosure.

[0064] Figure 4 Contour creation and bounding box detection is schematically depicted in accordance with some embodiments of the present disclosure.

[0065] Figure 5 Refinement of segments is schematically depicted in accordance with some embodiments of the present disclosure.

[0066] Figure 6 An object module is schematically depicted in accordance with some embodiments of the present disclosure.

[0067] Figure 7 A feature module is schematically depicted in accordance with some embodiments of the present disclosure.

[0068] Figure 8 A feature extraction model is schematically depicted in accordance with some embodiments of the present disclosure.

[0069] Figure 9 A comparison module is schematically depicted in accordance with some embodiments of the present disclosure.

[0070] Figure 10A And Figure 10B A product recommendation method is schematically depicted in accordance with some embodiments of the present disclosure.

[0071] Figure 11 A flowchart of a product recommendation system is schematically depicted in accordance with some embodiments of the present disclosure.

[0072] Figure 12 Processing and filtering of backup images is schematically depicted in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION

[0073] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations will be apparent to persons skilled in the art. Various embodiments of the present disclosure are now described in detail. Referring to the drawings, like numbers indicate like parts throughout the views. As used in the description of the disclosure and throughout the claims, the meaning of "a", "an", and "the" includes plural references unless the context clearly dictates otherwise. Also, as used in the description of the disclosure and the claims, the meaning of "in" includes "in" and "on" unless the context clearly dictates otherwise. As described herein, "a plurality" means two or more. As described herein, the terms "comprising", "carrying", "having", "containing", "involving", and the like are to be understood to be open-ended, i.e., to mean including but not limited to.

[0074] The terms used in this specification generally have their ordinary meanings in the art, in the context of the disclosure, and in the specific context where each term is used. Certain terms used to describe the present disclosure are discussed below or elsewhere in the specification, to provide additional guidance to the practitioner regarding the description of the present disclosure. It will be appreciated that the same thing can be said in more than one way. Consequently, alternative language and synonyms can be used for any one or more of the terms discussed herein. No special significance is to be given to the use of any particular term herein over others based on capitalization, italics, bolding, or other formatting. The use of any term herein is dependent on the intent of the practitioner and the explanation being given at the time of the use of that term. The use of a term here does not exclude the use of other synonyms or terms. The use of examples anywhere in this specification, including examples of any terms discussed herein, is illustrative only, and in no way limits the scope and meaning of the disclosure or of any exemplified term. Likewise, the disclosure is not limited to various embodiments given in this specification.

[0075] As described herein, the phrase at least one of A, B, and C should be interpreted to mean logical (A or B or C) using non-exclusive logical OR (OR). It will be understood that the methods can be performed in different orders (or concurrently) without altering the principles of the present disclosure. As described herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0076] As described herein, the term “module” can indicate, be part of, or include an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module can include memory (shared, dedicated, or group) that stores code executed by a processor.

[0077] The term “code,” as used herein, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. Shared, as used above with respect to the terms, means that parts or all of code from multiple modules can be executed using a single (shared) processor. Also, parts or all of code from multiple modules can be stored in a single (shared) memory. Grouped, as used above with respect to the terms, means that parts or all of code from a single module can be executed using a group of processors. Also, a group of memories can be used to store some or all code from a single module.

[0078] As described herein, the term“interface” generally refers to a communication tool or device used to perform data communication between components at an interaction point between the components. Generally, an interface can be applicable at both hardware and software levels, and can be a unidirectional or bidirectional interface. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. Components in communication with the interface can be, for example, multiple components or peripheral devices of a computer system.

[0079] The present disclosure relates to computer systems. As shown in the accompanying drawings, computer components can include physical hardware components, which are shown as solid line blocks, and virtual software components, which are shown as dashed line blocks. Those of ordinary skill in the art will appreciate that these computer components can be implemented in the form of software, firmware, or hardware components, or combinations thereof, but are not limited to these forms. The apparatuses, systems, and methods described herein can be implemented by one or more computer programs executed by one or more processors. A computer program includes processor-executable instructions stored on a non-transitory tangible computer readable medium. A computer program can also include stored data. Non-limiting examples of non-transitory tangible computer readable media are nonvolatile memory, magnetic storage, and optical storage.

[0080] The present disclosure will now be described more fully with reference to the accompanying drawings, in which embodiments of the present disclosure are shown. The present disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art.

[0081] There are extraction and analysis of image information, such as classification and detection, in the related art. However, they are mostly for natural images, not for commercial images; they only focus on a single topic, not a combined topic. There is no existing solution that can complete the entire process of e-commerce-specific tasks from saliency segmentation, instance-based feature extraction, layout inspection, cross-domain verification, and quality enhancement. Machine vision methods such as classification, detection, and segmentation in the related art only focus on the current image. There is a lack of comparison with other image features.

[0082] In certain aspects of the present disclosure, the e-commerce platform of the present disclosure requires that images or pictures from the same SKU only include products with the same color and the same type. Product images from different perspectives, different lighting conditions, different backgrounds, different zoom levels, or with / without post-processing, or only partial detailed views of the product make this task more challenging. Detailed views of the same product at different perspectives, lighting conditions, zoom levels, create greater feature differences than different types of products or the same type of product in different colors at the same perspective, same lighting condition, same zoom level. In these cases, common themes in the field of machine vision, such as finding duplicate images in the related art, all fail.

[0083] On the other hand, only text feature analysis in the field of natural language processing (NLP) is done in the text description. Correlation between query and title description is a common usage in the related art. Communication between image, numerical attributes, and text description is missing. In certain aspects, to automatically check whether an image is consistent with its description and its listed attributes, the present disclosure provides cross-domain correlation computation.

[0084] Image segmentation in the related art can be divided into two categories. The first category is to segment known object instances, such as people, vehicles, and bicycles. A labeled data set of these object instances must be provided first. The trained neural network cannot segment new types of instances. The second category is saliency detection and segmentation. This method attempts to approach the human perception method and generate a saliency map of the image. This method can segment unknown types of instances from the training data. But for commercial use, the accuracy is still unacceptable. In addition to this, this method usually deals with natural images, not artificial commercial images. In certain aspects, the present disclosure provides accurate object perception based on saliency map that is suitable for artificial commercial images.

[0085] Image enhancement in the related art can only handle the entire image. It does not have the function of instance segmentation of the image. In certain aspects, the present disclosure handles each segmented instance separately from the perspective of a commercial scale board and considers the relationship between different segmented instances.

[0086] This disclosure is superior to related technologies. In some embodiments, the advantages of this disclosure include: (1) providing an iterative segmentation process to segment product images to achieve the required accuracy. First, a fine-tuned saliency detection / segmentation neural network from e-commerce images is applied to the photograph. Then, normalization and adaptive thresholding are applied to the saliency map. Quality enhancement, subpixel smoothing, and noise removal are developed to further improve and enhance image quality on a region-by-region basis. The segmented and enhanced images can ultimately meet the high standards of e-commerce. (2) extracting generalized image features and artificial features for each segmented instance / region within the image. Fewer saliency instances / objects are automatically removed accordingly. High-level features such as aesthetics are summarized and extracted by analyzing the layout and relationships of each segment. Multimodal features such as numerical attribute features and texture features are combined with image features through a combination layer to verify and check image information from multiple domains. Feature conflicts, such as an attribute indicating that the product is blue while an image feature indicates that the product is green, are configured to assess whether there are errors in the product information. Incorrect images, images with incorrect label attributes or text descriptions, and images that conflict with other images from the same SKU can be automatically rejected without human interference.

[0087] Figure 1 A product recommendation system according to certain embodiments of this disclosure is schematically depicted. For example... Figure 1 As shown, system 100 includes computing device 110. In some embodiments, computing device 110 may be a server computer, cluster, cloud computer, general-purpose computer, or special-purpose computer that provides product recommendation services. In some embodiments, computing device 110 may communicate with other computing devices or services to obtain and provide product information. Product information may include text and images. Text may include product titles and descriptions, and images may include a main image and alternative images of the product. In some embodiments, communication is conducted over a network, which may be a wired or wireless network, or various forms such as public and private networks, or via non-transitory computer media, including but not limited to flash drives, USB drives, hard disk drives, floppy disks, SD cards, optical drives, or any other portable data storage media.

[0088] like Figure 1As shown, the computing device 110 can include, but is not limited to, a processor 112, a memory 114, and a storage device 116. In certain embodiments, the computing device 110 can include other hardware components and software components (not shown) to perform their respective tasks. Examples of these hardware and software components can include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices. The processor 112 can be a central processing unit (CPU) configured to control the operations of the computing device 110. The processor 112 can execute an operating system (OS) or other applications of the computing device 110. In some embodiments, the computing device 110 can have multiple CPUs as processors, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. The memory 114 can be a volatile memory, such as a random access memory (RAM), for storing data and information during the operations of the computing device 110. In certain embodiments, the memory 114 can be a volatile memory array. In certain embodiments, the computing device 110 can run on multiple memories 114. The storage device 116 is a non-volatile data storage medium for storing the operating system (not shown) and other applications of the computing device 110. Examples of the storage device 116 can include a non-volatile memory, such as a flash memory, a memory card, a USB drive, a hard disk drive, a floppy disk, an optical drive, a solid state drive (SSD), or any other type of data storage device. In certain embodiments, the storage device 116 can be a local storage, a remote storage, or a cloud storage. In certain embodiments, the computing device 110 can have multiple storage devices 116, which can be the same storage device or different types of storage devices, and the applications of the computing device 110 can be stored in one or more storage devices 116 of the computing device 110. In certain embodiments, the computing device 110 is a cloud computer, and the processor 112, the memory 114, and the storage device 116 are shared resources provided on-demand through the Internet.

[0089] As Figure 1As shown, the storage device 116 includes a product recommendation application 118, a product database 190, and a recommended product database 192. The product recommendation application 118 is configured to determine high-quality products and recommend high-quality products to customers. The product database 190 includes information of products or stock keeping units (SKUs) of an e-commerce platform. The information of a product can include a title, a description, a primary image, and a number of backup images of the product. The number of backup images can be, for example, about 5 to 10. The recommended product database 192 includes information of products that the system plans to recommend to customers. The product information for recommendation can include a title, a description, a primary image, and a predetermined number of enhanced backup images of the product. The predetermined number can be, for example, 1 to 3. In one example, the predetermined number is 2. In certain embodiments, the product database 190 and the recommended product database 192 can also be stored in a remote computing device in communication with the computing device 110, as long as these databases are accessible by the product recommendation application 118.

[0090] In addition, the product recommendation application 118 includes a product retriever 120, an image module 130, an object module 140, a feature module 150, a text and primary image module 160, a comparison module 170, a recommendation module 180, and a user interface 182. In certain embodiments, the product recommendation application 118 can include other applications or modules as needed for its operation. It is noted that each module is implemented by computer executable code or instructions, or data tables or databases, which collectively form an application. In certain embodiments, each module can also include sub-modules. Alternatively, some modules can be combined as a stack. In certain embodiments, some modules can be implemented as circuits instead of executable code. In certain embodiments, some or all of the modules of the product recommendation application 118 can be located at a remote computing device or distributed in the cloud.

[0091] Product retrieval 120 is configured to retrieve product information from product database 190 and provide the product information to image module 130 and text and main image module 160. Product information may include, for example, title text and description, main image, and alternative images. Product retrieval 120 is configured to send alternative images to image module 130 and text and main image to text and main image module 160. Alternate images may be sent one at a time or in batches. The number of alternative images may vary depending on the product. The number of alternative images may be, for example, 2 to 20. In some embodiments, the number of alternative images is 5 to 10. In some embodiments, the number of alternative images is approximately 7 or 8. In some embodiments, product retrieval 120 may be configured to retrieve product information from product database 190 product by product, such that all products in product database 190 can be analyzed by product recommendation application 118. In some embodiments, product database 190 may be analyzed by product recommendation application 118 at predetermined timelines, such as every 24 hours or weekly. In this configuration, the product retrieval unit 120 can record products that have already been analyzed and only provide information on newly stored products from the product database 190 to the image module 130 and the text and main image module 160. The main image and the backup image can be the same size. In some embodiments, when the main image and the backup image are not the same size, the product retrieval unit 120 can also be configured to resize the backup image to the size of the main image. In some embodiments, the aspect ratio of the backup image remains constant during resizing, and a white background is added so that the main image and the resized backup image have the same pixel size in both the horizontal and vertical directions. In the following description, this disclosure uses 500 pixels for both the main image and the backup image. The size is 500 pixels, but the size is not limited to this.

[0092] Image module 130 is configured to process each backup image upon receiving a backup image of the product to obtain an enhanced image, and then provide the enhanced image to object module 140. In some embodiments, image module 130 may process the backup images sequentially or in parallel. Processing each backup image includes segmenting the image to obtain segments in the image, defining bounding boxes surrounding the segments, thinning the segments, and enhancing the quality of the backup image. Figure 2 As shown, the image module 130 includes a saliency segmentation module 132, a bounding box detection module 134, a segmentation refinement module 136, and a quality enhancement module 138.

[0093] The saliency segmentation module 132 is configured to perform saliency detection and segmentation on each of the received auxiliary images to obtain a segmented image, and send the segmented image to the bounding box detection module 134. In some embodiments, saliency estimation is performed using static saliency, motion saliency, or objectness. In some embodiments, saliency estimation is performed using a neural network based method, such as TASED-Net, STRA-Net, or STAViS. In some embodiments, saliency detection and segmentation is performed using a convolutional neural network (CNN) model that is fine-tuned using e-commerce images. In some embodiments, the auxiliary image is a red-green-blue (RGB) image with three channels, and saliency detection and segmentation adds a fourth channel to the auxiliary image. The added channel is also referred to as the saliency map of the auxiliary image, and the value of each pixel in the added channel can be in the range of , in the range of , or in any other arbitrary range. The higher the saliency value, the more attention the pixel receives. In some embodiments, the four-channel auxiliary image can also be referred to as a matting image, where the pixels with high saliency values are foreground pixels and the pixels with low saliency values are background pixels. In some embodiments, the threshold value between foreground pixels and background pixels can be about . In some embodiments, the threshold value is set to 0.6, where the pixels with saliency values equal to or greater than 0.6 are considered foreground pixels. In some embodiments, the saliency channel or saliency map can also be referred to as a mask, where the foreground pixels and background pixels are contained in the mask, the foreground pixels are 1 in the mask, and the background pixels are 0 in the mask. The segmented image has the same size as the auxiliary image; each pixel has four channels, three RGB channels with respective color values, and one saliency channel with its saliency value. In some embodiments, the values in the three RGB channels of the background pixels are set to 255, so that the background pixels are white. However, the original auxiliary image is stored in memory, and the original RGB information of the auxiliary image is available for modules in the product recommendation application 118 to retrieve when needed.

[0094] In some embodiments, since one auxiliary image can have multiple objects, the saliency segmentation module 132 can find multiple segments. Figure 3 Saliency detection and segmentation of an auxiliary image according to some embodiments of the present disclosure is schematically depicted, where the left side is the auxiliary image before saliency processing and the right side is the auxiliary image after saliency processing. As shown in the left side, Figure 3 As shown in the right side, the two bottles have high saliency values and are considered foreground, while the rest of the auxiliary image has low saliency values and is considered background.

[0095] In some embodiments, after performing saliency detection and segmentation on one of the backup images, the saliency segmentation module 132 is also configured to process the next backup image in the same manner. In some embodiments, if there are no foreground pixels in a backup image, the saliency segmentation module 132 discards the backup image and then processes the next backup image. After processing all backup images of the product, the saliency segmentation module 132 can then process the next backup image of the product.

[0096] The bounding box detection module 134 is configured to, upon receiving a segmented image from the saliency segmentation module 132, create a contour for each segment, define a minimum bounding box that surrounds the contour of the segment, and send the image with the bounding box to the segmentation refinement module 136. The bounding box includes pixels within the contour of the segment, as well as pixels inside the bounding box but outside the contour.

[0097] Figure 4 The results of bounding box detection according to certain embodiments of this disclosure are schematically depicted. Figure 4 As shown, bounding box 402 is defined as the outline 404 surrounding the bottle. It should be noted that when there are multiple objects in the standby image, such as two bottles, the bounding box will surround the outlines of both bottles. In some embodiments, the bounding box may also be defined to surround only one object. In this case, if there are multiple objects in the image, a standby image may have multiple bounding boxes.

[0098] The segmentation refinement module 136 is configured to refine one or more segments enclosing the bounding box upon receiving a bounding box from the bounding box detection module 134 to obtain refined segments, and send the refined segments to the quality enhancement module 138. During this stage and before processing by the segmentation refinement module 136, the segmented image has the same size as the standby image, and each pixel has four channels. Bounding boxes are defined in the segmented image and enclose these segments. Each segment has a contour. Pixels outside the bounding box are white background pixels, while pixels inside the bounding box include background and foreground pixels. Since saliency detection may not be perfect, refinement can be performed to improve the quality of the segments. In some embodiments, the segmentation refinement module 136 is configured to define a segment bounding box for each segment and refine each segment. (1) First, for each segment within its segment bounding box, the segmentation refinement module 136 removes outliers from the saliency map obtained from the saliency detection module. In some embodiments, outlier removal is performed by erosion and dilation. (2) Next, the segmentation refinement module 136 normalizes the saliency map after removing outliers. In some embodiments, the normalization is similar to L-infinity normalization. In some embodiments, for the saliency map in the segment, the segmentation refinement module 136 performs Gaussian blurring on the saliency map. Rescale the significance values ​​within the range, then cut off by 1. In other words, within... All values ​​within the range are now set to 1. (3) After normalization, the salience values ​​of pixels in the fragment are now in Within the range. The segmentation refinement module 136 further applies an adaptive threshold to the normalized saliency value in the fragment bounding box. In some embodiments, each pixel has surrounding neighboring pixels, and the average or medium saliency value of the neighboring pixels is calculated. If the saliency value of a pixel is less than the average or medium saliency value, the pixel is defined as a background pixel. If the saliency value of a pixel is equal to or greater than the average or medium saliency value, the pixel is defined as a foreground pixel. (4) In addition, to achieve subpixel precision and smoothness, pixels within the contour and pixels around the contour are processed separately. The saliency pixels within the contour are set to 1 (or the maximum value of 255, 8 bits), and the saliency pixels around the edges are smoothly reduced to 0 to blend with the surrounding environment. (5) In each fragment bounding box, in addition to the main fragment within the fragment bounding box, there may be a small portion of one or more other fragments. In the final refinement step, any other instances or fragments that intersect within the fragment bounding box are erased to create a clean segmentation. In some embodiments, erasure is performed along the contour of the fragment. It should be noted that through thinning, the foreground and background pixels are further thinned and can be different from the foreground and background pixels determined by the saliency segmentation module 132. When a pixel changes from a foreground pixel to a background pixel, its color changes to white; when a pixel changes from a background pixel to a foreground pixel, its color value can be retrieved from the original standby image stored in memory 114. Figure 5 The illustration schematically shows the refinement of segments according to certain embodiments of the present disclosure, where the left side is the segment obtained by the saliency segmentation module 132, and the right side is the segment refined by the segmentation refinement module 136. From Figure 5 As can be seen, after the fragment is refined, the edges of the fragment are clearer, and some foreground parts of the fragment are restored.

[0099] In some embodiments, bounding box detection and segmentation refinement may be performed once or iteratively over several rounds. In some embodiments, segmentation refinement is performed before bounding box detection.

[0100] In some embodiments, image erosion and dilation are performed within the bounding boxes received from the bounding box detection module 134 (large bounding box) to coarsely remove image noise. In some embodiments, several segments (small bounding boxes) may be returned. In some embodiments, there is only one segment in the image, and the large bounding box and small bounding boxes are the same bounding box.

[0101] The quality enhancement module 138 is configured to, upon receiving the refined segments from the segmentation refinement module 136, enhance the image contrast, brightness, and sharpness at the pixel level to obtain enhanced segments in their segment bounding boxes, and send the segment bounding boxes with the enhanced segments to the object module 140. The segment bounding boxes with the enhanced segments are also referred to as enhanced segment bounding boxes. In certain embodiments, the enhancement is for aesthetic purposes. In certain embodiments, the image enhancement includes image sharpening to improve high frequency image information and adaptive image histogram equalization to improve contrast.

[0102] The object module 140 is configured to, upon receiving the enhanced segment bounding boxes from the quality enhancement module 138, extract the intra-frame attributes of the segments, rank the segments, filter the segments to obtain one segment, and send the enhanced segment bounding box with the enhanced segment to the feature module 150. Referring to Figure 6 , the object module 140 includes an intra-frame attribute module 142, a saliency ranking module 144, and a noise removal module 146.

[0103] The intra-frame attribute module 142 is configured to, upon receiving the enhanced segments, provide a pixel index for each enhanced segment, extract the intra-frame attributes of each enhanced segment in the segment bounding box, send the pixel index to the saliency ranking module 144, and send the intra-frame attributes to the noise removal module 146. In certain embodiments, the intra-frame attribute module 142 can process the enhanced segments in parallel or sequentially. Depending on the segmentation result, the number of enhanced segments can be, for example, 1 to 8. In certain embodiments, the number of enhanced segments is 1 to 3. In certain embodiments, the intra-frame attributes include image moments, central moments, aspect ratio, and optional orientation angle. The image moments of one of the enhanced segments are defined as:

[0104] (1),

[0105] where I(x, y) is the pixel intensity, the zeroth (0th) or 0° image moment of the enhanced segment (M0) represents the area of the enhanced segment in the bounding box. The intensity here is the saliency intensity.

[0106] The central moments of the enhanced segment are defined as:

[0107] (2),

[0108] where the centroid is: .

[0109] In certain embodiments, the intra-frame attribute module 142 is configured to detect the aspect ratio of each segment bounding box that encloses a corresponding one segment. The aspect ratio is defined as the ratio of the width to the height of the corresponding segment bounding box. ​

[0110] In some embodiments, the in-frame attribute module 142 can also be configured to detect the azimuth angle of the enhanced segment. In some embodiments, the azimuth angle is defined by forming an ellipse around the segment and determining the angle between the horizontal line and the major axis.

[0111] For each segment, the in-frame attribute module 142 is also configured to obtain the coordinates of the pixels in the segment contour, prepare a list of pixels with their coordinates, and provide the list of pixels of the segment or object to the saliency ranking module 144. In some embodiments, each pixel in the list of pixels also has an identification indicating which segment it belongs to, and the list of pixels is also called a pixel index. In some embodiments, the in-frame attribute module 142 can not extract the image moment, the central moment, the azimuth angle and the aspect ratio, but only obtain the pixel index and provide it to the saliency ranking module 144.

[0112] The saliency ranking module 144 is configured to, upon receiving the pixel index, calculate the average saliency value of the listed pixels of each enhanced segment according to the pixel index, rank the average saliency values of the segments, and provide the ranked segments to the noise removal module 146. In some embodiments, the saliency value used for calculation is the value after normalization by the segmentation refinement module 136 but before sub-pixel precision smoothing by the segmentation refinement module 136.

[0113] The noise removal module 146 is configured to, upon receiving the in-frame attributes from the in-frame attribute module 142 and the ranked segments from the saliency ranking module 144, discard segments that do not meet the in-frame attribute criteria, select the segment that meets the in-frame attribute criteria and has the highest ranking among the ranked segments, and send the selected segment to the feature module 150.

[0114] In some embodiments, the noise removal module 146 is configured to filter the segments using the image moment. The zero-order image moment of a segment represents the area of the segment, and the size of the zero-order image moment is compared with the size of the entire backup image, e.g. 500 pixels 500 pixels. When the ratio is equal to or greater than the 0th image moment threshold, the segment is retained, and when the ratio is less than the 0th image moment threshold, the segment is discarded. In some embodiments, the 0th image moment threshold is in the range of 0.1~0.2. In some embodiments, the 0th image moment threshold is in the range of 0.1~0.2. In some embodiments, the 0th image moment threshold is 0.15, i.e. the ratio of the area of the segment bounding box to the area of the backup image is equal to or greater than 0.15. By this criterion, too small segments will be removed as noise.

[0115] In certain embodiments, the noise removal module 146 is configured to filter segments using a centroid moment, where the centroid moment is a first order image moment. The centroid moment of a segment represents the center of the segment, the distance from the center of the segment to the center of the spare image is calculated, and then the ratio of the distance to the length of the diagonal of the spare image is calculated. When the ratio is less than a center deviation threshold, the segment is retained; when the ratio is equal to or greater than the center deviation threshold, the segment is discarded. In certain embodiments, the center deviation threshold is in the range of 0.1-0.6. In certain embodiments, the center deviation threshold is in the range of 0.2-0.3. In certain embodiments, the center deviation threshold is 0.25, i.e., the distance between the center of the segment bounding box and the center of the spare image should be less than 0.25 of the length of the diagonal of the spare image. By this criterion, segments far away from the center of the spare image will be removed as noise.

[0116] In certain embodiments, the noise removal module 146 is configured to filter segments using an aspect ratio. The aspect ratio of a segment bounding box represents the shape of the segment, which is preferably about 1:1. When the aspect ratio is within an aspect ratio range, the segment is retained; when the ratio is outside the aspect ratio range, the segment is discarded. In certain embodiments, the aspect ratio range is In certain embodiments, the aspect ratio range is In certain embodiments, the aspect ratio range is By this criterion, segments that are too thin or too thick will be removed as noise.

[0117] In certain embodiments, the noise removal module 146 is configured to filter segments using an azimuth angle. The azimuth angle indicates whether the segment is tilted in the spare image, and is preferably about 0 degrees or about 90 degrees. When the azimuth angle is within an azimuth threshold of 0 degrees or 90 degrees, the segment is retained; otherwise, the segment is discarded. In certain embodiments, the azimuth threshold is in the range of -20-20 of 0 or 90. In certain embodiments, the azimuth threshold is in the range of -10-10 of 0 or 90, i.e., the azimuth angle of the segment should be -10 to 10 degrees or 80-100 degrees. In certain embodiments, filtering using the azimuth angle is optional.

[0118] In certain embodiments, the noise removal module 146 is configured to filter segments using ranking segments. The segments are ranked by their average saliency values, the noise removal module 146 is configured to select the segment with the highest average saliency value, and send the selected segment to the feature module 150. When a segment is removed according to the image moment, the centroid moment, the aspect ratio, or the optional azimuth angle, the noise removal module 146 is configured to discard the segment, and select the segment with the highest saliency value from the remaining segments. In other words, the selected segment is the highest in the ranked segment list that meets the image moment, the centroid moment, the aspect ratio, and the optional azimuth angle criteria.

[0119] The denoising can be performed using the image moments, the centroid moments, the aspect ratio, the orientation angle, and the ranking saliency value in an order other than as described above, and some of the steps of denoising, such as the step based on the orientation angle, can not be necessary. In some embodiments, only the aspect ratio and the ranking saliency value can be used for noise removal. After noise removal, only one segment, and its segment bounding box (with the augmented segment bounding box of the augmented segment) remain. In some embodiments, when all segments are discarded during noise removal, the analysis of the standby image is completed, the standby image is discarded, and the product recommendation application 118 is configured to analyze the next standby image.

[0120] The feature module 150 is configured to, upon receiving the segment bounding box with the selected segment, extract image features and popular colors from the segment bounding box, and send the extracted image features and popular colors to the comparison module 170. As shown, the feature module 150 includes an image feature extraction module 151, an image feature embedding module 152, an artificial color histogram feature module 153, a color histogram augmentation module 154, a popular color module 155, and a feature combination module 156. Figure 7

[0121] The image feature extraction module 151 is configured to, upon receiving the segment bounding box containing the selected segment, extract image features of the segment bounding box containing the selected segment, and send the image features to the image feature embedding module 152. In some embodiments, the image feature extraction is performed using a convolutional neural network (CNN). In some embodiments, the CNN is a ResNet pre-trained in ImageNet. In some embodiments, the CNN model used for extracting the image features has a model structure as shown in Figure 8

[0122] The image feature embedding module 152 is configured to, upon receiving the image features from the image feature extraction module 151, embed the image features to obtain embedded image features, and send the embedded image features to the feature combination module 156. In some embodiments, the image feature embedding module 152 is a layer of a multi-layer perceptron (MLP) that compresses the 152-float-value vector into a smaller number, such as a 128-float-value vector. In some embodiments, the image feature embedding module 152 is optional.

[0123] ​​The artificial color histogram feature module 153 is configured to, upon receiving the segment bounding box containing the selected segment, collect color information within the segment, list the collected color information as a color histogram, and send the color histogram to the color histogram enhancement module 154. The normalized color histogram feature can be invariant to image scale, translation, and rotation. It can be considered as a color and texture feature of a given product.

[0124] The color histogram enhancement module 154 is configured to, upon receiving the color histogram of the segment, convert the color space of the color histogram from RGB space to hue-saturation-value (HSV) space to obtain an enhanced color histogram, and send the enhanced color histogram to the popular color module 155. Color features can be collected from different color spaces from traditional RGB space to HSV space. Compared to RGB space, HSV space is designed to be close to human perception and is more robust to lighting interference. In certain embodiments, the color histogram enhancement module 154 is further configured to truncate the color histogram to increase the robustness of this feature. In certain embodiments, the conversion from RGB space to HSV space reduces the difference under different lighting conditions.

[0125] The popular color module 155 is configured to, upon receiving the enhanced color histogram, apply a k-means clustering algorithm to the enhanced color histogram feature to obtain k primary colors selected via k-means, and send the k primary colors to the feature combination module 156. In certain embodiments, the number k is set in the range of 1 to 5. In certain embodiments, the number k is set to 2 or 3.

[0126] The feature combination module 156 is configured to, upon receiving the embedded color feature from the image feature embedding module 152 and the primary colors from the popular color module 155, combine the embedded color feature and the primary colors to form a combined image feature, and send the combined image feature to the comparison module 170. The combined image feature can be in the form of a vector with multiple dimensions, and the dimensions store values representing the embedded color feature and the primary colors.

[0127] The text and main image module 160 is configured to, upon receiving the text and main image of the product from the product retriever 120, embed the text to obtain a text feature of the product, embed the main image of the product to obtain a main image feature of the product, and send the text feature and the main image feature to the comparison module 170. In certain embodiments, the text feature is obtained using BERT, Word2Vec, GloVe, FastTex, or other types of pre-trained word embedding models, and the text feature feature is in the form of a vector. In certain embodiments, embedding the main image to obtain the main image feature is similar to the processing of the secondary images, and the result of the embedding includes general image features and primary colors.

[0128] The comparison module 170 is configured to, upon receiving the combined image features of the backup image (containing the enhanced segment boundary box of the selected enhanced segment), the text features of the product, and the image features of the main image, match the combined image features with the text features, match the combined image features with the image features of the main image, determine whether the product is a valid recommended product, and send the valid product to the recommendation module 180.

[0129] The stitching module 172 is configured to weight stitch the image features from the feature combination module 156 and the text features from the text and main image module 160 to obtain a stitching vector, pass the stitching vector through a classification / regression layer (or a transformer) to obtain the correlation between the image features of the backup image of the product and the text features of the product, and send the backup image-text correlation to the product verification module 176. In some embodiments, in order to compare the image features and the text features, the image features are embedded by imitating the embedding of the text. For example, the backup image can be divided into small grids, the small grids are sequentially aligned, each grid is provided to a CNN model to obtain the embedding of each grid, the position of the grid in the sequence of grids is embedded, and the embedding of the sequence of grids has a similar format to the sequence of words in a sentence. In some embodiments, as shown in FIG. 2, the comparison module 170 can include a stitching module 172, a feature comparison module 174, and a product verification module 176. Figure 9

[0130] The feature comparison module 174 is configured to, upon receiving the image features of the backup image from the feature combination module 156 and the image features of the main image from the text and main image module 160, compare the image features of the backup image with the image features of the main image to obtain the correlation between the backup image and the main image, and send the matching result to the product verification module 176. In some embodiments, the feature comparison module 174 is configured to use at least one of the following feature comparison methods to determine the correlation between the backup image (containing the segment boundary box of the selected segment) and the main image, such as cosine similarity, correlation, histogram intersection, and Kullback-Leibler divergence.

[0131] In some embodiments, the cosine similarity score is calculated for comparing two image features:

[0132] (3).

[0133] In some embodiments, the correlation score is calculated for comparing two image features:

[0134] (4),

[0135] wherein: . ​

[0136] In certain embodiments, a cross score is computed for comparing two image features:

[0137] (5).

[0138] In certain embodiments, a KL divergence is computed for comparing two image features:

[0139] (6).

[0140] In certain embodiments, the comparison is made using only the cosine similarity score.

[0141] The product verification module 176 is configured to, upon receiving the spare image-text correlation from the splicing module 172 and the spare image-main image correlation from the feature comparison module 174, determine whether the spare image is a qualified spare image, determine whether other spare images are qualified spare images, select a threshold number of qualified spare images if the number of qualified spare images is equal to or greater than a threshold number, and send the product with the text, the main image, and the threshold number of qualified spare images to the recommendation module 180.

[0142] In certain embodiments, the product verification module 176 is configured to determine that a spare image is a qualified spare image when there is a correlation between the spare image and the text and between the spare image and the main image. The correlation between the spare image and the text, and the correlation between the spare image and the main image can be in the form of a classification with a result of yes or no, or a correlation value. When the correlation is represented by a correlation value, the correlation can be determined by comparing the correlation value with a correlation threshold. In certain embodiments, the threshold value can be in the range of [0, 1]. In certain embodiments, the correlation threshold is 0.5. In certain embodiments, the correlation threshold is 0.7. In certain embodiments, the product verification module 176 is configured to place the augmented segment bounding box to the size of the original spare image to obtain a final spare image, where white background pixels can need to be added. In certain embodiments, since the main image usually has high quality, the main image from the product database 190 can be used directly in the recommended product without the need to enhance the main image. In certain embodiments, the product verification module 176 is configured to place the augmented segment bounding box to the size of the original spare image to obtain a final spare image, where white background pixels can need to be added. In certain embodiments, since the main image usually has high quality, the main image from the product database 190 can be used directly in the recommended product without the need to enhance the main image.

[0143] ​After determining whether a backup image is a qualified backup image, the product validation module 176 is further configured to instruct the image module 130 to process the next backup image of the product. For example, the product can have about seven or eight backup images. The product validation module 176 is further configured to count the number of qualified backup images, and if the number is equal to or greater than a threshold number, select the threshold number of qualified backup images with the highest average saliency values, and send the product with the text, the main image, and the threshold number of backup images to the recommendation module 180. The threshold number can be in the range of In certain embodiments, the threshold number is 2. In certain embodiments, the size of the backup images is the same as the size of the main image; in each backup image, the segment is the enhanced segment and the background is white.

[0144] The recommendation module 180 is configured to, after receiving the product with the text, the main image, and the threshold number of backup images, define the product as a recommended product, and recommend the recommended product to the customer or a selected customer. In certain embodiments, the recommendation is made after the product recommendation application 118 has processed all the products from the product database 190. In certain embodiments, the recommendation module 180 is configured to store the recommended product in a recommended product database 192, and retrieve the recommended product when needed for recommending to the customer. In certain embodiments, the product recommendation application 118 can process new products in the product database 190 or products in the product database 190 that have updated main images or updated backup images at predetermined time intervals to periodically update the recommended product database 192. The predetermined time intervals can be, for example, every 24 hours, every week, or every month.

[0145] The user interface 182 is configured to communicate with an administrator of the product recommendation application 118, for example, through a graphical user interface. The administrator can adjust parameters of the product recommendation application 118, or communicate with other applications to provide recommended products to customers.

[0146] In certain embodiments, the product recommendation application 118 can further include a scheduler to coordinate the data flow of the text and image information of the product. The coordination can be achieved by loading the product images into memory and invoking different functional modules of the product recommendation application 118 to process the images step by step.

[0147] In certain aspects, the present disclosure relates to a product recommendation method. Figure 10A and Figure 10B A product recommendation method according to certain embodiments of the present disclosure is schematically depicted. In certain embodiments, the method is implemented by the computing device 110 as shown in Figure 1 It is specifically noted that the steps of the method can be arranged in different orders unless otherwise specified by the present disclosure, and are not limited to the order shown in Figure 10A andFigure 10B the order shown.

[0148] At step 1002, the product retriever 120 retrieves product information from the product database 190. The product information includes text such as title and description, a main image, and multiple alternative images. The product retriever 120 sends the alternative images to the saliency segmentation module 132 and sends the text and the main image to the text and main image module 160.

[0149] At step 1004, for each alternative image, the saliency segmentation module 132 performs saliency detection and segmentation to obtain a segmented image and sends the segmented image to the bounding box detection module 134. In some embodiments, the saliency detection and segmentation is performed using a CNN fine-tuned on e-commerce images. Thus, in addition to the three RGB channels of the alternative image, the segmented image includes a fourth channel storing saliency values for the pixels. The saliency value of each pixel in the fourth channel can be in the range of [0, 1]. Pixels with saliency values equal to or greater than a threshold value are defined as foreground pixels, and pixels with saliency values less than the threshold value are defined as background pixels. In some embodiments, the threshold value is set to about 0.6. In some embodiments, the background pixels are assigned a white color. In some embodiments, the foreground can include one or more segments. Each segment can correspond to an object in the alternative image, and each segment can be named an instance.

[0150] At step 1006, upon receiving the segmented image, the bounding box detection module 134 creates a contour for each segment, defines a minimum bounding box that encloses the segment contour, and sends the image with the bounding box and the segment contour to the segmentation refinement module 136.

[0151] At step 1008, upon receiving the bounding box with the enclosed foreground segment, the segmentation refinement module 136 refines the segment and sends the refined segment bounding box with the refined segment to the quality enhancement module 138. Specifically, the segmentation refinement module 136 defines a small segment bounding box for each segment and then refines the segment in the small segment bounding box. In some embodiments, the segment refinement includes outlier removal of the segment contour, normalization of the saliency map of the segment, application of an adaptive threshold, sub-pixel smoothness, and erasing of segment intersections. The result is one or several small segment bounding boxes, each containing a corresponding refined segment. During refinement, some pixels can change from background pixels to foreground pixels, and their RGB values can be retrieved from the original alternative image located in the memory 114; some pixels can change from foreground pixels to background pixels and are assigned a white color.

[0152] ​At step 1010, upon receiving the refined segment boundary box with the refined segment, the quality enhancement module 138 improves the quality of the refined segment to obtain an enhanced segment boundary box with an enhanced segment, and sends the enhanced segment boundary box to the in- frame attribute module 142 of the object module 140. The enhancement can include image sharpening and adaptive image histogram equalization.

[0153] At step 1012, upon receiving the enhanced segment, the in-frame attribute module 142 extracts the pixel index of the enhanced segment, extracts the in-frame attributes of the enhanced segment, sends the pixel index to the saliency ranking module 144, and sends the extracted in-frame attributes to the noise removal module 146. The extracted in-frame attributes can include, for example, image moments, central moments, and aspect ratio, and optionally azimuth angle.

[0154] At step 1014, upon receiving the pixel index, the saliency ranking module 144 ranks the enhanced segment based on the average saliency value of the enhanced segment, and sends the ranked segment to the noise removal module 146.

[0155] At step 1016, upon receiving the in-frame attributes of the segment and the ranked segment, the noise removal module 146 removes noise, selects one enhanced segment, and sends the selected segment to the feature module 150. In certain embodiments, the noise removal module 146 first filters the segments using image moments to discard segments with too small image moments, then filters the remaining segments using central moments to discard segments far from the center of the image, then filters the remaining segments using aspect ratio to discard segments with large aspect ratio, then filters the remaining segments using azimuth angle to discard segments with too large tilt, and finally selects one of the remaining segments with the highest average saliency value. In certain embodiments, the filtering based on azimuth angle is optional and can be removed. In certain embodiments, only the aspect ratio and the ranked average saliency value are used for noise removal.

[0156] At step 1018, upon receiving the segment boundary box containing the selected segment from the noise removal module 146, the image feature extraction module 151 extracts image features from the enhanced segment boundary box with the enhanced segment, and sends the extracted image features to the image feature embedding module 152; the image feature embedding module 152 embeds the extracted image features to obtain embedded image features, and sends the embedded image features to the feature combination module 156.

[0157] At step 1020, upon receiving the segmented bounding box containing the selected segment from the noise removal module 146, the artificial color histogram feature module 153 generates a color histogram for the segment and sends the color histogram to the color histogram enhancement module 154; the color histogram enhancement module 154 converts the color histogram from RGB space to HSV space to obtain an enhanced color histogram and sends the enhanced color histogram to the popular color module 155. The popular color module 155 determines a plurality of popular colors from the enhanced histogram and sends the popular colors to the feature combination module 156. The number of popular colors can be in the range of 1-5. In some embodiments, the number of popular colors is 3.

[0158] At step 1022, the feature combination module 156 combines the embedded image features and the popular colors of the segment to obtain a combined image feature and sends the combined image feature to the comparison module 170. In some embodiments, the combined image feature is in the form of a vector.

[0159] At step 1024, upon receiving the text and the primary image of the product from the product retriever 120, the text and primary image module 160 embeds the text to obtain a text feature of the product, embeds the primary image of the product to obtain a primary image feature of the product, and sends the text feature and the primary image feature to the comparison module 170. In some embodiments, the primary image is directly used for image embedding using the feature module 150. In some embodiments, the primary image can also be pre-processed by the image module 130, the object module 140 before being embedded by the feature module 150.

[0160] At step 1026, upon receiving the combined image feature, the text feature, and the primary image feature of the alternate image, the comparison module 170 compares the combined image feature with the text feature and the primary image feature respectively, and determines the alternate image as a qualified alternate image if the combined image feature matches both the text feature and the primary image feature. The above steps 1004-1024 can be repeated for other alternate images. If there are a threshold number or more of alternate images that are qualified alternate images, the comparison module 170 selects the threshold number of alternate images with the highest average saliency values and sends the text, the primary image (or the enhanced primary image), and the threshold number of enhanced alternate images to the recommendation module 180. The threshold number can be, for example, 2.

[0161] At step 1028, upon receiving the product from the comparison module 170, the recommendation module 180 recommends the product to the customer or the selected customer. Each recommended product includes the text of the product, the primary image (or the enhanced primary image) of the product, and the threshold number of enhanced alternate images.

[0162] Figure 11A flowchart of a product recommendation system according to certain embodiments of the present disclosure is schematically depicted. As shown in FIG. 11, M products are to be processed through the recommendation process. For each product m with N alternative images, each alternative image n is processed and filtered, and matched with the text and the main image of product m. If alternative image n matches the text and matches the main image, the alternative image is stored, and the process continues with the next alternative image. After all alternative images are processed, if the product has a threshold number of alternative images, the product is determined to be a qualified product, and the product is determined to be a recommended product. After all M products are processed, the obtained recommended products are recommended to a customer. Figure 12 A processing and filtering of alternative images according to certain embodiments of the present disclosure is schematically depicted. As shown in FIG. 12, the steps of saliency segmentation, bounding box detection, segment refinement, segment enhancement, in-box attribute extraction, average saliency value sorting, and noise removal can be performed on each alternative image in sequence. Figure 12

[0163] In certain aspects, the present disclosure relates to a non-transitory computer- readable medium storing computer executable code. The code, when executed by a processor of a computing device, can perform the above-described method. In certain embodiments, the non-transitory computer-readable medium can include, but is not limited to, any physical or virtual memory including, for example, a computer hard drive, solid state drive, random access memory, database storage, a compact disk, a digital video disk, a tape drive, an optical storage device, a memory card, or any other suitable non-transitory memory. Figure 1 In certain embodiments, the non-transitory computer-readable medium can be implemented as the storage device 116 of the computing device 110 as shown in FIG. 13.

[0164] In summary, the present disclosure creates a complete machine vision processing pipeline to segment, extract features, enhance, and cross-domain verify product photos. There is currently no solution that can fully handle this task. In general, the present disclosure has four modules: (1) image block, (2) object block, (3) feature block, and (4) matching block.

[0165] The image block processes accurate image segmentation and quality improvement. This module cuts the image into several object instances and processes each object according to saliency priority.

[0166] The object block analyzes each segmented image region and extracts the following: (a) geometric features: centroid, bounding box, perspective ratio, area, and corresponding position within the image, (b) general image features: residual network backbone feature extraction, (c) color distribution features: saliency analysis with noise removal and color histogram features of texture information.

[0167] The feature block post-processes all image features: (a) embeds general image features; (b) color space transformation, normalization, k-means, truncation of color distribution features.

[0168] ​The matching block embeds all the differentiating features together and evaluates the correlation between these features. Images with low feature correlation scores or containing conflicting information are automatically deleted.

[0169] Certain embodiments of the present disclosure have the following advantageous benefits, among others: (1) the saliency detection of images is accurate, for example, by iterative refinement segmentation. (2) image feature extraction can be performed in segment level. (3) the backup images are enhanced using a variable method to obtain high quality backup images. (4) the contrast between images and text is provided.

[0170] The foregoing description of exemplary embodiments of the present disclosure has been presented for the purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings.

[0171] The embodiments were chosen and described in order to explain the principles of the disclosure and its practical application and to enable others skilled in the art to utilize the disclosure and various embodiments and with various modifications as are suited to the particular use contemplated. Alternative embodiments will become apparent to those skilled in the art to which the present disclosure pertains, without departing from the spirit and scope of the present disclosure. Accordingly, the scope of the present disclosure is defined by the appended claims rather than by the foregoing description and exemplary embodiments described therein.

Claims

1. A computer-implemented product recommendation method, comprising: Provide product information for the product, wherein the product information includes a product description, a main image, and multiple alternative images; Extract text features from the product description and extract main image features from the main image; For each of the plurality of backup images, image features are extracted from the backup image, and the image features are matched with the text features and the main image features. If the image features match the text features and the main image features, then the backup image is determined to be a qualified backup image; and When the number of qualified backup images is equal to or greater than a threshold number, the product with updated product information is recommended to the customer, wherein the updated product information includes the product description, the main image, and the threshold number of qualified backup images.

2. The method according to claim 1, wherein, The step of extracting image features from the backup image includes: The backup image is saliency segmented to obtain a mask for the backup image, the mask comprising a fragment and a background, wherein the saliency value of the fragment is greater than the saliency value of the background; Define the fragment bounding box that surrounds the fragment; and Extract the image features from the bounding box of the fragment.

3. The method according to claim 2, further comprising, before extracting the image features from the fragment bounding box: The fragment within the fragment bounding box is refined to obtain a refined fragment bounding box; as well as Enhance the image quality of the refined fragment bounding box to obtain an enhanced fragment bounding box. Specifically, the image features are extracted from the enhanced fragment bounding boxes, and the qualified alternative images are obtained based on the enhanced fragment bounding boxes.

4. The method according to claim 3, wherein, The refinement of the fragment within the fragment bounding box includes: Abnormal pixels are removed from the fragment by erosion and dilation; The saliency values ​​of the pixels in the segment are normalized to obtain normalized saliency values ​​in the range of 0 to 1; An adaptive threshold is applied to the normalized saliency value to define background and foreground pixels; Set the saliency value of the pixels in the outline of the segment to 1; and The saliency value around the edge of the contour is smoothly reduced to 0.

5. The method according to claim 4, wherein, Enhancing the image quality of the thinned fragment bounding box includes performing image sharpening and adaptive image histogram equalization on the thinned fragment bounding box.

6. The method according to claim 2, wherein, The mask comprises multiple segments, and the method further includes, before extracting the image features from the segment bounding boxes: Calculate the 0-degree image moment for each of the multiple fragment bounding boxes; When the 0-degree image moment of one of the plurality of fragment bounding boxes is less than the 0-degree image moment threshold, one of the plurality of fragment bounding boxes is deleted to obtain the retained fragment bounding boxes; as well as Calculate the average saliency value of each of the preserved fragment bounding boxes, and select the preserved fragment bounding box with the highest average saliency value as the fragment bounding box used to extract the image features.

7. The method according to claim 6, wherein, The 0-degree image moment threshold is 10% of the area of ​​the backup image.

8. The method according to claim 2, wherein, The step of extracting the image features from the fragment bounding box includes: Use a convolutional network to extract general image features from the fragment bounding box; Extract the color histogram from the fragment bounding box; Extract the first k colors from the color histogram; and The general image features and the first k colors are combined to form the image features.

9. The method according to claim 8, wherein, The k-means clustering was used to extract the top k colors, where k is 3.

10. The method according to claim 1, wherein, The step of determining the backup image as a qualified backup image includes: The image features are matched with the text features using a converter; and The image features are matched with the main image features using cosine similarity.

11. The method according to claim 1, wherein, The number of thresholds is 2.

12. A product recommendation system, wherein, The system includes a computing device, which includes a processor and a storage device storing computer-executable code, the computer-executable code being configured to: Provide product information for the product, wherein the product information includes a product description, a main image, and multiple alternative images; Extract text features from the product description and extract main image features from the main image; For each of the plurality of backup images, image features are extracted from the backup image, and the image features are matched with the text features and the main image features. If the image features match the text features and the main image features, then the backup image is determined to be a qualified backup image; and When the number of qualified backup images is equal to or greater than a threshold number, the product with updated product information is recommended to the customer, wherein the updated product information includes the product description, the main image, and the threshold number of qualified backup images.

13. The system according to claim 12, wherein, The computer-executable code is configured to extract the image features through the following steps: The backup image is saliency segmented to obtain a mask for the backup image, the mask comprising a fragment and a background, wherein the saliency value of the fragment is greater than the saliency value of the background; Define the fragment bounding box that surrounds the fragment; and Extract the image features from the bounding box of the fragment.

14. The system according to claim 13, wherein, The computer-executable code is also configured to, before extracting the image features from the fragment bounding box: The fragment within the fragment bounding box is thinned to obtain a thinned fragment bounding box; and Enhance the image quality of the refined fragment bounding box to obtain an enhanced fragment bounding box. The computer-executable code is configured to extract the image features from the enhanced fragment bounding boxes, and the qualified alternative image is obtained based on the enhanced fragment bounding boxes.

15. The system of claim 14, wherein the computer-executable code is configured to refine the fragment within the fragment bounding box by the following steps: Abnormal pixels are removed from the fragment by erosion and dilation; The saliency values ​​of the pixels in the segment are normalized to obtain normalized saliency values ​​in the range of 0 to 1; An adaptive threshold is applied to the normalized saliency value to define background and foreground pixels; Set the saliency value of the pixels in the outline of the fragment to 1; as well as The saliency value around the edge of the contour is smoothly reduced to 0.

16. The system according to claim 15, wherein, The computer-executable code is configured to enhance the image quality of the thinned fragment bounding box by performing image sharpening and adaptive image histogram equalization on the thinned fragment bounding box through the following steps: performing image sharpening and adaptive image histogram equalization on the thinned fragment bounding box.

17. The system according to claim 13, wherein, The segment of the backup image comprises multiple segments, and the computer-executable code is further configured to: Calculate the 0-degree image moment for each of the multiple fragment bounding boxes; When the 0-degree image moment of one of the plurality of fragment bounding boxes is less than the 0-degree image moment threshold, one of the plurality of fragment bounding boxes is deleted to obtain the retained fragment bounding boxes; as well as Calculate the average saliency value of each of the retained fragment bounding boxes, and select the retained fragment bounding box with the highest average saliency value as the fragment bounding box for extracting the image features. The 0-degree image moment threshold is 10% of the area of ​​the backup image.

18. The system according to claim 13, wherein, The computer-executable code is configured to extract image features from the fragment bounding box through the following steps: Use a convolutional network to extract general image features from the fragment bounding box; Extract the color histogram from the fragment bounding box; Extract the first k colors from the color histogram; and The general image features and the first k colors are combined to form the image features.

19. The system according to claim 12, wherein, The computer-executable code is configured to determine the backup image as a qualified backup image by means of the following steps: The image features are matched with the text features using a converter; and The image features are matched with the main image features using cosine similarity.

20. A non-transitory computer-readable medium for storing computer-executable code, wherein, The computer-executable code is configured to, when executed at the processor of the computing device: Provide product information for the product, wherein the product information includes a product description, a main image, and multiple alternative images; Extract text features from the product description and extract main image features from the main image; For each of the plurality of backup images, image features are extracted from the backup image, and the image features are matched with the text features and the main image features. If the image features match the text features and the main image features, then the backup image is determined to be a qualified backup image; and When the number of qualified backup images is equal to or greater than a threshold number, the product with updated product information is recommended to the customer, wherein the updated product information includes the product description, the main image, and the threshold number of qualified backup images.

Images