Ai-powered media analysis for item recognition

Abstract

To provide a method for media processing and item recognition.SOLUTION: A method includes: receiving digital media from a user; identifying objects depicted within the digital media; determining items currently available at a physical location by analyzing information collected by a set of cameras at the physical location; identifying a set of target items, from the items currently available at the physical location, that are similar to at least one of the objects based on the digital media and the collected information; and generating a guidance that navigates the user to at least one of the set of target items within the physical location.SELECTED DRAWING: Figure 6

Description

Background Art

[0001]

[0001] Social media platforms are becoming increasingly popular for promoting and advertising products. Social media posts designed for these purposes often include various visual media such as images, videos, and motion graphics to show different perspectives of the product being promoted. These posts mainly serve as digital advertisements, but some posts are integrated with e-commerce functions, and the integrated posts enable the viewer to directly purchase the product displayed online. When such an e-commerce option is activated, for example, an auto-purchase window may pop up at the end of the video, and the window guides the viewer to purchase the item featured. This system streamlines online transactions so that customers can complete their purchases with one click. Nevertheless, despite the convenience of online shopping, many customers still highly value the specific benefits provided by offline shopping, such as the ability to physically touch and examine products, receive immediate assistance from store staff, and purchase products without additional shipping costs. Such preferences often lead customers to visit physical stores after viewing products online. These customers may visit a physical store after viewing an online post and attempt to find the product being promoted or something similar in the store.

Brief Description of the Drawings

[0002]

Figure 1

[0002] Illustrates an exemplary environment in which embodiments of the present disclosure may be implemented.

Figure 2

[0003] Illustrates an example of a workflow for item recognition and guidance route generation according to some embodiments of the present disclosure.

Figure 3

[0004] Illustrates an exemplary method for media data processing and inventory search according to some embodiments of the present disclosure.

Figure 4

[0005] Illustrative methods for similarity matching according to some embodiments of the present disclosure are shown.

Figure 5

[0006] Illustrative methods for alert generation according to some embodiments of the present disclosure are shown.

Figure 6

[0007] FIG. is a flow diagram illustrating an illustrative method for identifying in-store items similar to objects featured in digital media and generating corresponding guidance routes according to some embodiments of the present disclosure.

Figure 7

[0008] Illustrative computing devices for item recognition and guidance route generation according to some embodiments of the present disclosure are shown. DETAILED DESCRIPTION

[0003]

[0009] For ease of understanding, the same reference numbers are used, where possible, to designate the same elements common to the drawings. It is contemplated that elements disclosed in one embodiment may be beneficially used in other embodiments without further recitation.

[0004]

[0010] Social media and other online platforms have become powerful tools for product promotion and often integrate e-commerce capabilities to facilitate direct online shopping from digital posts. However, for customers who prefer offline shopping, particularly when they view products advertised on social media and wish to purchase them in physical stores, challenges arise. This is because the current retail store setup typically lacks a streamlined mechanism for customers to quickly identify items seen in online posts and then locate these advertised products within the physical store. The gap between online discovery and offline purchase leads to a sub-optimal consumer experience and potentially reduces sales opportunities for retailers.

[0005]

[0011] Embodiments of the present disclosure provide methods and techniques for improving the shopping experience in a retail store by integrating online product discovery and in-store experiences. In some embodiments, the disclosed techniques include extracting and identifying products shown in various forms of digital media (e.g., images or videos within social media posts), searching the store inventory for similar or matching products, and generating guidance routes to lead customers to the locations of these identified similar or matching products within the store. As used herein, "digital media" can refer to images, videos, 3D models, motion graphics, or any other form of visual representation that presents an item or product to potential consumers. As used herein, "in-store inventory" can refer to a group of items available for purchase within the physical premises or other physical location of a store where items can be bought. In some embodiments, the in-store inventory can include a catalog of available items along with their categories, sizes, quantities, and other relevant information. In some embodiments, the in-store inventory can be determined at least in part using cameras installed within the store. These in-store cameras can capture visual data of the products available in the store. The visual data can then be processed by advanced AI-based algorithms to recognize the products and their associated attributes such as size, color, and quantity, which can be used for further item matching and searching.

[0006]

[0012] FIG. 1 illustrates an exemplary environment 100 in which embodiments of the present disclosure may be implemented. In the illustrated example, environment 100 includes one or more servers 110, a digital library 115, a database 125, one or more in-store cameras 130, one or more in-store devices 135, and one or more end-user devices 140. In some embodiments, one or more of the illustrated devices may be physical devices or systems. In other embodiments, one or more of the illustrated devices may be implemented using virtual devices and / or across several devices.

[0007]

[0013] In the illustrated example, the one or more servers 110, digital library 115, database 125, one or more in-store cameras 130, one or more in-store devices 135, and one or more end-user devices 140 are remote from each other and communicatively coupled to each other via network 105. That is, the one or more servers 110, digital library 115, database 125, one or more in-store cameras, one or more in-store devices 135, and one or more end-user devices 140 may each be implemented using separate hardware systems. Network 105 may include, or correspond to, any suitable combination of wide area networks (WANs), local area networks (LANs), the Internet, intranets, or other available communication media, including wired, wireless, or combinations of wired and wireless links. In some embodiments, servers 110, database 125, in-store devices 135, and in-store cameras 130 may be local to each other (e.g., within the same local network and / or the same hardware system), and may communicate with each other using any suitable local communication media, such as local area networks (LANs) (including wireless local area networks (WLANs)), hardware, wireless links, or intranets, etc.

[0008]

[0014] In the illustrated example, the digital library 115 comprises a plurality of online-posted digital media 120 that display, promote, or depict various items. In some embodiments, the digital media 120 may include a variety of data formats including, but not limited to, images, videos, motion graphics, three-dimensional (3D) models, and animations. In some embodiments, the digital media 120 may serve as a visual representation that provides details regarding the design, functionality, and usefulness of a product. In some embodiments, the digital media 120 may further comprise text data (e.g., headings, product descriptions, and customer reviews) to provide additional context or clarity regarding the product. In some embodiments, an end user 150 (who may also be referred to as a customer in some embodiments) may access, edit, download, and / or upload digital media 120 to the digital library via an end user device 140. In some embodiments, the digital library 115 may be a static repository. In some embodiments, the digital library 115 may function as a dynamic platform that updates based on various factors. For example, the library 115 may be continuously updated with content (e.g., user-generated social media posts including digital media) to reflect user preferences or the latest trends. In some embodiments, some advanced search functionality may be integrated into the digital library 115. These search functions that utilize AI-based models may efficiently process queries to understand the user's intent and preferences. Based on this understanding, future searches may be predicted. In some embodiments, the digital library 115 may interface with a recommendation system to push customized content to the user device 140 based on the user's browsing trends, preferences, or purchase history. In some aspects, the digital library 115 may correspond to one or more social media platforms.

[0009]

[0015] In the example shown, end user 150 (which may also be referred to as a customer in some embodiments) can stream, view, edit, upload, and / or download digital media 120 directly on their device 140. In some embodiments, when an end user or customer 150 enters a merchant location such as a store, they may scan digital media 120 displayed on their personal device 140 to an in-store device 135 (e.g., a kiosk terminal). As used herein, "scan" may refer to the process of directly capturing or recording media (e.g., an image or video) displayed on a user's device. To achieve this, the in-store device 135 may be equipped with a high-resolution camera 155 capable of taking an image or video of the content displayed on the user's device 140.

[0010]

[0016] For example, end user 150 can ensure that the item(s) of interest can be viewed or displayed on their device 140 (e.g., play a relevant section of a video, pause the video at the correct time, or output an image of the item(s)). The end user 150 can then place their user device 140 on or against a transparent (e.g., glass) surface where the camera 155 is positioned behind or directly below the surface to capture an image(s) or video of the content displayed on the user device 140. Once scanned, the in-store device 135 can forward the captured digital content (e.g., an image or video) to the server(s) 110 for further processing and analysis (e.g., item recognition, item matching, and guidance route generation). In addition to, or instead of, scanning via the in-store device 135, several other methods can be used to send the digital media 120 to the in-store device 135. For example, the in-store device 135 can be configured to support Near Field Communication (NFC), Bluetooth®, and / or Wi-Fi Direct transmission, through which a user can wirelessly send digital media (or a link to media in a social media platform such as a digital library 115) to the in-store device. In some embodiments, the in-store device 135 can include several direct wired ports, such as a UCB-C, to facilitate the direct transfer of media and / or links between the user device 140 and the in-store device 135.

[0011]

[0017] In some embodiments, to provide a more customized experience, end user 150 may log into the store's application on their device 140 when scanning or transmitting digital media 120 to in-store device 135. After the application processes the visual data and identifies potential matches (e.g., items currently available in the store that potentially match or are similar to the product featured in the digital media), the application may activate an in-store navigation function to guide end user or customer 150 in real time to the exact location of the matching or similar item in the store.

[0012]

[0018] In the illustrated example, in-store cameras 130 are strategically placed throughout the store. In some embodiments, the in-store cameras 130 may capture detailed visual data of products in the store (e.g., products arranged on shelves or racks 160 in the retail and / or storage areas of the store). In some embodiments, the in-store cameras 130 may generate or capture images of these products from various angles to provide a comprehensive, detailed representation of each product. In some embodiments, the visual data captured by the in-store cameras 130 may be transmitted to the centralized server 110 for further analysis. Upon receiving the visual data, the server 110 may perform item recognition using advanced AI-based algorithms. For example, the server 110 may identify products in the store and determine relevant attributes about the products, such as color, size, brand, quantity, and the like. In some embodiments, in-store cameras 130 with built-in analytical capabilities may be used. In such a configuration, the in-store cameras 130 may process the visual data to extract features and forward the refined information to the server 110. Based on the improved information, server 110 may perform advanced analytics to identify items currently available in the store. Following item recognition, server 110 may generate an updated in-store inventory, which may provide customers 150 and / or staff with real-time visibility into the store's stock levels. In some embodiments, the in-store inventory may be used to identify items in the store that match and / or are similar to items advertised or depicted in digital media 120 (e.g., content contained in social media posts). In some embodiments, the in-store inventory, digital media 120 scanned or transmitted by users or customers 150, and matching results may be stored in database 125 to ensure a consistent, retrievable record of product availability, customer search history, and digital media interactions.

[0013]

[0019] 2 illustrates an example workflow 200 for item recognition and guidance route generation according to some embodiments of the present disclosure. In some embodiments, workflow 200 may be performed by one or more computing systems, such as server 101, user device 140, in-store camera 130, and in-store device 135 as illustrated in FIG. 1, and / or computing device 700 as illustrated in FIG.

[0014]

[0020] In the illustrated example, digital media input 210 (e.g., an image or video in a social media post) viewed by a customer or end user 150 on their personal device 205 (e.g., a smartphone) (e.g., 140 in FIG. 1) is scanned on or received by an in-store device 215 (e.g., a kiosk terminal) (e.g., 135 in FIG. 1), which then forwards the digital media to a data processing component 220.

[0015]

[0021] In the illustrated example, upon receiving the digital media input 210, the data processing component 220 may utilize sophisticated algorithms to identify items covered within the content. For example, in some embodiments, the data processing component 220 may process the digital media to extract relevant features. This may involve identifying unique patterns, shapes, textures, and / or colors that can distinguish one item from another. After these features are extracted, the data processing component 220 may provide the extracted features to one or more trained machine learning models for the purpose of identifying the items displayed within the digital media input 210. A variety of ML models, including but not limited to random forest, support vector machine, and neural networks (e.g., conventional neural networks), may be used in this process. In some embodiments, the ML model may be trained using images or videos of products as input and corresponding product labels (e.g., names or categories) as output. Through training, the ML model may learn to correlate features from the images or videos with appropriate labels and adjust its internal parameters to more accurately predict labels based on the features. After training is complete, the model may be validated and / or tested using a separate set of labeled videos or images of products. Through the validation and / or testing process, the model's parameters may be adjusted and their accuracy improved. In some embodiments, the customer or end user 150 may provide feedback regarding the accuracy of these items identified or recognized from the digital media input 210. For example, the recognized items may be displayed on either an in-store device (e.g., a kiosk terminal) (e.g., 135 in FIG. 1) or the user's personal device (e.g., a smartphone) (e.g., 140 in FIG. 1). For each recognized item, the device may display an image of the item, its name, and possible additional details such as price or product description.Below or beside the recognized item details, the device may provide options for the user to rate the accuracy of recognition, such as using a star rating (ranging from 1 to 5 stars, where 1 represents not similar and 5 represents a perfect match), or using a thumbs-up button (representing similarity) and a thumbs-down button (representing dissimilarity). The feedback can then be integrated to further adjust the parameters of the model to improve item recognition accuracy.

[0016]

[0022] In some embodiments, the data processing component 220 may generate a list of recognized items 225 from a digital media input (also referred to as a digital media item in some embodiments). In some embodiments, when one or more digital media items 225 are identified, the data processing component 220 may generate a detailed profile for each item. The profile may include various attributes of the item such as its name, color, size, and the like. In some embodiments, the profile may further include context information such as the function of the item or its intended purpose. Such a detailed profile for the recognized item 225 from the digital media input may match or be similar to items seen by customers online (e.g., on a social media platform) and can be used in subsequent item matching to facilitate the efficient identification of in-store items that customers are interested in purchasing in the store.

[0017]

[0023] In the illustrated example, in-store cameras 230 (e.g., 130 in FIG. 1 ) actively collect inventory visual data 235 and then transmit the data 235 to data processing component 220. In some embodiments, inventory visual data 235 may represent inventory stock levels for a store. In some embodiments, inventory visual data 235 may be in the form of images or video to capture different attributes or characteristics (e.g., shape, color, size, and quantity) of items in the store. Upon receiving inventory visual data 235, data processing component 220 may perform item recognition to identify available items 240 (also referred to in-store items, in some embodiments) in the store and the locations of such items. As discussed above, in some embodiments, data processing component 220 may extract relevant features that define each item captured in inventory visual data 235. After extraction, data processing component 220 may provide the extracted features (along with their corresponding values) to an ML model, which is trained to predict product labels (e.g., names or categories) based on the extracted features. In some embodiments, after the in-store items 240 are identified, the data processing component 220 may further generate a profile for each in-store item 240, detailing its size, color, brand, quantity, location, and other relevant characteristics. In some embodiments, based on the identified in-store items 240, the data processing component 220 may generate and / or update an in-store inventory database. In some embodiments, the inventory database may include a comprehensive record of stock levels within a physical store, where each item's availability, its exact location within the store, and other relevant details may be systematically logged.

[0018]

[0024] In the example shown, both the digital media item 225 and the in-store item 240 are provided to the item matching component 245. In some embodiments, the item matching component 245 may utilize advanced algorithms to assess the similarity between both sets of items. For example, the item matching component 245 may compare various features of the digital media item 225, such as color, shape, design, texture, material, brand, and other unique identifiers, to the features of the in-store item 240. Through the comparison, the item matching component 245 may identify in-store items that match or are similar to (visually or functionally) the digital media item 225 (e.g., the item the customer viewed online). As shown in the example, the output of the item matching component 245 is the matching result 250. In some embodiments, the matching result 250 may include a list of in-store items that match or are similar to the item displayed in the digital media input 210. In some embodiments, the matching or similar items may also be referred to as target items. In some embodiments, for each matching or similar in-store item, the matching result 250 may provide additional contextual information such as the available size or color of the item, its current quantity in stock, and its exact location within the store. In some embodiments, when it is determined that the in-store item does not match or is not similar to the digital media item, the matching result 250 may include a response indicating that the item was not found or that the search did not yield corresponding results. In some embodiments, the matching result 250 may be sent to the user device 205 and / or displayed on the user device 205. In some embodiments, the matching result 250 may be sent to the in-store device 215 and / or displayed on the in-store device 205.

[0019]

[0025] In the illustrated example, after item matching is completed, the matching result 250 is then provided to the guidance generation component 255. The guidance generation component 255 is configured to generate a guidance route 260 that guides the customer to the location of the target item within the store. In some embodiments, the guidance route 260 may include an internal map of the store and may highlight the shortest or most convenient route for the customer to find each target item (e.g., starting from a known location where the in-store device 215 is located and leading to each desired item). The generated guidance route 260 may then be sent to the alert generation component 275. The alert generation component 275 may collect the customer's real-time location data 270 from the customer location detection component 265. By integrating the guidance route 260 with the customer's real-time location data 270, the alert generation component 275 may track the movement of the customer within the store and generate timely alerts 280. For example, in some embodiments, the customer may follow the guidance route 260 to find one of the target items. When the customer is within a predefined proximity (e.g., 10 meters) of the item, the alert generation component 275 may trigger an alert 280 and send the alert 280 to the customer's device 205. This mechanism ensures that all customers scanning digital media are promptly notified of nearby items of interest and can thus improve the customer's in-store shopping experience.

[0020]

[0026] In some embodiments, the matching result 250 may be provided to the end user / customer via either the in-store device(s) 215 (e.g., kiosk terminal) (e.g., 135 in FIG. 1) or the customer's personal device(s) 205 (e.g., smartphone) (e.g., 140 in FIG. 1). By displaying the matching result, it enables the user / customer to review, select, and / or confirm the items they wish to route to or further explore, facilitating a more personalized shopping experience.

[0021]

[0027] Figure 3 illustrates an exemplary method for media data processing and inventory search according to some embodiments of the present disclosure. In some embodiments, method 300 may be performed by one or more computing devices (e.g., a system that processes visual data and performs an inventory search), such as server 110 as illustrated in FIG. 1, data processing components 220 as illustrated in FIG. 2, item matching components 245, and guidance generation components 255, and / or computing device 700 as illustrated in FIG. 7.

[0022]

[0028] Method 300 begins at block 305, where a computing system (e.g., 110 of FIG. 1) receives a digital media input (e.g., 210 of FIG. 2). In some embodiments, the digital media input may include images, videos, graphics, or other media formats from various sources (e.g., social media platforms, online publications, e-commerce functional websites). In some embodiments, the digital media can serve as promotional materials and / or advertisements created to display one or more products from various perspectives. The digital media is posted online (e.g., on a social media platform) and can provide a visual representation of products that viewers may be interested in purchasing. After a customer or end user views this media on their personal device and identifies a particular product of interest, they may be more likely to further examine these products while visiting a store. The system can receive digital media that a customer has previously viewed or shown interest in through various means. In some embodiments, the system can receive digital media through an in-store device (e.g., 135 of FIG. 1). For example, in some embodiments, the in-store device can be configured with scanning capabilities (e.g., a high-resolution camera) through which the in-store device can quickly record or capture the content within the digital media. In some embodiments, the in-store device can be equipped with an interface (e.g., Bluetooth, NFC, or Wi-Fi Direct) that enables the in-store device to wirelessly receive and / or transmit digital media content. In some embodiments, the system can obtain digital media through a dedicated application installed on the customer's personal device. After the customer selects, views, or updates the media of interest within the application, the application can directly share the digital media with the system with the customer's explicit approval.

[0023]

[0029] In block 310, the computing system identifies items or objects (e.g., 225 in FIG. 2) displayed within the received digital media. For example, in some embodiments, the computing system may utilize sophisticated algorithms to process the media data and extract relevant features. These features may include attributes such as color, pattern, texture, shape, and other unique identifiers useful for determining the nature of the item. In some aspects, such features are extracted using trained machine learning model(s). After these features are extracted and their corresponding values are identified, the data may be provided to an ML model trained for item recognition. In some embodiments, the recognized items from the digital media may be displayed on either an in-store device (e.g., a kiosk terminal) (e.g., 135 in FIG. 1) or the user's personal device (e.g., a smartphone) (e.g., 140 in FIG. 1), and the user may be requested to provide feedback regarding the accuracy of these items. The feedback may then be used to fine-tune the ML model for the purpose of further improving their accuracy and performance.

[0024]

[0030] In some embodiments, in addition to identifying products displayed within digital media, the computing system may further extract context information from the media, such as the source URL of the media and / or the text description of the product (e.g., headings, product descriptions, user feedback). The extracted context information may provide additional details about the product, such as their brand, the collection or season to which they belong, any promotional events or discounts associated with them, and the like.

[0025]

[0031] At block 315, the computing system performs item matching to identify in-store products that match or are similar to the products identified from the digital media. In some embodiments, the real-time availability and stock levels of in-store items may be determined through a network of in-store cameras (e.g., 130 in FIG. 1 ). These in-store cameras may be configured to capture visual data of products in the store in real time. The collected inventory visual data may then be provided to a system to perform item recognition. As discussed above, the system may use advanced algorithms to extract features from the inventory visual data and then apply trained ML models to identify various items in the store (e.g., 240 in FIG. 2 ). Using the information (e.g., in-store items), the system may create or update an in-store inventory database. To identify in-store items that match or are similar to the items from the digital media, the system may search the in-store inventory database and compare the similarity between the extracted features from the digital media and the features of the items in the in-store inventory database. If no matching or similar in-store items are found, method 300 proceeds to block 330.

[0026]

[0032] In block 330, the computing system searches an off - store inventory database. In some embodiments, the off - store inventory database may comprise a group of items that are not currently present in the physical store where the customer is located but are available at either another store or a warehouse and / or can be purchased from a supplier. In block 335, upon determining that an item that matches or is similar to the item identified from the digital media is available in another store, the system provides an alternative purchase route to the customer. In some embodiments, the alternative purchase route may include options such as an online order followed by home delivery. In some embodiments, the alternative purchase route may be displayed within a pop - up window that appears on the customer's personal device (e.g., 140 in FIG. 1). In some embodiments, when the customer scans or transmits digital media to an in - store device (e.g., 135 in FIG. 1), a pop - up window with alternative purchase options may be displayed on that in - store device. The pop - up window (regardless of whether it appears on a personal device or an in - store device) can ensure that the customer is promptly notified of the alternative purchase options available to them.

[0027]

[0033] In block 315, when a matching or similar in - store item (also called the target item in some embodiments) is identified, method 300 proceeds to block 320, where the computing system generates a guidance route (e.g., 260 in FIG. 2) for each of the identified in - store items. In some embodiments, after a matching or similar in - store item is identified, their locations within the store may also be determined. Based on the locations and the internal map of the store, the system may create a guidance route. In some embodiments, the guidance route may provide the customer with the shortest or most convenient route to each of the identified in - store items. In some embodiments, the guidance route may be displayed on the customer's personal device (e.g., 140 in FIG. 1) and can guide the customer to find these identified in - store items.

[0028]

[0034] In block 325, the computing system monitors for real-time updates to determine if any of the identified in-store items (which match or are similar to the products displayed within the digital media) have been selected or purchased by another shopper while the customer is approaching the location of the item. In some embodiments, such as when the identified in-store item has limited availability (e.g., there is only one left in the store, a mug with a unique pattern), the system may detect (e.g., using in-store cameras) that the item has been selected or purchased by another shopper before the customer arrives. In such a configuration, the system may update the status of the item to "unavailable" and send a notification to the customer's personal device (e.g., 140 of FIG. 1). The notification may alert the customer about the change in the availability of the item. In some embodiments, the system may provide a suggestion about potential alternative items or other similar products available in the store. In some embodiments, the system may update the guidance route based on the detected change. The method then proceeds to block 330, where the item recognition and matching process ends.

[0029]

[0035] FIG. 4 illustrates an exemplary method 400 for similarity matching according to some embodiments of the present disclosure. In some embodiments, method 400 may be executed by one or more computing devices such as server 110 as illustrated in FIG. 1, item matching component 245 as illustrated in FIG. 2, and / or computing device 700 as illustrated in FIG. 7.

[0030]

[0036] In block 405, a computing system (e.g., 245 of FIG. 2) receives data related to identified items (e.g., 225 of FIG. 2) from a digital media input (e.g., 210 of FIG. 2). For each identified digital media item, the data may include characteristics, attributes, and / or descriptions that reflect the nature and characteristics of the item. The data may then be used to compare and match the identified digital media items with items available in a physical store.

[0031]

[0037] At block 410, the computing system receives data related to items currently available in a physical store (240 in FIG. 2 ). The data may include characteristics, attributes, and / or descriptions of the in-store items and may be used for subsequent comparison and matching with digital media items.

[0032]

[0038] At block 415, the computing system assesses the degree of similarity between the digital media item (e.g., 225 of FIG. 2 ) and the in-store item (e.g., 240 of FIG. 2 ). In some embodiments, the assessment may include determining whether any in-store item matches the digital media item. In some embodiments, to determine a match, the system may compare features, attributes, and / or descriptions of the item identified from the digital media to those of the in-store item. If an exact match is determined, such as when the in-store item is found to be identical to one of the digital media items, the method proceeds to block 430. If an exact match is not identified, such as when the in-store item is found to be identical to none of the digital media items, the method 400 proceeds to block 420.

[0033]

[0039] In block 420, the computing system evaluates whether any in-store item resembles a digital media item. In some embodiments, each item (either an in-store item or a digital media item) can be represented by a feature vector. To determine similarity, the system can calculate various metrics between these vectors, such as cosine similarity, Euclidean distance, etc. In some embodiments, the computing system can establish a certain similarity criterion, and when the calculated similarity meets these criteria, the two items can be considered similar. For example, when calculating cosine similarity where a value of 0 indicates not completely similar and a value of 1 indicates completely similar, the computing system can set a similarity threshold. If the cosine similarity between two items exceeds the defined threshold (e.g., 0.85), the system can determine that the two items are similar. In some embodiments, the computing system can focus on comparing similarity in functionality, such as whether an in-store item is functionally similar to one of the digital media items. In such a configuration, the feature vector can be generated from attributes related to the functionality of the item, such as practicality, purpose, user reviews, manufacturer descriptions, or any features unique to the product catalog. In some embodiments, the system can determine visual similarity, such as whether an in-store item is visually similar to one of the digital media items. For such an evaluation, the feature vector can be generated from attributes that reflect the visual characteristics of the item, such as color, shape, texture, pattern, and the like. By comparing these feature vectors, the system can determine how visually similar one in-store item is to one digital media item. When similarity is identified, such as when an in-store item is determined to be similar to a digital media item (either functionally or visually), method 400 proceeds to block 430.If no such similarity is identified, such as the calculated similarity between the in-store item and the digital media item does not meet the similarity criteria, method 400 proceeds to block 425, where the computing system returns a response indicating a lack of matching or similar items in the store.

[0034]

[0040] At block 430, the computing system identifies locations of in-store items that match or are similar to the digital media item.

[0035]

[0041] At block 435, the computing system assesses whether any digital media items remain unchecked. If such items are found, method 400 returns to block 415 to repeat the matching and comparison process. However, if all digital media items have been addressed, method 400 proceeds to block 440, where the system generates a response including the locations of the identified in-store items and provides the response to generate a guidance route.

[0036]

[0042] 5 illustrates an example method 500 for alert generation according to some embodiments of the present disclosure. In some embodiments, method 500 may be performed by one or more computing devices, such as server 110 as illustrated in FIG. 1, alert generation component 275 as illustrated in FIG. 2, and / or computing device 700 as illustrated in FIG. 7.

[0037]

[0043] Method 500 begins at block 505, where a computing system (e.g., 275 in FIG. 2 ) receives a guidance route. In some embodiments, the guidance route may be generated based on an internal map of the store and / or the locations of in-store items that match or are similar to the digital media item (e.g., an item recognized by processing the digital media input). In some embodiments, the guidance route may lead the customer to find identified in-store items (also referred to as target items in some embodiments). In some embodiments, the guidance route may highlight the shortest distance or most convenient route from the customer's current location to these identified in-store items.

[0038]

[0044] In block 510, a computing system monitors and tracks the real-time location of customers within the store.

[0039]

[0045] At block 515, the computing system evaluates whether the customer is in close proximity to one of the identified in-store items (also referred to in some embodiments as target items). In some embodiments, the evaluation may involve continuous checking of the spatial relationship between the customer's real-time location and the location of the target item. In some embodiments, the system may define a distance threshold (e.g., 2 meters) for the evaluation. If the calculated distance between the customer and the target item is less than or equal to the distance threshold (e.g., 10 meters), the system determines that the customer is near the target item, and method 500 proceeds to block 520. If the calculated distance exceeds the threshold, indicating that the customer is not sufficiently close to the target item, the system returns to block 510, where it continues to track the customer's location and movements within the store.

[0040]

[0046] In block 520, the computing system generates an alert to notify the customer about the item of interest. The alert may include details about the target item, such as its name, description, available colors and sizes, and the like.

[0041]

[0047] In block 525, the computing system sends the alert to a user device (e.g., 140 in FIG. 1). In some embodiments, the alert may be sent to an in-store device (e.g., 135 in FIG. 1).

[0042]

[0048] In block 530, the computing system checks whether the alert has been recognized by the customer (through the user device or the in-store device). If the alert is confirmed, the method proceeds to block 540, where the alert generation process ends. If the alert is not recognized by the customer, such as when the customer is near the target item but cannot find it successfully, method 500 returns to block 510, where the computing system continues to track the customer's location and / or calculate the spatial distance between the customer and the target item in real time.

[0043]

[0049] FIG. 6 is a flowchart illustrating an exemplary method 600 for identifying in-store items similar to objects picked up in digital media and generating corresponding guidance routes, according to some embodiments of the present disclosure.

[0044]

[0050] At block 605, a computing system (e.g., 110 in FIG. 1 ) receives digital media input (e.g., 210 in FIG. 2 ) from a user. In some embodiments, the process of receiving the digital media may comprise scanning the digital media displayed on the user's device (e.g., 140 in FIG. 1 ) by a device associated with the physical location (e.g., in-store device 135 in FIG. 2 ). In some embodiments, the device located at the physical location comprises a scanning camera.

[0045]

[0051] At block 610, the computing system identifies an object depicted in the digital media (eg, digital media item 225 of FIG. 2).

[0046]

[0052] In block 615, the computing system determines currently available items at the physical location (e.g., in-store items 240 in FIG. 2 ) by analyzing information collected by a set of cameras at the physical location (e.g., 225 in FIG. 2 ).

[0047]

[0053] In block 620, the computing system identifies a set of target items (e.g., matching results 250 in FIG. 2 ) from the currently available items at the physical location that are similar to at least one of the objects based on the digital media and collected information.

[0048]

[0054] At block 625, the computing system generates guidance (e.g., 260 in FIG. 2) that navigates the user to at least one of the set of items of interest within the physical location. According to some embodiments, the guidance may be displayed on at least one of (i) a device associated with the physical location or (ii) the user's device, where the guidance is displayed along with information related to at least one of the set of items of interest.

[0049]

[0055] In some embodiments, the computing system may further monitor, in real time, changes in the status of a set of target items via a set of cameras at a physical location (as shown in block 325 of FIG. 3), and update the guidance based on the changes.

[0050]

[0056] In some embodiments, the computing system may further access an inventory database to check the inventory of at least one of the objects at one or more other physical locations (as shown in blocks 330 and 335 of FIG. 3), and provide alternative purchase routes to the user.

[0051]

[0057] In some embodiments, the set of cameras at the physical location may be configured with an artificial intelligence-based algorithm to determine at least one of (i) the category or (ii) the quantity of each currently available item at the physical location.

[0052]

[0058] In some embodiments, the set of target items may comprise at least one of (i) currently available items that are the same as at least one of the objects, (ii) currently available items that are visually similar to at least one of the objects, or (iii) currently available items that are functionally similar to at least one of the objects.

[0053]

[0059] In some embodiments, the digital media may comprise at least one of an image, a video, a live stream, a 3D model, or motion graphics.

[0054]

[0060] In some embodiments, objects within digital media may be identified by using one or more neural networks, where the one or more neural networks are trained using as input the received digital media over time and as the target output the product identifiers labeled, and the one or more neural networks learn to correlate the features from each respective digital media of the received digital media over time with each respective product identifier of the labeled product identifiers.

[0055]

[0061] In some embodiments, the computing may further receive feedback from the user regarding the accuracy of the objects identified from the digital media and may improve the one or more neural networks based on the received feedback.

[0056]

[0062] In some embodiments, the computing system may further send an alert to the user's device when the user approaches the location of an item among a set of target items (as illustrated in block 530 of FIG. 5).

[0057]

[0063] FIG. 7 illustrates an exemplary computing device 700 for item recognition and guidance route generation according to some embodiments of the present disclosure. Although illustrated as a physical device, in some embodiments, the computing device 700 may be implemented using virtual device(s) and / or across several devices (e.g., within a cloud environment). The computing device 700 can be embodied as any computing device such as the server 110, user device 140, in-store device 135, in-store camera 130 as illustrated in FIG. 1.

[0058]

[0064] As illustrated, computing device 700 includes a CPU 705, memory 710, storage 715, one or more network interfaces 725, and one or more I / O interfaces 720. In the illustrated embodiment, CPU 705 retrieves and executes programming instructions stored in memory 710, as well as stores and retrieves application data residing in storage 715. CPU 705 generally represents a single CPU and / or GPU, multiple CPUs and / or GPUs, a single CPU and / or GPU with multiple processing cores, and the like. Memory 710 is included to generally represent random access memory. Storage 715 may be any combination of disk drives, flash-based storage devices, and the like, and may include fixed and / or removable storage devices such as fixed disk drives, removable memory cards, cache, optical storage, network attached storage (NAS), or storage area networks (SAN).

[0059]

[0065] In some embodiments, I / O devices 735 (such as a keyboard, monitor, etc.) are connected via I / O interface(s) 720. Additionally, via network interface 725, computing device 700 can be communicatively coupled to one or more other devices and components (e.g., via a network, which may include the Internet, local network(s), and the like). As illustrated, CPU 705, memory 710, storage 715, network interface(s) 725, and / or I / O interface(s) 720 are communicatively coupled by one or more buses 730.

[0060]

[0066] In the illustrated embodiment, memory 710 includes a data processing component 750, an item matching component 755, a guidance generation component 760, a customer location detection component 765, and an alert generation component 770. While illustrated as separate components for conceptual clarity, in some embodiments, the operations of the illustrated components (and others not illustrated) may be combined or distributed across any number of components. Furthermore, while illustrated as software residing in memory 710, in some embodiments, the operations of the illustrated components (and others not illustrated) may be implemented using hardware, software, or a combination of hardware and software.

[0061]

[0067] In the illustrated embodiment, the data processing component 750 may be configured to process various visual data. The visual data may include data viewed by customers online (e.g., digital media) and inventory visual data captured by in-store cameras. In some embodiments, the data processing component 750 may extract features by processing the visual data. In some embodiments, after these features are identified, the component may perform item recognition using a trained ML model. In some embodiments, such as when the visual data is digital media received from a customer (e.g., through an in-store device), the output of the data processing component may be a structured dataset of items featured in the digital media (also referred to in some embodiments as digital media items) and their respective attributes or characteristics. In some embodiments, such as when the visual data is received from an in-store camera, the output may be a real-time inventory database detailing available products in the store (also referred to in-store items in some embodiments) and their associated attributes (e.g., color, size, brand, quantity, and location).

[0062]

[0068] In an embodiment as illustrated, the item matching component 755 may determine a similarity between an in-store item and a digital media item. In some embodiments, the item matching component 755 may represent each item as a feature vector based on the attributes and characteristics of the item. The vector captures the essence of the item and projects its diverse characteristics into a multi-dimensional space. To evaluate the similarity between an in-store item and a digital media item, the item matching component 755 may calculate a metric such as the cosine similarity or Euclidean distance between the respective vectors of these items. The metric may quantify the relationship between items in the multi-dimensional space. For example, in some embodiments, a perfect match between an in-store item and a digital media item may be identified when the distance metric is 0. In some embodiments, two items may be determined to be similar when the distance metric between these items meets a certain criterion. For example, items may be considered similar if the cosine similarity between their vectors exceeds a defined threshold, or if the Euclidean distance between their vectors is below a defined threshold. The output of the item matching component may be a list of in-store items that match or are similar to the digital media item (also called the target item in some embodiments), and their associated attributes (e.g., color, size, brand, quantity, and location).

[0063]

[0069] In the illustrated embodiments, the guidance generation component 760 can be configured to generate guidance routes to lead the customer to find these target items. In the illustrated embodiments, the customer location detection component 765 can collect the real-time location of the customer within the store. In the illustrated example, the alert generation component 770 can receive the guidance route for the target item from the guidance generation component 760 and collect the real-time location data of the customer from the customer location detection component 765. Based on the received data, the alert generation component 770 can calculate the spatial distance between the customer and the set of target items. The alert generation component 770 can generate an alert when it determines that the distance between the customer and any target item falls below a defined threshold value. The alert can notify the customer of the proximity of the item of interest.

[0064]

[0070] In the illustrated example, the storage 715 can include digital media records 775, inventory data 780, and alert records 785. In some embodiments, the aforementioned data can be stored in a remote database (e.g., 125 in FIG. 1) connected to the computing device 700 via a network (e.g., 105 in FIG. 1).

[0065]

[0071] The description of the various embodiments of the present disclosure is presented for purposes of illustration but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, the practical application, or a technical improvement found in the marketplace, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0066]

[0072] Reference will be made below to embodiments presented in this disclosure. However, the scope of the disclosure is not limited to the described embodiments. Instead, any combination of the following features and elements, whether associated with different embodiments, is contemplated to implement and practice the contemplated embodiments. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or prior art, whether or not an advantage is achieved by a given embodiment does not limit the scope of the disclosure. For this reason, the following aspects, features, embodiments, and advantages are merely exemplary and should not be considered elements or limitations of the appended claims unless expressly recited in the claim(s). Similarly, references to "the present invention" should not be construed as a generalization of any inventive subject matter disclosed herein, nor should they be considered elements or limitations of the appended claims unless expressly recited in the claim(s).

[0067]

[0073] Aspects of the present disclosure may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which may be generally referred to herein as "circuits," "modules," or "systems."

[0068]

[0074] The present disclosure may be a system, a method, and / or a computer program product, which may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to implement aspects of the present disclosure.

[0069]

[0075] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical coding devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, should not be construed as being a transitory signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0070]

[0076] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them for storage in a computer-readable storage medium within the respective computing / processing device.

[0071]

[0077] The computer-readable program instructions for carrying out the operations of the present disclosure may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, or the like, and traditional procedural programming languages such as the "C" programming language or similar. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, an electronic circuit, including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuit to perform aspects of the present disclosure.

[0072]

[0078] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0073]

[0079] These computer-readable program instructions may be provided to a general-purpose computer, a special-purpose computer processor, or other programmable data processing apparatus to produce a machine, such that the instructions executing on the computer processor or other programmable data processing apparatus create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to function in a certain manner, such that a computer-readable storage medium having instructions stored therein comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0074]

[0080] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that a series of operational steps are executed on the computer, other programmable apparatus, or other device to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0075]

[0081] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of an instruction, comprising one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may not occur in the order noted in the figures. For example, depending on the functionality involved, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or that executes a combination of special-purpose hardware and computer instructions.

[0076]

[0082] In embodiments of the present disclosure, computing resources may be provided to end users through a cloud computing infrastructure. Cloud computing generally refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing may be defined as computing capabilities that provide abstraction between computing resources and their underlying technical architecture (e.g., servers, storage, networks), enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be quickly provisioned and released with minimal management effort or interaction with a service provider. From this, cloud computing enables users to access virtual computing resources (e.g., storage, data, applications, and even complete virtualized computing systems) in the "cloud," regardless of the underlying physical systems used to provide the computing resources (or the location of those systems).

[0077]

[0083] Typically, cloud computing resources are provided to users on a pay-per-use basis, where the user is charged only for the computing resources actually used (e.g., the amount of storage space consumed by the user or the number of virtualization systems instantiated by the user). A user can access any of the resources present in the cloud at any time from anywhere on the Internet. In the context of the present disclosure, a user may access an application (e.g., an item search application) or associated data available in the cloud. For example, an item search application can perform item recognition and matching through the cloud computing infrastructure and store associated data in a storage location in the cloud. Doing so enables a user to access this information from any computing system attached to a network connected to the cloud (e.g., the Internet).

[0078]

[0084] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which scope is determined by the following claims.

Claims

1. Receiving digital media from a user; Identifying an object depicted within the digital media; Determining current available items at the physical location by analyzing information collected by a camera at the physical location; Identifying a target item similar to at least one of the objects based on the digital media and the collected information from the current available items at the physical location; Generating guidance to navigate the user to at least one of the target items within the physical location A method comprising the steps of.

2. The guidance is displayed on at least one of (i) a device associated with the physical location or (ii) the user's device, and the guidance is displayed together with information related to at least one of the target items. The method according to claim 1.

3. Monitoring in real time a change in status of at least one of the target items via the camera at the physical location; Updating the guidance based on the change The method according to claim 1, further comprising the steps of.

4. Accessing an inventory database to check the inventory of at least one of the objects at one or more other physical locations; Providing the user with an alternative purchase route The method according to claim 1, further comprising the steps of.

5. The camera at the physical location is configured with an artificial intelligence-based algorithm to determine at least one of (i) the category or (ii) the quantity of each of the current available items at the physical location. The method according to claim 1.

6. The target item comprises at least one of (i) the same current available item as at least one of the objects, (ii) the current available item visually similar to at least one of the objects, or (iii) the current available item functionally similar to at least one of the objects. The method according to claim 1.

7. The method according to claim 1, wherein the digital media may include at least one of an image, a video, a live stream, a 3D model, or a motion graphic.

8. The object in the digital media is identified by using one or more neural networks, The one or more neural networks are trained using, as input, the received digital media over time and, as the target output, the labeled product identifiers, The one or more neural networks learn to correlate the features from each respective digital media of the received digital media over time with each respective product identifier of the labeled product identifiers. The method according to claim 1.

9. Receiving feedback from the user regarding the accuracy of the object identified from the digital media, and Improving the one or more neural networks based on the received feedback The method according to claim 8, further comprising.

10. The method according to claim 1, further comprising transmitting an alert to the user's device when the user approaches the location of an item among the target items.

11. Receiving the digital media from the user comprises scanning the digital media displayed on the user's device by a device associated with the physical location, the device located at the physical location comprising a scanning camera. The method according to claim 1.

12. One or more memories collectively storing computer-executable instructions, Collectively executing the computer-executable instructions to cause the system to Receive digital media from a user, Identify an object depicted in the digital media, Determine currently available items at the physical location by analyzing information collected by a camera at the physical location, and Identify a target item similar to at least one of the objects based on the digital media and the collected information from the currently available items at the physical location. Generating guidance to navigate the user to at least one of the target items within the physical location One or more processors configured to cause the above to be performed A system comprising the above

13. The guidance is displayed on at least one of (i) a device associated with the physical location or (ii) the user's device, and the guidance is displayed together with information related to at least one of the target items. The system according to claim 12

14. The one or more processors collectively execute the computer-executable instructions to cause the system to Monitor in real time a change in the status of at least one of the target items via the camera at the physical location Update the guidance based on the change The system according to claim 12, further configured to cause the above to be performed

15. The one or more processors collectively execute the computer-executable instructions to cause the system to Access an inventory database to check the inventory of at least one of the objects at one or more other physical locations Provide the user with an alternative purchase route The system according to claim 12, further configured to cause the above to be performed

16. The camera at the physical location is configured with an artificial intelligence-based algorithm to determine at least one of (i) the category or (ii) the quantity of each of the currently available items at the physical location. The system according to claim 12

17. The target item comprises at least one of (i) the currently available item that is the same as at least one of the objects, (ii) the currently available item that is visually similar to at least one of the objects, or (iii) the currently available item that is functionally similar to at least one of the objects. The system according to claim 12

18. The digital media may comprise at least one of an image, a video, a live stream, a 3D model, or motion graphics. The system according to claim 12

19. The object in the digital media is identified by using one or more neural networks, The one or more neural networks are trained using, as input, received digital media over time and, as target output, product identifiers labeled with labels, The one or more neural networks learn to correlate features from each of the received digital media over time with each of the product identifiers of the labeled product identifiers, The system according to claim 12.

20. When executed by the operation of a computing system, Receiving digital media from a user; Identifying an object depicted in the digital media; Determining currently available items at the physical location by analyzing information collected by a camera at the physical location; Identifying a target item similar to at least one of the objects based on the digital media and the collected information from the currently available items at the physical location; Generating guidance for navigating the user to at least one of the target items within the physical location One or more non-transitory computer-readable media including, in any combination, computer program code for performing operations comprising

Images