Method, apparatus, and recording medium for searching products

Abstract

A product search method performed by an electronic device, according to an embodiment of the present disclosure, may comprise the steps of: receiving an image corresponding to a product search query from a user terminal; in response to receiving the image, extracting at least one piece of text information and an image feature from the image, the image feature including information indicating a visual form of the image; on the basis of the extracted at least one piece of text information and analysis data stored in a memory, determining a first search result including at least one product corresponding to the image, the analysis data including a product, text information regarding the product, and data to which an image feature has been mapped; on the basis of the extracted at least one image feature and the analysis data, determining a second search result including at least one product corresponding to the image; and determining a final search result on the basis of the first search result and the second search result.

Description

Methods, devices and recording media for searching for products

[0001] The present disclosure relates to product search using image features and keywords extracted from an image.

[0002] Recent advancements in communication technology and societal changes have enabled users to search for and purchase a wide variety of products online via electronic devices. As the variety of products available in online stores has grown, the accuracy of algorithms that provide a list of products matching keywords or images entered by users has become increasingly important.

[0003] Traditionally, search engines would search within a database based on keywords entered by the user in the search bar, and then present the search results to the user. However, this method had limitations: it relied heavily on user-entered keywords and failed to accurately reflect the user's desired product image.

[0004] A technical problem to be solved through one embodiment of the present disclosure is to enable a user to search using an image of a desired product.

[0005] A technical problem to be solved through one embodiment of the present disclosure is to more accurately present a product desired by a user by synthesizing keyword search results and image search results.

[0006] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art of the present disclosure from the description below.

[0007] A product search method performed by an electronic device according to one embodiment of the present disclosure may include: receiving an image corresponding to a product search query from a user terminal; extracting, in response to receiving the image, at least one piece of text information and an image feature from the image, wherein the image feature includes information indicating a visual form of the image; determining a first search result including at least one product corresponding to the image based on the extracted at least one piece of text information and analysis data stored in a memory, wherein the analysis data includes data to which the product, text information about the product, and image feature are mapped; determining a second search result including at least one product corresponding to the image based on the extracted at least one image feature and the analysis data; and determining a final search result based on the first search result and the second search result.

[0008] In one embodiment, the step of determining the final search result may include the step of determining, as the final search result, a product that overlaps at least one product included in the first search result and at least one product included in the second search result.

[0009] In one embodiment, the step of extracting at least one text information and an image feature from the image may include the step of extracting the at least one text information and the image feature using a machine learning model.

[0010] In one embodiment, the step of extracting at least one text information and an image feature from the image may include the step of extracting the at least one text information from the image using natural language processing.

[0011] In one embodiment, the step of extracting the at least one text information from the image using the natural language processing may include the step of extracting the at least one text information from the image using text mining.

[0012] In one embodiment, the step of extracting at least one text information and image feature from the image may further include the step of identifying a plurality of products included in the image, and determining one product among the plurality of products for which a search is to be performed.

[0013] In one embodiment, the step of extracting at least one text information and an image feature from the image may include the step of extracting a first text information group including at least one text information from a first product included in the image; and the step of extracting a second text information group including at least one text information from a second product included in the image.

[0014] In one embodiment, the step of extracting at least one text information and image feature from the image may include the step of outputting a message to the user terminal instructing the user to select one of the first product and the second product; and the step of determining to perform a search based on a group of text information corresponding to the selected product based on a user input to the user terminal.

[0015] In one embodiment, the step of determining one product to be searched from among the plurality of products may include the step of comparing the size of an area where a first product is displayed and the size of an area where a second product is displayed in the image, and determining a product with a larger displayed area as the product to be searched.

[0016] In one embodiment, the step of determining one product to be searched from among the plurality of products may include the step of comparing the resolution of an area where a first product is displayed and the resolution of an area where a second product is displayed in the image, and determining a product with a higher resolution of the displayed area as the product to be searched.

[0017] In one embodiment, the step of extracting at least one text information and an image feature from the image may further include the step of performing preprocessing on the image before extracting at least one text information and an image feature from the image.

[0018] In one embodiment, preprocessing of the image may include image normalization, denoising, and feature enhancement.

[0019] In one embodiment, the step of determining the final search result may further include the step of transmitting the final search result to the user terminal.

[0020] In one embodiment, the at least one piece of text information may be transmitted to the user terminal together with the final search result.

[0021] An electronic device according to one embodiment of the present disclosure comprises one or more processors, one or more memories storing instructions to be executed by the one or more processors, and when the instructions are executed by the one or more processors, the one or more processors may be configured to execute a method according to the present disclosure.

[0022] In one embodiment of the present disclosure, a non-transitory computer-readable recording medium having recorded thereon instructions that, when executed by one or more processors, cause the one or more processors to perform an operation, the instructions may be configured to cause the one or more processors to perform a method according to the present disclosure.

[0023] The technical problem to be solved through one embodiment of the present disclosure is to improve the user experience by enabling a user to conveniently search for a desired product by inputting an image of the product.

[0024] A technical problem to be solved through one embodiment of the present disclosure is to provide search results for products desired by a user more accurately by synthesizing keyword search results and image search results.

[0025] The effects according to the technical idea of ​​the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the specification.

[0026] FIG. 1 is a diagram illustrating an environment in which an electronic device according to one embodiment of the present disclosure can be applied.

[0027] FIG. 2 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0028] FIG. 3 illustrates an embodiment of an electronic device identifying at least one product in an image according to one embodiment of the present disclosure.

[0029] FIG. 4 illustrates multiple layers of a process by which an electronic device performs an image-based product search according to one embodiment of the present disclosure.

[0030] FIG. 5 is a flowchart of a method for an electronic device to perform a product search based on an image according to one embodiment of the present disclosure.

[0031] FIG. 6 is a flowchart of a method for extracting keywords from an image by an electronic device according to one embodiment of the present disclosure.

[0032] FIG. 7 is a flowchart of a method for an electronic device to search for a product by performing image search and keyword search according to one embodiment of the present disclosure.

[0033] The embodiments of this disclosure are provided for the purpose of illustrating the technical concepts of this disclosure. The scope of rights under this disclosure is not limited to the embodiments presented below or the specific descriptions of these embodiments.

[0034] All technical and scientific terms used in this disclosure, unless otherwise defined, have the meanings commonly understood by those of ordinary skill in the art to which this disclosure pertains. All terms used in this disclosure have been selected for the purpose of more clearly explaining this disclosure and are not intended to limit the scope of rights under this disclosure.

[0035] Expressions such as “including,” “comprising,” “having,” and the like used in this disclosure should be understood as open-ended terms that imply the possibility of including other embodiments, unless otherwise stated in the phrase or sentence in which the expression is included.

[0036] The singular forms described in this disclosure may include plural meanings unless otherwise stated, and the same applies to the singular forms described in the claims.

[0037] The expressions “first,” “second,” etc. used in this disclosure are used to distinguish between multiple components, and do not limit the order or importance of the components.

[0038] As used herein, the expressions "A, B, and C," "A, B, or C," "at least one of A, B, and C," or "at least one of A, B, or C" may refer to each of the listed items or all possible combinations of the listed items. For example, "at least one of A or B" may refer to (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0039] The term "unit" as used herein refers to software or hardware components such as field-programmable gate arrays (FPGAs) and application specific integrated circuits (ASICs). However, "units" are not limited to hardware and software. "Units" may be configured to reside on an addressable storage medium, and may be configured to execute one or more processors. Thus, by way of example, "units" include components such as software components, object-oriented software components, class components, and task components, as well as processors, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within a component and "unit" may be combined into a smaller number of components and "units" or further separated into additional components and "units."

[0040] The expression "based on" as used in this disclosure is used to describe one or more factors that influence the decision, act of judgment, or action described in the phrase or sentence containing the expression, and this expression does not exclude additional factors that influence the decision, act of judgment, or action.

[0041] In this disclosure, when a component is referred to as being "connected" or "connected" to another component, it should be understood that the component can be directly connected or connected to the other component, or can be connected or connected via a new other component.

[0042] Hereinafter, embodiments of the present disclosure will be described with reference to the attached drawings. In the attached drawings, identical or corresponding components are assigned the same reference numerals. Furthermore, in the description of the embodiments below, redundant descriptions of identical or corresponding components may be omitted. However, even if a description of a component is omitted, it is not intended that such component is not included in any embodiment.

[0043] FIG. 1 is a diagram illustrating an environment in which an electronic device according to one embodiment of the present disclosure can be applied.

[0044] An electronic device (110) and a user terminal (120) are connected through a network and can communicate with each other. The electronic device (110) may be a server or a different terminal from the user terminal (120).

[0045] The electronic device (110) may be a server device that provides an e-commerce service. That is, the electronic device (110) may be a server device operated under the management of an operator of the e-commerce service.

[0046] An electronic device (110) can manage information about users using e-commerce services. The user information managed by the electronic device (110) may include, for example, the user's name, age, address, or contact information. Managing user information can encompass a series of actions involving controlling and processing that information. For example, managing user information can include storing, updating, or modifying that information.

[0047] An electronic device (110) can manage information about items provided by an e-commerce service. An e-commerce service may provide users with a service that allows them to purchase goods, such as items, food, or products. Alternatively, the e-commerce service provided by the electronic device (110) may include a service that mediates the sale of goods by some users to other users. Information about items managed by the electronic device (110) may include, for example, item identification information, item name, product information corresponding to the item, item price, item capacity, item size, item weight, item attributes, category, or unit price. An item may be a sales unit of a product, and even for the same product, it may be provided to users as different items. For example, even for the same product, it may be sold as different items with different sales unit quantities and prices. Therefore, different items may share product information. Managing information about an item can collectively refer to a series of actions that control and process that information. For example, managing information about an item may include storing, updating, or modifying that information.

[0048] An electronic device (110) can manage information about customers using e-commerce services. The customer information managed by the electronic device (110) may include, for example, the customer's name, age, address, payment method, purchase history, or contact information. Managing customer information can encompass a series of actions involving controlling and processing that information. For example, managing customer information can include storing, updating, or modifying that information.

[0049] Managing information about users or items can encompass all actions related to controlling and processing that information. For example, managing user information can include storing, updating, or modifying that information.

[0050] The electronic device (110) can manage various information to be provided to users using e-commerce services. Based on requests from an administrator managing the e-commerce service or a user using the e-commerce service, the electronic device (110) can manage various data necessary to provide e-commerce services. For example, the electronic device (110) can create, modify, store, or provide to users data related to various pages associated with the e-commerce service, such as an item list page, a product detail page, a search results page, a product sales page, an event page, and landing pages requested by users.

[0051] Regarding the management method of the electronic device (110), in one embodiment, the electronic device (110) can manage information accompanying the provision of an electronic commerce service based on a user's input.

[0052] In one embodiment, the electronic device (110) may provide a page related to an e-commerce service to a user terminal (120) based on a request from a user using the e-commerce service. Here, the page related to the e-commerce service provided to the user terminal (120) may include a page that provides various information regarding items that the user can purchase, a page regarding the purchase of an item requested by the user, a shopping cart page for the purchase of an item, a page providing a list of items searched in various ways, a page providing detailed information about an item, etc. The page providing operation may include an operation of providing the user with at least a portion of all data generated in the process of the user using the e-commerce service.

[0053] In addition, the electronic device (110) can manage transactions related to e-commerce services provided to users, i.e., transactions related to online products (e.g., goods or services). That is, the electronic device (110) can manage a series of processes such as enabling users to search for, select, purchase, pay for, and request delivery of items. As such processes, the electronic device (110) can provide customers with information regarding items and delivery so that users can search for, select, purchase, pay for, and request delivery of items, and can receive purchase, payment, and delivery requests from users. In addition, it should be noted that even if an operation that the electronic device (110) can perform is not mentioned in the present disclosure, if it is a general operation that a server device providing a known e-commerce service can perform, applying the technical ideas of the present disclosure with reference to that operation is not excluded from the scope of the present disclosure.

[0054] The electronic device (110) may be a server device that manages an e-commerce service. In other words, the electronic device (110) may be a server device that manages a platform that provides an e-commerce service. A user may subscribe to the e-commerce service and purchase an item through the e-commerce service. The user terminal (120) may transmit an input requesting item information to the electronic device (110), and the electronic device (110) may provide the item information to the user terminal (120).

[0055] The electronic device (110) described above may be implemented as one or more computing devices. For example, all functions of the electronic device (110) may be implemented in a single computing device. For another example, a first function of the electronic device (110) may be implemented in a first computing device, and a second function may be implemented in a second computing device. As another example, multiple computing devices may be used, each of which implements all or specific functions of the electronic device (110). The computing devices described above may be, but are not limited to, a desktop computer, a laptop computer, an application server, a proxy server, or a cloud server, and any type of device equipped with computing functions may be a computing device.

[0056] The user terminal (120) may be a device for a user to use an e-commerce service. The user terminal (120) may be implemented as a terminal capable of transmitting and receiving various information with the electronic device (110) via a network. For example, the user terminal (120) may be one of a computer, a laptop, a portable communication terminal (such as a smartphone), a portable multimedia device, a wearable device, or an HMD. However, the type of the user terminal (120) is not limited thereto, and the user terminal (120) may be any device that includes an input / output interface capable of receiving information from a user or outputting information to a user, and that can communicate with the electronic device (110) or other devices via a network.

[0057] The user terminal (120) can provide information received from the electronic device (110) to the user, and can receive input from the user and transmit it to the electronic device (110). Specifically, the user terminal (120) can obtain inputs from the user that instruct to call various pages, and generate commands that instruct to call various pages in response to the obtained inputs. The user terminal (120) can transmit commands that instruct to call various pages to the electronic device (110). The input obtained from the user can include various forms of input, such as a click using a mouse, a touch using a touchpad or a touch screen, voice recognition, and other electronic inputs. The user terminal (120) can receive various pages from the electronic device (110) and output the received various pages.

[0058] The user terminal (120) may be a device for a user to use a delivery order service. The user terminal (120) may be implemented as a terminal capable of transmitting and receiving various information with the electronic device (110) via a network. For example, the user terminal (120) may be one of a computer, a laptop, a portable communication terminal (such as a smartphone), a portable multimedia device, a wearable device, or an HMD. However, the type of the user terminal (120) is not limited thereto, and the user terminal (120) may be any device that includes an input / output interface capable of receiving information from a user or outputting information to a user, and that can communicate with the electronic device (110) or other devices via a network.

[0059] A network may serve to connect an electronic device (110) and a user terminal (120) or other external device. For example, the network may provide a connection path so that a user terminal (120) or other external device can be connected to the electronic device (110) and transmit and receive packet data with the electronic device (110). The network may be implemented as any type of wired or wireless network, such as a local area network (LAN), a wide area network (WAN), a mobile radio communication network, or Wibro (Wireless Broadband Internet).

[0060] In one embodiment, the electronic device (110) and the user terminal (120) may operate as a single device. The user terminal (120) may be included in the electronic device (110) as a component of all or part of the electronic device (110). In this case, for example, various types of information exchanged between the electronic device (110) and the user terminal (120) via a network may be various types of information exchanged between components within a single device.

[0061] Electronic devices (e.g., electronic device (110)) according to various embodiments of the present disclosure may be devices of various forms. For example, the electronic device (110) may be a portable communication device, a computer device, a portable multimedia device, a wearable device, a home appliance device, or a device according to a combination of one or more of the above-described devices. The electronic device (110) of the present disclosure is not limited to the above-described devices.

[0062] FIG. 2 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0063] An electronic device (200) can process information regarding an e-commerce service. In one embodiment, the electronic device (200) may include one or more processors (210), one or more memories (220), and a communication interface (230) as components. In one embodiment, at least one of the components of the electronic device (200) may be omitted, or another component may be added to the electronic device (200). In one embodiment, additionally or alternatively, some of the components may be implemented in an integrated manner or implemented as a single or multiple entities. In the present disclosure, one or more processors (210) may be referred to as a processor (210). The expression “processor (210)” may mean a set of one or more processors, unless the context clearly indicates otherwise. In the present disclosure, one or more memories (220) may be referred to as a memory (220). The expression "memory (220)" may mean a set of one or more memories, unless the context clearly indicates otherwise. In one embodiment, at least some of the components inside / outside the electronic device (200) may be connected to each other via a bus, a General Purpose Input / Output (GPIO), a Serial Peripheral Interface (SPI), or a Mobile Industry Processor Interface (MIPI), and may exchange information (data, signals, etc.).

[0064] The communication interface (communication interface, 230) can communicate with the user's terminal and an external device. The communication interface (230) can perform wireless or wired communication between the electronic device (200) and the terminal. For example, the communication interface (230) can perform wireless communication according to a method such as eMBB (enhanced Mobile Broadband), URLLC (Ultra Reliable Low-Latency Communications), MMTC (Massive Machine Type Communications), LTE (Long-Term Evolution), LTE-A (LTE Advance), NR (New Radio), UMTS (Universal Mobile Telecommunications System), GSM (Global System for Mobile communications), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), WiBro (Wireless Broadband), WiFi (Wireless Fidelity), Bluetooth (Bluetooth), NFC (Near Field Communication), GPS (Global Positioning System), or GNSS (Global Navigation Satellite System). For example, the communication interface (230) may perform wired communication according to a method such as Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), Recommended Standard-232 (RS-232), or Plain Old Telephone Service (POTS). In one embodiment, the electronic device (200) may be implemented by being integrated with another device. In this case, the communication interface (230) may function as a connection circuit or interface connecting the electronic device (200) and the other device.

[0065] The processor (210) can control at least one component of the electronic device (200) connected to the processor (210) by running software (e.g., commands, programs, etc.). In addition, the processor (210) can perform various operations related to the present disclosure, such as calculations, processing, data generation, and processing. In addition, the processor (210) can load data from the memory (220) or store data in the memory (220). Furthermore, the processor (210) can transmit and receive various information with the user's terminal and external devices through the communication interface (230). In one embodiment, the processor (210) can control the communication interface (230) to transmit various information, such as information on various pages, to the user's terminal.

[0066] The memory (220) can store various information (data). The information stored in the memory (220) is information acquired, processed, or used by at least one component of the electronic device (200), and may include software (e.g., commands, programs, etc.). The memory (220) may include volatile and / or non-volatile memory. In the present disclosure, the commands or programs are software stored in the memory (220), and may include an operating system for controlling the resources of the electronic device (200), an application, and / or middleware for providing various functions to the application so that the application can utilize the resources of the electronic device (200). In one embodiment, the memory (220) may store commands that cause the processor (210) to perform operations when executed by the processor (210). The memory (220) may store at least a portion of information received from the terminal via the communication interface (230) and / or information transmitted to the terminal via the communication interface (230). The processor (210) can store at least a portion of information received from the terminal through the communication interface (230) and / or information transmitted to the terminal through the communication interface (230) in the memory (220).

[0067] The memory (220) can store images and keywords corresponding to the images and image features by mapping them. According to one embodiment, the memory (220) can store analysis data that maps various images uploaded to the network and keywords corresponding to the images and image features. According to one embodiment, the keywords and image features corresponding to the images may be results analyzed through a machine learning model in the past. The memory (220) can check the size of the accumulated data of the images uploaded to the network and the keywords and image features corresponding to the images, and if the size of the accumulated data is greater than a predetermined value, the analysis data can be updated to include the data accumulated on the network.

[0068] The memory (220) can store a machine learning model. In one embodiment, the machine learning model can learn from analysis data stored in the memory (220). In one embodiment, when the analysis data in the memory (220) is updated based on a newly uploaded image on the network and keywords and image features corresponding to the image, the machine learning model can learn based on the updated analysis data.

[0069] In another embodiment, the machine learning model may perform preprocessing of the input image. For example, the machine learning model may preprocess the image using methods such as image normalization, denoising, and feature enhancement. The machine learning model may include at least one of an artificial neural network (ANN), a deep neural network (DNN), a convolutional neural network (CNN), and a recurrent neural network (RNN), but embodiments of the present invention are not limited thereto.

[0070] The processor (210) can obtain an image uploaded from a user terminal (120) via a network. The user terminal can upload the image to the network based on user input, and the processor (210) can obtain the image uploaded to the network using a communication interface (230). According to one embodiment, the image uploaded by the user may be related to a product the user wishes to purchase. For example, the image may include an image of a bag the user wishes to purchase. According to one embodiment, the image may include multiple products (e.g., a bag, a hat, and shoes).

[0071] The processor (210) can search for a product desired by the user using the acquired image. The processor (210) can extract keywords from the image and search for a product corresponding to the extracted keyword using analysis data stored in the memory (220). The processor (210) can extract image features from the image and search for a product corresponding to the extracted image feature using analysis data stored in the memory (220). The processor (210) can synthesize the keyword search results and the image search results to determine a final search result. Hereinafter, the method by which the processor (210) searches for a product based on a keyword is referred to as a keyword search, and the method by which the processor (210) searches for a product based on an image feature is referred to as an image search. The search result may include at least one product. For example, the keyword search result may include at least one product determined to be similar to the product being searched as a result of the keyword search, and the image search result may include at least one product determined to be similar to the product being searched as a result of the image search. The final search result may include at least one product determined by synthesizing keyword search results and image search results. The processor (210) may transmit the final search result to the user terminal (120). The process by which the processor (210) searches for a product based on an image is described in detail below.

[0072] According to one embodiment, the processor (210) may perform preprocessing on the acquired image. The processor (210) may perform preprocessing using a machine learning model to increase the resolution of the acquired image and better extract image features. For example, the processor (210) may perform preprocessing on the acquired image using methods such as image normalization, denoising, and feature enhancement.

[0073] The processor (210) can identify products from the acquired image (or an image after preprocessing). The processor (210) can identify products to be searched from the image. In one embodiment, if the image includes a product and a background, the processor (210) can identify the product from the image regardless of the background. For example, the processor (210) can identify the same product from two images containing the same product but with different backgrounds.

[0074] In one embodiment, an image may include multiple products. In response to determining that the acquired image includes multiple products, the processor (210) may set regions of interest corresponding to each of the multiple products. In one embodiment, the regions of interest set by the processor (210) may be regions that include images of a single product. For example, if the image acquired by the processor (210) includes images of a first product, a second product, and a third product, the processor (210) may set a first region of interest where the first product is located on the image, a second region of interest where the second product is located, and a third region of interest where the third product is located. The processor (210) may identify the products included in each region of interest. If the acquired image includes multiple products, the processor (210) may select one target product from among the multiple products to produce search results. In one embodiment, the processor (210) may select the target product based on the image sizes of the multiple products. For example, the processor (210) may select a product with the largest image size among multiple products as the target product. According to another embodiment, the processor (210) may select a target product based on the resolution of the multiple product images. For example, the processor (210) may select a product with the highest resolution image among multiple products as the target product. According to another embodiment, the processor (210) may select a target product based on user input. For example, the processor (210) may control a message box instructing the user to select one product among multiple products included in the image to be displayed on an output device of a user terminal, and may select one target product based on user input to the output device. Hereinafter, "product" may mean "target product."

[0075] The processor (210) may determine keywords for performing a product search based on an image. Keywords may refer to contextual information such as the shape, color, size, texture, and logo of a product expressed in text format. According to one embodiment, the processor (210) may extract at least one keyword from an image. According to one embodiment, the processor (210) may extract keywords related to a product using a natural language processing technique (e.g., text mining, keyword extraction algorithm). According to one embodiment, the processor (210) may extract keywords related to a product from an image using a machine learning model. According to one embodiment, when one target product is determined among multiple products included in an image, the processor (210) may extract only keywords related to the target product being searched.

[0076] According to one embodiment, the processor (210) may determine a keyword for product search from among at least one keyword extracted from an image. For example, the processor (210) may extract a first product keyword, a second product keyword, and a third product keyword as keywords for an image containing various products. The processor (210) may select one of these keywords and determine it as a keyword to perform a search. According to another embodiment, the processor (210) may extract a shape keyword, a color keyword, and a logo keyword from an image containing one product. The processor (210) may determine the shape keyword and the logo keyword as keywords to search for. In this case, the search results of the keyword search may include products of the same brand and model as the product included in the image uploaded by the user, but with different colors.

[0077] According to one embodiment, the processor (210) may determine a keyword to search among a plurality of keywords based on a user input. For example, the processor (210) may extract a first keyword, a second keyword, and a third keyword from an image, control a message box to be displayed on an output device of a user terminal, instructing the user to select one of the first to third keywords, and select a keyword to search based on a user input to the output device.

[0078] According to one embodiment, the processor (210) can directly obtain keywords from the user. According to one embodiment, the processor (210) can obtain keywords based on user input for the user terminal through a network. The processor (210) can perform a keyword search for a product using the keywords obtained from the user. According to one embodiment, if a keyword extracted from an image and a keyword obtained directly from the user conflict, the processor (210) can give priority to the keyword obtained directly from the user. For example, if the keyword extracted by the processor (210) from the image is a first keyword (e.g., white) and the keyword obtained directly from the user is a second keyword (e.g., black), the processor (210) can perform a keyword search using the second keyword rather than the first keyword.

[0079] According to one embodiment, when an image includes multiple products, the processor (210) may group keywords extracted for each of the multiple products into a single keyword group and select one keyword group to perform a search among them. For example, when an image includes a first product and a second product, the processor (210) may extract a first keyword group related to the first product and a second keyword group related to the second product. The processor (210) may control an output device of a user terminal to display a message box instructing the user to select one of the first and second products, and may select a keyword group to search based on a user input to the output device.

[0080] The processor (210) can search for products based on the determined keyword. The processor (210) can compare the keyword with analysis data stored in the memory (220) and determine keyword search results that include products corresponding to the keyword. In one embodiment, the processor (210) can determine that the keyword search results include products corresponding to keywords similar to the determined keyword.

[0081] The processor (210) can extract image features by parsing the acquired image. Image features may be data containing visual information related to the image. In one embodiment, the image features may be written in machine language. For example, the image features may include information about the shape, color, size, and texture of the image.

[0082] The processor (210) can search for products based on image features extracted from an image. The processor (210) can compare the image features with analysis data stored in the memory (220) and store image search results including products corresponding to the image features. In one embodiment, the processor (210) can determine that the image search results include products having image features similar to the extracted image features.

[0083] The processor (210) can perform keyword searches and image searches in parallel. For example, the processor (210) may include a first processor and a second processor, with the first processor performing keyword searches and the second processor performing image searches. However, embodiments of the present invention are not limited thereto, and the keyword searches and image searches may also be performed sequentially.

[0084] The processor (210) may determine final search results based on keyword search results and image search results. According to one embodiment, the processor (210) may determine products commonly included in the keyword search results and image search results as final search results.

[0085] The processor (210) can transmit the final search results to the user terminal and control them to be displayed on the output device of the user terminal. According to one embodiment, the processor (210) can provide the user with search keywords along with the final search results.

[0086] The processor (210) may change the search results based on user input regarding the final search results. In one embodiment, the processor (210) may update the final search results to include at least one product corresponding to a newly entered keyword among the products included in the final search results. For example, if the final search results include a first product having a first color, a second product having a second color, and a third product having a third color, and the user adds the third color as a new keyword after checking the final search results, the processor (210) may update the final search results to include only the third product. In another embodiment, the processor (210) may update the final search results based on the search results with the changed keywords in response to receiving a user input indicating a change to some of the search keywords.

[0087] Hereinafter, the operations described as being performed by the electronic device in FIGS. 3 to 7 can be understood as being performed by the processor (210) of the electronic device (200) described in FIG. 2.

[0088] FIG. 3 illustrates an embodiment of an electronic device identifying at least one product in an image according to one embodiment of the present disclosure.

[0089] The processor can obtain an image uploaded to the network by a user terminal. In one embodiment, the image may include objects other than the area containing the product (300). For example, the image may include the product (300) and a background. The processor can identify the product (300) to be searched in the image.

[0090] In one embodiment, the processor may perform preprocessing on the image to better identify the product (300). The processor may increase the resolution of the image using methods such as image normalization, denoising, and feature enhancement.

[0091] In one embodiment, the processor may determine a product to be searched from among multiple products included in an image. In one embodiment, the processor may determine a product with the largest area in the image as the search target from among multiple products included in the image. In another embodiment, the processor may determine a product with the highest image resolution as the search target from among multiple products included in the image. In another embodiment, the processor may determine a product to be searched from among multiple products based on user input. For example, the processor may instruct an output device of a user terminal to output a message prompting the user to select one product to be searched from among multiple products, and may select one product from among the multiple products based on the user input in response to the message.

[0092] The processor may analyze an image of an identified product (300) to extract at least one keyword and image feature. The processor may perform a keyword search and an image search using the extracted at least one keyword and image feature, and determine a final search result based on the keyword search result and the image search result.

[0093] FIG. 4 illustrates multiple layers of a process by which an electronic device performs an image-based product search according to one embodiment of the present disclosure.

[0094] Referring to FIG. 4, an algorithm for a processor to perform product search based on an image may be configured with an architecture having multiple layers. The multiple layers may include a user interface layer (400), an image processing layer (410), a database layer (420), a matching layer (430), and a matching result return layer (440).

[0095] The user interface layer (400) is the front-end layer of the architecture, allowing users to upload images to their user terminals. The user interface layer (400) may include a mobile application or an Internet web interface.

[0096] The image processing layer (410) can perform operations for preprocessing an image and extracting keywords and image features. In the image processing layer (410), the processor can perform preprocessing to increase the resolution of the acquired image, enhance features, and extract image features and keywords containing visual information of the image. According to one embodiment, the processor in the image processing layer (410) can extract image features from the image using a machine learning model (e.g., CNN).

[0097] The database layer (420) may correspond to a memory (e.g., memory (220) of FIG. 2). The database layer (420) may store analysis data that maps product images and keywords corresponding to the images to image features. In one embodiment, the database layer (420) may efficiently store analysis data using a distributed storage system.

[0098] The matching layer (430) is a layer that searches for products to determine the final search results, and may include a keyword matching layer (432) and an image feature matching layer (434). In the keyword matching layer (432), the processor may search for products using extracted keywords and determine keyword search results. In the image feature matching layer (434), the processor may search for products using extracted image features and determine image search results. The processor may determine the final search results by synthesizing the keyword search results and the image search results. According to one embodiment, the processor may determine products commonly included in the keyword search results and the image search results as the final search results.

[0099] The matching result return layer (440) is a layer that returns the final search results to the user terminal. The processor can control the transmission of the final search results to the user terminal via a network and display them on the user terminal's output device.

[0100] FIG. 5 is a flowchart of a method for an electronic device to perform a product search based on an image according to one embodiment of the present disclosure.

[0101] In step 500, the electronic device may obtain an image corresponding to a product search query. The electronic device may obtain an image uploaded by a user via a network. Since the user-uploaded image corresponds to the product query, the processor may search for a product corresponding to the obtained image in memory. To search for a product, the electronic device may identify at least one product in the image. For example, the electronic device may remove the background contained in the image and obtain only the product image. For example, if the image includes multiple products, the electronic device may determine one product among the multiple products to be searched.

[0102] In step 510, the electronic device, in response to receiving the image, may extract keywords and image features for the image. The electronic device may extract keywords and image features for the product for which the search is being performed. In one embodiment, the electronic device may extract keywords and image features for the product using a machine learning model.

[0103] In step 520, the electronic device can search for corresponding products by comparing the extracted keywords and image features with analysis data stored in the memory. The memory of the electronic device can store analysis data that maps keywords and image features corresponding to the products and a plurality of products. The electronic device can search for products corresponding to the keywords and image features extracted from the received image based on the analysis data stored in the memory. According to one embodiment, the electronic device can perform a search using keywords to determine keyword search results, perform a search using image features to determine image search results, and determine a final search result that synthesizes the keyword search results and the image search results. For example, the electronic device can determine products that are commonly included in the keyword search results and the image search results as the final search result.

[0104] In step 530, the electronic device may transmit the product search results to the user terminal. The electronic device may transmit the final search results to the user terminal. In one embodiment, the electronic device may transmit search keywords to the user terminal along with the final search results. The electronic device may update the final search results based on user input regarding the search keywords. For example, if the user changes some of the search keywords, the electronic device may update the final search results by reflecting the keyword search results using the changed search keywords.

[0105] FIG. 6 is a flowchart of a method for extracting keywords from an image by an electronic device according to one embodiment of the present disclosure.

[0106] In step 600, the electronic device may extract multiple keywords from the received image. In one embodiment, the image may include multiple products. The electronic device may extract multiple keywords for each product. The electronic device may generate multiple keyword groups including the multiple keywords extracted from each product. In another embodiment, if the image includes a single product, the electronic device may extract multiple keywords representing various levels of information about the product.

[0107] In step 610, the electronic device may determine at least one keyword for which a search will be performed among a plurality of keywords based on a user input. The electronic device may control a message box prompting the user to select one of the plurality of keywords to be output to an output device of the user terminal. The electronic device may determine at least one keyword for which a search will be performed among the plurality of keywords based on the user input to the message box.

[0108] FIG. 7 is a flowchart of a method for an electronic device to search for a product by performing image search and keyword search according to one embodiment of the present disclosure.

[0109] At step 700, the electronic device can obtain an image corresponding to the product search query.

[0110] In step 710, the electronic device may extract image features from the image. In step 712, the electronic device may compare the extracted image features with analysis data stored in memory. In step 720, the electronic device may extract keywords from the image. In step 722, the electronic device may compare the extracted keywords with analysis data stored in memory.

[0111] In one embodiment, steps 710 to 712 and steps 720 to 722 may be performed in parallel.

[0112] In step 730, the electronic device may determine the intersection of the search results using image features and the search results using keywords as the final search results. The electronic device may determine products commonly included in the keyword search results and the image search results as the final search results, and transmit the final search results to the user terminal.

[0113] Although the process steps, method steps, algorithms, etc., are described in a sequential order in the schematic diagrams and flowcharts illustrated in FIGS. 5 through 7, such processes, methods, and algorithms may be configured to operate in any suitable order. In other words, the steps of the processes, methods, and algorithms described in various embodiments of the present disclosure need not be performed in the order described herein. Furthermore, even if some steps are described as being performed asynchronously, in other embodiments, such some steps may be performed concurrently. Furthermore, the illustration of a process by depiction in the drawings does not imply that the illustrated process excludes other variations and modifications thereof, nor does it imply that the illustrated process or any of its steps is essential to one or more of the various embodiments of the present disclosure, nor does it imply that the illustrated process is preferred.

[0114] Various embodiments of the present disclosure may be implemented as software on a machine-readable storage medium. The software may be software for implementing various embodiments of the present disclosure. The software may be inferred from various embodiments of the present disclosure by programmers skilled in the art to which the present disclosure pertains. For example, the software may be a program including machine-readable instructions (e.g., code or code segments). The device may be a device capable of operating according to instructions called from a storage medium, such as a computer. In one embodiment, the device may be an electronic device (200) according to embodiments of the present disclosure. In one embodiment, the processor of the device may execute the called instructions, causing components of the device to perform functions corresponding to the instructions. In one embodiment, the processor may be a processor (210) according to embodiments of the present disclosure. The storage medium may refer to any type of recording medium that stores data and can be read by the device. The storage medium may include, for example, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, etc. In one embodiment, the storage medium may be memory (220). In one embodiment, the storage medium may be implemented in a distributed form, such as in a network-connected computer system. The software may be distributed and stored and executed in a computer system, etc. The storage medium may be a non-transitory storage medium. A non-transitory storage medium means a tangible medium regardless of whether data is stored semi-permanently or temporarily, and does not include a signal that is propagated transitorily.

[0115] While the technical concept of the present disclosure has been described through various embodiments, it should be understood that the technical concept of the present disclosure encompasses various substitutions, modifications, and variations that can be made within the scope of those skilled in the art to which the present disclosure pertains. Furthermore, it should be understood that such substitutions, modifications, and variations are encompassed within the scope of the appended claims.

Claims

1. In a product search method performed by an electronic device, A step of receiving an image corresponding to a product search query from a user terminal; In response to receiving the image, extracting at least one textual information and an image feature from the image, wherein the image feature includes information indicating a visual form of the image; A step of determining a first search result including at least one product corresponding to the image based on at least one extracted text information and analysis data stored in a memory, wherein the analysis data includes data to which a product, text information about the product, and an image feature are mapped; A step of determining a second search result including at least one product corresponding to the image based on at least one extracted image feature and the analysis data; A method comprising a step of determining a final search result based on the first search result and the second search result.

2. In paragraph 1, The step of determining the final search result is as follows: A method comprising the step of determining, as the final search result, a product that overlaps at least one product included in the first search result and at least one product included in the second search result.

3. In paragraph 1, The step of extracting at least one text information and image feature from the image is: A method comprising the step of extracting at least one piece of text information and the image feature using a machine learning model.

4. In paragraph 3, The step of extracting at least one text information and image feature from the image is: A method characterized in that at least one piece of text information is extracted from the image using natural language processing.

5. In paragraph 4, The step of extracting at least one piece of text information from the image using the above natural language processing is: A method characterized in that at least one piece of text information is extracted from the image using text mining.

6. In paragraph 1, The step of extracting at least one text information and image feature from the image is: Identify multiple products included in the above image, and A method further comprising the step of determining one product among the plurality of products to be searched.

7. In paragraph 6, The step of extracting at least one text information and image feature from the image is: A step of extracting a first text information group including at least one text information from a first product included in the image; and A method comprising the step of extracting a second text information group including at least one text information from a second product included in the image.

8. In paragraph 7, The step of extracting at least one text information and image feature from the image is: A step of outputting a message to the user terminal instructing the user to select one of the first product and the second product; A method comprising a step of determining to perform a search based on a group of text information corresponding to a selected product based on a user input for the user terminal.

9. In paragraph 6, The step of determining one product to search among the above multiple products is as follows: A method comprising the step of comparing the size of an area where a first product is displayed and the size of an area where a second product is displayed in the image, and determining a product with a larger displayed area as the product to be searched.

10. In paragraph 6, The step of determining one product to search among the above multiple products is as follows: A method comprising the step of comparing the resolution of an area where a first product is displayed and the resolution of an area where a second product is displayed in the image, and determining a product with a higher resolution of the displayed area as the product for which a search is to be performed.

11. In paragraph 1, The step of extracting at least one text information and image feature from the image is: A method further comprising the step of performing preprocessing on the image before extracting at least one text information and image feature from the image.

12. In paragraph 11, A method characterized in that the preprocessing of the above image includes image normalization, denoising, and feature enhancement.

13. In paragraph 1, The step of determining the final search result is as follows: A method further comprising the step of transmitting the final search result to the user terminal.

14. In paragraph 13, A method characterized in that it transmits at least one piece of text information together with the final search result to the user terminal.

15. In electronic devices, One or more processors, comprising one or more memories storing instructions executed by the one or more processors; An electronic device, wherein when the instructions are executed by the one or more processors, the one or more processors are configured to execute a method according to any one of claims 1 to 14.

16. In a non-transitory computer-readable recording medium, instructions are recorded that cause one or more processors to perform an operation when executed by one or more processors. A non-transitory computer-readable recording medium, wherein the instructions are configured to cause the one or more processors to execute a method according to any one of claims 1 to 14.

Images