Electronic device for providing image search result, and operation method thereof

Abstract

An electronic device for providing an image search result, and an operation method thereof are provided. The electronic device can acquire a natural language query for image search, identify, from the natural language query, a word or a phrase for indicating at least one from among an object action, relationship, attribute and quantity, acquire at least one alternative natural language query in which the identified word or phrase is substituted with an alternative word or an alternative phrase, and display an image search result by means of the at least one alternative natural language query.

Description

Electronic device providing image search results and method of operation thereof

[0001] The present disclosure relates to an electronic device for searching for images using natural language and providing search results, and a method of operation thereof. Specifically, the present disclosure discloses an electronic device and a method of operation thereof that receives a natural language query from a user, searches for at least one image matching the natural language query among a plurality of images stored in memory, and displays the search results.

[0002] As the frequency of taking photos with cameras on mobile devices such as smartphones increases, the number of images stored in memory storage is growing. Users can search for specific images among those already stored in the storage. Even if a user enters a search query into a photo application (e.g., a gallery), images matching the query may not be found. In the case of photo applications that perform keyword searches, if no search results are displayed, suggested search terms are displayed or alternative searches are performed. However, if a natural language query is entered, the photo application does not provide search results because it is difficult to provide suggested search terms or perform alternative searches. Reasons why image search may fail via natural language search may include, for example, typos or ambiguity in the search (e.g., photos that are good but bad, or sad but happy), the absence of matching images, or limitations of the search algorithm.

[0003] If search results are not provided, users lose the opportunity to search for images, user convenience decreases, and users may experience dissatisfaction.

[0004] One aspect of the present disclosure discloses a method for an electronic device to provide image search results. A method of operation of an electronic device according to one embodiment of the present disclosure may include the step of obtaining a natural language query for image search. The method of operation of the electronic device may include the step of identifying a word or phrase representing at least one of an action, relationship, attribute, or count of objects from the obtained natural language query. The method of operation of the electronic device may include the step of obtaining at least one alternative natural language query in which the identified word or phrase is replaced with an alternative word or alternative phrase. The method of operation of the electronic device may include the step of displaying image search results based on the at least one alternative natural language query obtained.

[0005] One aspect of the present disclosure discloses an electronic device that provides image search results. An electronic device according to one embodiment of the present disclosure may include a user input interface; a display; at least one processor comprising a processing circuit; and a memory that stores one or more instructions. By executing the one or more instructions individually or collectively by the at least one processor, the electronic device may obtain a natural language query for image search through the user input interface and identify a word or phrase representing at least one of an action, relationship, attribute, or count of objects from the obtained natural language query. By executing the one or more instructions individually or collectively by the at least one processor, the electronic device may obtain at least one alternative natural language query in which the identified word or phrase is replaced with an alternative word or alternative phrase. By executing the above one or more instructions individually or collectively by the above at least one processor, the electronic device can display image search results based on at least one acquired alternative natural language query on the display.

[0006] One aspect of the present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may include instructions readable by the electronic device to perform the operations of: acquiring a natural language query for image search; identifying a word or phrase representing at least one of an action, relationship, attribute, or count of objects from the acquired natural language query; acquiring at least one alternative natural language query in which the identified word or phrase is replaced with an alternative word or alternative phrase; and displaying an image search result based on the acquired at least one alternative natural language query.

[0007] The present disclosure can be easily understood from the combination of the following detailed description and the accompanying drawings, where reference numerals denote structural elements.

[0008] FIG. 1 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure that provides image search results and alternative natural language queries regarding a natural language query.

[0009] FIG. 2 is a flowchart illustrating a method in which an electronic device according to one embodiment of the present disclosure provides image search results.

[0010] FIG. 3 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to obtain an alternative natural language query from an input natural language query.

[0011] FIG. 4 is a block diagram illustrating the components of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 5 is a flowchart illustrating a method in which an electronic device according to one embodiment of the present disclosure identifies a replacement target word or replacement target phrase from a natural language query and obtains a replacement natural language query by replacing the replacement target word or replacement target phrase with a replacement search term.

[0013] FIG. 6 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure identifying the part of speech of vocabulary included in a natural language query.

[0014] FIG. 7 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to obtain alternative search terms for a natural language query.

[0015] FIG. 8 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to obtain an alternative search query by changing the arrangement order of vocabulary included in a natural language query.

[0016] FIG. 9 is a flowchart illustrating a method in which an electronic device according to one embodiment of the present disclosure identifies the part of speech and lexical hierarchy of a natural language query and obtains an alternative natural language query based on the identification result.

[0017] FIG. 10 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure to identify the part of speech and lexical hierarchy of a natural language query and to obtain an alternative natural language query based on the identification result.

[0018] FIG. 11 is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure displaying image search results.

[0019] FIG. 12a is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure displaying a user interface (UI) for providing image search results by alternative natural language query.

[0020] FIG. 12b is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure displaying a user interface (UI) for providing image search results by alternative natural language query.

[0021] FIG. 12c is a diagram illustrating the operation of an electronic device according to one embodiment of the present disclosure displaying a user interface (UI) for providing image search results by alternative natural language query.

[0022] The terms used in the embodiments of this specification have been selected to be as widely used as possible, taking into account the functions of the present disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description section of the relevant embodiments. Therefore, terms used in this specification should be defined not merely by their names, but based on their meanings and the overall content of the present disclosure.

[0023] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art as described in this specification.

[0024] Throughout this disclosure, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "...module," etc., as used in this specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0025] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “capable of” in conjunction with other devices or components. For example, the phrase “processor configured to perform A, B, and C” may mean a dedicated processor for performing the said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0026] In addition, when a component is described in the present disclosure as being "connected" or "connected" to another component, it should be understood that the component may be directly connected to or directly connected to the other component, but unless otherwise specifically stated, it may also be connected or connected through another component in between.

[0027] All functions or operations described in this disclosure may be processed individually by a single processor and / or collectively by a plurality of processors. A single processor or a combination of a plurality of processors may include circuitry that performs processing, such as an Application Processor (AP), Communication Processor (CP), Graphical Processing Unit (GPU), Neural Processing Unit (NPU), Microprocessor Unit (MPU), System on Chip (SoC), Integrated Chip (IC), etc.

[0028] It should be understood that the blocks and combinations of flowcharts in the flowcharts illustrated in the present disclosure may be performed by one or more computer programs comprising computer-executable instructions. The one or more computer programs may be stored all in a single memory or may be divided and stored in multiple different memories.

[0029] In the present disclosure, a 'search query' refers to a signal requested by a user to search for an object they wish to search for from storage or a database in which a plurality of images are stored. In one embodiment of the present disclosure, the search query may include keywords representing the content to be searched or may be composed of natural language.

[0030] In the present disclosure, a "natural language query" refers to a search query composed of natural language, which is the language people use in everyday life. A natural language query may be composed of one or more vocabulary words. For example, a natural language query may include vocabulary words representing objects, actions, relationships, attributes, or counts.

[0031] In the present disclosure, 'vocabulary' means a unit for expressing meaning, such as a word or phrase used in a sentence or syntax. Vocabulary may mean, for example, words, idioms, phrases, phrasal verbs, etc.

[0032] In the present disclosure, functions related to 'Artificial Intelligence' are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or AI-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or AI models stored in memory. Alternatively, if the one or more processors are AI-dedicated processors, the AI-dedicated processors may be designed with a hardware structure specialized for processing a specific AI model.

[0033] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform a desired characteristic (or objective) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0034] In the present disclosure, an 'artificial intelligence model' may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values ​​and performs neural network operations through operations between the results of operations of a previous layer and the plurality of weights. The plurality of weights possessed by the plurality of neural network layers may be optimized by the learning results of the artificial intelligence model. For example, the plurality of weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network model may include a Deep Neural Network (DNN), such as a Convolutional Neural Network, a Recurrent Neural Network, a Restricted Boltzmann Machine, a Deep Belief Network, a Bidirectional Recurrent Deep Neural Network, or Deep Q-Networks, but is not limited to the examples described above.

[0035] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein.

[0036] Embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0037] FIG. 1 is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure providing an image search result (20) and an alternative natural language query (31, 32, 33) regarding a natural language query (10).

[0038] Referring to FIG. 1, the electronic device (100) may be implemented as a smartphone. However, it is not limited to what is shown in the drawings, and in one embodiment of the present disclosure, the electronic device (100) may be a mobile device such as a tablet PC, a laptop computer, a digital camera, an e-book terminal, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation device, or an MP3 player. In one embodiment of the present disclosure, the electronic device (100) may be a home appliance such as a TV, an air conditioner, a robot vacuum cleaner, or a clothing care device. In one embodiment of the present disclosure, the electronic device (100) may be implemented as a wearable device such as a smart watch, a glasses-type augmented reality device (e.g., AR glasses), a head-mounted device (HMD), or a body-attached device (e.g., a skin pad).

[0039] Referring to FIG. 1, an electronic device (100) may receive a natural language query (10) regarding an image to be searched from a user. In one embodiment of the present disclosure, the electronic device (100) may receive user input in which a natural language query (10) composed of natural language commonly used by people is entered via a touchscreen or keyboard. However, it is not limited thereto, and the electronic device (100) may also receive a voice signal of a search query composed of natural language spoken by a user via a microphone.

[0040] As a natural language query (10) is input, the electronic device (100) can display image search results (20) regarding images that match the natural language query among a plurality of images already stored in a storage space in memory (e.g., image storage (138, see FIG. 4)). The electronic device (100) can obtain at least one alternative natural language query (31, 32, 33) in which vocabulary such as words or phrases included in the natural language query (10) is replaced with other alternative search terms, and can display the obtained at least one alternative natural language query (31, 32, 33).

[0041] In one embodiment of the present disclosure, an electronic device (100) may identify a word or phrase representing at least one of the action, relationship, attribute, or number of objects from text constituting an input natural language query (10), and obtain at least one alternative natural language query (31, 32, 33) by replacing the identified word or phrase with an alternative search term (e.g., alternative word or alternative phrase). In the embodiment illustrated in FIG. 1, when the natural language query (10) input by a user is "man next to pig in Vietnam," the electronic device (100) may recognize objects included in the text constituting the natural language query (10), e.g., 'man', 'pig', or 'Vietnam', and identify 'next to' and 'in' representing the positional relationship of the recognized objects. The electronic device (100) can obtain a first alternative natural language query (31) and a second alternative natural language query (32) by replacing the identified 'next to' with an alternative search term representing an action, such as 'riding' or 'feeding'. In one embodiment of the present disclosure, the electronic device (100) may obtain a third alternative natural language query (33) by replacing 'pig' among the objects with an alternative search term, such as 'buffalo'.

[0042] The electronic device (100) can display image search results (41, 42, 43) for each of at least one alternative natural language query (31, 32, 33). In the embodiment illustrated in FIG. 1, the electronic device (100) can display images (41) retrieved by the first alternative natural language query (31) among a plurality of images already stored in a storage space in memory. Likewise, the electronic device (100) can display images (42, 43) retrieved by the second alternative natural language query (32) and the third alternative natural language query (33), respectively, among a plurality of images.

[0043] The electronic device (100) can display image search results (41, 42, 43) based on at least one alternative natural language query (31, 32, 33) together with image search results (20) based on a natural language query (10) entered by a user. In one embodiment of the present disclosure, the electronic device (100) can obtain an alternative natural language query when there is no image searched by the natural language query (10) among a plurality of previously stored images. In this case, the image search results (20) based on the natural language query (10) do not include the searched image, and the electronic device (100) can display a message such as, for example, "No photo matching the search term found." on the image search results (20).

[0044] FIG. 2 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure provides an image search result.

[0045] FIG. 3 is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure obtaining an alternative natural language query (320) from an input natural language query (300).

[0046] Hereinafter, with reference to FIG. 2 and FIG. 3 together, the function and / or operation of the electronic device (100) of the present disclosure obtaining an alternative natural language query from a natural language query input by a user and providing an image search result based on the alternative natural language query will be described in detail.

[0047] In step S210 of FIG. 2, the electronic device (100) obtains a natural language query for image search from a user. The electronic device (100) may receive user input that inputs a natural language query composed of natural language through keyboard input, mouse input, or touch input via a touch screen. However, it is not limited thereto, and in one embodiment of the present disclosure, the electronic device (100) may receive the natural language query by receiving a voice signal composed of natural language spoken by the user through a microphone. Referring together to operation 1 of the embodiment illustrated in FIG. 3, the electronic device (100) may receive a natural language query (300) "man next to black pig in Vietnam" through the user's touch input or voice input.

[0048] Referring again to FIG. 2, in step S220, the electronic device (100) identifies a word or phrase representing at least one of the action, relationship, attribute, or number of objects from an acquired natural language query. In the present disclosure, 'action' may mean an action or movement of an object (e.g., jumping, running, waving, etc.). In the present disclosure, 'relationship' may mean a position (e.g., front, back, etc.) or direction (e.g., right, left, etc.) in which objects are placed on an image. In the present disclosure, 'attribute' may mean the color (e.g., white, red, blue, etc.) or pattern (e.g., grid, stripe, etc.) of the objects. The electronic device (100) analyzes an input natural language query to identify the parts of speech of vocabulary included in the text constituting the natural language query, and based on the identified parts of speech, determines a word or phrase representing at least one of the behavior, relationship, attribute, or number of objects as a replacement target word or replacement target phrase. In one embodiment of the present disclosure, the electronic device (100) performs Part-Of-Speech tagging on the text constituting the natural language query to identify the parts of speech of vocabulary included in the natural language query.

[0049] Referring together with operation ② of the embodiment illustrated in FIG. 3, the electronic device (100) can analyze a natural language query (300) through POS tagging to identify the parts of speech of a plurality of vocabulary words (301 to 306) included in the text constituting the natural language query (300). For example, the electronic device (100) can recognize, as a result of POS analysis, that the first vocabulary word ('man'), the fourth vocabulary word ('pig'), and the sixth vocabulary word ('Vietnam') are nouns, the second vocabulary word ('next to') and the fifth vocabulary word ('in') are prepositions, and the third vocabulary word ('black') is an adjective. The electronic device (100) can determine, based on the part-of-speech identification results of the vocabulary (301 to 306), the second vocabulary (302, 'next to') and the fifth vocabulary (305, 'in'), which are prepositions indicating the relationship of objects among the vocabulary (301 to 306), and the third vocabulary (303, 'black'), which are adjectives indicating the attributes of objects, as replacement target words or replacement target phrases.

[0050] In one embodiment of the present disclosure, when receiving voice input by a user's speech, the electronic device (100) performs automatic speech recognition (ASR) to convert the acoustic signal of the received voice input into text, and analyzes the text through a natural language understanding (NLU) model to identify the part of speech or lexical hierarchy of vocabulary included in the natural language query. Based on the identified part of speech or lexical hierarchy, the electronic device (100) can determine a replacement target word or replacement target phrase from the natural language query.

[0051] In step S230 of FIG. 2, the electronic device (100) obtains at least one alternative natural language query in which an identified word or phrase is replaced with an alternative word or alternative phrase. The electronic device (100) may generate at least one alternative natural language query by replacing a word or phrase determined as a replacement target with an alternative word or alternative phrase as vocabulary representing at least one of the behavior, relationship, attribute, or number of objects from the natural language query. In one embodiment of the present disclosure, the electronic device (100) may identify at least one image among a plurality of images stored in a storage space in memory whose similarity to the natural language query is greater than or equal to a preset threshold, and may input the identified at least one image into a captioning model or a large vision language model (LVLM) to obtain an alternative search term from the natural language output as an inference result. The alternative search term may consist of a word or phrase. However, it is not limited thereto, and in one embodiment of the present disclosure, the electronic device (100) may obtain an alternative search term based on a tag for each of the plurality of images stored. The electronic device (100) can generate at least one alternative natural language query by replacing a target word or target phrase with an acquired alternative search term.

[0052] Referring together to operation ③ of the embodiment illustrated in FIG. 3, the electronic device (100) can obtain 'riding', 'feeding', and 'in front of' as alternative search terms (312) for the second vocabulary (302) determined as the replacement target phrase. Additionally, the electronic device (100) can obtain 'white', 'brown', and 'stripe' as alternative search terms (313) for the third vocabulary (303) determined as the replacement target word, and 'outside' and 'at' as alternative search terms (315) for the fifth vocabulary (305), respectively. However, the alternative search terms (312, 313, 315) illustrated in FIG. 3 are exemplary, and the alternative search terms of the present disclosure are not limited to the examples illustrated in FIG. 3.

[0053] Referring to operation ④ of FIG. 3, the electronic device (100) can obtain at least one alternative natural language query (320) by replacing a target word or phrase with an alternative search term. For example, the electronic device (100) can obtain an alternative natural language query "man riding black pig in Vietnam" by replacing the second vocabulary (302) in the natural language query (300) with 'riding' among the alternative search terms. For example, the electronic device (100) can obtain an alternative natural language query "man next to white pig in Vietnam" by replacing the third vocabulary (303) in the natural language query (300) with 'white' among the alternative search terms. The electronic device (100) can also obtain alternative natural language queries by replacing multiple words or phrases with alternative search terms. For example, the electronic device (100) can obtain an alternative natural language query "man riding white pig at Vietnam" by replacing the second vocabulary (302), third vocabulary (303), and fifth vocabulary (305) of the natural language query (300) with 'riding', 'white', and 'at', respectively.

[0054] Referring again to FIG. 2, in step S240, the electronic device (100) displays image search results based on at least one alternative natural language query obtained. In one embodiment of the present disclosure, the electronic device (100) may display at least one alternative natural language query and images searched by at least one alternative natural language query together with at least one alternative natural language query.

[0055] In one embodiment of the present disclosure, the electronic device (100) may display together an image search result based on at least one alternative natural language query and an image search result based on a natural language query entered by a user. The electronic device (100) may obtain an alternative natural language query when there is no image searched by the natural language query among a plurality of previously stored images. In this case, the image search result based on the natural language query does not include the searched image, and the electronic device (100) may display a message such as, for example, "No photo matching the search term was found." as an image search result.

[0056] A user may want to search for a specific image among multiple images stored in the memory of an electronic device (100). However, even if the user enters a search query composed of natural language into a photo application (e.g., a gallery), an image matching the search query may not be found. Cases where image search is not possible through natural language search may include, for example, typos or ambiguity in the search (e.g., photos that are good but bad, sad but happy), the absence of matching images, or limitations of the search algorithm. If no search results are provided, the user loses the opportunity to search for images, the reliability of natural language search decreases, and the user experiences dissatisfaction. Consequently, the user may give up on image search, leading to a decrease in the usage rate of image search using natural language.

[0057] The present disclosure aims to provide an electronic device (100) and a method of operation thereof, which obtains an alternative natural language query using an alternative search term and provides an image search result based on the alternative natural language query as a fallback for cases where an image matching a natural language query entered by a user is not found.

[0058] An electronic device (100) according to an embodiment illustrated in FIGS. 1 to 3 may obtain at least one alternative natural language query by replacing a word or phrase representing at least one of the behavior, relationship, attribute, or number of objects included in a natural language query entered by a user with an alternative word or alternative phrase, and may display an image search result based on the alternative natural language query. An electronic device (100) according to an embodiment of the present disclosure may provide a continuous search experience by providing image search results based on the alternative natural language query even when a user searches for images by entering a natural language query that is ambiguous or inaccurate in meaning, thereby improving the reliability of the search based on the natural language query and providing a technical effect of increasing user convenience. In addition, an electronic device (100) according to an embodiment of the present disclosure may provide a technical effect of increasing the usage time of a photo application (e.g., gallery) by providing at least one alternative natural language query.

[0059] The method of providing image search results through alternative natural language queries by the electronic device (100) according to the embodiment illustrated in FIGS. 1 to 3 is not limited to searching for images in a photo application, but can be applied in the same way to other search services that include images, such as e-commerce.

[0060] FIG. 4 is a block diagram illustrating the components of an electronic device (100) according to one embodiment of the present disclosure.

[0061] Referring to FIG. 4, the electronic device (100) may include a user input interface (110), a processor (120), a memory (130), and a display (140). The user input interface (110), the processor (120), the memory (130), and the display (140) may each be electrically and / or physically connected to each other. FIG. 4 illustrates only essential components for explaining the function and / or operation of the electronic device (100), and the components included in the electronic device (100) are not limited to those illustrated in FIG. 4. In one embodiment of the present disclosure, the electronic device (100) may further include a camera configured to capture an object to obtain a still image or video, comprising a lens module, an image sensor, and an image processing module. In one embodiment of the present disclosure, when the electronic device (100) is implemented as a portable device or a mobile device, the electronic device (100) may further include a battery that supplies driving power to a user input interface (110), a processor (120), and a display (140).

[0062] The user input interface (110) is configured to receive user input, such as a search query, from a user. The user input interface (110) may include a touch panel (112) and a microphone (114). The touch panel (112) may receive touch input from the user. The touch panel (112) may be integrated with a display (140) and configured as a touchscreen that displays a graphical user interface (GUI) receiving touch input.

[0063] The user input interface (110) can receive user input that inputs a natural language query consisting of a word, phrase, and / or sentence to be searched. For example, the user input interface (110) can receive a search query entered via a keyboard or touchscreen.

[0064] A microphone (114) can receive voice input from a user speaking a natural language query. The microphone (114) can receive voice input spoken by the user and obtain a voice signal from the received voice input. A processor (120) can convert the sound components of the voice input received through the microphone (114) into an acoustic signal and obtain a voice signal by removing noise (e.g., non-voice components) from the acoustic signal.

[0065] However, the user input interface (110) is not limited to a touch panel (112) and a microphone (114), and may be implemented with hardware components such as a keyboard, key pad, mouse, trackball, jog dial, jog switch, or touch pad.

[0066] The processor (120) can execute one or more instructions of a program stored in memory (130). The processor (120) may be composed of hardware components that perform arithmetic, logic, and input / output operations and image processing. Although the processor (120) is depicted as a single element in FIG. 4, it is not limited thereto. In one embodiment of the present disclosure, the processor (120) may be composed of one or more elements.

[0067] The processor (120) may include a processing circuit and / or a plurality of processors. For example, the term "processor" as used in the present disclosure, including in the claims, may include at least one processor and various processing circuits. In at least one processor, one or more processors may be configured to perform the various functions and / or operations described in the present disclosure in a distributed manner, individually and / or collectively. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms cover, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor can perform all functions. Additionally, at least one processor may include a combination of processors performing various functions of the disclosed functions in a distributed manner. At least one processor may execute program code or instructions to achieve or perform various functions.

[0068] One or more processors included in the processor (120) may be circuitry such as a system on chip (SoC) or an integrated circuit (IC). The processor (120) may be implemented as a general-purpose processor such as a CPU (Central Processing Unit), AP (Application Processor), or DSP (Digital Signal Processor), a graphics-dedicated processor such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit), or an artificial intelligence-dedicated processor such as an NPU (Neural Processing Unit). The processor (120) may be controlled to process input data according to predefined operation rules or an artificial intelligence model. Alternatively, if the processor (120) is an artificial intelligence-dedicated processor, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0069] The memory (130) may be composed of at least one type of storage medium, such as a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), or an optical disk.

[0070] The memory (130) may store instructions related to functions and / or operations in which the electronic device (100) identifies a replacement target word or replacement target phrase from a natural language query, replaces the replacement target word or replacement target phrase with a replacement search term to obtain at least one replacement natural language query, and displays an image search result based on at least one replacement natural language query. In one embodiment of the present disclosure, the memory (130) may store at least one of instructions, an algorithm, a data structure, program code, and an application program that can be read by the processor (120). The instructions, algorithm, data structure, and program code stored in the memory (130) may be implemented in a programming or scripting language such as, for example, C, C++, Java, assembler, etc.

[0071] The memory (130) may store instructions, algorithms, data structures, or program codes related to a natural language query identification module (132), an alternative natural language query acquisition module (134), and an image search module (136). A 'module' included in the memory (130) refers to a unit that processes a function or operation performed by the processor (120), and this may be implemented as software such as instructions, algorithms, data structures, or program code. In one embodiment of the present disclosure, the memory (130) may include an image storage (138) that stores a plurality of images.

[0072] Functions and / or operations of the electronic device (100) can be performed by the processor (120) executing instructions or program codes stored in memory (130). Hereinafter, the functions and / or operations performed by the processor (120) by executing instructions or program codes of each of the plurality of modules stored in memory (130) will be described in detail.

[0073] The natural language query identification module (132) is composed of instructions or program code for executing a function and / or operation to identify a word or phrase representing at least one of the action, relationship, attribute, or count of objects from a natural language query input by a user. In one embodiment of the present disclosure, the processor (120) can identify a replacement target word or replacement target phrase, such as the action, relationship, attribute, or count of objects, from a natural language query received through a user input interface (110) by executing the program code or instructions of the natural language query identification module (132). In the present disclosure, 'action' may mean an action or operation of an object (e.g., jumping, running, waving, etc.). In the present disclosure, 'relationship' may mean a position (e.g., front, back, etc.) or direction (e.g., right, left, etc.) in which objects are placed on an image. In the present disclosure, 'attribute' may refer to the color (e.g., white, red, blue, etc.) or pattern (e.g., grid, stripe, etc.) of objects.

[0074] In one embodiment of the present disclosure, the processor (120) performs Part-Of-Speech tagging on text constituting a natural language query to identify the parts of speech of vocabulary included in the natural language query, and based on the identified parts of speech, determines a word or phrase representing at least one of the behavior, relationship, attribute, or number of objects as a replacement target word or replacement target phrase. The processor (120) may, for example, determine verbs, adjectives, and prepositions among the identified parts of speech of words or phrases included in the natural language query as replacement target words or replacement target phrases.

[0075] When receiving voice input from a user speaking natural language through a microphone (114), the processor (120) can perform automatic speech recognition (ASR) to convert the acoustic signal of the received voice input into text, and analyze the text through a natural language understanding (NLU) model to identify the part of speech or lexical hierarchy of vocabulary included in the natural language query. Based on the identified part of speech or lexical hierarchy, the processor (120) can determine the replacement target word or replacement target phrase from the natural language query.

[0076] The replacement natural language query acquisition module (134) is composed of instructions or program code for executing a function and / or operation to acquire at least one replacement natural language query by replacing a word or phrase identified from a natural language query with a replacement word or replacement phrase. By executing the program code or instructions of the replacement natural language query acquisition module (134), the processor (120) can acquire at least one replacement natural language query by replacing a word or phrase representing at least one of the behavior, relationship, attribute, or number of objects included in the natural language query with a replacement word or replacement phrase.

[0077] The processor (120) can obtain alternative search terms for replacing a word or phrase to be replaced from a plurality of images stored in the image storage (138). In one embodiment of the present disclosure, the processor (120) can obtain a first embedding vector by vector embedding text constituting an input natural language query, and obtain a plurality of second embedding vectors by vector embedding each of the plurality of images stored in the image storage (138). The processor (120) can calculate the similarity between the first embedding vector and the plurality of second embedding vectors, and compare the calculated similarity with a preset threshold to identify at least one image among the plurality of images that has a similarity exceeding the preset threshold. The processor (120) can input the identified at least one image into a captioning model or a large vision language model (LVLM) and obtain alternative search terms from the natural language output as an inference result. The alternative search terms may consist of a word or a phrase. A specific embodiment in which the processor (120) obtains a replacement search term for replacing a replacement target word or replacement target phrase of a natural language query from a plurality of images will be described in detail in FIG. 7.

[0078] In one embodiment of the present disclosure, the processor (120) may obtain alternative search terms based on tags for each of a plurality of images already stored in the image storage (138). The processor (120) may identify the tag attached to the image with the largest number of tags among the tags for each of the plurality of images and determine the identified tag as an alternative search term. A specific embodiment in which the processor (120) obtains alternative search terms using the tags of each of the plurality of images will be described in detail in FIG. 10.

[0079] The processor (120) can generate at least one replacement natural language query by replacing a replacement target word or replacement target phrase using the acquired replacement search term.

[0080] The image search module (136) is composed of instructions or program code for executing a function and / or operation of performing image search using a natural language query or an alternative natural language query. The processor (120) can perform image search using natural language by executing the program code or instructions of the image search module (136). In one embodiment of the present disclosure, the processor (120) can obtain at least one third embedding vector by vector embedding text constituting at least one alternative natural language query. The processor (120) can measure similarity by comparing at least one third embedding vector with a plurality of second embedding vectors embedded by each of a plurality of images already stored in the image storage (138). The processor (120) can calculate the similarity between the plurality of second embedding vectors and at least one third embedding vector using, for example, a cosine similarity or Euclidean similarity measurement method. However, it is not limited thereto, and the processor (120) may calculate the similarity between a plurality of second embedding vectors and at least one third embedding vector using, for example, Jaccard similarity, Manhattan similarity, or other known similarity measurement algorithms. The processor (120) may compare the calculated similarity with a preset threshold and identify at least one image having a similarity exceeding the threshold and output it as an image search result. Since the method by which the processor (120) performs an image search for a natural language query is the same as the method described above except that the target for text embedding is a natural language query, a redundant description is omitted.

[0081] Image storage (138) is a storage device within a memory (130) that stores image data including a plurality of images. In one embodiment of the present disclosure, the image storage (138) may store a tag for each of the plurality of images together with the plurality of images.

[0082] The image storage (138) may be composed of non-volatile memory. Non-volatile memory refers to a storage medium that stores and maintains information even when power is not supplied, and can use the stored information again when power is supplied. Non-volatile memory may include, for example, at least one of flash memory, hard disk, SSD (Solid State Drive), multimedia card micro type, card type memory (e.g., SD or XD memory), ROM (Read Only Memory; ROM), magnetic memory, magnetic disk, and optical disk.

[0083] Although the image storage (138) in FIG. 4 is depicted as a component included within the memory (130), the present disclosure is not limited to what is shown in the drawings. In one embodiment of the present disclosure, the image storage (138) may be configured as a database within the electronic device (100), which is a component separate from the memory (130). However, it is not limited thereto, and in one embodiment of the present disclosure, the image storage (138) may be implemented as a web storage or cloud server that is accessible via a network and performs a storage function. In this case, the electronic device (100) further includes a communication interface configured to perform wired or wireless data communication, and can perform data transmission and reception by establishing a communication connection with the web storage or cloud server through the communication interface. The processor (120) can access a plurality of images from the web storage or cloud server and search for at least one image among the plurality of images that matches an alternative natural language query.

[0084] The display (140) is configured to display image search results by natural language query and alternative natural language query under the control of the processor (120). The display (140) may be implemented as at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.

[0085] The processor (120) may display at least one image retrieved by an alternative natural language query on the display (140) along with the image search results by the natural language query. In one embodiment of the present disclosure, the processor (120) may display the number of images retrieved by the alternative natural language query through the display (140). Specific embodiments in which the image search results by the alternative natural language query are displayed through the display (140) will be described in detail in FIG. 11.

[0086] In one embodiment of the present disclosure, the display (140) may display a drop-down menu user interface for receiving user input to replace words or phrases included in an alternative natural language query with other alternative search terms. The drop-down menu UI may include at least one alternative search term candidate for each word or phrase included in the alternative natural language query. Specific embodiments of how the display (140) displays the drop-down menu UI will be described in detail in FIGS. 12a through 12c.

[0087] FIG. 5 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure identifies a replacement target word or replacement target phrase from a natural language query and obtains a replacement natural language query by replacing the replacement target word or replacement target phrase with a replacement search term.

[0088] Steps S510 and S520 illustrated in FIG. 5 are operations that embody the operation of step S220 of FIG. 2. Steps S530 to S550 illustrated in FIG. 5 are operations that embody the operation of step S230 of FIG. 2. After step S550 of FIG. 5 is performed, step S240 of FIG. 2 may be performed.

[0089] In step S510, the electronic device (100) performs Part-Of-Speech tagging on a natural language query to identify the parts of speech of vocabulary included in the natural language query. In the present disclosure, 'POS tagging' refers to the operation of analyzing text to identify the parts of speech of vocabulary included in the text.

[0090] FIG. 6 is a diagram illustrating an operation in which an electronic device (100) according to one embodiment of the present disclosure identifies the part of speech of vocabulary included in a natural language query (600). Referring to step S510 of FIG. 5 in conjunction with the embodiment illustrated in FIG. 6, the POS analyzer (610) may be composed of instructions, program code, or algorithms for executing a function and / or operation of identifying the part of speech of vocabulary (e.g., words or phrases) included in the text by analyzing the input text. The POS analyzer (610) may be included in a natural language query identification module (132, see FIG. 4) within a memory (130, see FIG. 4), but is not limited thereto. A processor (120, see FIG. 4) of the electronic device (100) may input the natural language query (600) into the POS analyzer (610) and identify the part of speech of vocabulary included in the natural language query (600) by analyzing the text of the natural language query (600) using the POS analyzer (610). For example, if the natural language query (600) is "man next to black pig in Vietnam," when the natural language query (600) is input into the POS analyzer (610), output values ​​can be obtained indicating that the first vocabulary (601, 'man'), the fourth vocabulary (604, 'pig'), and the sixth vocabulary (606, 'Vietnam') are nouns, the second vocabulary (602, 'next to') and the fifth vocabulary (605, 'in') are prepositions, and the third vocabulary (603, 'black') is an adjective. The processor (120) can identify 'man', 'pig', and 'Vietnam' as nouns, 'next to' and 'in' as prepositions, and 'black' as an adjective, respectively, through the output values ​​of the POS analyzer (610).

[0091] Referring again to FIG. 5, in step S520, the electronic device (100) determines a word or phrase representing at least one of the actions, relationships, attributes, or counts of objects based on the identified parts of speech as a replacement target word or replacement target phrase. In one embodiment of the present disclosure, the electronic device (100) may determine a verb, an adjective, and a preposition among the identified parts of speech of a word or phrase included in a natural language query as a replacement target word or replacement target phrase. Referring together to the embodiment illustrated in FIG. 6, the processor (120) may determine a second vocabulary (602, 'next to'), a fifth vocabulary (605, 'in'), which is a preposition representing the relationship of an object among the vocabularys (601 to 606), and a third vocabulary (603, 'black'), which is an adjective representing the attribute of an object, as a replacement target word or replacement target phrase based on the part of speech identification results of the vocabularys (601 to 606) included in the natural language query (600).

[0092] In step S530 of FIG. 5, the electronic device (100) identifies at least one image among a plurality of images that is similar to the embedding vector of a natural language query. In one embodiment of the present disclosure, the electronic device (100) may search for at least one image among a plurality of images stored in a storage space in memory that has a similarity to the natural language query greater than or equal to a preset threshold.

[0093] FIG. 7 is a diagram illustrating an operation in which an electronic device (100) according to an embodiment of the present disclosure obtains an alternative search term (720) for a natural language query (700). Referring to step S530 of FIG. 5 in conjunction with the embodiment illustrated in FIG. 7, the processor (120) uses the natural language query (700) to obtain at least one image (i) among a plurality of images (i1, i2, i3, ...) already stored in an image storage (138) in which the similarity with the natural language query (700) is higher than a preset threshold. k...can be searched. In one embodiment of the present disclosure, the processor (120) can obtain a first embedding vector by vector embedding text constituting a natural language query (700). The processor (120) can obtain a plurality of second embedding vectors by vector embedding each of a plurality of images (i1, i2, i3, ...) already stored in the image storage (138). The processor (120) can calculate the similarity between the first embedding vector and the plurality of second embedding vectors. The processor (120) can calculate the similarity between the first embedding vector and the plurality of second embedding vectors using, for example, a cosine similarity or Euclidean similarity measurement method. However, it is not limited thereto, and the processor (120) may calculate the similarity between a first embedding vector and a plurality of second embedding vectors using, for example, Jaccard similarity, Manhattan similarity, or other known similarity measurement algorithms. The processor (120) compares the calculated similarity with a preset threshold and at least one image (i) having a similarity that exceeds the threshold. k Can identify ).

[0094] Referring again to FIG. 5, in step S540, the electronic device (100) inputs at least one identified image into a captioning model or a large language model (LVLM) to obtain a replacement word or replacement phrase from the natural language output as an inference result. In this disclosure, the 'captioning model' is an artificial intelligence model trained to generate and output a natural language description of an input image using computer vision technology and natural language processing (NLP) technology when an image is input. In this disclosure, the 'large language model (LLM)' is a language model composed of an artificial neural network having numerous parameters (usually billions of weights or more), and is a model trained on a large amount of unlabeled text using self-supervised or semi-self-supervised learning methods. The large language model can analyze the input text to generate and output an answer, or perform creative writing, coding, etc. In the present disclosure, a 'large vision language model (LVLM)' is a model trained to simultaneously analyze and process text and images to output results, and, for example, when an image is input, it can generate and output a description of the input image.

[0095] Referring to step S540 in conjunction with the embodiment illustrated in FIG. 7, the processor (120) retrieves at least one image (i k() can be input into a captioning model or a large vision language model (710), and alternative search terms can be obtained as a result of performing inference using the captioning model or the large vision language model (710). In one embodiment of the present disclosure, 'alternative search terms' may mean a word or phrase included in the natural language output by the captioning model or the large vision language model (710). In the embodiment illustrated in FIG. 7, the processor (120) can obtain alternative search terms such as 'man', 'riding', 'black', or 'buffalo' from the natural language output by the captioning model or the large vision language model (710).

[0096] In step S550 of FIG. 5, the electronic device (100) substitutes the target word or target phrase using the acquired replacement word or replacement phrase. Referring together with the embodiment illustrated in FIG. 7, the processor (120) may substitute the target word or target phrase determined in step 520 using the acquired replacement search terms, e.g., 'man', 'riding', 'black', or 'buffalo'. Referring together with the embodiment illustrated in FIG. 6, the processor (120) may substitute the second vocabulary (602, 'next to') determined as the target word or target phrase with 'riding'.

[0097] The electronic device (100) can obtain at least one alternative natural language query by replacing the alternative target word or alternative target phrase with an alternative search term.

[0098] FIG. 8 is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure to obtain an alternative search query by changing the arrangement order of vocabulary included in a natural language query (800).

[0099] Referring to FIG. 8, a processor (120, see FIG. 4) of an electronic device (100) inputs a natural language query (800) input by a user into an alternative natural language query acquisition module (134), and by executing program code or commands of the alternative natural language query acquisition module (134), can change the order of vocabulary representing objects among the vocabulary included in the input natural language query (800). In one embodiment of the present disclosure, the processor (120) can perform POS tagging on the natural language query (800) to identify the part of speech of vocabulary (e.g., words or phrases) included in the text constituting the natural language query (800). Based on the identified part of speech, the processor (120) can recognize words representing objects included in the natural language query (800). The processor (120) can change the ordering of the words recognized as representing objects to obtain at least one alternative natural language query (810).

[0100] For example, if the natural language query (800) is "man next to girl with a dog," the processor (120) can identify the words 'man', 'girl', and 'dog' representing objects from the natural language query (800) by performing POS tagging on the natural language query (800) and analyzing the text constituting the natural language query (800). The processor (120) can generate at least one alternative natural language query (820) by changing the order of the identified 'man', 'girl', and 'dog'. For example, the processor (120) can generate an alternative natural language query "man next to dog with a girl" by changing the order of 'girl' and 'dog'. For example, the processor (120) can generate an alternative natural language query "girl next to man with a dog" by changing the order of 'man' and 'girl'.

[0101] When performing image search using a natural language query, if the arrangement order of words representing objects is changed, the vector value of the embedding vector may change. The electronic device (100) according to the embodiment illustrated in FIG. 8 can obtain an alternative natural language query (810) by changing the arrangement order of words representing objects included in the natural language query (800), thereby enabling images that are not searched by the natural language query (800) to be searched through the alternative natural language query (810). The electronic device (100) according to one embodiment of the present disclosure provides a continuous search experience to the user through the alternative natural language query (810) and provides a technical effect of improving the reliability of the natural language search.

[0102] FIG. 9 is a flowchart illustrating a method in which an electronic device (100) according to one embodiment of the present disclosure identifies the part of speech and lexical hierarchy of a natural language query and obtains an alternative natural language query based on the identification result.

[0103] Steps S910 and S920 illustrated in FIG. 9 are operations that embody the operation of step S220 of FIG. 2. Steps S930 and S940 illustrated in FIG. 9 are operations that embody the operation of step S230 of FIG. 2. After step S940 of FIG. 9 is performed, step S240 of FIG. 2 may be performed.

[0104] FIG. 10 is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure identifying the part of speech and lexical hierarchy structure (1030) of a natural language query (1000) and obtaining an alternative natural language query (1040) based on the identification result.

[0105] Hereinafter, with reference to FIG. 9 and FIG. 10 together, the function and / or operation of the electronic device (100) obtaining an alternative natural language query will be described in detail.

[0106] In step S910 of FIG. 9, the electronic device (100) performs Part-Of-Speech tagging on a natural language query to identify the part of speech and the lexical hierarchy of the natural language query. The electronic device (100) can analyze a natural language query entered by a user through POS tagging to identify the part of speech of multiple vocabularys included in the text constituting the natural language query, and identify the lexical hierarchy based on the identified part of speech. Referring together to the embodiment illustrated in FIG. 10, the processor (120, see FIG. 4) of the electronic device (100) can convert the natural language query (1000) into an embedding vector and use the converted embedding vector to search for a corresponding image among multiple images stored in an image storage (138, see FIG. 4) and output an image search result (1010).

[0107] Images that exceed a preset threshold for similarity with the embedding vector converted from the natural language query (1000) may not be found. That is, if no images are displayed in the image search results (1010), the processor (120) inputs the natural language query (1000) into the POS analyzer (1020) and can identify the parts of speech of the vocabulary included in the natural language query (1000) by analyzing the text constituting the natural language query (1000) using the POS analyzer (1020). For example, if the natural language query (1000) is "man next to pig in Vietnam," when the natural language query (1000) is input into the POS analyzer (1020), an output value can be obtained indicating that 'man', 'pig', and 'Vietnam' are nouns, and 'next to' and 'in' are prepositions. The processor (120) can identify 'man', 'pig', and 'Vietnam' as nouns and 'next to' and 'in' as prepositions, respectively, through the output values ​​of the POS analyzer (1020).

[0108] The processor (120) can identify a vocabulary hierarchy structure (1030), which is a structure between vocabularys based on the identified parts of speech. In the embodiment illustrated in FIG. 10, the vocabulary hierarchy structure (1030) may be a syntactic structure composed of a noun, a preposition, a noun, a preposition, and a noun.

[0109] Referring again to FIG. 9, in step S920, the electronic device (100) determines a replacement target word or replacement target phrase from a natural language query based on the identified part of speech and lexical hierarchy. In one embodiment of the present disclosure, the electronic device (100) may determine a replacement target word or replacement target phrase by part of speech while maintaining the lexical hierarchy. Referring together to the embodiment illustrated in FIG. 10, the processor (120) may determine from the lexical hierarchy (1030) words or phrases corresponding to nouns, prepositions, nouns, prepositions, and nouns, respectively, such as 'man', 'next to', 'pig', and 'Vietnam', as replacement target words or replacement target phrases.

[0110] In step S930 of FIG. 9, the electronic device (100) obtains alternative search terms based on tags for each of the previously stored multiple images. In one embodiment of the present disclosure, image storage (138, see FIG. 4) in memory (130, see FIG. 4) may store multiple images as well as tags for each of the multiple images. In the present disclosure, a 'tag' is information for distinguishing an image from another image and may include, for example, information regarding a person, animal, object, action, behavior, location, relationship, attribute, number, situation, or event. The electronic device (100) may obtain alternative search terms from tags for each of the multiple images. Referring together to the embodiment illustrated in FIG. 10, the processor (120) can obtain alternative search terms 'man', 'dog', and 'Seoul' from a first tag (tag1) for a first image (i1), obtain alternative search terms 'woman', 'cat', and 'Seoul' from a second tag (tag2) for a second image (i2), and obtain alternative search terms 'cat', 'person', and 'Japan' from a third tag (tag3) for a third image (i3).

[0111] In step S940 of FIG. 9, the electronic device (100) obtains at least one replacement natural language query by substituting a replacement target word or replacement target phrase with the obtained replacement search term. Referring together to the embodiment illustrated in FIG. 10, the processor (120) can obtain at least one replacement natural language query (1040) based on the replacement result by substituting a replacement target word or replacement target phrase using a replacement search term obtained from tags (tag1, tag2, tag3, ...) for a plurality of images (i1, i2, i3, ...). For example, the processor (120) can obtain a first replacement natural language query by substituting 'buffalo', which is determined as a replacement target word or replacement target phrase in the natural language query (1000) "man next to buffalo in Vietnam", with 'dog' obtained from the first tag (tag1) for the first image (i1), and substituting 'Vietnam' with 'Seoul' obtained from the first tag (tag1). For example, the processor (120) can obtain a second replacement natural language query by replacing 'man', which is determined as the replacement target word or replacement target phrase in the natural language query (1000) "man next to buffalo in Vietnam," with 'woman' obtained from the second tag (tag2) for the second image (i2), replacing 'buffalo' with 'cat' obtained from the second tag (tag2), and replacing 'Vietnam' with 'Seoul' obtained from the second tag (tag2). The processor (120) can obtain at least one replacement natural language query (1040) by replacing the replacement target word or replacement target phrase of the natural language query (1000) using replacement search terms obtained from a plurality of tags (tag1, tag2, tag3, ...) through the method described above.

[0112] An electronic device (100) according to the embodiment illustrated in FIGS. 9 and 10 can provide a fallback option for image search to a user by obtaining a fallback option from tags (tag1, tag2, tag3, ...) for a plurality of previously stored images (i1, i2, i3, ...) when no image is found as a result of an image search by a natural language query (1000, see FIG. 10), and by obtaining at least one fallback natural language query (1040) by replacing a target word or target phrase of the natural language query (1000) using the fallback option. Accordingly, an electronic device (100) according to one embodiment of the present disclosure provides a technical effect of improving the user's satisfaction with the search of a photo application (e.g., gallery) by using a fallback option obtained from tags (tag1, tag2, tag3, ...) for a plurality of previously stored images (i1, i2, i3, ...) in the user's device when the user has a photo they want to find but cannot remember exactly which photo it is.

[0113] FIG. 11 is a drawing illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure displaying image search results.

[0114] Referring to FIG. 11, the electronic device (100) can display image search results (1110) by a natural language query (1100) and image search results (1130-1, 1130-2, 1130-3) by at least one alternative natural language query (1120-1, 1120-2, 1120-3) on a display (140). The electronic device (100) can display image search results (1110) regarding images that match the natural language query (1100) among a plurality of images already stored in an image storage (138, FIG. 4) within a memory (130, FIG. 4) as the natural language query (1100) is input. For example, if no images are found by the natural language query (1100), the electronic device (100) may not display images in the image search results (1110) and may display a text message saying, "No photos matching the search term could be found. How about these photos in XX's gallery?"

[0115] The electronic device (100) can display at least one alternative natural language query (1120-1, 1120-2, 1120-3) and can display image search results (1130-1, 1130-2, 1130-3) by at least one alternative natural language query (1120-1, 1120-2, 1120-3). In one embodiment of the present disclosure, the electronic device (100) can display the number of images searched by at least one alternative natural language query (1120-1, 1120-2, 1120-3). For example, the number of images searched by the first alternative natural language query (1120-1) is 5, and in this case, the electronic device (100) can display the number '5' together with the first alternative natural language query (1120-1). For example, the number of images retrieved by the second alternative natural language query (1120-2) is 8, and in this case, the electronic device (100) can display the number '8' along with the second alternative natural language query (1120-2).

[0116] FIG. 12a is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure displaying a user interface (UI) for providing image search results by an alternative natural language query (1200).

[0117] Referring to FIG. 12a, the electronic device (100) can display image search results by natural language query and image search results by at least one alternative natural language query (1200) on the display (140). As a natural language query is input, the electronic device (100) can display image search results regarding images that match the natural language query among a plurality of images stored in image storage (138, FIG. 4) within memory (130, FIG. 4). For example, if no images are found by the natural language query, the electronic device (100) does not display images in the image search results and can display a text message saying, "No photos matching the search term could be found. How about these photos in XX's gallery?"

[0118] The electronic device (100) may display a drop-down menu user interface (1220) for receiving user input to replace a replacement target word or replacement target phrase included in the replacement natural language query (1200) with another replacement search term. The drop-down menu UI (1220) may include at least one replacement search term candidate for each replacement target word or replacement target phrase included in the replacement natural language query (1200). As in the embodiment illustrated in FIG. 12a, when the replacement natural language query (1200) is "man next to buffalo in Vietnam" and the replacement target phrase (1210) is 'next to', the electronic device (100) may display a drop-down menu UI (1220) including 'riding', 'feeding', and 'in front of' as replacement search term candidates for 'next to'.

[0119] The electronic device (100) can receive user input selecting one of alternative search term candidates, e.g., 'riding', 'feeding', and 'in front of', through a dropdown menu UI (1220). The electronic device (100) can obtain additional alternative natural language queries by using the alternative search term candidate selected by the user input to replace the replacement target phrase (1210), e.g., 'next to'.

[0120] FIG. 12b is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure displaying a user interface (UI) for providing image search results by an alternative natural language query (1200).

[0121] Referring to FIG. 12b, the electronic device (100) displays an alternative natural language query (1200) and a dropdown menu UI (1220), and the dropdown menu UI (1220) may include a numeric UI (1230-1, 1230-2, 1230-3) indicating the number of images searched by each of a plurality of alternative search term candidates (1220-1, 1220-2, 1220-3). Since the embodiment illustrated in FIG. 12b is identical to FIG. 12a except that the dropdown menu UI (1220) includes a numeric UI (1230-1, 1230-2, 1230-3), a redundant description is omitted.

[0122] The electronic device (100) can count the number of images searched by an alternative natural language query obtained when a plurality of alternative search term candidates (1220-1, 1220-2, 1220-3) included in a dropdown menu UI (1220) each substitute a target phrase (1210). The electronic device (100) can display a numeric UI (1230-1, 1230-2, 1230-3) indicating the number of counted images together with the corresponding plurality of alternative search term candidates (1220-1, 1220-2, 1220-3). For example, when the target phrase (1210) 'next to' is replaced with 'riding', which is the first alternative search term candidate (1220-1) among multiple alternative search term candidates (1220-1, 1220-2, 1220-3), the number of images searched by the acquired alternative natural language query may be 8. The electronic device (100) may display a first numeric UI (1230-1) representing the number '8'. For example, when the target phrase (1210) 'next to' is replaced with 'feeding', which is the second alternative search term candidate (1220-2), the number of images searched by the acquired alternative natural language query is 3, and the electronic device (100) may display a second numeric UI (1230-2) representing the number '3'.

[0123] FIG. 12c is a diagram illustrating the operation of an electronic device (100) according to one embodiment of the present disclosure displaying a user interface (UI) for providing image search results by an alternative natural language query (1200).

[0124] Referring to FIG. 12c, the electronic device (100) displays an alternative natural language query (1200) and a dropdown menu UI (1220), and the dropdown menu UI (1220) includes a numeric UI (1230-1, 1230-2, 1230-3) indicating the number of images searched by each of the multiple alternative search term candidates (1220-1, 1220-2, 1220-3), and the sorting order of the multiple alternative search term candidates (1220-1, 1220-2, 1220-3) can be changed based on the number of images searched. The embodiment illustrated in FIG. 12c is identical to FIG. 12a and FIG. 12b except that the sorting order of the multiple alternative search term candidates (1220-1, 1220-2, 1220-3) in the dropdown menu UI (1220) is changed, so a redundant description is omitted.

[0125] The electronic device (100) counts the number of images searched by a replacement natural language query obtained when a plurality of replacement search term candidates (1220-1, 1220-2, 1220-3) included in a dropdown menu UI (1220) each replace a replacement target phrase (1210), and can display the plurality of replacement search term candidates (1220-1, 1220-2, 1220-3) sorted in descending order based on the number of images counted. In the embodiment illustrated in FIG. 12c, the number of images searched by the alternative natural language query obtained as a result of replacing the target phrase (1210) using the first alternative search term candidate (1220-1) 'riding' is 8, the number of images searched by the alternative natural language query obtained as a result of replacing the target phrase (1210) using the second alternative search term candidate (1220-2) 'feeding' is 3, and the number of images searched by the alternative natural language query obtained as a result of replacing the target phrase (1210) using the third alternative search term candidate (1220-3) 'in front of' is 5. In this case, the electronic device (100) can display multiple alternative search term candidates (1220-1, 1220-2, 1220-3) arranged in order of the number of images searched: a first alternative search term candidate (1220-1) with 8 images searched, a third alternative search term candidate (1220-3) with 5 images searched, and a second alternative search term candidate (1220-2) with 3 images searched.

[0126] One aspect of the present disclosure discloses a method in which an electronic device (100) provides image search results. A method of operation of an electronic device (100) according to one embodiment of the present disclosure may include a step (S210) of obtaining a natural language query for image search. A method of operation of the electronic device (100) may include a step (S220) of identifying a word or phrase representing at least one of an action, relationship, attribute, or count of objects from the obtained natural language query. A method of operation of the electronic device (100) may include a step (S230) of obtaining at least one alternative natural language query in which the identified word or phrase is replaced with an alternative word or alternative phrase. A method of operation of the electronic device (100) may include a step (S240) of displaying image search results based on the obtained at least one alternative natural language query.

[0127] In one embodiment of the present disclosure, the step of identifying the word or phrase (S220) may include: a step of identifying the part of speech of vocabulary included in the text constituting the natural language query by performing POS tagging on the input natural language query (S510); and a step of determining the word or phrase representing at least one of the behavior, relationship, attribute, or number of objects as the replacement target word or replacement target phrase based on the identified part of speech (S520).

[0128] In one embodiment of the present disclosure, the step of obtaining at least one alternative natural language query (S230) may include: a step of identifying at least one image in which the similarity between a first embedding vector obtained by vector embedding text constituting the natural language query among a plurality of images and a plurality of second embedding vectors obtained by vector embedding each of the plurality of images is greater than or equal to a preset threshold (S530); and a step of obtaining an alternative word or alternative phrase from the natural language output as an inference result by inputting the identified at least one image into a captioning model or a large vision language model (LVLM) (S540). The step of obtaining at least one alternative natural language query (S230) may include a step of replacing a target word or target phrase using the obtained alternative word or alternative phrase (S550).

[0129] In one embodiment of the present disclosure, the step of obtaining at least one alternative natural language query (S230) may include: a step of recognizing objects included in the natural language query based on identified parts of speech; and a step of obtaining at least one alternative natural language query in which the ordering of the objects has been changed.

[0130] In one embodiment of the present disclosure, the step of identifying the word or phrase (S220) may include: a step of identifying the part of speech and lexical hierarchy of the text constituting the natural language query by performing POS tagging on the input natural language query (S910); and a step of determining the replacement target word or replacement target phrase from the natural language query based on the identified part of speech and lexical hierarchy (S920). The step of obtaining at least one replacement natural language query (S230) may include: a step of obtaining a replacement search term based on a tag for each of a plurality of images stored in memory (130) (S930); and a step of obtaining at least one replacement natural language query by substituting the replacement target word or replacement target phrase with the obtained replacement search term (S940).

[0131] In one embodiment of the present disclosure, the step (S240) of displaying the image search result by the at least one alternative natural language query may include the step of displaying at least one image searched by the at least one alternative natural language query among a plurality of images stored in memory (130) together with the image search result by the natural language query.

[0132] In one embodiment of the present disclosure, the step (S240) of displaying image search results by at least one alternative natural language query may include the step of displaying the number of images searched by at least one alternative natural language query.

[0133] In one embodiment of the present disclosure, the step (S240) of displaying image search results by at least one alternative natural language query may include the step of displaying a drop-down menu user interface (UI) for receiving user input to select an alternative search term that replaces a word or phrase. The drop-down menu UI may include at least one alternative search term candidate.

[0134] In one embodiment of the present disclosure, the dropdown menu UI may further include a numeric UI indicating the number of images searched by each of at least one alternative search term candidates.

[0135] In one embodiment of the present disclosure, the at least one alternative search term candidate is a plurality of, and the dropdown menu UI can display the plurality of alternative search term candidates sorted in descending order based on the number of images searched by each of the plurality of alternative search term candidates.

[0136] Another aspect of the present disclosure discloses an electronic device (100) that provides image search results. An electronic device (100) according to one embodiment of the present disclosure may include a user input interface (110); a display (140); at least one processor (120) comprising a processing circuit; and a memory (130) that stores one or more instructions. By executing the one or more instructions individually or collectively by the at least one processor (120), the electronic device (100) may obtain a natural language query for image search through the user input interface (110) and identify a word or phrase representing at least one of an action, relationship, attribute, or count of objects from the obtained natural language query. By executing the above one or more commands individually or collectively by the at least one processor (120), the electronic device (100) can obtain at least one alternative natural language query in which an identified word or phrase is replaced with an alternative word or alternative phrase. By executing the above one or more commands individually or collectively by the at least one processor (120), the electronic device (100) can display an image search result based on the obtained at least one alternative natural language query on the display (140).

[0137] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (120), the electronic device (100) performs Part-Of-Speech tagging on an input natural language query to identify the parts of speech of vocabulary included in the text constituting the natural language query, and based on the identified parts of speech, determine a word or phrase representing at least one of the behavior, relationship, attribute, or number of objects as a replacement target word or replacement target phrase.

[0138] In one embodiment of the present disclosure, the one or more instructions are executed individually or collectively by the at least one processor (120), thereby the electronic device (100) can identify at least one image in which the similarity between a first embedding vector obtained by vector embedding text constituting a natural language query among a plurality of images and a plurality of second embedding vectors obtained by vector embedding each of the plurality of images is greater than or equal to a preset threshold, and input the identified at least one image into a captioning model or a large vision language model (LVLM) to obtain a replacement word or replacement phrase from the natural language output as an inference result. The one or more instructions are executed individually or collectively by the at least one processor (120), thereby the electronic device (100) can obtain at least one replacement natural language query by replacing a target word or target phrase with the obtained replacement word or replacement phrase.

[0139] In one embodiment of the present disclosure, the one or more instructions are executed individually or collectively by the at least one processor (120), so that the electronic device (100) can recognize objects included in the natural language query based on the identified part of speech and obtain at least one alternative natural language query in which the ordering of the objects is changed.

[0140] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (120), the electronic device (100) can perform Part-Of-Speech tagging on an input natural language query to identify the part of speech and lexical hierarchy of the text constituting the natural language query, and determine a replacement target word or replacement target phrase from the natural language query based on the identified part of speech and lexical hierarchy. By executing one or more instructions individually or collectively by the at least one processor (120), the electronic device (100) can obtain a replacement search term based on a tag for each of a plurality of images stored in memory (130), and obtain at least one replacement natural language query by substituting a replacement target word or replacement target phrase with the obtained replacement search term.

[0141] In one embodiment of the present disclosure, the one or more instructions are executed individually or collectively by the at least one processor (120), so that the electronic device (100) can display at least one image, which is retrieved by at least one alternative natural language query among a plurality of images stored in memory (130), on the display (140) together with the image search result by the natural language query.

[0142] In one embodiment of the present disclosure, by executing one or more instructions individually or collectively by the at least one processor (120), the electronic device (100) can display the number of images retrieved by at least one alternative natural language query.

[0143] In one embodiment of the present disclosure, as the one or more instructions are executed individually or collectively by the at least one processor (120), the electronic device (100) may display a drop-down menu user interface (UI) through a display (140) for receiving user input selecting an alternative search term that replaces a word or phrase. The drop-down menu UI may include at least one alternative search term candidate.

[0144] In one embodiment of the present disclosure, the at least one alternative search term candidate is a plurality of, and the dropdown menu UI can display the plurality of alternative search term candidates sorted in descending order based on the number of images searched by each of the plurality of alternative search term candidates.

[0145] The present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may include instructions readable by the electronic device (100) for the electronic device (100) to perform the operations of: acquiring a natural language query for image search; identifying a word or phrase representing at least one of an action, relationship, attribute, or count of objects from the acquired natural language query; acquiring at least one alternative natural language query in which the identified word or phrase is replaced with an alternative word or alternative phrase; and displaying an image search result based on the acquired at least one alternative natural language query.

[0146] A program executed by the electronic device (100) described in the present disclosure may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. The program may be executed by any system capable of executing computer-readable instructions.

[0147] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively.

[0148] Software can be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). Computer-readable recording media can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The medium is readable by a computer, stored in memory, and can be executed by a processor.

[0149] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory' means only that the storage medium does not contain a signal and is tangible, and does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a 'non-transitory storage medium' may include a buffer in which data is stored temporarily.

[0150] In addition, the program according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0151] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically through the manufacturer of the electronic device (100) or an electronic market (e.g., Samsung Galaxy Store™). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer of the electronic device (100), a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0152] A computer program product may include a storage medium of a server or a storage medium of an electronic device (100) in a system composed of an electronic device (100) and / or a server. Alternatively, if there is a third device that is communicationally connected to the electronic device (100), the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the electronic device (100) to the third device or from the third device to the electronic device.

[0153] In this case, either the electronic device (100) or one of the third devices may execute a computer program product to perform the method according to the disclosed embodiments. Alternatively, at least one of the electronic device (100) and the third device may execute a computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0154] For example, an electronic device (100) can execute a computer program product stored in memory (130, see FIG. 4) to control another electronic device that is connected to the electronic device (100) to perform a method according to the disclosed embodiments.

[0155] As another example, a third device may execute a computer program product to control an electronic device connected to the third device in communication to perform the method according to the disclosed embodiment.

[0156] When the third device executes a computer program product, the third device may download the computer program product from the electronic device (100) and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.

[0157] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or module are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

Claims

1. In a method in which an electronic device (100) provides image search results, Step of obtaining a natural language query for image search (S210); A step (S220) of identifying a word or phrase representing at least one of the action, relationship, attribute, or number of objects from the above-mentioned natural language query; A step (S230) of obtaining at least one alternative natural language query in which the identified word or phrase is replaced with an alternative word or alternative phrase; and A step (S240) of displaying image search results based on at least one alternative natural language query obtained above; A method including 2. In Paragraph 1, The step (S220) of identifying the above word or phrase is, A step (S510) of performing POS tagging (Part-Of-Speech tagging) on ​​the input natural language query to identify the parts of speech of vocabulary included in the text constituting the natural language query; and A step (S520) of determining a word or phrase representing at least one of the behavior, relationship, attribute, or number of the objects based on the above-identified part of speech as a replacement target word or replacement target phrase; A method including 3. In Paragraph 2, The step (S230) of obtaining at least one alternative natural language query is, A step (S530) of identifying at least one image in which the similarity between a first embedding vector obtained by vector embedding text constituting the natural language query among a plurality of images and a plurality of second embedding vectors obtained by vector embedding each of the plurality of images is greater than or equal to a preset threshold; A step (S540) of inputting at least one identified image into a captioning model or a large vision language model (LVLM) to obtain the replacement word or replacement phrase from the natural language output as an inference result; and A step (S550) of replacing the target word or target phrase using the acquired replacement word or replacement phrase; A method including 4. In Paragraph 2, The step (S230) of obtaining at least one alternative natural language query is, A step of recognizing objects included in the natural language query based on the identified part of speech; and A step of obtaining at least one alternative natural language query in which the ordering of the above objects has been changed; A method including 5. In Paragraph 1, The step (S220) of identifying the above word or phrase is, A step (S910) of performing POS tagging (Part-Of-Speech tagging) on ​​the input natural language query to identify the part of speech and lexical hierarchy of the text constituting the natural language query; and A step (S920) of determining a replacement target word or replacement target phrase from the natural language query based on the identified part of speech and lexical hierarchy above; Includes, The step (S230) of obtaining at least one alternative natural language query is, A step (S930) of obtaining alternative search terms based on tags for each of the plurality of images stored in the memory (130) of the electronic device (100); and A step (S940) of obtaining at least one alternative natural language query by replacing the alternative target word or the alternative target phrase with the alternative search term obtained above; A method including 6. In any one of paragraphs 1 through 5, The step (S240) of displaying image search results based on at least one alternative natural language query is, A method comprising the step of displaying at least one image searched by at least one alternative natural language query among a plurality of images stored in the memory (130) of the electronic device (100), together with the image search result by the natural language query.

7. In any one of paragraphs 1 through 6, The step (S240) of displaying image search results based on at least one alternative natural language query is, The method includes the step of displaying a drop-down menu user interface for receiving user input to select an alternative search term that replaces the above word or phrase, and The above dropdown menu UI includes at least one alternative search term candidate, a method.

8. In an electronic device (100) that provides image search results by natural language query, User input interface (110); Display (140); At least one processor (120) including a processing circuit; and Memory (130) for storing one or more instructions; Includes, By executing the above one or more instructions individually or collectively by the above at least one processor (120), the electronic device (100) A natural language query for image search is obtained through the above user input interface (110), and Identify a word or phrase representing at least one of the action, relationship, attribute, or count of objects from the above-mentioned natural language query, and Obtain at least one alternative natural language query in which the above-mentioned identified word or phrase is replaced with an alternative word or alternative phrase, and An electronic device (100) that displays image search results based on at least one alternative natural language query obtained above on the display (140).

9. In Paragraph 8, By executing the above one or more instructions individually or collectively by the above at least one processor (120), the electronic device (100) Part-Of-Speech tagging is performed on the above input natural language query to identify the parts of speech of vocabulary included in the text constituting the above natural language query, and An electronic device (100) that determines a word or phrase representing at least one of the behavior, relationship, attribute, or number of the objects as a replacement target word or replacement target phrase based on the above-identified part of speech.

10. In Paragraph 9, By executing the above one or more instructions individually or collectively by the above at least one processor (120), the electronic device (100) Identifying at least one image in which the similarity between a first embedding vector obtained by vector embedding the text constituting the natural language query among a plurality of images and a plurality of second embedding vectors obtained by vector embedding each of the plurality of images is greater than or equal to a preset threshold, and The above-mentioned at least one identified image is input into a captioning model or a large vision language model (LVLM) to obtain the above-mentioned alternative word or above-mentioned alternative phrase from the natural language output as an inference result, and An electronic device (100) that obtains at least one replacement natural language query by replacing the replacement target word or replacement target phrase using the above-mentioned replacement word or replacement phrase.

11. In Paragraph 9, By executing the above one or more instructions individually or collectively by the above at least one processor (120), the electronic device (100) Recognize objects included in the natural language query based on the parts of speech identified above, and An electronic device (100) for obtaining at least one alternative natural language query in which the ordering of the above objects has been changed.

12. In Paragraph 8, By executing the above one or more instructions individually or collectively by the above at least one processor (120), the electronic device (100) Part-Of-Speech tagging is performed on the above input natural language query to identify the part of speech and lexical hierarchy of the text constituting the above natural language query, and Based on the above-identified part of speech and lexical hierarchy, determine the replacement target word or replacement target phrase from the above natural language query, and Based on the tags for each of the plurality of images stored in the memory (130), alternative search terms are obtained, and An electronic device (100) that obtains at least one alternative natural language query by replacing the alternative target word or the alternative target phrase with the alternative search term obtained above.

13. In any one of paragraphs 8 through 12, By executing the above one or more instructions individually or collectively by the above at least one processor (120), the electronic device (100) An electronic device (100) that displays on the display (140) at least one image among a plurality of images stored in the memory (130) that is searched by the at least one alternative natural language query, along with the image search result by the natural language query.

14. In any one of paragraphs 8 through 13, By executing the above one or more instructions individually or collectively by the above at least one processor (120), the electronic device (100) A drop-down menu UI (drop-down menu user interface) for receiving user input to select an alternative search term to replace the above word or phrase is displayed through the display (140), and The above dropdown menu UI includes at least one alternative search term candidate, electronic device (100).

15. In a computer program product comprising a computer-readable storage medium, The above storage medium is, Operation of obtaining a natural language query for image search; An operation to identify a word or phrase representing at least one of the action, relationship, attribute, or count of objects from the above-mentioned natural language query; The operation of obtaining at least one alternative natural language query in which the identified word or phrase is replaced with an alternative word or alternative phrase; and The operation of displaying image search results based on at least one alternative natural language query obtained above; A computer program product comprising instructions executed by said electronic device (100) for the electronic device (100) to perform.

Images