Electronic device and method for searching for section of image on basis of query

Abstract

Disclosed are an electronic device and method for searching for a section of an image on the basis of a query. The electronic device and method for searching for a section of an image on the basis of a query may comprise the steps of: acquiring a plurality of keywords of a query for searching for a section of a target image; acquiring a plurality of keywords of the target image indicating the context of the target image; filtering at least one keyword from among the plurality of keywords of the query on the basis of the plurality of keywords of the target image; and determining at least one frame to be extracted from the target image on the basis of a plurality of keywords of the filtered query.

Description

Electronic device and method for searching a section of an image based on a query

[0001] An electronic device and method for searching a section of an image based on a query are disclosed. Specifically, an electronic device and method for filtering keywords of a query for searching a section of a target image based on the context of the target image and extracting frames from the target image based on the keywords of the filtered query are disclosed.

[0002] Users can use queries to search for specific sections of a video. Some keywords in these queries may be unnecessary for performing segment searches or may reduce the accuracy of the search. Therefore, a technique for filtering keywords in queries is needed to achieve more accurate segment searches.

[0003] According to one aspect of the present disclosure, a method for searching a section of an image based on a query is disclosed. In one embodiment, the method may include obtaining a plurality of keywords of a query for searching a section of a target image. In one embodiment, the method may include obtaining a plurality of keywords of the target image that represent a context of the target image. In one embodiment, the method may include filtering at least one keyword of the plurality of keywords of the query based on the plurality of keywords of the target image. In one embodiment, the method may include determining at least one frame to be extracted from the target image based on the plurality of keywords of the filtered query.

[0004] According to one aspect of the present disclosure, an electronic device for searching a section of an image based on a query is disclosed. In one embodiment, the electronic device may include a memory storing one or more instructions and at least one processor for executing one or more instructions stored in the memory. In one embodiment, the electronic device may obtain a plurality of keywords of a query for searching a section of a target image by the at least one processor executing one or more instructions. In one embodiment, the electronic device may obtain a plurality of keywords of a target image representing a context of the target image by the at least one processor executing one or more instructions. In one embodiment, the electronic device may filter at least one keyword among the plurality of keywords of the query based on the plurality of keywords of the target image by the at least one processor executing one or more instructions. In one embodiment, the electronic device may determine at least one frame to be extracted from the target image based on the plurality of keywords of the filtered query by the at least one processor executing one or more instructions.

[0005] According to one aspect of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing an operation of an electronic device, one of the methods described above and below.

[0006] FIG. 1 is a drawing schematically illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0007] FIG. 2 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0008] FIG. 3 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to obtain a keyword of a target image.

[0009] FIG. 4A is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to filter at least one keyword based on a first threshold value.

[0010] FIG. 4b is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to filter at least one keyword based on a second threshold value.

[0011] FIG. 5 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to filter at least one keyword that does not correspond to a plurality of keywords of a target image.

[0012] FIG. 6 is a diagram illustrating a method for an electronic device according to an embodiment of the present disclosure to determine at least one frame to be extracted from a target image based on a plurality of keywords of a filtered query.

[0013] FIG. 7 is a drawing for explaining modules included in an electronic device according to one embodiment of the present disclosure.

[0014] FIG. 8 is a diagram illustrating a keyword extraction module according to one embodiment of the present disclosure.

[0015] FIG. 9 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to search for a section of a target image based on a query.

[0016] FIG. 10 is a detailed configuration diagram of an electronic device according to one embodiment of the present disclosure.

[0017] FIG. 11 is a detailed configuration diagram of a server according to one embodiment of the present disclosure.

[0018] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0019] Throughout this disclosure, unless specifically stated otherwise, "or" is inclusive and not exclusive. Thus, unless explicitly stated otherwise or the context dictates otherwise, "A or B" can mean "A, B, or both." As used herein, the phrases "at least one of" or "one or more of" can mean that different combinations of one or more of the listed items can be used, or that only any one of the listed items is required. For example, "at least one of A, B, and C" can include any of the following combinations: A, B, C, A and B, A and C, B and C, or A and B and C.

[0020] Unless the context clearly dictates otherwise, the singular forms "a," "an," and "the" are to be understood to include plural referents. Thus, for example, the description "a constituent surface" may also include reference to one or more of such surfaces.

[0021] Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art described herein.

[0022] When a part of this disclosure is said to "include" a component, unless otherwise specifically stated, this does not exclude other components but rather implies the inclusion of other components. Furthermore, terms such as "part," "module," and the like used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0023] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system, in conjunction with other devices or components, is "capable of." For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) that can perform the operations by executing one or more software programs stored in memory.

[0024] It should be understood that the blocks and combinations of flowcharts in each of the flowcharts in this disclosure can be implemented by one or more computer programs containing computer-executable instructions. The one or more computer programs may be stored entirely in a single memory, or may be divided and stored across multiple different memories.

[0025] All functions or operations described in the present disclosure may be processed by a single processor or a combination of processors. A single processor or a combination of processors may include circuitry that performs processing, such as an Application Processor (AP), a Communication Processor (CP), a Graphical Processing Unit (GPU), a Neural Processing Unit (NPU), a Microprocessor Unit (MPU), a System on Chip (SoC), or an Integrated Chip (IC).

[0026] In the present disclosure, a processor is a component that controls a series of processes so that an electronic device operates according to the embodiments described below, and may be composed of one or more processors. One or more processors included in the processor may be circuitry such as a System on Chip (SoC), an Integrated Circuit (IC), etc. One or more processors included in the processor may be a general-purpose processor such as a Central Processing Unit (CPU), a Micro Processor Unit (MPU), an Application Processor (AP), a Digital Signal Processor (DSP), a graphics-only processor such as a Graphics Processing Unit (GPU), a Vision Processing Unit (VPU), an artificial intelligence-only processor such as a Neural Processing Unit (NPU), or a communication-only processor such as a Communication Processor (CP). When one or more processors included in the processor are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0027] In the present disclosure, a processor may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits, including at least one processor. At least one processor, one or more processors may be configured to perform various functions described herein, individually and / or collectively, in a distributed fashion. As used herein, “processor,” “at least one processor,” and “one or more processors” may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0028] The processor can write data to memory, read data stored in memory, and process data according to predefined operating rules or artificial intelligence models, particularly by executing a program or at least one instruction stored in memory. Accordingly, the processor can perform the operations described in the following embodiments, and operations described as performed by electronic devices or detailed components included in the electronic devices in the following embodiments can be considered to be performed by the processor, unless otherwise specified.

[0029] When a component is referred to as being "connected" or "connected" to another component in this disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0030] In the present disclosure, a "model" or "artificial intelligence (AI) model" may refer to a set of functions or algorithms that are set to perform a desired characteristic (or purpose) by being learned using a plurality of learning data by a learning algorithm. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. In one embodiment, the AI ​​model may be stored in the memory of an electronic device. However, the AI ​​model is not limited thereto, and the electronic device may transmit data input to the AI ​​model to the server and receive data output from the AI ​​model from the server.

[0031] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, portions irrelevant to the description have been omitted for clarity of explanation, and similar reference numerals have been used throughout the specification to designate similar parts.

[0032] The present disclosure will be described below with reference to the attached drawings.

[0033] FIG. 1 is a drawing schematically illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0034] Referring to FIG. 1, the electronic device (1000) may acquire a target image (110). In one embodiment, the target image (110) may include an image captured by a camera of the electronic device (1000) or an image received from an external electronic device. In one embodiment, the target image (110) acquired by the electronic device (1000) may be stored in the memory of the electronic device (1000) and used to perform various operations and functions of the electronic device (1000) of the present disclosure. In one embodiment, the target image (110) may refer to an image that is the target of a query for searching a section of the target image, which will be described later. In one embodiment, the electronic device (1000) may extract at least one frame from the target image (110) based on the query for searching a section of the target image.

[0035] In one embodiment, the electronic device (1000) may obtain a query for searching a section of a target image (110). Here, searching for a section may mean searching for a specific frame from the target image (110) or for searching for an image composed of multiple frames including a specific frame. Meanwhile, the query for searching a section of the target image (110) may be replaced with various expressions representing the same / similar concept. For example, the query for searching a section of the target image (110) may be replaced with expressions such as a query for editing the target image (110), a query for the target image (110), etc., and is not limited to the examples described above. In one embodiment, the query for searching a section of the target image (110) may include data related to a request for extracting a specific frame from the target image (110). Meanwhile, the term "extract" may be replaced with various expressions representing the same / similar concept. Extraction may be replaced with expressions such as "obtain" or "acquire", for example, and is not limited to the examples set forth above.

[0036] In one embodiment, a query for searching a section of a target image (110) may include at least one of text, an image, and a video related to the query. For example, a query for searching a section of a target image (110) may include an image query (120) in the form of an image data. In one embodiment, the electronic device (1000) may obtain a query for searching a section of a target image (110) based on a user input of at least one of text, an image, and a video related to the query. However, the present invention is not limited to the above-described example, and the electronic device (1000) may also obtain a query for searching a section of a target image (110) from an external electronic device.

[0037] In one embodiment, the electronic device (1000) may obtain a plurality of keywords of the query based on a query for searching a section of the target image (110). Here, the keywords may be replaced with various expressions representing the same / similar concepts. The keywords may be replaced with expressions such as, for example, “word,” “term,” “main word,” “search term,” “core term,” “primary term,” “related term,” “recommendations,” etc., and are not limited to the examples described above. In one embodiment, the plurality of keywords of the query may include a plurality of keywords representing the context of the query for searching a section of the target image (110). In one embodiment, the plurality of keywords representing the context of the query may correspond to a plurality of elements forming the context of the query.

[0038] For example, the electronic device (1000) may obtain an image query (120) as a query for searching a section of a target image (110). Here, the image query (120) may be an image of people having fun toasting with glasses at a bar, and elements such as people, objects, background, and atmosphere included in the image query (120) may form the context of the image query (120). In this case, the electronic device (1000) may obtain 'people', 'toast', 'joy', and 'bar', which are keywords representing the context of the image query (120), as multiple keywords of the query based on the image query (120).

[0039] In one embodiment, the electronic device (1000) may obtain a plurality of keywords of the target image (110) indicating the context of the target image (110). In one embodiment, the plurality of keywords of the target image (110) indicating the context of the target image (110) may correspond to a plurality of elements forming the context of the target image (110). In one embodiment, the plurality of keywords of the target image may include a keyword indicating the context of each of a plurality of unit sections of the target image. Here, the plurality of unit sections of the target image (110) may mean each of a plurality of frames constituting the target image (110), or may mean each section (or frames of each section) formed by dividing into preset time intervals (for example, 60 frames or 1 second).

[0040] For example, the electronic device (1000) may be configured such that the plurality of unit sections of the target image (110) may be a plurality of frames constituting the target image (110). In this case, the plurality of keywords of the target image (110) may include keywords representing the context of each of the plurality of unit sections of the target image (110), namely the first frame (110-1), the second frame (110-2), the third frame (110-3), the fourth frame (110-4), and the fifth frame (110-5). Here, the first frame (110-1) may include a context formed by elements such as a bar with a Christmas tree and a red-toned interior. In this case, the electronic device (1000) may obtain 'bar', 'red', and 'Christmas' from the first frame (110-1) as keywords representing the context of the first frame (110-1). The electronic device (1000) can obtain keywords indicating the context of each frame for each of the second frame (110-2), the third frame (110-3), the fourth frame (110-4), and the fifth frame (110-5).

[0041] In one embodiment, the electronic device (1000) can filter a plurality of keywords of a query based on a plurality of keywords of a target image (110). For example, the electronic device (1000) can filter out 'people' and 'bar' among the plurality of keywords of the query, 'people', 'toast', 'joy', and 'bar', based on the plurality of keywords of the target image (110). Then, the electronic device (1000) can obtain 'toast' and 'joy', which are the plurality of keywords of the filtered query.

[0042] In one embodiment, the electronic device (1000) may determine at least one frame to be extracted from the target image (110) based on a plurality of keywords of the filtered query. In one embodiment, the at least one frame to be extracted from the target image (110) may mean at least one frame corresponding to a query for searching a section of the target image. For example, the electronic device (1000) may determine the fourth frame (110-4) corresponding to the query of the image query (120) in the target image (110) as at least one frame to be extracted from the target image (110). In this case, the context of the fourth frame (110-4) and the context of the image query (120) may correspond to each other. In one embodiment, the electronic device (1000) may extract the determined at least one frame from the target image (110) and display the extracted at least one frame through the display of the electronic device (1000).

[0043] An electronic device (1000) according to an embodiment of the present disclosure can filter at least one keyword among a plurality of keywords of a query for searching a section of a target image (110) based on a plurality of keywords of the target image (110), thereby excluding keywords having a low relevance to the intention of a user who inputs the query. In other words, through filtering, keywords that are not helpful in performing a search of the target image (110) based on the query or an editing based on the search, or keywords having a low importance, can be removed from the plurality of keywords of the query.

[0044] For example, the intention of a user who inputs an image query (120) to search for a section of a target image (110) that captures people in a bar may not be to simply obtain a frame containing only a bar and people in the target image (110), but may be to obtain a frame containing scenes of people toasting and enjoying themselves in the bar. In this case, among the plurality of keywords of the query obtained from the image query (120), 'bar' and 'people' may be keywords that are obtained with a higher frequency or probability in the entire section of the target image (110) than 'toast' and 'joy'. That is, the electronic device (1000) may filter out 'bar' and 'people', which are keywords that form a context included in most sections of the target image (110), based on the plurality of keywords of the query, thereby removing 'bar' and 'people', which are unnecessary keywords for performing a section search, and obtaining 'toast' and 'joy'.

[0045] In this way, the electronic device (1000) according to one embodiment of the present disclosure can determine at least one frame to be extracted from the target image (110) based on a plurality of keywords of a query filtered based on the context of the target image (110). Accordingly, a search for a target image (110) that better matches the intention of a user who inputted a query or editing of the target image (110) based on the search can be performed. In addition, since the user does not have to directly filter keywords of the query, user convenience can be increased.

[0046] FIG. 2 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0047] Referring to FIG. 2, the operation of the electronic device (1000) will be schematically described, and a detailed description of each operation will be described with reference to the drawings that follow. In addition, the operations of the electronic device (1000) described in the present disclosure may be understood as the operations of the electronic device (1000) and the processor (1600) of the electronic device (1000) illustrated in FIG. 10, and the server (2000) and the processor (2300) of the server (2000) illustrated in FIG. 11. In one embodiment, some of the steps illustrated in FIG. 2 may be omitted or other steps may be added.

[0048] In step S210, the electronic device (1000) can obtain multiple keywords of a query for searching a section of a target image.

[0049] In one embodiment, the electronic device (1000) may obtain at least one of text, image, and video related to a query for searching a section of a target image. For example, the electronic device (1000) may obtain at least one of a text query, which is text-format data including the query, an image query, which is image data format including the query, and an image query, which is image data format including the query.

[0050] In one embodiment, the electronic device (1000) may obtain a query for searching a section of a target image through an input interface of the electronic device (1000). For example, the electronic device (1000) may obtain a text input from a user through the input interface and obtain the obtained text input as a text query. As another example, the electronic device (1000) may obtain a user input for selecting an image or video stored in the memory of the electronic device (1000) through the input interface and obtain the image or video selected based on the user input as an image query or a video query. However, the present invention is not necessarily limited to the above-described example, and the electronic device (1000) may also obtain an image or video captured by a camera of the electronic device (1000) as an image query or a video query.

[0051] In one embodiment, the electronic device (1000) can obtain a plurality of keywords of the query from at least one of text, image, and video related to the obtained query. In one embodiment, a plurality of keywords that can be obtained as keywords of the query may be predetermined. In one embodiment, the plurality of keywords that can be obtained as keywords of the predetermined query may be referred to as a plurality of candidate keywords, and the electronic device (1000) can store information about the plurality of candidate keywords in a memory. In one embodiment, the plurality of candidate keywords may also correspond to a plurality of keywords that can be obtained as keywords of a target image.

[0052] In one embodiment, a plurality of candidate keywords that can be obtained as keywords of a query and a plurality of candidate keywords that can be obtained as keywords of a target image may be different from each other. In this case, the plurality of candidate keywords of the query and the plurality of candidate keywords of the target image may form a correspondence relationship with each other, and information about the plurality of candidate keywords may include information indicating the correspondence relationship between the plurality of candidate keywords of the query and the plurality of candidate keywords of the target image. The electronic device (1000) may identify a correspondence relationship between the plurality of keywords of the target image and the plurality of keywords of the query based on information about the plurality of candidate keywords stored in advance.

[0053] In one embodiment, the electronic device (1000) may parse a text related to a query to obtain a plurality of components (e.g., words, phrases, clauses, etc.) constituting the text, and may obtain at least one of the obtained plurality of components as a plurality of keywords of the query. In one embodiment, the electronic device (1000) may obtain a candidate keyword that is included in the obtained plurality of components among a plurality of candidate keywords stored in advance, or has a corresponding relationship, such as a synonym or similar word, with the obtained plurality of components as a plurality of keywords of the query.

[0054] In one embodiment, the electronic device (1000) may obtain keywords representing the context of an image or video related to a query as a plurality of keywords of the query. In one embodiment, the electronic device (1000) may identify the probability that a plurality of candidate keywords will be obtained from the image or video related to the query. In one embodiment, the electronic device (1000) may obtain candidate keywords whose identified probability is greater than or equal to a preset probability as a plurality of keywords of the query.

[0055] In one embodiment, the electronic device (1000) may obtain a user input including a plurality of keywords through an input interface, and obtain the plurality of keywords included in the user input as a plurality of keywords of a query for searching a section of a target image. In one embodiment, the electronic device (1000) may receive data including a plurality of keywords from an external electronic device through a communication interface, and obtain the plurality of keywords included in the received data as a plurality of keywords of a query for searching a section of a target image. In other words, the electronic device (1000) may obtain data including only a plurality of keywords of a query instead of a query for searching a section of a target image, and obtain the plurality of keywords of the query based on the obtained data.

[0056] In one embodiment, the electronic device (1000) may acquire a plurality of keywords of the query based on the output acquired by inputting at least one of text, image, and video related to the acquired query into a keyword extraction module. In one embodiment, the keyword extraction module may be an artificial intelligence model trained to output a probability that a plurality of keywords representing the context of the training text, training image, or training video can be acquired based on the training text, training image, or training video. A specific description of the keyword extraction module will be described again below with reference to FIGS. 7 and 8.

[0057] At step S220, the electronic device (1000) can obtain a plurality of keywords of the target image representing the context of the target image.

[0058] In one embodiment, the plurality of keywords of the target image representing the context of the target image may include keywords representing the context of each of the plurality of unit sections of the target image. In one embodiment, if the unit section of the target image includes a plurality of frames, the electronic device (1000) may obtain a keyword for each of the plurality of unit sections based on a keyword obtained from at least one frame included in each of the plurality of unit sections. For example, the electronic device (100) may obtain a keyword representing the context of at least one frame among the plurality of frames constituting the unit section of the target image, and may obtain the obtained keyword as a keyword representing the context of the unit section including the plurality of frames. In one embodiment, the electronic device (1000) may map or associate the plurality of keywords of the target image to each of the plurality of unit sections of the target image from which the plurality of keywords are obtained, and store the keywords in the memory of the electronic device (1000).

[0059] In one embodiment, the electronic device (1000) may identify the probability of obtaining a plurality of candidate keywords of each of a plurality of unit sections of the target image. In one embodiment, the electronic device (1000) may obtain a candidate keyword identified as having an obtaining probability greater than or equal to a preset probability as a keyword representing the context of each of the plurality of unit sections. In one embodiment, if a unit section of the target image includes a plurality of frames, the electronic device (1000) may identify the probability of obtaining a plurality of candidate keywords of each of a plurality of frames constituting the unit section, and may obtain a candidate keyword for which an average of the identified probabilities for the plurality of frames is greater than or equal to a preset probability as a keyword representing the context of the unit section including the plurality of frames. In one embodiment, the electronic device (1000) may store information about the probability of obtaining a plurality of candidate keywords in each of the plurality of unit sections in a memory of the electronic device (1000) by mapping or associating the information with each of the plurality of unit sections. In one embodiment, the electronic device (1000) can identify the probability that the plurality of keywords of the target image are obtained in each of the plurality of unit sections of the target image among the plurality of candidate keywords based on information about the probability that the plurality of candidate keywords are obtained in each of the plurality of unit sections of the target image.

[0060] In one embodiment, the electronic device (1000) may acquire a plurality of keywords of the target image based on an output of the keyword extraction module obtained by inputting the target image into the keyword extraction module. In one embodiment, the keyword extraction module may be an artificial intelligence model trained to output a probability of acquiring a plurality of keywords representing the context of the training image based on the training image. In one embodiment, the probability of acquiring a plurality of keywords output by the keyword extraction module may include a probability of acquiring a plurality of keywords for each of a plurality of unit sections of the training image.

[0061] A detailed description of the operation of the electronic device (1000) to obtain multiple keywords of the target image representing the context of the target image will be described again below with reference to FIG. 3.

[0062] At step S230, the electronic device (1000) can filter at least one keyword among the plurality of keywords of the query based on the plurality of keywords of the target image.

[0063] In one embodiment, the electronic device (1000) can identify the frequency or probability that a plurality of keywords of the target image are acquired in the entire section of the target image based on keywords representing the context of each of the plurality of unit sections. In one embodiment, the electronic device (1000) can identify the probability that a plurality of keywords of the target image are acquired in the entire section of the target image based on an average of the probabilities that the plurality of keywords of the target image are acquired in each of the plurality of unit sections. In one embodiment, the electronic device (1000) can identify the frequency that a plurality of keywords of the target image are acquired in the entire section of the target image based on the number of times that the plurality of keywords of the target image are acquired in each of the plurality of unit sections.

[0064] In one embodiment, the electronic device (1000) may filter at least one keyword based on the identified frequency or the identified probability. In one embodiment, the electronic device (1000) may filter at least one keyword among a plurality of keywords of a query based on both the identified probability and the identified frequency. In this case, the electronic device (1000) may first filter at least one keyword based on the identified probability, and then secondarily filter the keyword filtered in the first step based on the identified frequency. However, the present invention is not limited to the above-described example, and the filtering order may be interchanged.

[0065] In one embodiment, the electronic device (1000) may filter at least one keyword corresponding to a keyword whose identified probability or identified frequency is greater than or equal to a first threshold value among a plurality of keywords of the target image. In other words, the electronic device (1000) may filter a keyword whose identified probability or identified frequency is greater than or equal to a first threshold value among a plurality of keywords of the query. In one embodiment, the first threshold value is a preset value, and the first threshold value for filtering based on the identified probability and the first threshold value for filtering based on the identified frequency may be different from each other. Specific details regarding an operation in which the electronic device (1000) performs filtering based on the first threshold value will be described again below with reference to FIG. 4A.

[0066] In one embodiment, the electronic device (1000) may filter at least one keyword corresponding to a keyword whose identified probability or identified frequency is less than a second threshold value among a plurality of keywords of the target image. In other words, the electronic device (1000) may filter out a keyword whose identified probability or identified frequency is less than a second threshold value among a plurality of keywords of the query. In one embodiment, the second threshold value is a preset value, and the second threshold value for filtering based on the identified probability and the second threshold value for filtering based on the identified frequency may be different from each other. Specific details regarding an operation in which the electronic device (1000) performs filtering based on the second threshold value will be described again below with reference to FIG. 4B.

[0067] In one embodiment, the electronic device (1000) may filter out at least one keyword that does not correspond to a plurality of keywords of the target image. In other words, the electronic device (1000) may filter out a keyword among the plurality of keywords of the query that is not included in the plurality of keywords of the target image. Specific details regarding the operation of the electronic device (1000) filtering out at least one keyword that does not correspond to the plurality of keywords of the target image will be described again below with reference to FIG. 5.

[0068] In one embodiment, the above-described filtering may be performed in an overlapping manner. For example, the electronic device (1000) may first filter at least one keyword among a plurality of keywords of a query based on a first threshold value, secondarily filter at least one keyword among the keywords filtered in the first filtering based on a second threshold value, and secondarily filter at least one keyword that does not correspond to the plurality of keywords of the target image among the keywords filtered in the second filtering. However, the present invention is not limited to the above-described example, and some of the above-described filtering may be omitted or the order of the filtering may be reversed.

[0069] In step S240, the electronic device (1000) may determine at least one frame to be extracted from the target image based on a plurality of keywords of the filtered query. In one embodiment, the electronic device (1000) may determine a frame among the plurality of frames of the target image whose similarity to the query is greater than or equal to a preset similarity as at least one frame to be extracted from the target image. In one embodiment, the similarity of the frame to the query may be determined based on a keyword indicating the context of the frame and a plurality of keywords of the filtered query.

[0070] In one embodiment, the electronic device (1000) may calculate a similarity for a query based on the number or probability that keywords corresponding to the plurality of keywords of the filtered query are acquired in the plurality of unit sections. In one embodiment, the electronic device (1000) may calculate a similarity based on the number of times that the plurality of keywords of the filtered query are acquired in each of the plurality of unit sections of the target image, based on the plurality of keywords of the target image acquired in each of the plurality of unit sections of the target image. In one embodiment, the electronic device (1000) may calculate a similarity based on the probability that the plurality of keywords of the filtered query are acquired in each of the plurality of unit sections of the target image, based on the probability that the plurality of keywords of the target image are acquired in each of the plurality of unit sections of the target image.

[0071] In one embodiment, the electronic device (1000) may identify a unit section among a plurality of unit sections, the unit section having a calculated similarity greater than or equal to a third threshold value. In one embodiment, the third threshold value may be a preset value. In one embodiment, the electronic device (1000) may determine at least one frame from the identified unit section. In one embodiment, if the unit section of the target image includes a plurality of frames, the electronic device (1000) may determine at least one frame from among the plurality of frames of the identified unit section as at least one frame to be extracted from the target image. In one embodiment, if the unit section of the target image includes a plurality of frames, the electronic device (1000) may calculate a similarity to a query for each of the plurality of frames of the identified unit section, and determine a frame having the highest similarity to the query as at least one frame to be extracted from the target image. In this case, the operation of calculating the similarity of the frame to the query may correspond to the operation of calculating the similarity of the unit section to the query.

[0072] In one embodiment, the electronic device (1000) may determine at least one frame to be extracted from the target image between the first and second sections if the time interval between the identified unit sections, i.e., the first and second sections, is less than or equal to a preset time interval. In one embodiment, if the unit section includes a plurality of frames, the time interval between the first and second sections may mean between frames located at the center of each section, or may mean between frames of each section that are closest to another section. In one embodiment, if a plurality of frames are included between the first and second sections, the electronic device (1000) may determine at least one frame among the plurality of frames as at least one frame to be extracted from the target image.

[0073] A detailed description of the operation of the electronic device (1000) to determine at least one frame to be extracted from the target image based on a plurality of keywords of the filtered query will be described again below with reference to FIG. 6.

[0074] In one embodiment, the electronic device (1000) can extract at least one determined frame from the target image (110) and display the extracted at least one frame through the display of the electronic device (1000).

[0075] In one embodiment, if there are multiple extracted frames, the electronic device (1000) may generate an output image by combining the multiple extracted frames, and display the generated output image through the display of the electronic device (1000). In one embodiment, if the multiple extracted frames are continuous frames, the electronic device (1000) may generate and display an output image, and if the multiple extracted frames are not continuous frames, the electronic device (1000) may display an image of each of the determined multiple frames. For example, if multiple frames included in a unit section composed of multiple frames are determined as at least one frame to be extracted from a target image, the electronic device (1000) may generate an output image corresponding to the unit section including the multiple frames, and display the generated output image through the display. As another example, if each of the frames included in different sections is determined as at least one frame to be extracted as a target image, the electronic device (1000) may display images of the frames included in the determined different sections through the display.

[0076] Specific details regarding the operation of the electronic device (1000) to display at least one frame extracted from the target image will be described again below with reference to FIG. 9.

[0077] FIG. 3 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to obtain a keyword of a target image.

[0078] Referring to FIG. 3, the electronic device (1000) may obtain a plurality of keywords (310) of the target image representing the context of the target image (110) from the target image (110). In one embodiment, the plurality of keywords (310) of the target image may include keywords representing the context of each of the plurality of unit sections of the target image (110). For example, each of the plurality of unit sections of the target image (110) may include a first frame (110-1), a second frame (110-2), a third frame (110-3), a fourth frame (110-4), and a fifth frame (110-5). In this case, the electronic device (1000) can obtain 'bar', 'red', 'Christmas', 'people', 'toast', 'joy', and 'drink' as multiple keywords (310) of the target image, including the keyword (210-1) of the first frame, the keyword (210-2) of the second frame, the keyword (210-3) of the third frame, the keyword (210-4) of the fourth frame, and the keyword (210-5) of the fifth frame.

[0079] In one embodiment, the electronic device (1000) can identify the probability that a plurality of candidate keywords including a plurality of keywords (310) of the target image are acquired in each of a plurality of unit sections of the target image (110). In one embodiment, the electronic device (1000) can acquire candidate keywords identified with an identified probability greater than a preset probability as keywords representing the context of each of the plurality of unit sections. For example, if the preset probability is 0.5, the electronic device (1000) can acquire 'bar' with a probability of 0.6, 'Red' with a probability of 0.7, and 'Christmas' with a probability of 0.6, which are acquired from the keyword (210-1) of the first frame, as the keyword (210-1) of the first frame.

[0080] In one embodiment, the electronic device (1000) can identify the probability that a plurality of keywords (310) of the target image are acquired in each of a plurality of unit sections of the target image (110). The probability that the plurality of keywords (310) of the target image are acquired may include the probability of acquiring a keyword that is not acquired as a keyword in a specific unit section but is acquired as a keyword in another unit section. For example, the plurality of keywords (310) of the target image may include 'bar', 'red', 'Christmas', 'people', 'toast', 'joy', and 'drink', which are keywords acquired in the entire section of the target image (110). In this case, only the keywords 'bar', 'red', and 'Christmas' are keywords obtained from the first frame (110-1), but the probability that multiple keywords (310) of the target image are obtained may include the probability that 'bar', 'red', and 'Christmas' are obtained from the first frame (110-1) and the probability that 'people', 'toast', 'joy', and 'drink' are obtained from the first frame (110-1).

[0081] In one embodiment, the electronic device (1000) may identify a frequency (320) at which a plurality of keywords of the target image are acquired in the entire section of the target image (110) based on the number of times the plurality of keywords (310) of the target image are acquired in each of the plurality of unit sections of the target image (110). For example, 'red', which is one of the plurality of keywords (310) of the target image, may be a keyword acquired only in the first frame (110-1) among the first to fifth frames (110-1 to 110-5). In this case, the electronic device (1000) may identify the frequency at which 'red' is acquired in the entire section of the target image (110) as 0.2, which is obtained by dividing the number of times 'red' is acquired by the number of times the first to fifth frames (110-1 to 110-5) are acquired. The electronic device (1000) can identify the frequency (320) at which multiple keywords of the target image are acquired in the entire section of the target image (110) based on the number of times the keywords 'bar', 'Christmas', 'people', 'toast', 'joy', and 'drink' are acquired in the first to fifth frames (110-1 to 110-5) in the same manner.

[0082] In one embodiment, the electronic device (1000) may identify a probability (330) of obtaining a plurality of keywords of the target image in the entire section of the target image (110) based on an average of the probabilities of obtaining a plurality of keywords (310) of the target image in each of a plurality of unit sections of the target image (110). For example, 'bar', which is one of the plurality of keywords (310) of the target image, may have a probability of being obtained in the first frame (110-1) of 0.6, a probability of being obtained in the second frame (110-2) of 0.7, a probability of being obtained in the third frame (110-3) of 0.8, a probability of being obtained in the fourth frame (110-4) of 0.8, and a probability of being obtained in the fifth frame (110-5) of 0.8. In this case, the electronic device (1000) can identify the probability that 'bar' is acquired in the entire section of the target image (110) as 0.74, which is the average of the probabilities of 'bar' being acquired in the first to fifth frames (110-1 to 110-5). In the same manner, the electronic device (1000) can identify the probability (330) that multiple keywords of the target image are acquired in the entire section of the target image (110) based on the probabilities that other keywords, 'red', 'Christmas', 'people', 'toast', 'joy', and 'drink', are acquired in the first to fifth frames (110-1 to 110-5).

[0083] In one embodiment, the electronic device (1000) may store in memory information regarding the frequency (320) at which multiple keywords of the identified target image are acquired and the probability (330) at which multiple keywords of the identified target image are acquired. The information stored in the electronic device (1000) may be used by the electronic device (1000) to filter at least one keyword, as will be described later with reference to FIGS. 4A and 4B .

[0084] FIG. 4A is a diagram illustrating an operation of an electronic device according to one embodiment of the present disclosure to filter at least one keyword based on a first threshold value.

[0085] In one embodiment, the electronic device (1000) may filter the plurality of keywords (410-1) of the query based on the frequency (320) at which the plurality of keywords of the target image are obtained or the probability (330) at which the plurality of keywords of the target image are obtained. For the convenience of explanation of the invention, filtering based on the frequency (320) at which the plurality of keywords of the target image are obtained may be referred to as frequency-based filtering, and filtering based on the probability (330) at which the plurality of keywords of the target image are obtained may be referred to as probability-based filtering.

[0086] In one embodiment, the electronic device (1000) may filter at least one keyword corresponding to a keyword whose frequency obtained in the entire section of the target image is greater than or equal to a first threshold value among the plurality of keywords (310) of the target image. In one embodiment, the electronic device (1000) may identify a keyword whose frequency obtained in the entire section of the target image is greater than or equal to the first threshold value among the plurality of keywords (310) of the target image based on the frequency (320) at which the plurality of keywords of the target image are obtained. In one embodiment, the electronic device (1000) may filter a keyword corresponding to the identified keyword among the plurality of keywords (410-1) of the query. For example, if the first threshold value is 0.7 as shown in FIG. 4A, the electronic device (1000) may identify keywords 'bar' and 'people' whose frequency obtained in the entire section of the target image is greater than or equal to 0.8 among the plurality of keywords (310) of the target image. And, the electronic device (1000) can obtain the plurality of keywords (421-1) 'toast' and 'joy' of the query for which frequency-based filtering has been performed by filtering the keywords 'bar' and 'people' identified in the plurality of keywords (410-1) of the query.

[0087] In one embodiment, the electronic device (1000) may filter out at least one keyword among the plurality of keywords (310) of the target image, the probability of which is acquired in the entire section of the target image is greater than or equal to a first threshold value. In one embodiment, the electronic device (1000) may identify a keyword among the plurality of keywords (310) of the target image, the probability of which is acquired in the entire section of the target image is greater than or equal to the first threshold value, based on the probability (330) of which the plurality of keywords are acquired. In one embodiment, the electronic device (1000) may filter out a keyword corresponding to the identified keyword among the plurality of keywords (410-1) of the query. For example, as shown in FIG. 4A, if the first threshold value is 0.7, the electronic device (1000) may identify the keyword 'bar' among the plurality of keywords (310) of the target image, the probability of which is acquired in the entire section of the target image is greater than or equal to 0.7. And, the electronic device (1000) can obtain the plurality of keywords (422-1) 'people', 'toast', and 'joy' of the query for which probability-based filtering has been performed by filtering the keyword 'bar' identified from the plurality of keywords (410-1) of the query.

[0088] In Fig. 4a, both the frequency-based filtering and the probability-based filtering are illustrated as having a first threshold of 0.7, but the first threshold of the frequency-based filtering and the first threshold of the probability-based filtering may be different from each other. In addition, the electronic device (1000) performs both the frequency-based filtering and the probability-based filtering, thereby filtering out keywords whose frequency obtained in the entire section of the target image is greater than or equal to the first threshold value and keywords whose probability obtained in the entire section of the target image is greater than or equal to the first threshold value from among the plurality of keywords (410-1) of the query.

[0089] In this way, the electronic device (1000) according to one embodiment of the present disclosure can filter out keywords (410-1) of a query among a plurality of keywords (310) of a target image, the keywords having a probability or frequency of being acquired in the entire section of the target image that is greater than or equal to a first threshold value. Here, the keywords having a probability or frequency of being acquired in the entire section of the target image that is greater than or equal to the first threshold value may be keywords indicating the context of most frames of the target image. Therefore, by including the corresponding keyword in the plurality of keywords of the query, the similarity of each frame to the query may be unnecessarily increased.

[0090] That is, the electronic device (1000) can filter out keywords that have a significantly high probability or frequency of being acquired across the entire section of the target image, thereby excluding keywords that are not helpful in searching the section corresponding to the query from among multiple keywords of the query. Accordingly, the electronic device (1000) can more accurately extract frames corresponding to the query input by the user from the target image.

[0091] FIG. 4b is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to filter at least one keyword based on a second threshold value.

[0092] In one embodiment, the electronic device (1000) may filter at least one keyword corresponding to a keyword whose frequency obtained in the entire section of the target image is less than a second threshold value among the plurality of keywords (310) of the target image. In one embodiment, the electronic device (1000) may identify a keyword whose frequency obtained in the entire section of the target image is less than a second threshold value among the plurality of keywords (310) of the target image based on the frequency (320) at which the plurality of keywords of the target image are obtained. In one embodiment, the electronic device (1000) may filter a keyword corresponding to the identified keyword among the plurality of keywords (410-2) of the query. For example, if the second threshold value is 0.3 as shown in FIG. 4B , the electronic device (1000) may identify keywords 'red', 'Christmas', and 'drink' whose frequency obtained in the entire section of the target image is less than 0.3 among the plurality of keywords (310) of the target image. And, the electronic device (1000) can obtain the plurality of keywords (421-2) 'bar' of the query for which frequency-based filtering has been performed by filtering the keywords 'red' and 'Christmas' identified in the plurality of keywords (410-2) of the query.

[0093] In one embodiment, the electronic device (1000) may filter out at least one keyword corresponding to a keyword whose probability of being acquired in the entire section of the target image is less than a second threshold value among the plurality of keywords (310) of the target image. In one embodiment, the electronic device (1000) may identify a keyword whose probability of being acquired in the entire section of the target image is less than a second threshold value among the plurality of keywords (310) of the target image based on the probability (330) of acquiring the plurality of keywords of the target image. In one embodiment, the electronic device (1000) may filter out a keyword corresponding to the identified keyword among the plurality of keywords (410-2) of the query. For example, if the second threshold value is 0.3 as shown in FIG. 4B , the electronic device (1000) may identify keywords 'red' and 'joy' whose frequencies of being acquired in the entire section of the target image are less than 0.3 among the plurality of keywords (310) of the target image. And, the electronic device (1000) can obtain the plurality of keywords (422-2) 'Christmas' and 'bar' of the query for which probability-based filtering has been performed by filtering the keyword 'red' identified from the plurality of keywords (410-2) of the query.

[0094] In Fig. 4b, both the frequency-based filtering and the probability-based filtering are illustrated as having a second threshold of 0.3, but the second threshold of the frequency-based filtering and the second threshold of the probability-based filtering may be different from each other. In addition, the electronic device (1000) may perform both the frequency-based filtering and the probability-based filtering, thereby filtering out keywords whose frequency obtained in the entire section of the target image is less than the second threshold and keywords whose probability obtained in the entire section of the target image is less than the second threshold from among the plurality of keywords (410-2) of the query.

[0095] In this way, the electronic device (1000) according to one embodiment of the present disclosure may filter out keywords (410-2) of a query, among a plurality of keywords (310) of a target image, from among a plurality of keywords of a query, keywords whose probability or frequency of being acquired in the entire section of the target image is less than a second threshold value. Here, the keywords whose probability or frequency of being acquired in the entire section of the target image is less than the second threshold value may be keywords indicating the context of a frame that is not related to the context of the target image itself. Therefore, by including the keyword in the plurality of keywords of the query, the similarity of each frame to the query may be unnecessarily lowered.

[0096] That is, the electronic device (1000) can filter out keywords that have a significantly low probability or frequency of being acquired across the entire section of the target image, thereby excluding keywords that are not helpful in searching the section corresponding to the query from among the multiple keywords of the query. Accordingly, the electronic device (1000) can more accurately extract frames corresponding to the query input by the user from the target image.

[0097] Meanwhile, the filtering of the plurality of keywords of the query described in FIGS. 4A and 4B may be performed simultaneously. In other words, the electronic device (1000) may filter at least one keyword corresponding to a keyword whose frequency or probability obtained in the entire section of the target image is greater than or equal to a first threshold value among the plurality of keywords (310) of the target image, and at least one keyword corresponding to a keyword whose frequency or probability obtained in the entire section of the target image is less than or equal to a second threshold value, from the plurality of keywords of the query. Accordingly, the electronic device (1000) may more accurately extract a frame corresponding to a query input by a user from the target image.

[0098] FIG. 5 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to filter at least one keyword that does not correspond to a plurality of keywords of a target image.

[0099] Referring to FIG. 5, the electronic device (1000) may filter out at least one keyword that does not correspond to the plurality of keywords (310) of the target image among the plurality of keywords (510) of the query. In one embodiment, the electronic device (1000) may identify a keyword that does not correspond to the plurality of keywords (310) of the target image among the plurality of keywords (510) of the query. In one embodiment, the electronic device (1000) may filter the identified keyword from the plurality of keywords (510) of the query.

[0100] For example, the plurality of keywords (310) of the target image may include 'bar', 'red', 'Christmas', 'people', 'toast', 'joy', and 'drink', and the plurality of keywords (510) of the query may include 'can', 'food', 'toast', and 'drink'. In this case, among the plurality of keywords (510) of the query, 'food', 'toast', and 'drink' are also included in the plurality of keywords (310) of the target image, but 'can' may not be included in the plurality of keywords (310) of the target image. In this case, the electronic device (1000) may obtain the plurality of keywords (520) of the filtered query, 'toast' and 'drink', by filtering 'can' from the plurality of keywords (510) of the query.

[0101] In FIG. 5, the electronic device (1000) is illustrated as performing filtering only on keywords that do not correspond to the plurality of keywords (310) of the target image among the plurality of keywords (510) of the query, but the filtering of FIG. 5 and the filtering of FIGS. 4A and 4B may be performed together. For example, the filtering of FIG. 5 may be performed on the plurality of keywords of the filtered query of FIG. 4A or the plurality of filtered keywords of FIG. 4B, or on the plurality of keywords of the filtered query for which both the filtering of FIGS. 4A and 4B have been performed.

[0102] In this way, the electronic device (1000) according to one embodiment of the present disclosure can filter out keywords that do not correspond to the plurality of keywords (310) of the target image among the plurality of keywords (510) of the query. Here, a keyword that is not included in the plurality of keywords (310) of the target image among the plurality of keywords (510) of the query may be a keyword that cannot be considered when extracting a frame corresponding to the query from the target image, but by being included in the plurality of keywords of the query, the keyword may affect the similarity of each frame to the query.

[0103] That is, the electronic device (1000) can filter out keywords that are not included in the plurality of keywords of the target image, thereby excluding keywords that are not helpful in searching for a section corresponding to the query from the plurality of keywords of the query. Accordingly, the electronic device (1000) can more accurately extract frames corresponding to the query input by the user from the target image.

[0104] FIG. 6 is a diagram illustrating a method for an electronic device according to an embodiment of the present disclosure to determine at least one frame to be extracted from a target image based on a plurality of keywords of a filtered query.

[0105] In one embodiment, the electronic device (1000) may determine at least one frame to be extracted from the target image (110) based on a plurality of keywords (610) of the filtered query. In one embodiment, the electronic device (1000) may calculate a similarity for the query of each of the plurality of unit sections of the target image (110) based on a frequency or probability that a keyword corresponding to the plurality of keywords (610) of the filtered query is obtained in the plurality of unit sections of the target image (110).

[0106] In one embodiment, the electronic device (1000) may calculate a similarity based on the number of keywords corresponding to the plurality of keywords (610) of the filtered query obtained in each of the plurality of unit sections of the target image, based on the plurality of keywords (310) of the target image obtained in each of the plurality of unit sections of the target image (110). For the convenience of description of the invention, the similarity based on the number of keywords corresponding to the plurality of keywords (610) of the filtered query obtained in each of the plurality of unit sections of the target image may be referred to as a count-based similarity. In one embodiment, the electronic device (1000) may calculate a value obtained by dividing the number of the plurality of keywords of the filtered query included in the keywords obtained from the unit section by the total number of the plurality of keywords of the filtered query as the count-based similarity.

[0107] For example, among the plurality of keywords (310) of the target image, the keywords obtained from the third frame (110-3) of the target image (110) may be 'bar', 'people', and 'toast', and among these, the keyword included in the plurality of keywords (610) of the filtered query may be 'toast'. In this case, the electronic device (1000) may calculate the number-based similarity for the query of the third frame (110-3) as 0.5, which is the value obtained by dividing 1, which is the number of the plurality of keywords (610) of the filtered query included in the keyword obtained from the third frame (110-3), by 2, which is the total number of the plurality of keywords (610) of the filtered query.

[0108] As another example, among the plurality of keywords (310) of the target image, the keywords obtained from the fourth frame (110-4) of the target image (110) may be 'bar', 'people', and 'toast', and among them, the keywords included in the plurality of keywords (610) of the filtered query may be 'toast' and 'joy'. In this case, the electronic device (1000) may calculate the number-based similarity for the query of the fourth frame (110-4) as 1, which is the value obtained by dividing the number 2 of the plurality of keywords (610) of the filtered query included in the keywords obtained from the fourth frame (110-4) by the total number 2 of the plurality of keywords (610) of the filtered query. The electronic device (1000) can calculate the number-based similarity (621) for each query of a plurality of unit sections of the target image (110) by calculating the similarity for the queries of the first frame (110-1), the second frame (110-2), and the fifth frame (110-5) in the same manner.

[0109] In one embodiment, the electronic device (1000) may calculate a similarity based on a probability that a keyword corresponding to the plurality of keywords (610) of the filtered query is acquired in each of the plurality of unit sections of the target image, based on a probability that the plurality of keywords (310) of the target image are acquired in each of the plurality of unit sections of the target image (110). For the convenience of description of the invention, the similarity based on the probability that a keyword corresponding to the plurality of keywords (610) of the filtered query is acquired in each of the plurality of unit sections of the target image may be referred to as a probability-based similarity. In one embodiment, the electronic device (1000) may calculate a value obtained by dividing a sum of probabilities that the plurality of keywords (610) of the filtered query included in the keywords acquired from the unit section by the total number of the plurality of keywords (610) of the filtered query as the probability-based similarity.

[0110] For example, the probability that the plurality of keywords (610) of the filtered query among the plurality of keywords (310) of the target image are obtained in the second frame (110-2) of the target image (110) may be 0.2 for 'toast' and 0.1 for 'joy'. In this case, the electronic device (1000) may calculate the probability-based similarity of the second frame (110-2) as 0.15, which is the value obtained by dividing the sum of the probabilities of the plurality of keywords (610) of the filtered query being obtained in the second frame (110-2), which is 0.3, by the total number of the plurality of keywords (610) of the filtered query, which is 2.

[0111] As another example, the probability that the plurality of keywords (610) of the filtered query among the plurality of keywords (310) of the target image are obtained in the fourth frame (110-4) of the target image (110) may be 0.9 for 'toast' and 0.7 for 'joy'. In this case, the electronic device (1000) may calculate the probability-based similarity of the second frame (110-2) as 0.8, which is the value obtained by dividing the sum of the probabilities of the plurality of keywords (610) of the filtered query being obtained in the second frame (110-2), which is 1.6, by the total number of the plurality of keywords (610) of the filtered query, which is 2. The electronic device (1000) can calculate the probability-based similarity (622) for the query of each of the plurality of unit sections of the target image (110) by calculating the similarity for the query of the first frame (110-1), the third frame (110-3), and the fifth frame (110-5) in the same manner.

[0112] In one embodiment, the electronic device (1000) can identify a unit section among a plurality of unit sections of the target image (110) in which the calculated similarity is greater than or equal to a third threshold value. In one embodiment, if the calculated similarity includes both a count-based similarity (621) and a probability-based similarity (622), the electronic device (1000) can identify a unit section in which at least one of the count-based similarity (621) and the probability-based similarity (622) is greater than or equal to the third threshold value. In one embodiment, the third threshold value compared with the count-based similarity (621) and the third threshold value compared with the probability-based similarity (622) may be different from each other. For example, if the third threshold value is 0.5, the electronic device (1000) can identify the third frame (110-3) and the fourth frame (110-4) among the first to fifth frames (110-1 to 110-5) of the target image (110) having a calculated similarity of 0.5 or greater based on at least one of the calculated number-based similarity (621) and the probability-based similarity (622).

[0113] In one embodiment, the electronic device (1000) may determine at least one frame to be extracted from the target image (110) in a unit section of the identified target image (110). In one embodiment, if the unit section of the identified target image (110) is a frame, the electronic device (1000) may determine the identified frame as at least one frame to be extracted, or may determine a frame included in a preset time section (e.g., from 1 second before to 1 second after the identified frame) based on the identified frame as at least one frame to be extracted. In one embodiment, if the unit section of the identified target image (110) includes a plurality of frames, the electronic device (1000) may determine the plurality of frames included in the identified unit section as at least one frame to be extracted, or may determine a frame obtained according to a preset criterion (e.g., a frame located in the middle of the unit section or a frame sampled at a preset time interval, etc.) among the plurality of frames as at least one frame to be extracted.

[0114] For example, the electronic device (1000) can identify the unit sections of the target image (110) whose calculated similarity is greater than or equal to a third threshold value as the third frame (110-3) and the fourth frame (110-4). Then, the electronic device (1000) can determine the identified third frame and fourth frame (110-4) as at least one frame to be extracted from the target image (110). In addition, a plurality of frames of the target image (110) included in a time section within 1 second from the third frame (110-3) and the fourth frame (110-4) can also be further determined as at least one frame to be extracted from the target image (110).

[0115] In one embodiment, the electronic device (1000) may determine at least one frame to be extracted between the first and second sections if the time interval between the first and second sections, which are unit sections of the identified target image (110), is less than or equal to a preset time interval. For example, the electronic device (1000) may identify the unit sections of the target image (110) whose calculated similarity is greater than or equal to a third threshold value as the third frame (110-3) and the fourth frame (110-4). And, if the electronic device (1000) identifies that the interval between the third frame (110-3) and the fourth frame (110-4) is less than or equal to a preset time interval of 7 seconds, the electronic device (1000) can determine the third frame (110-3) and the fourth frame (110-4) and the sixth frame (110-6) and the seventh frame (110-7) between the third frame (110-3) and the fourth frame (110-4) as at least one frame to be extracted from the target image (110).

[0116] In one embodiment, the electronic device (1000) may extract at least one determined frame from the target image (110) and display the at least one extracted frame through the display of the electronic device (1000). In one embodiment, if there are multiple extracted at least one frames, the electronic device (1000) may combine the extracted frames to generate an output image and display the generated output image. For example, the electronic device (1000) may determine the at least one frame to be extracted from the target image (110) as the third frame (110-3), the fourth frame (110-4), the sixth frame (110-6), and the seventh frame (110-7). In this case, although not illustrated, the surrounding frames of the third frame (110-3), the fourth frame (110-4), the sixth frame (110-6), and the seventh frame (110-7) may also be further determined as at least one frame to be extracted from the target image (110). In one embodiment, the electronic device (1000) may extract a third frame (110-3), a fourth frame (110-4), a sixth frame (110-6), and a seventh frame (110-7) from the target image (110), combine the extracted frames to generate an output image (630), and display the generated output image (630) through a display. However, the present invention is not necessarily limited to the above-described example, and each of the third frame (110-3), the fourth frame (110-4), the sixth frame (110-6), and the seventh frame (110-7) may be displayed as separate images.

[0117] FIG. 7 is a diagram illustrating modules included in an electronic device according to one embodiment of the present disclosure. An electronic device (1000) according to one embodiment of the present disclosure may include a keyword extraction module (722), a keyword filtering module (724), and a frame extraction module (726).

[0118] The keyword extraction module (722), the keyword filtering module (724), and the frame extraction module (726) included in the electronic device (1000) of FIG. 7 may be components classified based on their functions or roles. The keyword extraction module (722), the keyword filtering module (724), and the frame extraction module (726) included in the electronic device (1000) of FIG. 7 may be software components implemented by the processor of the electronic device (1000) executing a program stored in a memory, or may be virtual components in which an actual matching hardware device exists. In other words, the operations performed by the processor of the electronic device (1000) executing a program or instructions stored in a memory may be classified into a plurality of groups based on their functions or purposes, and the entities performing the operations included in each of the classified groups may be expressed as the keyword extraction module (722), the keyword filtering module (724), and the frame extraction module (726) included in the electronic device (1000) of FIG. 7. Accordingly, the operations and functions described as being performed by the keyword extraction module (722), keyword filtering module (724), and frame extraction module (726) included in the electronic device (1000) of FIG. 7 can be seen as actually being performed by the electronic device (1000) or the processor of the electronic device (1000) executing a program or instruction stored in the memory.

[0119] In one embodiment, the target image (110) may be input to the keyword extraction module (722). In one embodiment, the keyword extraction module (722) may obtain a plurality of keywords of the target image (110) based on the target image (110). In one embodiment, the plurality of keywords of the target image (110) may include keywords indicating the context of each of the plurality of unit sections of the target image (110). In one embodiment, information about the plurality of keywords of the target image (110) including keywords indicating the context of each of the plurality of unit sections of the target image (110) and information about the probability that the plurality of keywords of the target image (110) are obtained in each of the plurality of unit sections may be output.

[0120] In one embodiment, the keyword extraction module (722) can obtain a plurality of keywords of the query (710) based on the query (710) for section search of the target image (110). In one embodiment, the query (710) for section search of the target image (110) can include at least one of a text query (712) which is text related to the query, an image query (714) which is an image related to the query, and an image query (716) which is an image related to the query. In one embodiment, the keyword extraction module (722) can output information on the plurality of keywords of the identified query (710).

[0121] In one embodiment, data output from the keyword extraction module (722) may be input to the keyword filtering module (724). Since the method by which the keyword extraction module (722) obtains multiple keywords of the target image (110) or multiple keywords of the query (710) may correspond to the operations of the electronic device (1000) described in the present disclosure, a redundant description will be omitted.

[0122] In one embodiment, the keyword filtering module (724) can filter at least one keyword among the plurality of keywords of the query (710) based on the plurality of keywords of the target image (110). In one embodiment, the keyword filtering module (724) can identify the frequency or probability at which the plurality of keywords of the target image are obtained in the entire section of the target image (110) based on information about the plurality of keywords of the target image (110) and information about the probability at which the plurality of keywords of the target image (110) are obtained. In one embodiment, the keyword filtering module (720) can filter at least one keyword among the plurality of keywords of the query (710) based on the identified frequency or probability. In one embodiment, the keyword filtering module (724) can output information about the plurality of keywords of the filtered query (710).

[0123] In one embodiment, data output from the keyword filtering module (724) may be input to the frame extraction module (726). Since the operation of the keyword filtering module (724) filtering multiple keywords of the query (710) may correspond to the operations of the electronic device (1000) described in the present disclosure, a redundant description will be omitted.

[0124] In one embodiment, the target image (110) may be input to the frame extraction module (726). In one embodiment, the frame extraction module (726) may determine at least one frame to be extracted from the target image (110) based on the target image (110) and a plurality of keywords of the filtered query (710). In one embodiment, the frame extraction module (726) may calculate a similarity for each query of a plurality of unit sections of the target image (110) based on the plurality of keywords of the filtered query (710), and determine at least one frame (730) to be extracted from the target image (110) based on the calculated similarity. In one embodiment, the keyword extraction module (726) may extract the determined at least one frame (730) from the target image (110) or output information for identifying the determined at least one frame (730).

[0125] In one embodiment, at least one extracted frame (730) output from the keyword extraction module (726) may be displayed through the display of the electronic device (1000). In one embodiment, an output image including at least one frame (730) determined based on information for identifying at least one frame (730) output from the keyword extraction module (726) may be generated, and the generated output image may be displayed through the display of the electronic device (1000). Since the operation of the keyword extraction module (726) to determine at least one frame to be extracted from the target image (110) may correspond to the operations of the electronic device (1000) described in the present disclosure, a redundant description will be omitted.

[0126] FIG. 8 is a diagram illustrating a keyword extraction module according to one embodiment of the present disclosure.

[0127] Referring to FIG. 8, the keyword extraction module (722) can obtain a plurality of keywords (810) of the query based on the query (710) for searching a section of the target image (110).

[0128] In one embodiment, the keyword extraction module (722) can identify the probability that a plurality of candidate keywords will be obtained from a query (710) for searching a section of a target image (110). In one embodiment, the keyword extraction module (722) can obtain candidate keywords identified as having a probability greater than or equal to a preset probability obtained from the query (710) for searching a section of a target image (110) as a plurality of keywords (810) of the query for searching a section of a target image (110).

[0129] For example, when a text query (712) "a photo of people toasting" is input, the keyword extraction module (722) can identify the probability that a plurality of candidate keywords are obtained from the text query (712) and output candidate keywords 'people' and 'toast' whose identified probabilities are greater than or equal to a preset probability as keywords (812) of the text query. As another example, when an image query (714) is input, the keyword extraction module (722) can identify the probability that a plurality of candidate keywords are obtained from the image query (714) and output candidate keywords 'people', 'toast', 'joy' and 'bar' whose identified probabilities are greater than or equal to a preset probability as keywords (814) of the image query. As another example, when an image query (716) is input, the keyword extraction module (722) can identify the probability that a plurality of candidate keywords are obtained from the image query (716), and output candidate keywords 'can', 'food', 'toast', and 'drink' whose identified probabilities are greater than or equal to a preset probability as keywords (816) of the image query. Here, the keyword extraction module (722) can identify the probability that a plurality of candidate keywords are obtained from each of a plurality of frames constituting the image query (716), and output candidate keywords whose average of the identified probabilities is greater than or equal to a preset probability as keywords (816) of the image query.

[0130] In the examples described above, the keyword extraction module (722) is described as outputting keywords (812) of the text query, keywords (814) of the image query, and keywords (816) of the image query based on the text query (712), the image query (714), and the image query (716), respectively, but is not necessarily limited thereto. For example, the keyword extraction module (722) may obtain a plurality of keywords (810) of the query based on two or more of the text query (712), the image query (714), and the image query (716). In this case, the keyword extraction module (722) may identify the probability that a plurality of candidate keywords are obtained from each of the input data, and may identify candidate keywords whose average of the identified probabilities is greater than or equal to a preset probability as the plurality of keywords (810) of the query, or may identify all keywords of each of the input data as the plurality of keywords (810) of the query. As another example, if the keyword extraction module (722) is an artificial intelligence model, it can obtain multiple keywords (810) of a query based on data of different formats by including a layer that independently processes data of different formats to obtain feature vectors of the data of different formats and performs combination steps such as vector concatenation, vector averaging, and weighted summing of the obtained feature vectors.

[0131] In one embodiment, the keyword extraction module (722) may obtain a plurality of keywords (818) of the target image from the target image (110). In one embodiment, the plurality of keywords (818) of the target image may include keywords representing the context of each of the plurality of unit sections of the target image (110).

[0132] In one embodiment, the keyword extraction module (722) can identify the probability that a plurality of candidate keywords are obtained from each of a plurality of unit sections of the target image (110). In one embodiment, the keyword extraction module (722) can identify a candidate keyword, which is identified as having a probability greater than or equal to a preset probability obtained from each of a plurality of unit sections of the target image (110), as a keyword for each of the plurality of unit sections of the target image (110). In one embodiment, the keyword extraction module (722) can obtain information about a plurality of keywords (818) of the target image obtained from each of the plurality of unit sections of the target image (110) and information about a probability that the plurality of keywords (818) of the target image are obtained from each of the plurality of unit sections of the target image (110).

[0133] For example, when a target image (110) is input, the keyword extraction module (722) can identify the probability that a plurality of candidate keywords are obtained in each of a plurality of frames, which are a plurality of unit sections of the target image (110), and output candidate keywords whose average of the identified probabilities is greater than or equal to a preset probability as a plurality of keywords (818) of the target image. In addition, the keyword extraction module (722) can output information about the plurality of keywords (818) of the target image obtained in each of the plurality of frames and information about the probability that the plurality of keywords (818) of the target image are obtained in each of the plurality of frames.

[0134] In one embodiment, the keyword extraction module (722) may be an artificial intelligence model that outputs the probability of obtaining a keyword representing the context of the text, image, and video based on the input text, image, and video. Here, the keyword for which the keyword extraction module (722) outputs the probability of being obtained may correspond to a plurality of keyword candidates. In other words, the keyword extraction module (722) may be an artificial intelligence model that outputs the probability of obtaining a plurality of keyword candidates from the input data based on the input data. However, the present invention is not necessarily limited to the above-described example, and the keyword extraction module (722) may be an artificial intelligence model that outputs a keyword among the plurality of keyword candidates whose probability of being obtained from the input data is greater than or equal to a preset probability.

[0135] In one embodiment, the keyword extraction module (722) may be trained based on a training data set. In one embodiment, the training data set may be stored in the memory of the electronic device (1000) and used for training the keyword extraction module (722). In one embodiment, the training data set may be stored in an external electronic device, and training of the keyword extraction module (722) may be performed in the external electronic device, and the trained keyword extraction module (722) may be stored in the memory of the electronic device (1000).

[0136] In one embodiment, the training data set may include training text and ground-truth keywords representing the context of the training text. The keyword extraction module (722) may be trained to output a probability of obtaining a plurality of keywords representing the context of the training text based on the training text and / or a plurality of keywords representing the context of the training text by performing training based on the training data set.

[0137] In one embodiment, the training data set may include training images and correct keywords representing the context of the training images. The keyword extraction module (722) may be trained to output a probability of obtaining a plurality of keywords representing the context of the training text based on the training data set and / or a plurality of keywords representing the context of the training text by performing training based on the training data set.

[0138] In one embodiment, the training data set may include training images and correct keywords representing the context of the training images. Here, the training images and correct keywords representing the context of the training images may include correct keywords representing the context of each of a plurality of frames of the training images and the context of each of the plurality of frames of the training images. The keyword extraction module (722) may be trained to output a probability of obtaining a plurality of keywords representing the context of the training images and / or a plurality of keywords representing the context of the training images by performing training based on the training data set.

[0139] In one embodiment, the keyword extraction module (722) may be trained using a loss function calculated based on the difference between data output during the training process and the correct keyword. In one embodiment, the loss function may include various loss functions such as cross-entropy loss and cosine similarity loss, and is not necessarily limited to the examples described above.

[0140] FIG. 9 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to search for a section of a target image based on a query.

[0141] Referring to FIG. 9, the electronic device (1000) can display an image output UI (910), a query input UI (920), and a search result UI (930) through the display of the electronic device (1000).

[0142] In one embodiment, the image output UI (910) may include a preset area where the image is displayed and various UI elements. In one embodiment, the electronic device (1000) may display a target image (912) in the preset area of ​​the image output UI (910). In one embodiment, the image output UI (910) may include a media controller (914) for controlling various operations related to the displayed image. In one embodiment, the electronic device (1000) may include a plurality of buttons corresponding to operations such as Play, Pause, Fast Forward, and Rewind for the displayed image. In one embodiment, the electronic device (1000) may obtain a user input for selecting one of the plurality of buttons of the media controller (914) and perform an operation corresponding to the selected button. For example, the electronic device (1000) can play a target image (912) on the image output UI (910) based on obtaining a user input that selects a button corresponding to a playback operation of the media controller (914).

[0143] In one embodiment, the image output UI (910) may include a button (916) for entering a query to search for a section of the target image (912) (e.g., "Find a desired scene"). In one embodiment, the electronic device (1000) may display a query input UI (920) based on obtaining a user input selecting the button (916) for entering a query. In one embodiment, the electronic device (1000) may include a button (918) for entering a still image of the target image (912) as a query (e.g., "Find a similar scene").

[0144] In one embodiment, when the electronic device (1000) obtains a user input of selecting a button (918) for inputting a still image of the target image (912) as a query, the electronic device (1000) may obtain a still image of the target image (912) displayed on the image output UI (910) (or an image at the time when the user input is obtained). In one embodiment, the electronic device (1000) may obtain a plurality of keywords representing the context of the obtained image as a plurality of keywords of a query for searching a section of the target image (110), and may filter the obtained plurality of keywords to obtain a plurality of keywords (913) of a filtered image query. In one embodiment, when the electronic device (1000) obtains the plurality of keywords (913) of the filtered image query, the electronic device (1000) may determine at least one frame to be extracted from the target image (912) based on the plurality of keywords (913) of the obtained filtered image query, and may extract the determined at least one frame and display it through a search result UI (930).

[0145] In one embodiment, the query input UI (920) may include a text field (922) for receiving text related to a query. In one embodiment, the electronic device (1000) may obtain a text query (923) input through the text field (922). In one embodiment, the electronic device (1000) may obtain a plurality of keywords indicating a context of the text query (923) based on the text query (923). In one embodiment, the electronic device (1000) may filter the plurality of keywords of the text query (923) to obtain a plurality of keywords (944) of the filtered text query.

[0146] In one embodiment, the query input UI (920) may include a button (924) for receiving an image related to the query (e.g., “Select Photo”). In one embodiment, when the electronic device (1000) obtains a user input for selecting the button (924) for receiving an image, the electronic device (1000) may display a plurality of images stored in the memory of the electronic device (1000) and obtain a user input for selecting an image query (925) that is one of the displayed plurality of images. In one embodiment, the electronic device (1000) may obtain a plurality of keywords of the image query (925) based on the image query (925). In one embodiment, the electronic device (1000) may filter the plurality of keywords of the image query (925) to obtain a plurality of keywords (946) of the filtered image query.

[0147] In one embodiment, the query input UI (920) may include a button (926) for receiving an image related to the query (e.g., “Select Image”). In one embodiment, when the electronic device (1000) obtains a user input for selecting the button (926) for receiving an image, the electronic device (1000) may display a plurality of images stored in the memory of the electronic device (1000) and obtain a user input for selecting an image query (927) among the displayed plurality of images. In one embodiment, the electronic device (1000) may obtain a plurality of keywords of the image query (927) based on the image query (927). In one embodiment, the electronic device (1000) may filter the plurality of keywords of the image query (927) to obtain a plurality of keywords (948) of the filtered image query.

[0148] In one embodiment, the query input UI (920) may include a button (928) (e.g., “OK”) for executing an image search based on a query. In one embodiment, when the electronic device (1000) obtains a user input of selecting the button (928) for executing an image search, the electronic device (1000) may determine at least one frame to be extracted from the target image (110) based on a plurality of keywords of a filtered query obtained through the query input UI (920), and may extract the determined at least one frame and display it through the search result UI (930).

[0149] In one embodiment, the search result UI (930) may include a preset area where at least one frame (932) extracted from the target image (912) is displayed and various UI elements. In one embodiment, the electronic device (1000) may display an output image composed of a plurality of frames extracted from the target image (912) in the preset area of ​​the search result UI (930), or may display an image of a plurality of frames extracted from the target image (912) in the preset area of ​​the search result UI (930). In one embodiment, the electronic device (1000) may obtain a user input for controlling various operations related to an image or video displayed through the search result UI (930) through the media controller (914). For example, the electronic device (1000) may play an output image composed of a plurality of frames extracted from the target image (912) on the search result UI (930) based on obtaining a user input for selecting a button corresponding to a play operation of the media controller (914). Additionally, the electronic device (1000) can display an image of a frame that is not displayed on the search result UI (930) among the images of a plurality of frames extracted from the target image (912) based on obtaining a user input for selecting a button corresponding to the operation of the next image confirmation of the media controller (914).

[0150] FIG. 10 is a detailed configuration diagram of an electronic device according to one embodiment of the present disclosure.

[0151] Referring to FIG. 10, an electronic device (1000) may include a memory (1100), a display (1200), a communication interface (1300), an input interface (1400), an output interface (1500), and a processor (1600). The memory (1100), the display (1200), the communication interface (1300), the input interface (1400), the output interface (1500), and the processor (1600) may each be electrically and / or physically connected to each other.

[0152] The components illustrated in FIG. 10 are merely according to one embodiment of the present disclosure, and the components included in the electronic device (1000) are not limited to those illustrated in FIG. 10. The electronic device (1000) according to one embodiment of the present disclosure may not include some of the components illustrated in FIG. 10, and may further include components not illustrated in FIG. 10.

[0153] The memory (1100) may store instructions or program codes for performing functions or operations of the electronic device (1000). In one embodiment, at least one instruction, algorithm, data structure, program code, and application program stored in the memory (1100) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0154] In one embodiment, the memory (1100) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a Mask ROM, a Flash ROM, etc.), a hard disk drive (HDD), or a solid state drive (SSD). The memory (1100) may not exist separately and may be configured to be included in the processor (1700). The memory (1100) may be configured as a volatile memory, a nonvolatile memory, or a combination of a volatile memory and a nonvolatile memory. A program or at least one instruction for performing operations according to embodiments described below may be stored in the memory (1100). The memory (1100) may provide stored data to the processor (1600) at the request of the processor (1700). In one embodiment, the memory (1100) may include at least one of a target image, a query for searching a section of the target image, information on a plurality of candidate keywords, a plurality of candidate keywords obtained from each of a plurality of unit sections of the target image, and information on a probability of obtaining the plurality of candidate keywords. In one embodiment, the memory (1100) may include at least one of a keyword extraction module (722), a keyword filtering module (724), and a frame extraction module (726). In one embodiment, the memory (1100) may include a training data set for training the keyword extraction module (722).However, it is not necessarily limited to the above-described examples, and the memory (1100) may further include various data necessary to perform the operations and functions of the electronic device (1000) disclosed in this specification.

[0155] The display (1200) is a component for displaying images and / or videos. In one embodiment, the display (1200) may be configured as a physical device including at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode (OLED), a flexible display, a 3D display, and an electrophoretic display. In one embodiment, the display (1200) may display at least one of the image output UI (910), the query input UI (920), and the search result UI (930) of FIG. 9 based on a signal received from the processor (1600). However, the present invention is not limited to the above-described example, and the display (1200) may display various UIs and information necessary for performing operations and functions of the electronic device (1000) disclosed herein based on a signal received from the processor (1600).

[0156] The communication interface (1300) is a component for the electronic device (1000) to communicate with an external electronic device. In one embodiment, the communication interface (1300) may perform data communication between the electronic device (1000) and the external electronic device using at least one of data communication methods including wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication.

[0157] In one embodiment, the communication interface (1300) may receive from an external electronic device or transmit to an external electronic device at least one of a target image, a query for searching a section of the target image, information about a plurality of candidate keywords, a plurality of candidate keywords obtained from each of a plurality of unit sections of the target image, and information about a probability of obtaining the plurality of candidate keywords, based on a signal received from the processor (1600). In one embodiment, the communication interface (1300) may receive from an external electronic device or transmit to an external electronic device data input to or output from at least one of a keyword extraction module (722), a keyword filtering module (724), and a frame extraction module (726), based on a signal received from the processor (1600). In one embodiment, the communication interface (1300) may receive from an external electronic device or transmit to an external electronic device at least one frame extracted from the target image, based on a signal received from the processor (1600). However, it is not necessarily limited to the above-described examples, and the communication interface (1300) can receive from or transmit to an external electronic device various data necessary to perform the operation and function of the electronic device (1000) disclosed in this specification.

[0158] The input interface (1400) is a component for receiving various user inputs. In one embodiment, the input interface (1400) may include a touch panel, a physical button, a microphone, etc. In one embodiment, information input through the input interface (1400) may be provided to the processor (1600). In one embodiment, a query for searching a section of a target image may be obtained through the input interface (1400). In one embodiment, UI elements included in the image output UI (910), query input UI (920), and search result UI (930) of FIG. 9 may be selected through the input interface (1400), or various user inputs related to the selected UI elements may be obtained. However, the present invention is not limited to the examples described above, and the input interface (1400) may obtain various data necessary for performing the operations and functions of the electronic device (1000) disclosed herein.

[0159] The output interface (1500) is a component for the electronic device (1000) to provide various information to the user. In one embodiment, the electronic device (1000) may include a speaker, which is a component that outputs sound. In one embodiment, the output interface (1500) may output a voice corresponding to text displayed through the display (1200) based on a signal received from the processor (1600). However, the present invention is not necessarily limited to the above-described example, and the output interface (1500) may output various voices or information for performing the operations and functions of the electronic device (1000) disclosed herein.

[0160] The processor (1600) can control the overall operations of the electronic device (1000). In one embodiment, the processor (1600) can include multiple processors. In one embodiment, at least one processor (1600) can perform the operations and functions of the electronic device (1000) disclosed herein by executing one or more instructions of a program stored in the memory (1100).

[0161] The processor (1600) may be configured as at least one of, for example, a Central Processing Unit, a microprocessor, a Graphic Processing Unit, Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), an Application Processor, a Neural Processing Unit, or an artificial intelligence processor designed with a hardware structure specialized for processing artificial intelligence models, but is not limited thereto.

[0162] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by a single processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first and second operations may be performed by a first processor and the third operation may be performed by a second processor. However, the embodiments of the present disclosure are not limited thereto.

[0163] One or more processors according to the present disclosure may be implemented as a single-core processor or a multi-core processor. If a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by a single core or by multiple cores included in one or more processors.

[0164] In one embodiment, at least one processor (1600) may obtain a plurality of keywords of a query for searching a section of a target image by executing at least one command. In one embodiment, at least one processor (1600) may obtain a plurality of keywords of a target image indicating a context of the target image by executing at least one command. In one embodiment, at least one processor (1600) may filter at least one keyword among the plurality of keywords of a query based on the plurality of keywords of the target image by executing at least one command. In one embodiment, at least one processor (1600) may determine at least one frame to be extracted from the target image based on the plurality of keywords of the filtered query by executing at least one command.

[0165] In one embodiment, the plurality of keywords of the target image may include keywords representing the context of each of the plurality of unit sections of the target image. In one embodiment, at least one processor (1600) may, by executing at least one command, identify a frequency or probability at which the plurality of keywords of the target image are acquired in the entire section of the target image based on the keywords representing the context of each of the plurality of unit sections. In one embodiment, at least one processor (1600) may, by executing at least one command, filter at least one keyword based on the identified frequency or identified probability.

[0166] In one embodiment, at least one processor (1600) may filter at least one keyword corresponding to a keyword whose identified frequency or identified probability is greater than or equal to a first threshold value among a plurality of keywords of a target image by executing at least one command.

[0167] In one embodiment, at least one processor (1600) may filter at least one keyword corresponding to a keyword having an identified frequency or identified probability less than a second threshold value among a plurality of keywords of the target image by executing at least one command.

[0168] In one embodiment, at least one processor (1600) can filter out at least one keyword that does not correspond to a plurality of keywords of a target image by executing at least one instruction.

[0169] In one embodiment, at least one processor (1600) may calculate a similarity for each query of a plurality of unit sections based on a plurality of keywords of a filtered query by executing at least one command. In one embodiment, at least one processor (1600) may identify a unit section among the plurality of unit sections whose calculated similarity is greater than or equal to a third threshold value by executing at least one command. In one embodiment, at least one processor (1600) may determine at least one frame in the identified unit section by executing at least one command.

[0170] In one embodiment, at least one processor (1600) may execute at least one command to calculate a similarity for a query based on the number or probability that keywords corresponding to a plurality of keywords of a filtered query are obtained in a plurality of unit intervals.

[0171] In one embodiment, at least one processor (1600) can determine at least one frame between the first interval and the second interval by executing at least one instruction, if the time interval between the identified unit intervals, the first interval and the second interval, is less than or equal to a preset time interval.

[0172] In one embodiment, at least one processor (1600) may obtain at least one of text, image, and video related to a query by executing at least one command. In one embodiment, at least one processor (1600) may obtain a plurality of keywords of the query from at least one of the obtained text, image, and video by executing at least one command.

[0173] However, it is not necessarily limited to the above-described examples, and at least one processor (1600) can perform the operation and function of the electronic device (1000) disclosed in the present specification by executing at least one instruction.

[0174] FIG. 11 is a detailed configuration diagram of a server according to one embodiment of the present disclosure.

[0175] Referring to FIG. 11, the server (2000) may include a memory (2100), a communication interface (2200), and a processor (2300). Each may be electrically and / or physically connected to each other.

[0176] The components illustrated in FIG. 11 are merely in accordance with one embodiment of the present disclosure, and the components included in the server (2000) are not limited to those illustrated in FIG. 11. The server (2000) according to one embodiment of the present disclosure may not include some of the components illustrated in FIG. 11, and may further include components not illustrated in FIG. 11.

[0177] The memory (2100) may store instructions or program codes for performing functions or operations of the server (2000). In one embodiment, at least one instruction, algorithm, data structure, program code, and application program stored in the memory (2100) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or assembler.

[0178] In one embodiment, the memory (2100) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a mask ROM, a flash ROM, etc.), a hard disk drive (HDD), or a solid state drive (SSD).

[0179] In one embodiment, the data stored in the memory (2100) may correspond to the data stored in the memory (2100) of the electronic device (1000) of FIG. 10, so redundant description will be omitted.

[0180] The communication interface (2200) is a component for the server (2000) to communicate with an external electronic device. In one embodiment, the communication interface (2200) may perform data communication between the server (2000) and the external electronic device using at least one of data communication methods including wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication.

[0181] In one embodiment, the operation and function of the communication interface (2200) may correspond to the operation and function of the communication interface (1300) of the electronic device (1000) of FIG. 10, so redundant description will be omitted.

[0182] The processor (2300) can control the overall operations of the server (2000). In one embodiment, the processor (2300) may include multiple processors. In one embodiment, at least one processor (2300) can perform the operations and functions of the electronic device (1000) disclosed herein by executing one or more instructions of a program stored in the memory (2100).

[0183] In one embodiment, the operations and functions performed by at least one processor (2300) by executing at least one instruction stored in the memory (2100) may correspond to the operations and functions of the processor (1600) of the electronic device (1000) of FIG. 10, so redundant descriptions will be omitted.

[0184] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include computer-readable instructions, data structures, or other data in a modulated data signal, such as program modules.

[0185] Additionally, a computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0186] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0187] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. A method for searching a section of an image based on a query, Step (S210) of obtaining multiple keywords of a query for searching a section of a target image; A step (S220) of obtaining a plurality of keywords of the target image representing the context of the target image; A step (S230) of filtering at least one keyword among the plurality of keywords of the query based on the plurality of keywords of the target image; and A method comprising: a step (S240) of determining at least one frame to be extracted from the target image based on a plurality of keywords of the filtered query.

2. In paragraph 1, Multiple keywords of the above target image are: Includes keywords representing the context of each of the multiple unit sections of the target image, The step of filtering at least one keyword among multiple keywords of the above query is: A step of identifying the frequency or probability at which a plurality of keywords of the target image are acquired in the entire section of the target image based on keywords representing the context of each of the plurality of unit sections; and A method comprising: a step of filtering the at least one keyword based on the identified frequency or the identified probability.

3. In paragraph 2, The step of filtering at least one keyword based on the identified frequency or the identified probability is: A method comprising: a step of filtering at least one keyword corresponding to a keyword whose identified frequency or identified probability is greater than or equal to a first threshold value among a plurality of keywords of the target image.

4. In paragraph 2, The step of filtering at least one keyword based on the identified frequency or the identified probability is: A method comprising: a step of filtering at least one keyword corresponding to a keyword among a plurality of keywords of the target image whose identified frequency or identified probability is less than a second threshold value.

5. In any one of paragraphs 1 to 4, The step of filtering at least one keyword among multiple keywords of the above query is: A method comprising: a step of filtering out at least one keyword that does not correspond to a plurality of keywords of the target image; 6. In any one of paragraphs 1 to 5, The step of determining at least one frame to be extracted from the target image comprises: A step of calculating the similarity for each of the plurality of unit sections based on the plurality of keywords of the filtered query; A step of identifying a unit section among the plurality of unit sections in which the calculated similarity is greater than or equal to a third threshold value; and A method comprising: determining at least one frame in the identified unit section; 7. In paragraph 6, The step of calculating the similarity for each of the above queries for each of the above multiple unit sections is: A method comprising: calculating a similarity for the query based on the number or probability of keywords corresponding to the plurality of keywords of the filtered query being obtained in the plurality of unit sections.

8. In an electronic device (2000) for searching a section of an image based on a query, A memory (2100) storing one or more instructions; and At least one processor (2300) that executes one or more instructions stored in the memory (2100); The electronic device (2000) executes the one or more instructions by the at least one processor (2300), Obtain multiple keywords for the query to search for a section of the target video, Obtaining multiple keywords of the target image representing the context of the target image, Filtering at least one keyword among the plurality of keywords of the query based on the plurality of keywords of the target image, An electronic device that determines at least one frame to be extracted from the target image based on a plurality of keywords of the filtered query.

9. In paragraph 8, Multiple keywords of the above target image are: Includes keywords representing the context of each of the multiple unit sections of the target image, The electronic device, wherein the at least one processor executes the one or more instructions, Identifying the frequency or probability that multiple keywords of the target image are acquired in the entire section of the target image based on keywords representing the context of each of the multiple unit sections, An electronic device that filters at least one keyword based on the identified frequency or the identified probability.

10. In paragraph 9, The electronic device, wherein the at least one processor executes the one or more instructions, An electronic device that filters at least one keyword corresponding to a keyword among a plurality of keywords of the target image whose identified frequency or identified probability is greater than or equal to a first threshold value.

11. In paragraph 9, The electronic device, wherein the at least one processor executes the one or more instructions, An electronic device that filters at least one keyword corresponding to a keyword among a plurality of keywords of the target image whose identified frequency or identified probability is less than a second threshold value.

12. In any one of paragraphs 8 to 11, The electronic device, wherein the at least one processor executes the one or more instructions, An electronic device that filters out at least one keyword that does not correspond to a plurality of keywords of the target image.

13. In any one of paragraphs 8 to 12, The electronic device, wherein the at least one processor executes the one or more instructions, Calculate the similarity for each of the plurality of unit sections based on the plurality of keywords of the filtered query, Identifying a unit section among the above multiple unit sections whose calculated similarity is greater than or equal to a third threshold value, An electronic device comprising: a step of determining at least one frame in the identified unit section; 14. In paragraph 13, The electronic device, wherein the at least one processor executes the one or more instructions, An electronic device that calculates a similarity to the query based on the number or probability of keywords corresponding to the plurality of keywords of the filtered query being obtained in the plurality of unit sections.

15. A computer-readable recording medium having recorded thereon a program for performing the method of any one of clauses 1 to 7 on a computer.

Images