Information processing device, information processing method and information processing program

Abstract

To provide an information processing device, an information processing method and an information processing program for performing retrieval using position information of an object included in a scene in a moving image.SOLUTION: In an information processing system in which a terminal device, an information processing device and a storage device are communicably connected via a wired or wireless network, an information processing device 10 includes an acquisition unit 30 for acquiring retrieval information including object information representing features of an object and position information of the object, a first extraction unit 32 and a second extraction unit 34 for extracting an image being the retrieval information including a candidate image suitable for object information and the object suitable for the position information from a plurality of retrieval object images, and an output unit 36 for outputting a retrieval result based on the retrieval information.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] A user selects and downloads an image with a desired color coordinate from videos managed in a service on a network such as the Internet, and watches the video on a terminal device. Various methods have been proposed for a user to search for a desired video from a large number of videos managed in such a service. For example, Patent Document 1 proposes a method of associating metadata related to a search for a scene included in a video with the video, and searching for a scene in the video that satisfies a desired search condition using the metadata. Patent Document 2 also proposes a method of adding time information to the metadata and further searching for a video using the time information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-076078 [Patent Document 2] JP 2008-005010 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the techniques of Patent Documents 1 and 2 cannot perform a search using the position of an object included in a scene in a video.

[0005] The present disclosure has been made in consideration of the above circumstances, and has an object to enable searches to be performed using position information of objects. [Means for solving the problem]

[0006] The information processing device according to the present disclosure includes: an acquisition unit that acquires search information including object information indicating a feature of an object and position information of the object; an extraction unit that extracts a result image that matches the search information from a plurality of search target images; and an output unit that outputs a search result based on the search information.

[0007] The information processing method according to the present disclosure includes: acquiring search information including object information indicating a feature of an object and position information of the object; extracting result images that match the search information from a plurality of search target images; A search result based on the search information is output.

[0008] An information processing program according to the present disclosure includes a step of causing a computer to acquire search information including object information indicating a feature of an object and position information of the object; extracting a result image matching the search information from a plurality of search target images; and outputting a search result based on the search information. Effect of the Invention

[0009] According to the present disclosure, a search can be performed using object position information. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of a schematic configuration of an information processing system according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to an embodiment of the present invention. [Diagram 3] 2 is a block diagram showing an example of a functional configuration of the information processing device according to the present embodiment. FIG. [Figure 4] FIG. 2 is a diagram illustrating a data structure of a moving image. [Diagram 5] FIG. 13 is a diagram showing candidate images. [Figure 6]FIG. 13 is a diagram showing a candidate image divided into a plurality of mesh-like small regions. [Figure 7] FIG. 13 is a diagram showing a candidate image in which a region corresponding to the center is set. [Figure 8] FIG. 13 is a diagram showing candidate images. [Figure 9] FIG. 13 is a diagram showing a candidate image whose center of gravity has been identified. [Figure 10] 4 is a flowchart showing a process performed in the present embodiment. [Figure 11] 4 is a flowchart showing a process performed in the present embodiment. [Figure 12] 4 is a flowchart showing a process performed in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Fig. 1 is a schematic configuration diagram of an information processing system 1 to which an information processing device 10 according to the present embodiment is applied. As shown in Fig. 1, the information processing system 1 includes a terminal device 4, an information processing device 10, and a storage device 11. The terminal device 4 and the information processing device 10 are connected in a communicable state via a wired or wireless network 6.

[0012] The terminal device 4 is at least one device having a microphone for voice input and a display on which the user 2 can play back videos, and may be, for example, a smartphone, a tablet terminal, a wearable terminal, or a personal computer, as appropriate. The information processing device 10 is at least one device installed, for example, at a provider that provides a video distribution service, and the service provider manages videos, searches for videos in response to a search request from the terminal device 4, and distributes them to the terminal device 4 of the user 2. The storage device 11 stores a plurality of videos managed by the service provider.

[0013] The user 2 accesses the information processing device 10 and searches for a video that the user 2 wishes to view on his / her own terminal device 4. Specifically, the user 2 inputs search information for identifying the desired video to the terminal device 4 by voice. The terminal device 4 transmits the search information to the information processing device 10, and receives a result image, which is the search result, from the information processing device 10. The functions of the information processing device 10 will be described below.

[0014] First, an example of a hardware configuration of the information processing device 10 will be described with reference to Fig. 2. As shown in Fig. 2, the information processing device 10 includes a CPU (Central Processing Unit) 21, a non-volatile storage unit 22, and a memory 23 as a temporary storage area. The information processing device 10 also includes a display 24 such as a liquid crystal display, an input unit 25 such as a touch panel, a keyboard, and a mouse, and a network I / F (InterFace) unit 26. The network I / F unit 26 performs wired or wireless communication with the terminal device 4 and other external devices. The CPU 21, the storage unit 22, the memory 23, the display 24, the input unit 25, and the network I / F unit 26 are connected to each other via a bus 28 such as a system bus and a control bus so that various information can be exchanged between them.

[0015] The storage unit 22 is realized by a storage medium such as, for example, a hard disk drive (HDD), a solid state drive (SSD), and a flash memory. An information processing program 27 for the information processing device 10 is stored in the storage unit 22. The CPU 21 reads the information processing program 27 from the storage unit 22, expands it in the memory 23, and executes the expanded information processing program 27. For example, a personal computer, a server computer, or the like can be appropriately applied as the information processing device 10.

[0016] Next, an example of a functional configuration of the information processing device 10 will be described with reference to Fig. 3. As shown in Fig. 3, the information processing device 10 includes an acquisition unit 30, a first extraction unit 32, a second extraction unit 34, and an output unit 36. The CPU 21 executes the information processing program 27, causing the CPU 21 to function as each of the functional units of the acquisition unit 30, the first extraction unit 32, the second extraction unit 34, and the output unit 36. Note that the information processing device 10 may be implemented with a function as a storage unit for storing and managing moving images in the storage device 11. The first extraction unit 32 and the second extraction unit 34 are examples of the extraction unit of the present disclosure.

[0017] The acquisition unit 30 acquires search information transmitted from the terminal device 4 of the user 2 via the network 6. The search information includes object information expressing the characteristics of a desired object contained in the video (hereinafter referred to as a search target image) stored in the storage device 11, and object position information. The object information includes, for example, at least one of the object type, object color, and object shape, but is not limited to this. The object type includes, for example, information such as a person, an animal, and what the object is. The position information includes information expressing the position of the object in the video frame.

[0018] The position information may represent the absolute position of an object within a frame, or may represent the positional relationship of two or more objects within a frame, i.e., the relative positions of the objects. An absolute position represents a position such as "center", "top right", or "bottom right" in one frame of a moving image. An example of a relative position represents the positional relationship of two or more objects in one frame of a moving image. An example of a relative position is "desired object A is located at the bottom left of another object B". Note that a frame is an example of one unit of an image in this disclosure.

[0019] In this embodiment, the user 2 inputs search information by voice from the terminal device 4. Therefore, the search information by voice is transmitted from the terminal device 4 to the information processing device 10. The acquisition unit 30 converts the acquired search information by voice into text. The acquisition unit 30 divides the search information converted into text into words using a natural language processing technique, and generates search tags for searching for videos. For example, if the search information is "the green object was in the middle", "green" and "middle" are generated as search tags.

[0020] It should be noted that the search information input by the user 2 at the terminal device 4 is not limited to voice, and may of course be text input using a keypad or keyboard.

[0021] FIG. 4 is a diagram showing a schematic data structure of a video stored in the storage device 11 in this embodiment. In this specification, a video is an image that is composed of a plurality of frames that represent image information, and that allows the movement of an object represented by the image information to be reproduced by continuously reproducing the plurality of frames. As shown in FIG. 4, a video 40 is divided into a plurality of chapters 40A-40D, each of which is composed of one or more frames, and metadata 42A-42D is assigned to each chapter. Note that the metadata 42A-42D is not limited to those assigned to the video 40. The metadata 42A-42D may be stored in the storage device 11 separately from the video 40, in association with the chapters 40A-40D of the video 40.

[0022] The metadata 42A to 42D include, but are not limited to, information on objects included in frames constituting the chapters 40A to 40D, timestamps of frames including objects, length of chapters, and information on the creation date and time of video. The information on objects includes at least one of object type, object color, and object shape, but is not limited to, the information on objects. For example, the metadata 42A for chapter 40A includes information indicating that a green object A is included in a frame with a timestamp such as 00:10 or 05:12, that a red triangular object B is included in a frame with a timestamp such as 05:12, that the length of chapter 40A is 15:00, and that the creation date and time is 2023 / 8 / 11. The metadata 42A includes information on each of the chapters 40A to 40D, but includes the timestamps of frames including objects. For this reason, the metadata is associated with one unit of image.

[0023] The first extraction unit 32 extracts candidate images matching object information included in the search information from the multiple search target images stored in the storage device 11. Specifically, the first extraction unit 32 compares a search tag generated from the search information with metadata assigned to each chapter of the multiple search target images, and extracts, as candidate images, video images including chapters to which metadata including information corresponding to the search tag is assigned from the multiple search target images. The first extraction unit 32 extracts candidate images from all of the multiple search target images. If a candidate image cannot be extracted, the first extraction unit 32 outputs information to that effect to the output unit 36.

[0024] Here, if the search information is "a green object was in the middle", the search tag representing the object information is "green". In this case, the first extraction unit 32 extracts, as a candidate image, a video including a chapter to which metadata including information representing that the object is green is added. FIG. 5 is a diagram showing candidate images. Note that FIG. 5 shows three candidate images 50A to 50C including green objects 52A to 52C, respectively, as an example. In FIG. 5, the objects 52A to 52C are green, as indicated by diagonal hatching. Note that the candidate image 50A includes a red triangular object 54A, and the candidate image 50C includes a red triangular object 54C that is smaller than the object 54A. In FIG. 5, the red color is indicated by horizontal hatching.

[0025] The second extraction unit 34 divides a frame containing an object in a candidate image into a plurality of small regions in a mesh pattern, and identifies a small region containing an object from among the plurality of small regions. The second extraction unit 34 then extracts a result image from the candidate image based on whether the position of the identified small region matches the position information included in the search information. In this embodiment, if a small region contains a predetermined proportion (e.g., 50%) of objects, the small region is identified as a small region containing an object.

[0026] Fig. 6 is a diagram showing a candidate image divided into a plurality of mesh-like small regions. In Fig. 6, the boundaries of the small regions are indicated by dashed lines. The second extraction unit 34 identifies a small region including the objects 52A to 52C from among the plurality of small regions. The position of the small region can be represented by, for example, a two-dimensional coordinate position with the lower left corner of the frame as the origin, but is not limited to this. In Fig. 6, the boundaries of the small regions including the objects 52A to 52C are indicated by thick solid lines.

[0027] In this embodiment, the search tag representing the position information included in the search information is "middle", which represents the absolute position of the object in one frame. Examples of absolute positions in the frame of the candidate image include the middle, edge, top left, bottom left, top right, and bottom right. In this embodiment, which of the multiple small regions these absolute positions correspond to is set in advance. For this reason, the second extraction unit 34 sets an area corresponding to the position represented by the search tag in the frame including the object in the candidate image. As in this embodiment, when the search tag representing the position information is "middle", the second extraction unit 34 sets an area corresponding to the middle of the frame of the candidate image among the multiple small regions. FIG. 7 is a diagram showing a candidate image in which an area corresponding to the middle is set. As shown in FIG. 7, areas 56A to 56C consisting of 9×4 small areas corresponding to the middle are set in the candidate images 50A to 50C. In FIG. 7, the boundaries of the areas 56A to 56C are indicated by double solid lines.

[0028] The second extraction unit 34 judges whether the small region including the objects 52A to 52C matches the position information based on whether the small region including the objects 52A to 52C is included in the region 56A to 56C corresponding to the center. In this embodiment, it is determined that the small region including the objects 52A to 52C matches the position information when a predetermined ratio (for example, 80%) or more of the small region including the objects 52A to 52C is included in the region 56A to 56C corresponding to the center. As shown in FIG. 7, in the candidate image 50A, all of the small regions including the object 52A are included in the region 56A. In the candidate image 50B, all of the small regions including the object 52B are not included in the region 56B. In the candidate image 50C, a part of the small region including the object 52C is included in the region 56C, but the included small region is less than 80% of the small region including the object 52C. Therefore, the second extraction unit 34 extracts the candidate image 50A as the result image 58 from the three candidate images 50A to 50C shown in FIG. 7.

[0029] In the above, the search information is "the green object was in the middle", and the position of the object is represented by the absolute position in the frame of one object. On the other hand, in the search information, the position information may be represented by the positional relationship of two or more objects. For example, if the search information is "there was a red triangle at the lower left of the green object", the position of the green object is represented relative to the position of the red triangular object. Hereinafter, the search process in which the second extraction unit 34 extracts a result image based on the relative positional relationship of two or more objects will be described. Note that the above-mentioned search based on the absolute position of the object is referred to as a search process based on the absolute position, and the search based on the relative position of the object is referred to as a search process based on the relative position of the object.

[0030] When the search information is "There was a red triangle at the bottom left of a green object," the acquisition unit 30 generates "green," "bottom left," and "red triangle" as search tags. In this case, the object information is "green" and "red triangle," and the position information is "a red triangle at the bottom left of a green object," which indicates the relative position of the red triangle object with respect to the green object.

[0031] The first extraction unit 32 extracts moving images including a green object and a red triangular object as candidate images from the plurality of search target images stored in the storage device 11. In this case, as shown in Fig. 8, the first extraction unit 32 extracts candidate images 50A and 50C from the plurality of search target images among the candidate images 50A to 50C shown in Fig. 5.

[0032] The second extraction unit 34 divides frames including objects in the candidate images 50A and 50C into a plurality of small regions in a mesh pattern, and specifies small regions including the green objects 52A and 52C and the red triangular objects 54A and 54C from among the plurality of small regions. In Fig. 8, the boundaries of the specified small regions are indicated by solid lines.

[0033] Next, the second extraction unit 34 identifies the centers of gravity of the small regions including the green objects 52A and 52C and the red triangular objects 54A and 54C. Fig. 9 is a diagram showing a candidate image with the centers of gravity identified. As shown in Fig. 9, in the candidate image 50A, the center of gravity G1 of the four small regions including the green object 52A and the center of gravity G2 of the five small regions including the red triangular object 54A are identified. In the candidate image 50C, the center of gravity G3 of the nine small regions including the green object 52C and the center of gravity G4 of the two small regions including the red triangular object 54C are identified.

[0034] The second extraction unit 34 identifies the object that is the origin from among the green object and the red object. The origin is identified by natural language processing the search information. For example, the origin is identified by determining the modifier when the search tag indicating the position is used as the subject. In this embodiment, since "there was a red triangle in the lower left of the green object", the green object is identified as the origin.

[0035] Then, the second extraction unit 34 sets a vector from the center of gravity of the object corresponding to the origin to the center of gravity of the object not corresponding to the origin. That is, in the candidate image 50A, a vector V1 is set from the center of gravity G1 of the green object 52A to the center of gravity G2 of the red triangular object. In the candidate image 50C, a vector V2 is set from the center of gravity G3 of the green object 52C to the center of gravity G4 of the red triangular object. The second extraction unit 34 determines the directions of the vectors V1 and V2, and extracts the candidate image in which the vector is set to the lower left, the direction of which matches the position information included in the search information, as the result image. As shown in FIG. 9, the direction of the vector V1 is the lower right, and the direction of the vector V2 is the lower left. Therefore, the second extraction unit 34 extracts the candidate image 50C from the candidate images 50A and 50C as the result image.

[0036] If the resultant image cannot be extracted, the second extraction unit 34 outputs information to that effect to the output unit 36.

[0037] The output unit 36 ​​outputs the search result based on the search information. That is, when a result image 58 is extracted, the output unit 36 ​​outputs the result image 58 as the search result. Specifically, a frame including an object included in the search information is specified in the result image 58, and the result image 58 paused at the specified frame is transmitted to the terminal device 4 via the network 6. Note that it is also possible to transmit only the frame including the object included in the search information to the terminal device 4, and transmit the result image 58 to the terminal device 4 after an instruction to play the image is given in the terminal device 4.

[0038] On the other hand, if the first extraction unit 32 does not extract a candidate image, and if the second extraction unit 34 does not extract a result image, a notice that no video corresponding to the search information exists is output as a search result. The notice may be, for example, a message such as "No corresponding image exists" or a voice message.

[0039] Next, the processing performed in this embodiment will be described. Fig. 10 to Fig. 12 are flow charts showing the processing performed in this embodiment. The processing is started when the acquisition unit 30 receives an instruction to search for a video from the terminal device 4, and first the acquisition unit 30 acquires voice search information transmitted from the terminal device 4 (step ST1). The acquisition unit 30 converts the voice search information into text (step ST2), and generates a search tag using a natural language processing technique (step ST3).

[0040] Next, the first extraction unit 32 judges whether or not a candidate image matching the object information included in the search information is extracted from the multiple search target images stored in the storage device 11 (step ST4). If step ST4 is negative, information to that effect is output to the output unit 36. The output unit 36 ​​notifies the terminal device 4 that there is no video image matching the search information (notification of no match: step ST5), and returns to a standby state. If step ST4 is positive, the first extraction unit 32 extracts a candidate image from the multiple search target images (step ST6).

[0041] Next, the second extraction unit 34 divides the frame including the object in the candidate image into a plurality of small regions in a mesh pattern, and identifies the small region including the object (step ST7). Next, the second extraction unit 34 determines whether the search information includes the relative positional relationship of two or more objects (step ST8). If the result of step ST8 is negative, the second extraction unit 34 performs a search process based on the absolute position of the object (step ST9). If the result of step ST8 is positive, the second extraction unit 34 performs a search process based on the relative position of the object (step ST10).

[0042] 11 is a flowchart of a search process based on an absolute position of an object. The second extraction unit 34 acquires a search tag that indicates the position of the object (step ST21), and sets an area corresponding to the position indicated by the search tag in the candidate image being processed (area setting: step ST22). Then, the second extraction unit 34 determines whether the small area including the object matches the position information based on whether the small area including the object is included in the set area (step ST23).

[0043] If step ST23 is negative, the second extraction unit 34 sets the search result for the candidate image being processed to NG (step ST24). If step ST23 is positive, the second extraction unit 34 sets the search result for the candidate image being processed to OK (step ST25). Following steps ST24 and ST25, the second extraction unit 34 determines whether or not there is a candidate image that has not been processed in the search process based on the absolute position (step ST26), and if step ST26 is positive, the process returns to step ST21 and repeats the processes from step ST21 onward. If step ST26 is negative, the search process based on the absolute position is terminated.

[0044] 12 is a flowchart of the search process based on the relative position of an object. The second extraction unit 34 acquires a search tag indicating the positions of two or more objects (step ST31), and specifies one of the two or more objects as the origin (step ST32). Then, the second extraction unit 34 specifies the center of gravity of the object included in the candidate image being processed (step ST33), and sets a vector from the center of gravity of the object corresponding to the origin to the center of gravity of the object not corresponding to the origin (step ST34). Then, the second extraction unit 34 determines whether the direction of the vector matches the position information (step ST35).

[0045] If step ST35 is negative, the second extraction unit 34 sets the search result for the candidate image being processed to NG (step ST36). If step ST35 is positive, the second extraction unit 34 sets the search result for the candidate image being processed to OK (step ST37). Following steps ST36 and ST37, the second extraction unit 34 determines whether or not there is a candidate image that has not been processed in the search process based on the relative position (step ST38), and if step ST38 is positive, the process returns to step ST31 and repeats the processes from step ST31 onwards. If step ST38 is negative, the search process based on the relative position is terminated.

[0046] Returning to Fig. 10, following steps ST9 and ST10, the second extraction unit 34 determines whether or not a result image has been extracted from the candidate images (step ST11), and if step ST11 is negative, the process proceeds to step ST5. If step ST11 is positive, the output unit 36 ​​outputs the search results. That is, the output unit 36 ​​identifies a frame in the result image that includes an object included in the search information (step ST12), transmits the result image paused at the identified frame to the terminal device 4 (step ST13), and ends the process.

[0047] In this manner, in the present embodiment, a result image that matches the search information is extracted from a plurality of search target images, and a search result based on the search information is output. Specifically, a candidate image that matches the object information included in the search information is extracted from a plurality of search target images, and a result image including an object that matches the position information included in the search information is extracted from the candidate image. Therefore, a desired video image can be searched for from a plurality of search target images using the position of an object included in the video image.

[0048] In addition, in this embodiment, one frame of a candidate image is divided into a plurality of small regions, a small region that includes an object is identified from the plurality of small regions, and a result image is extracted from the candidate image based on whether the position of the identified small region matches the position information. Therefore, the result image can be extracted relatively easily, and as a result, a desired video image can be searched for quickly.

[0049] In the above embodiment, one unit of images to be searched is one frame of a video, but this is not limiting. For example, one chapter of a video may be used as one unit to perform object search processing.

[0050] In the above embodiment, the search target images are moving images, but still images may be search target images, in which case one unit of the search target images is the still image itself.

[0051] In the above embodiment, the following various processors can be used as the hardware structure of the processing unit that executes various processes, such as the acquisition unit 30, the first extraction unit 32, the second extraction unit 34, and the output unit 36. As described above, the above various processors include a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units, as well as a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processes, such as an ASIC (Application Specific Integrated Circuit), etc.

[0052] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0053] As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as represented by computers such as client and server. Second, there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as represented by System on Chip (SoC). In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0054] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0055] In the above embodiment, the information processing program 27 in the information processing device 10 is stored in advance in the storage unit 22, but the present invention is not limited to this. The information processing program 27 may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. The information processing program 27 may also be downloaded from an external device via a network. Furthermore, the technology of the present disclosure extends to a storage medium that non-temporarily stores a program in addition to the program.

[0056] The following are appended claims to this disclosure. (Additional note 1) an acquisition unit that acquires search information including object information representing a feature of an object and position information of the object; an extraction unit that extracts a result image that matches the search information from a plurality of search target images; An information processing device comprising an output unit that outputs a search result based on the search information. (Additional note 2) The extraction unit includes a first extraction unit that extracts a candidate image that matches the object information from the plurality of search target images; and a second extraction unit that extracts an image including an object that matches the position information from the candidate images as the result image. (Additional note 3) The information processing device described in Appendix 2, wherein the first extraction unit refers to metadata associated with a unit of the plurality of search target images and extracts the candidate image based on whether the metadata contains information that matches the object information. (Additional note 4) The information processing device described in Appendix 3, wherein the second extraction unit divides one unit of the candidate image into a plurality of small regions, identifies a small region among the plurality of small regions that includes the object, and extracts the resultant image from the candidate image based on whether the position of the identified small region matches the position information. (Additional note 5) When the image to be searched is a video, the one unit is one frame of the video; 5. The information processing device according to claim 3, wherein the output unit outputs, as a search result, the result image paused in a frame including the object. (Additional note 6) 6. The information processing device according to any one of claims 1 to 5, wherein the output unit notifies the user that there is no search target image matching the search information when the result image cannot be extracted. (Additional note 7) 7. The information processing device according to claim 1, wherein the position information includes an absolute position of the object. (Additional note 8) 7. The information processing device according to claim 1, wherein the position information includes a relative position of the object. (Additional note 9) When the acquisition unit acquires the search information by voice, the acquisition unit converts the search information by voice into text; 9. The information processing device according to any one of claims 1 to 8, wherein the extraction unit extracts the result image by using the search information converted into the text. (Additional note 10) A computer acquires search information including object information representing a characteristic of an object and position information of the object; extracting result images that match the search information from a plurality of search target images; An information processing method that outputs a search result based on the search information. (Additional note 11) A step of acquiring, in a computer, search information including object information representing a feature of an object and position information of the object; extracting a result image matching the search information from a plurality of search target images; and outputting a search result based on the search information. [Explanation of symbols]

[0057] 1. Information Processing Systems 2. Users 4 Terminal Equipment 6 Network 10. Information processing device 21 CPU 22 Memory section 23 Memory 24 Display 25 Input section 26 Network Interface 27 Information Processing Programs 28 Bus 30 Acquisition Department 32 1st extraction part 34 Second extraction part 36 Output section

Claims

1. an acquisition unit that acquires search information including object information representing a feature of an object and position information of the object; an extraction unit that extracts a result image that matches the search information from a plurality of search target images; An information processing device comprising an output unit that outputs a search result based on the search information.

2. The extraction unit includes a first extraction unit that extracts a candidate image that matches the object information from the plurality of search target images; The information processing apparatus according to claim 1 , further comprising: a second extraction unit configured to extract, as the resultant image, an image including an object matching the position information from the candidate images.

3. The information processing device according to claim 2 , wherein the first extraction unit refers to metadata associated with one unit of the plurality of search target images, and extracts the candidate images based on whether the metadata includes information matching the object information.

4. The information processing device according to claim 3, wherein the second extraction unit divides one unit of the candidate image into a plurality of small regions, identifies a small region among the plurality of small regions that includes the object, and extracts the resultant image from the candidate image based on whether the position of the identified small region matches the position information.

5. When the image to be searched is a video, the one unit is one frame of the video; The information processing apparatus according to claim 3 , wherein the output unit outputs, as a search result, the resultant image paused in a frame including the object.

6. The information processing apparatus according to claim 1 , wherein the output unit notifies the user that there is no search target image matching the search information when the result image cannot be extracted.

7. The information processing apparatus according to claim 1 , wherein the position information includes an absolute position of the object.

8. The information processing apparatus according to claim 1 , wherein the position information includes a relative position of the object.

9. When the acquisition unit acquires the search information by voice, the acquisition unit converts the search information by voice into text; The information processing apparatus according to claim 1 , wherein the extraction unit extracts the result image by using the search information converted into the text.

10. A computer acquires search information including object information representing a characteristic of an object and position information of the object; extracting result images that match the search information from a plurality of search target images; An information processing method that outputs a search result based on the search information.

11. A step of acquiring, in a computer, search information including object information representing a feature of an object and position information of the object; extracting a result image matching the search information from a plurality of search target images; and outputting a search result based on the search information.

Images