Information processing device and information processing program

Abstract

To provide a video desired by a user to the user even if the user cannot verbalize a target to be searched.SOLUTION: An information processing apparatus includes: a receiving unit which receives a search text input by a user for searching for a video; a generation unit which inputs the search text to an image generation model, which is a machine learning model configured to generate, from a text, an image corresponding to the text, to generate a search image corresponding to the search text; and a video search unit which searches for a video corresponding to the search image and the search text based on the search image and the search text.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing program. [Background technology]

[0002] A technique for receiving a search keyword from a user and searching for a video corresponding to the search keyword from among a plurality of videos is known. For example, a technique for searching for a video in response to a video search request including the search keyword from a client terminal is known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-128710 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with the above-mentioned conventional technology, there are cases where a user cannot find a desired video because the user cannot think of an appropriate search keyword. Therefore, with the above-mentioned conventional technology, even if the user cannot effectively verbalize the search target, it is not necessarily possible to provide the user with the video that the user wants.

[0005] The purpose of this application is to [Means for solving the problem]

[0006] The information processing device of the present application includes a reception unit that receives search text input by a user to search for videos, a generation unit that inputs the search text into an image generation model, which is a machine learning model that generates an image corresponding to the text from the text, and generates a search image corresponding to the search text, and a video search unit that searches for a video corresponding to the search image and the search text based on the search image and the search text. [Effects of the Invention]

[0007] According to one aspect of the embodiment, even if a user is unable to effectively verbalize what he or she is searching for, it is possible to provide the user with the video that the user desires. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of information processing according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of information processing according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of information processing according to the embodiment. [Figure 6] FIG. 6 is a flowchart showing the procedure of information processing by the information processing device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of information processing according to the modified example. [Figure 8] FIG. 8 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a detailed description will be given of an information processing device and an information processing program according to the present application (hereinafter referred to as an "embodiment") with reference to the drawings. Note that the information processing device and the information processing program according to the present application are not limited to the embodiment. Furthermore, the same components in the following embodiments are denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] (Embodiment) 1. Introduction Conventionally, users search for videos by inputting a character string (hereinafter, referred to as text). Note that the video according to this embodiment may include animation or video. However, if the user is unable to effectively express the target of their search using text (i.e., unable to effectively verbalize the search target), the video that matches the text entered by the user may differ from the video desired by the user. Furthermore, even if the user is unable to effectively verbalize the search target, the user may be able to visualize the target in their mind.

[0011] In contrast, the information processing device 100 according to the embodiment accepts search text input by a user to search for a video. The information processing device 100 inputs the search text into an image generation model, which is a machine learning model that generates an image corresponding to the text from the text, and generates a search image corresponding to the search text. The information processing device 100 searches for videos corresponding to the search image and the search text based on the search image and the search text. The information processing device 100 outputs information related to the searched video. As a result, even if a user is unable to effectively verbalize a search target, the information processing device 100 can visualize the search target that the user has in mind with an image and then search for a video corresponding to the image representing the visualized search target. Therefore, the information processing device 100 can provide the user with a video that the user desires, even if the user is unable to effectively verbalize a search target.

[0012] [2. Information Processing System Configuration] An example of the configuration of an information processing system 1 according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of the configuration of the information processing system 1 according to an embodiment. As shown in FIG. 1, the information processing system 1 includes a terminal device 10 and an information processing device 100. The terminal device 10 and the information processing device 100 are connected to each other via a network N so as to be able to communicate with each other via a wired or wireless connection. The network N is, for example, a wide area network (WAN) such as the Internet. Note that the information processing system 1 shown in FIG. 1 may include a plurality of terminal devices 10 and a plurality of information processing devices 100.

[0013] The terminal device 10 is an information processing device used by a user. The terminal device 10 is realized by, for example, a smartphone, a tablet terminal, a notebook PC (Personal Computer), a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), etc. The terminal device 10 displays information received from the information processing device 100, etc., using a web browser or an application.

[0014] The information processing device 100 is an information processing device that performs information processing according to the embodiment, and is realized by, for example, a server device or a cloud system. The terminal device 10 transmits search text input by a user to the information processing device 100. The information processing device 100 accepts the search text from the terminal device 10. The information processing device 100 also inputs the search text to an image generation model to generate a search image corresponding to the search text. The information processing device 100 also outputs the search image to the terminal device 10. The terminal device 10 displays the search image. The terminal device 10 also transmits edited text input by a user to the information processing device 100. The information processing device 100 accepts the edited text from the terminal device 10. The information processing device 100 also inputs the search image and the edited text to an image editing model to generate an edited image. The information processing device 100 also searches for a video corresponding to the edited image and the search text. The information processing device 100 also outputs information related to the searched video.

[0015] 3. Configuration of Information Processing Device An example of the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. The information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0016] (Communication unit 110) The communication unit 110 is realized by a NIC (Network Interface Card), an antenna, etc. The communication unit 110 is connected to various networks by wire or wirelessly, and transmits and receives information to and from the terminal device 10, for example.

[0017] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores various data. For example, the storage unit 120 may store various pieces of information received by the reception unit 131. The storage unit 120 may also store various pieces of information generated by the generation unit 132. The storage unit 120 also stores various programs. For example, the storage unit 120 stores an information processing program according to the embodiment. The storage unit 120 may also store information related to various machine learning models. For example, the storage unit 120 stores information related to an image generation model M1. The storage unit 120 also stores information related to an image editing model M2. The storage unit 120 also has a video database 121. The video database 121 stores information related to videos to be searched.

[0018] (control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs stored in a storage device inside the information processing device 100 using RAM as a work area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0019] The control unit 130 has a reception unit 131, a generation unit 132, a video search unit 133, and an output control unit 134 as functional units, and may realize or execute the information processing operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 4, and may have other configurations as long as they perform the information processing described below. Furthermore, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

[0020] (Reception Department 131) The receiving unit 131 receives search text input by a user to search for videos. Here, the search text refers to a search query, which is text. For example, the receiving unit 131 may receive a search request for videos including the search text from the user's terminal device 10. FIG. 3 is a diagram for explaining an example of information processing according to an embodiment. In FIG. 3, the receiving unit 131 receives a search request for videos including search text T1, "dark brown, cute, medium-sized dog," input by the user.

[0021] (Generation unit 132) The generation unit 132 inputs search text to an image generation model, which is a machine learning model that generates an image corresponding to the text from the text, and generates a search image corresponding to the search text. For example, the image generation model may be DALLE-2, DALLE-3, Stable Diffusion, or the like. For example, when the reception unit 131 receives a video search request, the generation unit 132 may acquire the search text accepted by the reception unit 131. Furthermore, when the generation unit 132 acquires the search text, the generation unit 132 may input the search text to the image generation model and cause the image generation model to generate a search image corresponding to the search text. The generation unit 132 may generate the search image by causing the image generation model to generate the search image. The generation unit 132 may input the search text to the image generation model and acquire the search image output from the image generation model. In FIG. 3, the generation unit 132 inputs search text T1, "dark brown, cute, medium-sized dog," accepted by the reception unit 131, to the image generation model M1 and generates search images G11 to G13 corresponding to the search text T1. In FIG. 3, the generation unit 132 generates search images G11 to G13, which are images of dogs having characteristics according to the search text T1, "dark brown, cute, medium-sized dog."

[0022] Furthermore, when the generation unit 132 generates a search image, it may output the search image to the terminal device 10. The generation unit 132 may transmit the search image to the terminal device 10. The terminal device 10 receives the search image from the information processing device 100. When the terminal device 10 receives the search image, it may display the search image on the screen.

[0023] The receiving unit 131 also receives edited text, which is text input by a user to edit a search image. For example, the receiving unit 131 may receive edited text from the user's terminal device 10. The receiving unit 131 may receive edited text input from the user's terminal device 10 as a prompt for an image editing model. FIG. 4 is a diagram for explaining an example of information processing according to an embodiment. In FIG. 4, the receiving unit 131 receives edited text T2, which is input by a user to edit a search image G11, and which reads, "Please make the image realistic, not cartoon-style. Also, the dog should face left, have long ears, and be a darker brown."

[0024] Furthermore, the generation unit 132 inputs the search image and the edited text into an image editing model, which is a machine learning model that performs editing processing on an image according to the text, and generates an edited image by performing editing processing on the search image according to the edited text. For example, the image editing model may be DALLE-2, DALLE-3, Stable Diffusion, or the like. For example, the generation unit 132 may acquire the edited text accepted by the acceptance unit 131. Furthermore, when acquiring the edited text, the generation unit 132 may input the search image and the edited text into the image editing model and cause the image editing model to generate an edited image by performing editing processing on the search image according to the edited text. Furthermore, the generation unit 132 may input the search image and the edited text into the image editing model and cause the image editing model to generate an edited image corresponding to the search image and the edited text. The generation unit 132 may generate the edited image by causing the image editing model to generate the edited image. The generation unit 132 may input the edited text into the image editing model and acquire an edited image output from the image editing model. 3, the generation unit 132 inputs the edited text T2 and the searched image G11 received by the reception unit 131 into the image editing model M2, and generates an edited image G21 by performing an editing process on the searched image G11 in accordance with the edited text T2. The generation unit 132 generates an edited image G21 by performing an editing process on the searched image G11 in accordance with the edited text T2, which says, "Please make the image realistic, not cartoon-style. Also, the dog should face left, have long ears, and be a darker brown."

[0025] (Video Search Section 133) The video search unit 133 searches for videos corresponding to the search image and search text based on the search image and search text. For example, the video search unit 133 may search for videos corresponding to the edited image and search text based on the edited image and search text. Specifically, for example, the video search unit 133 may search for videos based on scores related to the similarity of the search image and search text to the video. The video search unit 133 may calculate an image score related to the similarity between video features related to the search target video and image features of the edited image. The video search unit 133 may calculate a text score related to the similarity between video features related to the search target video and text features of the search text. Here, the video features related to the search target video may be a matrix in which frame features (generally, vectors) of each of multiple frames constituting the search target video are arranged as elements. Note that if the frame features are numerical values, the video features may be a vector in which the frame features are arranged as elements. The video search unit 133 may calculate an image score and a text score for each of the search target videos. The video search unit 133 may search for videos based on the image score and the text score, an example of which will be described below.

[0026] The video search unit 133 calculates an image score relating to the similarity between each frame feature and the image feature of an edited image for multiple frames included in the video to be searched. The video search unit 133 also calculates a text score relating to the similarity between each frame feature of multiple frames and the text feature of the search text. The video search unit 133 searches for videos based on the image score and the text score. For example, the video search unit 133 may acquire an edited image generated by the generation unit 132. Furthermore, when acquiring an edited image, the video search unit 133 may input the edited image to an image encoder that generates image features indicating image characteristics from the image, thereby generating image features of the edited image. For example, the video search unit 133 may generate image features using an image encoder used in CLIP (Contrastive Language-Image Pre-training) or EVA-CLIP, or an image encoder included in CoCa (Contrastive Captioners).

[0027] Furthermore, the video search unit 133 may refer to the video database 121 to acquire the video to be searched. Furthermore, when the video search unit 133 acquires the video to be searched, the video search unit 133 may input each of the frames constituting the video to be searched into an image encoder to generate frame features indicating characteristics of each of the frames constituting the video to be searched. For example, the video search unit 133 may generate frame features using an image encoder used in CLIP or EVA-CLIP or an image encoder included in CoCa. Furthermore, the video search unit 133 may calculate an image score related to the similarity between frame features indicating characteristics of each of the frames constituting the video to be searched and image features of the edited image. For example, the video search unit 133 may calculate an image score related to the cosine similarity between frame features indicating characteristics of each of the frames constituting the video to be searched and image features of the edited image.

[0028] For example, let X be the video feature quantity for a video i to be searched, which consists of M (M is a natural number greater than or equal to 2) frames. i Then, Xi =(x1 i , …, x M i ) where x j i (j=1, 2, ..., M) indicates the frame feature of the j-th frame of the video i. The generation unit 132 may generate a plurality of search images corresponding to the search text. The generation unit 132 may generate a plurality of edited images corresponding to each of the plurality of search images. Furthermore, the video search unit 133 calculates an image score relating to the similarity between the image feature amount and the frame feature amount of each of the edited images. For example, the generation unit 132 generates N edited images (N is a natural number equal to or greater than 2). Furthermore, the generation unit 132 calculates the image feature amount of the l (l=1, 2, ..., N)-th edited image among the N edited images as q l At this time, the generation unit 132 generates image feature quantities (q1, ..., q N ) and video feature X for video i i =(x1 i , …, x M i ) image similarity score score i gen For example, the generation unit 132 may calculate the image score Score expressed by the following formula (1): i gen may be calculated.

[0029]

number

[0030] In the above equation (1), cos(x j i , q l ) is the frame feature x of the jth frame of video i j i and the image feature q of the l-th edited image among the N edited images. lThe above formula (1) indicates that the cosine similarity between the M frames constituting the video i and the N edited images is selected from the M frames constituting the video i, and the cosine similarity of the frame with the maximum cosine similarity is selected for each of the N edited images. The above formula (1) also indicates that the average value of the maximum cosine similarities selected for each of the N edited images is used as the image score, i gen This indicates that the calculation is as follows.

[0031] Furthermore, the video search unit 133 may acquire the search text accepted by the accepting unit 131. Furthermore, when acquiring the search text, the video search unit 133 may input the search text to a text encoder that generates text features indicating the characteristics of the text from the text, thereby generating text features of the search text. For example, the video search unit 133 may generate text features using a text encoder used in CLIP or EVA-CLIP. Furthermore, the video search unit 133 may calculate a text score relating to the similarity between frame features indicating the characteristics of each of multiple frames constituting the video to be searched and the text features of the search text.

[0032] For example, the text feature of the search text is represented by q0. In this case, the generation unit 132 calculates the text feature q0 of the search text and the video feature X i =(x1 i , …, x M i ) text score for similarity to i text For example, the generation unit 132 may calculate the text score Score expressed by the following formula (2): i text may be calculated.

[0033]

number

[0034] In the above equation (2), cos(x j i, q0) is the frame feature x of the jth frame of video i j i The above formula (2) shows the cosine similarity between the text feature q0 of the search text and the M frames constituting the video i. The cosine similarity of the frame with the maximum cosine similarity is selected, and the cosine similarity of the frame with the maximum cosine similarity is calculated as the text score score i text This indicates that:

[0035] Furthermore, the video search unit 133 searches for videos based on an added score, which is a score obtained by adding together the image score and the text score. The video search unit 133 may calculate an added score for each video to be searched. For example, the video search unit 133 may calculate an image score Score expressed by the above formula (1) for a video i. i gen The value obtained by multiplying by α (α is a real number) and the text score expressed by the above formula (2) is i text For example, the video search unit 133 may calculate an additive score for a video i, expressed by the following formula (3):

[0036]

number

[0037] Furthermore, the video search unit 133 may select a predetermined number of videos from among the videos to be searched in order of the summed score. Furthermore, the video search unit 133 may acquire a predetermined number of videos from among the videos to be searched in order of the summed score. For example, the video search unit 133 may acquire a predetermined number of videos from among the videos to be searched in order of the summed score calculated by the above formula (3) as search results.

[0038] FIG. 5 is a diagram illustrating an example of information processing according to an embodiment. In FIG. 5, the video search unit 133 may acquire search text T1 accepted by the accepting unit 131. The video search unit 133 may also acquire a text encoder by referring to the storage unit 120. The video search unit 133 may also input the search text T1 to a text encoder to generate a text feature F1 of the search text T1. The video search unit 133 may also acquire an edited image G21 generated by the generating unit 132. The video search unit 133 may also acquire an image encoder by referring to the storage unit 120. The video search unit 133 may also input the edited image G21 to an image encoder to generate an image feature F21 of the edited image G21. The video search unit 133 may also acquire a video to be searched that is stored in the video database 121. Furthermore, when the video search unit 133 acquires a video to be searched, it may input each of the multiple frames constituting the video to be searched into an image encoder to generate frame features indicating the characteristics of each of the multiple frames constituting the video to be searched. Furthermore, the video search unit 133 may calculate a text score relating to the similarity between the video feature for each of the videos to be searched and the text feature F1. Furthermore, the video search unit 133 may calculate an image score relating to the similarity between the video feature for each of the videos to be searched and the image feature F21. Furthermore, the video search unit 133 may search for videos based on an added score which is a score obtained by adding the image score and the text score.

[0039] (output control unit 134) The output control unit 134 outputs information related to the videos searched for by the video search unit 133. For example, the output control unit 134 may output information related to the videos searched for by the video search unit 133 to the terminal device 10. For example, the output control unit 134 may output information related to the videos acquired by the video search unit 133 as search results.

[0040] [4. Processing Procedure] FIG. 6 is a flowchart showing the procedure of information processing by the information processing device 100 according to the embodiment. In FIG. 6, the receiving unit 131 of the information processing device 100 receives search text input by a user (step S101). The generation unit 132 of the information processing device 100 inputs the search text to an image generation model to generate a search image corresponding to the search text (step S102). The receiving unit 131 of the information processing device 100 also receives edited text, which is text input by the user to edit the search image (step S103). The generation unit 132 of the information processing device 100 inputs the search image and the edited text to an image editing model to generate an edited image by performing editing processing on the search image according to the edited text (step S104). The generation unit 132 also calculates an image score related to the similarity between the frame features of each of a plurality of frames constituting the video to be searched and the image features of the edited image, and a text score related to the similarity between the frame features of each of the plurality of frames and the text features of the search text (step S105). Furthermore, the video search unit 133 of the information processing device 100 searches for videos based on the image score and the text score (step S106). Furthermore, the output control unit 134 of the information processing device 100 outputs information related to the videos searched for by the video search unit 133 (step S107).

[0041] [5. Modifications] The processing according to the above-described embodiment may be implemented in various different forms other than the above embodiment.

[0042] In the above-described embodiment, the generation unit 132 inputs a search image and edited text into an image editing model to generate an edited image by performing editing processing on the search image according to the edited text. In a modified example, the generation unit 132 further inputs a mask image representing a binary mask into the image editing model to generate an edited image by performing editing processing on a region of the search image excluding the region masked by the mask image according to the edited text. Specifically, the generation unit 132 may generate the mask image based on the search image. For example, the generation unit 132 may identify a main object (e.g., an object located in the center) from among the objects included in the search image. Furthermore, the generation unit 132 may generate a mask image that is a binary mask image in which the pixel value of the region of the main object in the search image is 0 and the pixel value of the region excluding the region of the main object is 1. Furthermore, the generation unit 132 inputs the search image, the mask image, and the edited text into the image editing model to generate an edited image by performing editing processing on a region of the search image excluding the region masked by the mask image according to the edited text.

[0043] FIG. 7 is a diagram illustrating an example of information processing according to a modified example. The search image G31 illustrated in FIG. 7 is an image of a dog sitting on a park bench. The mask image G32 is a binary mask image in which the pixel value of the imaged region of the dog in the search image G31 is 0 and the pixel value of the background region other than the dog is 1. The reception unit 131 receives edited text T3, "face of a yellow cat, high resolution, sitting on a park bench," input by a user. The generation unit 132 inputs the search image G31, the mask image G32, and the edited text T3 into the image editing model M2 and generates an edited image G41 by performing an editing process according to the edited text T3 on the region of the search image G31 excluding the region masked by the mask image G32. For example, the generation unit 132 generates an edited image G41 that appears to be an image of a cat sitting on a park bench.

[0044] [6. Effects] As described above, the information processing device 100 according to the embodiment includes the receiving unit 131, the generating unit 132, and the video search unit 133. The receiving unit 131 receives search text input by a user to search for a video. The generating unit 132 inputs the search text to an image generation model, which is a machine learning model that generates an image corresponding to the text from the text, and generates a search image corresponding to the search text. The video search unit 133 searches for a video corresponding to the search image and the search text based on the search image and the search text.

[0045] As a result, even if a user is unable to effectively verbalize a search target, the information processing device 100 can visualize the search target that the user has in mind as an image and then search for a video corresponding to the image of the visualized search target. Therefore, even if a user is unable to effectively verbalize a search target, the information processing device 100 can provide the user with a video that the user desires. Furthermore, even if a user is unable to effectively verbalize a search target, the information processing device 100 can provide the user with a video that the user desires, thereby contributing to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization, and build resilient technological infrastructure."

[0046] Furthermore, the receiving unit 131 receives edited text, which is text input by a user to edit the searched image. The generating unit 132 inputs the searched image and the edited text into an image editing model, which is a machine learning model that performs editing processing on an image according to the text, and generates an edited image by performing editing processing on the searched image according to the edited text. The video searching unit 133 searches for videos corresponding to the edited image and the searched text based on the edited image and the searched text.

[0047] This allows the information processing device 100 to visualize the search target that the user has in mind with an image that is closer to the user's image, and then search for a video that corresponds to the image that is closer to the user's image.

[0048] In addition, the video search unit 133 calculates an image score relating to the similarity between the frame features of each of the multiple frames constituting the video to be searched and the image features of the edited image, and a text score relating to the similarity between the frame features of each of the multiple frames and the text features of the search text, and searches for the video based on the image score and the text score.

[0049] This enables the information processing device 100 to search for videos that are similar to an image that visualizes the search target that the user has in mind, and that are similar to the search text entered by the user to search for videos.

[0050] Furthermore, the video search unit 133 searches for videos based on an added score, which is a score obtained by adding together the image score and the text score.

[0051] This enables the information processing device 100 to search for videos that are similar to an image that visualizes the search target that the user has in mind, and that are similar to the search text entered by the user to search for videos.

[0052] Furthermore, the video search unit 133 calculates an image score relating to the degree of similarity between the image feature amount and the frame feature amount of each of the multiple edited images.

[0053] This allows the information processing device 100 to appropriately search for videos similar to an image that visualizes the search target that the user has in mind, based on the similarity between the image feature and the frame feature.

[0054] Furthermore, the generation unit 132 further inputs a mask image indicating a binary mask to the image editing model, and generates an edited image by performing editing processing according to the edited text on the area of ​​the search image excluding the area masked by the mask image.

[0055] This allows the information processing device 100 to more easily generate an image that visualizes the search target that the user has in mind.

[0056] [7. Hardware Configuration] The information processing device 100 according to the embodiment described above is realized by, for example, a computer 1000 configured as shown in Fig. 8. Fig. 8 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0057] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0058] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0059] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0060] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0061] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0062] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have undergone various modifications and improvements based on the knowledge of those skilled in the art.

[0063] [8. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0064] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0065] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content. [Explanation of symbols]

[0066] 100 Information processing device 110 Communications Department 120 Storage section 130 Control Unit 131 Reception 132 Generation part 133 Video Search Section 134 Output control section

Claims

1. a reception unit that receives search text input by a user to search for a video; a generation unit that inputs the search text into an image generation model, which is a machine learning model that generates an image corresponding to the text from the text, and generates a search image corresponding to the search text; a video search unit that searches for a video corresponding to the search image and the search text based on the search image and the search text; An information processing device comprising:

2. The reception unit accepting edited text, the edited text being text entered by the user to edit the retrieved image; The generation unit inputting the searched image and the edited text into an image editing model that is a machine learning model that performs editing processing on an image according to the text, and generating an edited image by performing editing processing on the searched image according to the edited text; The video search unit searching for the video corresponding to the edited image and the search text based on the edited image and the search text; The information processing device according to claim 1 .

3. The video search unit calculating an image score relating to the degree of similarity between the frame features of each of a plurality of frames constituting the video to be searched and the image features of the edited image, and a text score relating to the degree of similarity between the frame features of each of the plurality of frames and the text features of the search text, and searching the video based on the image score and the text score; The information processing device according to claim 2 .

4. The video search unit searching for the video based on an added score obtained by adding the image score and the text score; The information processing device according to claim 3 .

5. The video search unit calculating the image score relating to the degree of similarity between the image feature and the frame feature of each of the plurality of edited images; The information processing device according to claim 3 .

6. The generation unit a mask image representing a binary mask is further input to the image editing model, and an edited image is generated by performing an edit process on a region of the searched image excluding a region masked by the mask image in accordance with the edited text. The information processing device according to claim 2 .

7. a receiving step for receiving search text entered by a user to search for a video; a generation step of inputting the search text into an image generation model, which is a machine learning model that generates an image corresponding to the text from the text, and generating a search image corresponding to the search text; a video search procedure for searching for a video corresponding to the search image and the search text based on the search image and the search text; An information processing program that causes a computer to execute the above.

Images