Scene retrieval system, scene retrieval method and recording medium

Abstract

The present invention provides a scene retrieval system, a scene retrieval method, and a recording medium capable of accurately extracting from a large amount of video data only a video image in which a desired scene is shown.　The scene retrieval system includes: a search means that can access a first storage means for storing a first video to which a tag is attached in advance and a second storage means for storing the tag and a second video composed of a compressed version of the video stored in the first storage means, receives input of a search condition, and searches the second storage means for a second video to which a tag corresponding to the search condition is attached; a scene estimation means that estimates a scene captured in the second video and determines whether or not the second video matches the search condition; and a presentation means that identifies and presents the first video corresponding to the second video from the first storage means when it is determined that the second video matches the search condition.

Description

Scene search system, scene search method, and recording medium

[0001] The present invention relates to a scene search system, a scene search method, and a recording medium.

[0002] Patent Literature 1 discloses a video distribution device that allows multiple users to easily find scenes that are important to them when viewing video stored in the video distribution device. The video distribution device described in this publication extracts important scenes from the video based on metadata that indicates the characteristics of the video, and creates user interface data customized for each user. The publication also describes using tagging information added by the users themselves as the metadata.

[0003] JP 2012-129724 A

[0004] There is a need to accurately extract only videos that show desired scenes from a large amount of video content. Although various scene detection techniques are known, there is a problem in that extracting only videos that show desired scenes from a large amount of video content requires considerable computational cost and time.

[0005] An object of the present disclosure is to provide a scene search system, a scene search method, and a recording medium that can accurately extract a video containing a desired scene from a large amount of video data.

[0006] According to a first aspect, there is provided a scene search system including: a first storage means for storing a first video to which a tag has been assigned in advance; a second storage means for storing a second video obtained by compressing the video stored in the first storage means and the tag; a search means that is capable of accessing the first storage means and that accepts input of search criteria and searches the second storage means for second video to which a tag corresponding to the search criteria has been assigned; a scene estimation means that estimates a scene captured in the second video and determines whether the second video matches the search criteria; and a presentation means that, if it is determined that the second video matches the search criteria, identifies and presents a first video from the first storage means that corresponds to the second video.

[0007] According to a second aspect, a scene search method is provided in which a computer that can access a first storage means that stores first video to which a tag has been assigned in advance, a second video that is a compressed version of the video stored in the first storage means, and a second storage means that stores the tag, receives input of search criteria, searches the second storage means for second video to which a tag corresponding to the search criteria has been assigned, estimates a scene depicted in the second video, and determines whether the second video matches the search criteria, and if it is determined that the second video matches the search criteria, identifies and presents a first video corresponding to the second video stored in the first storage means.

[0008] According to a third aspect, there is provided a recording medium having recorded thereon a program that causes a computer that can access a first storage means that stores first video to which a tag has been assigned in advance, a second video that is a compressed version of the video stored in the first storage means, and a second storage means that stores the tag, to receive input of search criteria and search the second storage means for second video to which a tag corresponding to the search criteria has been assigned, estimate a scene captured in the second video, and determine whether the second video matches the search criteria, and, if it is determined that the second video matches the search criteria, identify and present a first video corresponding to the second video stored in the first storage means.

[0009] According to the present disclosure, it is possible to provide a scene search system, a scene search method, and a recording medium that can accurately extract a video containing a desired scene from a large amount of video data.

[0010] FIG. 1 is a diagram showing one configuration of the present disclosure. FIG. 2 is a flow chart showing the operation of the present disclosure. FIG. 3 is a diagram for explaining the operation of the present disclosure. FIG. 4 is a diagram showing one configuration of the present disclosure. FIG. 5 is a functional block diagram showing an example configuration of a scene search system of the present disclosure. FIG. 6 is a sequence diagram for explaining the operation of the present disclosure. FIG. 7 is a diagram showing another configuration of the present disclosure. FIG. 8 is a diagram showing another configuration of the present disclosure. FIG. 9 is a sequence diagram for explaining the operation of the other configuration of the present disclosure. FIG. 10 is a diagram showing another configuration of the present disclosure. FIG. 11 is a diagram showing the configuration of a computer constituting a scene search system of the present disclosure.

[0011] First, an overview of one embodiment of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. The reference numerals in the drawings attached to this overview are attached to each element for convenience as an example to facilitate understanding, and are not intended to limit the present disclosure to the illustrated form. Furthermore, connecting lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of main signals (data) and do not exclude bidirectionality. A program is executed via a computer device, which includes, for example, a processor, a storage device, an input device, a communication interface, and, if necessary, a display device. Furthermore, this computer device is configured to be able to communicate with internal or external devices (including computers) via the communication interface, whether wired or wireless. Although ports or interfaces are present at the input / output connection points of each block in the drawings, they are not shown.

[0012] In one embodiment, the present disclosure can be realized by a scene search system 10 that can access a first storage unit 11 and a second storage unit 13, as shown in FIG. 1 . The scene search system 10 also includes a search unit 14, a scene estimation unit 15, and a presentation unit 16. More specifically, the first storage unit 11 stores a first video to which a tag has been previously assigned. The second storage unit stores a second video obtained by compressing the video stored in the first storage unit 11 and the tag. The search unit 14 then receives input of search criteria and searches the second storage unit 13 for a second video to which a tag corresponding to the search criteria has been assigned. The scene estimation unit 15 estimates a scene captured in the second video and determines whether the second video matches the search criteria. If the presentation unit 16 determines that the second video matches the search criteria, it identifies and presents a first video corresponding to the second video from the first storage unit 11.

[0013] The scene retrieval system 10 configured as described above operates as follows. First, the scene retrieval system 10 accepts input of search conditions (step S01 in FIG. 2). For example, a user inputs a search condition such as "I'm looking for a video of a car and a bicycle traveling side by side." This search condition may be in natural language format, may follow a predetermined format, or may be input by keyword. If the search condition is input as a keyword, the keyword can be used as a search tag as is.

[0014] Next, the scene retrieval system 10 searches the second storage unit 13 for second videos tagged with tags corresponding to the search criteria (step S02 in FIG. 2). For example, the scene retrieval system 10 searches for videos tagged with "car" and "bicycle" from the second videos stored in the second storage unit 13.

[0015] Next, the scene retrieval system 10 estimates the scene captured in the second video (step S03 in FIG. 2), and determines whether the second video matches the retrieval conditions (step S04 in FIG. 2).

[0016] If it is determined that the second video matches the search criteria (Yes in step S04 in FIG. 2 ), the scene retrieval system 10 identifies a first video corresponding to the second video stored in the first storage unit 11 (step S05 in FIG. 2 ). Then, the scene retrieval system 10 presents, for example, the location of the identified first video to the user who input the search criteria (step S06 in FIG. 2 ). The location of the first video may be, for example, link information that allows the user to download the first video from the first storage unit 11. Alternatively, instead of sending the location of the first video to the user, the scene retrieval system 10 may retrieve the first video from the first storage unit 11 and transmit it to the user.

[0017] If it is determined that the second video does not match the search conditions (No in step S04 in Figure 2), the scene search system 10 notifies the user that there is no corresponding first video (step S07 in Figure 2).

[0018] FIG. 3 is a diagram illustrating the operation of the present disclosure. The first video stored in the first storage unit 11 is compressed by the compression unit 12, for example, when the first video is newly saved in the first storage unit 11, and then stored in the second storage unit 13. When the search unit 14 receives search criteria input from a user, it searches the second storage unit 13 for second video tagged with the search criteria. The scene estimation unit 15 then estimates the scene depicted in the compressed second video. Since the second video is compressed, computational costs and time can be significantly reduced. If a match with the search criteria is confirmed, the presentation unit 16 presents the location of the first video to the user. The compression parameters used to compress the first video to generate the second video are set to a level that enables scene estimation by the scene estimation unit 15. As described above, according to the present disclosure, scene estimation is performed on the compressed second video, rather than the first video, thereby significantly reducing computational costs and time. Furthermore, if a match with the search conditions is confirmed, the first video is presented to the user, so that high-quality video can be provided to the user.

[0019] As described above, according to the present disclosure, it is possible to accurately extract only videos that show desired scenes from a large amount of video data. Note that, in the example of Fig. 1, the first storage unit 11 and the second storage unit 13 are provided independently of the scene retrieval system 10, but the first storage unit 11 and the second storage unit 13 may be configured in an auxiliary storage device or the like within the scene retrieval system 10.

[0020] [First Embodiment] Next, a first embodiment will be described, which enables a search for a video showing a desired scene from a large amount of video data collected from a vehicle or the like equipped with a camera or the like. Fig. 4 is a diagram showing one configuration of the present disclosure. Referring to Fig. 4, a configuration including an in-vehicle terminal 200, a first video DB (database) 300, a scene search system 100, a second video DB (database) 400, and a user terminal 500 is shown.

[0021] The in-vehicle terminal 200 transmits video captured by a camera or the like mounted on a moving object such as a vehicle to the first video DB 300. It is desirable that the video captured by this camera or the like has high image quality that meets a certain standard. If the video captured by the camera or the like has high image quality and is expected to place a load on the network, the in-vehicle terminal 200 may reduce the data size of the video captured by this camera or the like by re-encoding the video.

[0022] The first video DB 300 is a database that stores video transmitted from the in-vehicle terminal 200. The video stored in the first video DB 300 is tagged according to a predetermined rule. The tags include the subject of the video and the shooting conditions, such as pedestrian, sudden steering, sudden braking, right turn, lane change, and rain. The tags may be added by the in-vehicle terminal, or may be added by the first video DB 300 using a generation AI (Artificial Intelligence) that can handle images. The first video DB 300 may be a database that stores video data for creating a dynamic map.

[0023] The second video DB 400 is a database that stores video with reduced data size compared to the video stored in the first video DB 300. Methods for reducing this data size include lowering the resolution, lowering the frame rate, and lowering the image quality (including control of increasing the image quality of important areas in the video and decreasing the image quality of other areas). Hereinafter, this reduction in the data size of video will also be referred to as "compression." In the following explanation, the first video DB 300 and the second video DB 400 correspond to the above-mentioned first storage unit 11 and second storage unit 13, respectively.

[0024] The scene retrieval system 100 receives search conditions for searching for a desired scene from the user terminal 500, searches for a corresponding second video from the second video DB 400, and analyzes the scene. As a result of the scene analysis, it is determined that the retrieved second video contains a scene that matches the search conditions specified in the user terminal 500. In this case, the scene retrieval system 100 retrieves a first video with higher image quality from the first video DB 300 and presents it to the user terminal 500.

[0025] The user terminal 500 is a terminal for searching for specific scenes, for example, for the development of an image analysis engine for autonomous driving. To develop and evaluate an image analysis engine, a large amount of high-quality, scene-specific video is required as training data for machine learning. The user terminal 500 can be a personal computer or a workstation that can be used as a terminal for collecting video of specific scenes.

[0026] 5 is a functional block diagram showing an example configuration of a scene retrieval system 100 according to the present disclosure. Referring to FIG. 5, the system includes a compression unit 102, a retrieval unit 104, a scene estimation unit 105, and a presentation unit 106.

[0027] The compression unit 102 compresses the first video stored in the first video DB 300 and stores the compressed first video in the second video DB 400. At this time, the compression unit 102 also assigns the tag assigned to the first video to the second video.

[0028] The search unit 104 displays a search condition input screen on the user terminal 500 for searching for a desired scene, and accepts input of the search conditions from the user terminal 500. The search unit 104 also uses the input search conditions to search the second video DB 400 for second videos having tags that match the search conditions. If one or more second videos are found as a result of the search, the search unit 104 sends the search results to the scene estimation unit 105 and requests scene estimation of the corresponding videos. Therefore, the search unit 104 corresponds to the above-mentioned search means 14.

[0029] The search criteria may be in the form of tag selection or natural language format. For example, assume that a search criterion in natural language format such as "a scene in which a truck traveling in front of the vehicle in the left lane changes lanes into the vehicle's lane" is input. In this case, the search unit 104 extracts keywords from the input sentence and searches the second video DB 400 for second videos having tags that match the search criteria.

[0030] The scene estimation unit 105 estimates a scene captured in the second video specified by the search unit 104, determines whether the scene matches the search criteria specified by the search unit 104, and outputs the result to the presentation unit 106. Various scene estimation technologies can be used for this scene estimation. Furthermore, a large-scale language model capable of performing scene analysis of video or a VLM (Vision-Language Model) can also be used as the scene estimation unit 105. Known examples of VLMs capable of scene analysis include LLaVA (Large Language and Vision Assistant). When the scene estimation unit 105 is configured with an LLM or VLM capable of handling video, the second video may be specified, search criteria in natural language format may be processed to create a prompt, and the prompt may be input to the scene estimation unit 105. Therefore, the scene estimation unit 105 corresponds to the scene estimation means 15 described above.

[0031] When the presentation unit 106 receives a determination result from the scene estimation unit 105 indicating that the second video contains a scene that matches the search criteria, the presentation unit 106 retrieves the first video corresponding to the second video from the first video DB 300 and presents the first video to the user terminal 500. Note that various forms of presentation of the first video from the presentation unit 106 to the user terminal 500 are possible. For example, the presentation unit 106 may transmit the first video to the user terminal 500 as a file, or may transmit a link to the first video to the user terminal 500. The presentation unit 106 corresponds to the presentation means 16 described above.

[0032] Next, the operation of this embodiment will be described in detail with reference to the drawings. Fig. 6 is a sequence diagram for explaining the operation of the present disclosure. Referring to Fig. 6, first, the scene retrieval system 100 displays a search condition input screen on the display device of the user terminal 500 (step S001).

[0033] When the user of the user terminal 500 inputs search criteria (step S002), the scene retrieval system 100 searches the second video DB 400 for second videos that match the search criteria (step S003). It is anticipated that, for example, when the search criteria are broad, multiple second videos will be found as a result of the search. In this case, the scene retrieval system 100 may select these multiple second videos and perform scene estimation. Conversely, the search may result in no second video being found. In this case, the scene retrieval system 100 may respond to the user that no videos match the search criteria and request the user to modify the search criteria, etc.

[0034] Next, the scene retrieval system 100 estimates the scene of the second video retrieved in step S003 (step S004), and then determines whether the estimated scene satisfies the search conditions (step S005).

[0035] If the estimated scene matches the search criteria (Yes in step S005), the scene retrieval system 100 extracts the first video corresponding to the corresponding second video from the first video DB 300 (step S006). Next, the scene retrieval system 100 transmits the first video to the user terminal 500 (step S007). The user terminal 500 receives the first video as a video matching the search criteria (step S008).

[0036] On the other hand, if the scene estimated in step S004 does not match the search criteria (No in step S005), the scene retrieval system 100 returns to step S001. The scene retrieval system 100 then responds to the user that there is no video that matches the search criteria, and requests the user to modify the search criteria, etc.

[0037] As described above, according to this embodiment, the user can receive videos that meet the search criteria and that are stored in the first video DB 300 and have higher quality. Meanwhile, the search for videos that meet the user's search criteria and the estimation of scenes are performed using compressed videos stored in the second video DB 400. This has the advantage of significantly reducing the calculation cost and time while maintaining the accuracy of the search and scene estimation compared to analyzing high-quality videos. Furthermore, according to this embodiment, because the first video DB 500 is not the target of the search process, the first video DB 500 can be configured using low-speed storage such as a hard disk, which has a low unit cost per capacity.

[0038] Second Embodiment Next, a second embodiment will be described in which a compression parameter optimization function is provided in the compression unit 102. FIG. 7 is a diagram showing another configuration of the present disclosure. The difference from the first embodiment shown in FIG. 5 is that a compression parameter determination unit 107 is provided in the compression unit 102a of the scene retrieval system 100a. The other configurations are almost the same as those of the first embodiment, so the following description will focus on the differences in operation.

[0039] If the parameters (compression parameters) used when compressing the first video into the second video are fixed, the scene estimation capability of the scene estimation unit 105 may vary depending on factors such as the brightness and darkness of the image and the speed of the subject. Therefore, the compression parameter determination unit 107 determines compression parameters that can achieve the desired scene estimation accuracy for the first video to be compressed. Such a compression parameter determination unit 107 can be implemented by inquiring of a generation AI capable of handling video about compression parameters that can reduce the data size while achieving the required scene estimation accuracy. Alternatively, such a compression parameter determination unit 107 may be implemented using AI that learns changes in the estimation results of the scene estimation unit 105 when compression parameters are varied in various ways, and derives appropriate compression parameters.

[0040] Compression unit 102 a compresses the first video stored in first video DB 300 using the compression parameters determined by compression parameter determination unit 107 , and stores the compressed first video in second video DB 400 .

[0041] According to this embodiment, in addition to the effects of the first embodiment, the compression unit 102a creates the second video using compression parameters that can satisfy the desired scene estimation accuracy, which has the advantage of improving the accuracy of scene estimation in the scene estimation unit 105.

[0042] [Third Embodiment] Next, a third embodiment will be described, which adds a video recompression function based on scene estimation accuracy. Fig. 8 is a diagram showing another configuration of the present disclosure. The difference from the first embodiment shown in Fig. 5 is the operation of the scene estimation unit 105b and the compression unit 102b of the scene retrieval system 100b. Since the other configurations are almost the same as those of the first embodiment, the following description will focus on the differences in operation.

[0043] The scene estimation unit 105b of the scene retrieval system 100b calculates the accuracy of the estimation result (scene estimation accuracy) when performing scene estimation. If the scene estimation accuracy is below a predetermined threshold, the scene estimation unit 105b notifies the compression unit 102b of the second video and the scene estimation accuracy, and requests that the second video be recompressed.

[0044] The compression unit 102b recompresses the first video based on the scene estimation accuracy notified by the scene estimation unit 105b and stores the recompressed first video in the second video DB 400.

[0045] The scene estimation unit 105 b performs scene estimation using the recompressed second video stored in the second video DB 400 .

[0046] According to this embodiment, it is possible to recover from a failure in scene estimation due to a problem with the compression parameters or the state of the second video, and it is also possible to maintain the scene estimation accuracy of the scene estimation unit 105b at a certain level or above.

[0047] [Fourth Embodiment] Next, a fourth embodiment will be described, which realizes a video recompression function based on scene estimation accuracy in a different form. Fig. 9 is a diagram showing another configuration of the present disclosure. The difference from the second embodiment shown in Fig. 7 is the operation of the scene estimation unit 105c of the scene retrieval system 100c and the compression parameter determination unit 107c of the compression unit 102c. Since the other configurations are almost the same as those of the second embodiment, the following description will focus on the differences in operation.

[0048] The scene estimation unit 105c of the scene retrieval system 100c calculates the accuracy of the estimation result (scene estimation accuracy) when performing scene estimation, and then notifies the compression unit 102c of the second video and the scene estimation accuracy.

[0049] The compression parameter determination unit 107c of the compression unit 102c is configured with a generation AI capable of handling video, as in the second embodiment. The compression parameter determination unit 107c additionally learns the second video and the scene estimation accuracy notified by the scene estimation unit 105c.

[0050] The compression unit 102 c compresses the first video stored in the first video DB 300 using the compression parameters determined by the compression parameter determination unit 107 c and stores the compressed first video in the second video DB 400 .

[0051] According to this embodiment, in addition to the effects of the second embodiment, the compression unit 102c adjusts the compression parameters by receiving feedback of the scene estimation accuracy of the scene estimation unit 105c, which has the advantage of being able to improve the accuracy of scene estimation for equivalent video in the scene estimation unit 105c.

[0052] [Fifth Embodiment] In the above-described embodiments, the compression unit compresses the video from the first video DB 300 in advance and stores it in the second video DB 400. However, a configuration in which the video is compressed before scene estimation can also be employed. FIG. 10 is a diagram illustrating another configuration of the present disclosure. The first difference from the first embodiment illustrated in FIG. 5 is that the search unit 104d of the scene search system 100d searches the first video DB 300 for video to be used for scene estimation. The second difference from the first embodiment is that the compression unit 102d compresses the first video searched by the search unit 104d and stores it in the second video DB 400. The third difference from the first embodiment is that the scene estimation unit 105d performs scene estimation on the compressed second video. Since the other configurations are substantially the same as those of the first embodiment, the following description will focus on the operational differences.

[0053] 11 is a sequence diagram for explaining the operation of the present disclosure. Referring to FIG. 11, first, the scene retrieval system 100 displays a search condition input screen on the display device of the user terminal 500 (step S001).

[0054] When the user of the user terminal 500 inputs search conditions (step S002), the scene search system 100 searches the first video DB 300 for a first video that matches the search conditions (step S103).

[0055] Next, the scene retrieval system 100 compresses the first video retrieved in step S103 and stores it in the second video DB 400 (step S104). Next, the scene retrieval system 100 estimates the scene of the stored second video (step S104).

[0056] Then, the scene retrieval system 100 determines whether the estimated scene matches the retrieval conditions (step S005). The subsequent operations are the same as those in the first embodiment, and therefore will not be described further.

[0057] As described above, in comparison with the first embodiment, this embodiment has the advantage that the storage capacity used as the second video DB 400 can be reduced because it adopts a configuration in which the video that is the subject of scene estimation is compressed.

[0058] [Sixth Embodiment] Next, a sixth embodiment will be described, which provides a different solution to the situation where the desired scene estimation accuracy cannot be achieved. Fig. 12 is a diagram showing another configuration of the present disclosure. The difference from the second embodiment shown in Fig. 7 is that the scene estimation unit 105e of the scene retrieval system 100e is able to access the first video in the first video DB 300. Since the other configurations are almost the same as those of the second embodiment, the following description will focus on the operational differences.

[0059] The scene estimation unit 105e of the scene retrieval system 100e calculates the accuracy of the estimation result (scene estimation accuracy) when performing scene estimation. If the scene estimation accuracy is below a predetermined threshold, the scene estimation unit 105e performs scene estimation on the first video in the first video DB 300.

[0060] According to this embodiment, similarly to the third and fourth embodiments, it becomes possible to recover from failure of scene estimation due to problems with compression parameters or the state of the second video.

[0061] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and further modifications, substitutions, and adjustments can be made without departing from the basic technical concept of the present disclosure. For example, the network configurations, element configurations, and data representation formats shown in the drawings are examples intended to aid in understanding the present disclosure, and are not limited to the configurations shown in these drawings.

[0062] For example, in the first to sixth embodiments described above, an example was given in which the video stored in the first video DB 300 was video captured by a camera mounted on a moving body such as a vehicle, but the application of the present disclosure is not limited to this.

[0063] (UC1) Scene collection at a construction site, etc. For example, video captured by a camera installed at the construction site is stored in the first video DB 300. Examples of tags include construction machinery, workers, and high altitudes. Using the scene search system of the present disclosure, it is possible to extract from a large volume of video footage video that matches search criteria, such as "a worker working at a high altitude without wearing a safety harness." Using such video footage as training data, it is possible to build a system that issues appropriate warnings to workers working at high altitudes. (UC2) Scene collection at a production line, etc. For example, video captured by a work status monitoring camera installed on the production line is stored in the first video DB 300. Examples of tags include product name, tool name, and type of work. Using the scene search system of the present disclosure, it is possible to extract from a large volume of video footage video that matches search criteria, such as "a scene where soldering failed during the production of product A." Using such video footage as training data, it is possible to build a system that issues appropriate warnings when a situation that could lead to a failure in the soldering process of product A occurs. (UC3) Scene collection in a retail store, etc. For example, video captured by a surveillance camera installed in a retail store is stored in the first video DB 300. Examples of tags include people (store clerks or customers (which may be further classified by attributes such as gender and age)), types of products, and carts used for stocking products. Using the scene search system disclosed herein makes it possible to extract from a large amount of video footage those videos that meet search criteria such as "a scene showing a man of approximately XX years old holding product B." Using such video as learning data, a system can be constructed to analyze the behavior of the target demographic for each product.

[0064] Furthermore, images searchable by the scene search system of the present disclosure are not limited to images captured by a camera using visible light. For example, images captured by an infrared video camera can be searched by storing them in the first image DB 300. Similarly, images of point cloud data generated by a 3D scanner or images captured by a synthetic aperture radar (SAR) can also be searched by storing them in the first image DB 300.

[0065] (Hardware Configuration) In each embodiment of the present disclosure, each component of each device represents a functional unit block. Some or all of the components of each device are realized by an arbitrary combination of an information processing device 900 and a program, for example, as shown in FIG. 13. FIG. 13 is a block diagram showing an example of the hardware configuration of the information processing device 900 that realizes each component of each device. The information processing device 900 includes, as an example, the following configuration: - CPU (Central Processing Unit) 901 - ROM (Read Only Memory) 902 - RAM (Random Access Memory) 903 - Program 904 loaded into RAM 903 - Storage device 905 that stores the program 904 - Drive device 907 that reads and writes to a recording medium 906 - Communication interface 908 that connects to a communication network 909 - Input / output interface 910 that inputs and outputs data - Bus 911 that connects each component

[0066] Each component of each device in each embodiment is realized by the CPU 901 acquiring and executing a program 904 that realizes the function. That is, the CPU 901 in FIG. 13 executes a video search program and a scene estimation program, and performs an update process for each calculation parameter stored in the RAM 903, the storage device 905, etc. The program 904 that realizes the function of each component of each device is stored in the storage device 905 or the ROM 902 in advance, for example, and is read by the CPU 901 as needed. The program 904 may be supplied to the CPU 901 via the communication network 909, or may be stored in advance on the recording medium 906, and the drive device 907 may read the program and supply it to the CPU 901.

[0067] There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a separate information processing device 900 and a program for each component. Furthermore, multiple components of each device may be realized by any combination of a single information processing device 900 and a program. In other words, each unit (processing means, function) of the scene retrieval system described above can be realized by a computer program that causes a processor installed in the device to execute each of the above processes using its hardware.

[0068] In addition, some or all of the components of each device may be realized by other general-purpose or dedicated circuits, processors, etc., or a combination of these. These may be configured by a single chip, or by multiple chips connected via a bus.

[0069] Some or all of the components of each device may be realized by a combination of the above-mentioned circuits and programs.

[0070] When some or all of the components of each device are realized by multiple information processing devices, circuits, etc., the multiple information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in a form in which each device is connected via a communication network.

[0071] It should be noted that the above-described embodiments are preferred embodiments of the present disclosure, and the scope of the present disclosure is not limited to only the above-described embodiments. In other words, those skilled in the art can modify or substitute the above-described embodiments to construct various modified forms without departing from the gist of the present disclosure.

[0072] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0073] [Supplementary Note 1] A scene retrieval system including: a first storage means for storing first video to which tags have been assigned in advance; a second storage means for storing second video obtained by compressing the video stored in the first storage means and the tags; a search means for accepting input of search criteria from a user requesting video showing a specific scene and for searching the second storage means for second video to which tags corresponding to the search criteria have been assigned; a scene estimation means for estimating a scene shown in the second video and determining whether the second video matches the search criteria; and a presentation means for, if it is determined that the second video matches the search criteria, identifying a first video corresponding to the second video from the first storage means and presenting the first video to the user who input the search criteria. [Supplementary Note 2] The above-mentioned scene retrieval system may further include a compression means for compressing the first video with predetermined compression parameters. [Supplementary Note 3] The compression means of the above-mentioned scene retrieval system can be configured to determine compression parameters that can satisfy a desired scene estimation accuracy for the input video, and compress the first video using the compression parameters. [Supplementary Note 4] The scene estimation means of the above-mentioned scene retrieval system can be configured to output a scene estimation accuracy when scene estimation is performed, and if the scene estimation accuracy is lower than a desired accuracy, the compression means changes the compression parameters, re-compresses the video, and stores the re-compressed video in the second database as a second video, and re-perform scene estimation on the re-compressed second video. [Supplementary Note 5] The scene estimation means of the above-mentioned scene retrieval system can be configured to output a scene estimation accuracy when scene estimation is performed, and the compression means additionally learns a relationship model between the compression parameters and the scene estimation accuracy, and determines compression parameters that can satisfy a predetermined scene estimation accuracy using the relationship model.[Supplementary Note 6] The above-mentioned scene search system may be configured such that, instead of the search means that receives input of search conditions from a user to request a video showing a specific scene and searches the second storage means for a second video to which a tag corresponding to the search conditions has been assigned, the search means receives input of search conditions from a user to request a video showing a specific scene and searches the first storage means for a first video to which a tag corresponding to the search conditions has been assigned, the compression means compresses the searched first video using predetermined compression parameters and stores it in the second storage means, and the scene estimation means determines whether the second video matches the search conditions when the second video is stored in the second storage means. [Supplementary Note 7] The scene estimation means of the above-mentioned scene search system can be configured to output the scene estimation accuracy when performing scene estimation, and if the scene estimation accuracy is lower than a desired accuracy, the scene estimation means retrieves a first video corresponding to the second video from the first storage means and performs scene estimation again, and if the presentation means determines that the first video matches the search conditions, it presents the first video to the user who input the search conditions. [Supplementary Note 8] A scene search method, comprising: a computer that can access a first storage means that stores a first video to which a tag has been assigned in advance; and a second storage means that stores a second video obtained by compressing the video stored in the first storage means and the tag; the computer accepts input of search criteria from a user to request a video showing a specific scene; searches the second storage means for a second video to which a tag corresponding to the search criteria has been assigned; estimates a scene shown in the second video and determines whether the second video matches the search criteria; and, if it is determined that the second video matches the search criteria, identifies a first video corresponding to the second video stored in the first storage means and presents the first video to the user who input the search criteria.[Supplementary Note 9] A recording medium having recorded thereon a program that causes a computer that can access a first storage means that stores first video to which tags have been added in advance, and a second storage means that stores second video obtained by compressing the video stored in the first storage means and the tags, to execute the following processes: accepting input of search criteria from a user to request video containing a specific scene, and searching the second storage means for second video to which tags corresponding to the search criteria have been added; estimating a scene contained in the second video and determining whether the second video matches the search criteria; and, if it is determined that the second video matches the search criteria, identifying a first video corresponding to the second video stored in the first storage means and presenting the first video to the user who input the search criteria. Note that the aspects described in the above supplementary notes can be combined with each other after making necessary modifications. For example, the scope of disclosure of this specification also includes a configuration that combines the contents of Supplementary Note 2 and the contents of Supplementary Note 3, includes a compression means, and determines compression parameters that can satisfy a desired scene estimation accuracy for an input video. Note that the forms of Supplementary Note 8 to Supplementary Note 9 can be expanded to the forms of Supplementary Note 2 to Supplementary Note 7, similar to Supplementary Note 1.

[0074] The disclosures of the above-cited patent documents are incorporated herein by reference and may be used as the basis or part of this disclosure, as necessary. Modifications and adjustments of the embodiments and examples are possible within the scope of this disclosure (including the claims), and further based on its basic technical concept. Furthermore, various combinations and selections (including partial deletions) of various disclosed elements (including elements of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the scope of this disclosure. In other words, this disclosure naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure, including the claims, and the technical concept. In particular, with regard to the numerical ranges described herein, any numerical value or subrange within that range should be construed as specifically described, even if not otherwise specified. Furthermore, the disclosures of the above-cited documents, when used in part or in whole in combination with the disclosures herein as part of this disclosure, in accordance with the spirit of this disclosure, are also deemed to be included in the disclosures of this application.

[0075] 10 Scene retrieval system 11 First storage means 13 Second storage means 14 Retrieval means 15 Scene estimation means 16 Presentation means 100, 100a to 100e Scene retrieval system 200 In-vehicle terminal 300 First video DB (database) 400 Second video DB (database) 500 User terminal 102, 102a, 102b, 102c, 102d Compression unit 104, 104d Search unit 105, 105b, 105c, 105d, 105e Scene estimation unit 106 Presentation unit 107, 107c Compression parameter determination unit 900 Information processing device 901 CPU (Central Processing Unit) 902 ROM (Read Only Memory) 903 RAM (Random Access Memory) 904 Program 905 Storage device 906 Recording medium 907 Drive device 908 Communication interface 909 Communication network 910 Input / output interface 911 Bus

Claims

a first storage means for storing a first video to which a tag has been added in advance; a second storage means for storing a second image obtained by compressing the image stored in the first storage means and the tag; a search means for receiving an input of a search condition and searching the second storage means for a second video to which a tag corresponding to the search condition has been added; a scene estimation means for estimating a scene captured in the second video and determining whether the second video matches the search criteria; a presentation means for identifying and presenting a first video corresponding to the second video from the first storage means when it is determined that the second video matches the search condition; A scene search system including:

2. The scene retrieval system of claim 1, further comprising compression means for compressing said first video with predetermined compression parameters. 　 the compression means determines compression parameters for the input video that can satisfy a desired scene estimation accuracy; 3. The scene retrieval system of claim 2, wherein the first video is compressed using the compression parameters. 　 the scene estimation means outputs scene estimation accuracy when performing scene estimation; If the scene estimation accuracy is lower than a desired accuracy, the compression means changes the compression parameters, re-compresses the video, and stores the re-compressed video in the second storage means as a second video; the scene estimation means re-performs scene estimation on the recompressed second video. The scene retrieval system of claim 2. the scene estimation means outputs scene estimation accuracy when performing scene estimation; 4. The scene retrieval system according to claim 3, wherein said compression means additionally learns a relationship model between said compression parameters and said scene estimation accuracy, and determines compression parameters that can satisfy a predetermined scene estimation accuracy using said relationship model. 　 Instead of the search means receiving an input of search criteria for requesting a video showing a specific scene from a user, and searching the second storage means for a second video to which a tag corresponding to the search criteria has been added, a search means for receiving an input of a search condition for requesting a video showing a specific scene from a user, and searching the first storage means for a first video to which a tag corresponding to the search condition has been added; the compression means compresses the retrieved first video using predetermined compression parameters and stores the compressed first video in the second storage means; the scene estimation means determines whether or not the second video matches the search criteria when the second video is stored in the second storage means; The scene retrieval system of claim 2. the scene estimation means outputs scene estimation accuracy when performing scene estimation; If the scene estimation accuracy is lower than a desired accuracy, the scene estimation means retrieves a first video corresponding to the second video from the first storage means and performs scene estimation again; When it is determined that the first video matches the search conditions, the presenting means presents the first video to the user who input the search conditions.

7. A scene retrieval system according to any one of claims 1 to 6. 　 a first storage means for storing a first video to which a tag has been added in advance; a computer that can access a second storage means that stores a second video obtained by compressing the video stored in the first storage means and the tag, and receives input of a search condition, and searches the second storage means for a second video that has been tagged with a tag corresponding to the search condition; a scene captured in the second video is estimated, and it is determined whether the second video matches the search criteria; If it is determined that the second video matches the search condition, a first video corresponding to the second video stored in the first storage means is identified and presented. How to search for scenes. 　 a first storage means for storing a first video to which a tag has been added in advance; a computer that can access a second storage means for storing a second image obtained by compressing the image stored in the first storage means and the tag; receiving an input of a search condition and searching the second storage means for a second video to which a tag corresponding to the search condition has been added; a process of estimating a scene captured in the second video and determining whether the second video matches the search criteria; a process of identifying and presenting a first video corresponding to the second video stored in the first storage means when it is determined that the second video matches the search condition; A recording medium on which a program for executing the above is recorded.

Images