Video retrieval method and device

Abstract

The invention discloses a video retrieval method and device, and belongs to the technical field of electronic equipment. The video retrieval method comprises the following steps: acquiring text information which is input by a user and is used for video retrieval; determining a first similarity between the text information and description information of each video in the plurality of videos; and taking the first video corresponding to the first similarity greater than a first threshold as a video retrieval result.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to a video retrieval method and a device thereof. Background Art

[0002] With the rapid development and popularization of electronic devices, people are increasingly accustomed to recording work or life through video recording. Recording various scenes has become an indispensable part of our life and work. However, with the increasing number of videos recorded, how to quickly and accurately retrieve the videos that users need from the videos recorded by users has become an urgent problem to be solved.

[0003] In the related art, recorded videos are usually classified according to video file type, video shooting time or video shooting location to obtain a video list arranged according to the video classification. When searching for videos, users can obtain the required videos by browsing the video list. However, this video retrieval method has low retrieval efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a video retrieval method and device thereof, which can solve the problem of low video detection efficiency.

[0005] In a first aspect, an embodiment of the present application provides a video retrieval method, comprising:

[0006] Obtaining text information input by the user for video retrieval;

[0007] Determining a first similarity between the text information and the description information of each of the plurality of videos;

[0008] The first video corresponding to the first similarity being greater than the first threshold is used as the video retrieval result.

[0009] In a second aspect, an embodiment of the present application provides a video retrieval device, including:

[0010] A first acquisition module, used to acquire text information input by a user for video retrieval;

[0011] A first determination module, used to determine a first similarity between the text information and the description information of each of the multiple videos;

[0012] The second determination module is used to take the first video corresponding to the first similarity being greater than the first threshold as the video retrieval result.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the video retrieval method provided in the embodiment of the present application are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the video retrieval method provided in the embodiment of the present application are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the video retrieval method provided in the embodiment of the present application.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the video retrieval method provided in an embodiment of the present application.

[0017] In the embodiment of the present application, the text information input by the user for video retrieval is obtained; the first similarity between the text information and the description information of each video in multiple videos is determined; and the first video corresponding to the first similarity greater than the first threshold is used as the video retrieval result. In this way, the user only needs to input the text information for video retrieval, and the similarity between the text information input by the user and the description information of each video in multiple videos is determined, and then the first video corresponding to the similarity greater than the first threshold is used as the video retrieval result. The user does not need to browse the video list, which can improve the efficiency of video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of a video retrieval method provided in an embodiment of the present application;

[0019] Figure 2 It is a schematic diagram of a photo provided in an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of a screenshot provided in an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a text message provided in an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of a video frame provided in an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of the structure of a video retrieval device provided in an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Figure 8 It is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] The video retrieval method and device provided by the embodiment of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0029] Figure 1 : is a flow chart of a video retrieval method provided in an embodiment of the present application. The video retrieval method may include:

[0030] Step 101: Obtaining text information input by a user for video retrieval;

[0031] In some possible implementations of the embodiments of the present application, in step 101, the user can input text information for video retrieval through a keyboard (including but not limited to a physical keyboard or a virtual keyboard) or a touch screen. For example, the text information input by the user for video retrieval is: video of changing mouse batteries.

[0032] Step 102: Determine a first similarity between the text information and the description information of each of the multiple videos;

[0033] In some possible implementations of the embodiments of the present application, in step 102, the text information and the description information of each video can be respectively input into the vector model, and the vector model outputs the vector corresponding to the text information and the vector corresponding to the description information of each video. Then, by calculating the similarity of the two vectors, the similarity of the text information and the description information of each video is obtained. Among them, the similarity of the two vectors can be measured using the cosine value of the angle between the two vectors.

[0034] In some possible implementations of the embodiments of the present application, the description information of each video in the embodiments of the present application may be generated in advance based on the description information of the objects included in the video and the media files related to the video. Based on this, before step 101, the video retrieval method provided by the embodiments of the present application may also include: determining a first media file related to a second video from multiple media files, wherein the second video is any one of the multiple videos; and generating description information of the second video based on the objects included in the second video and the description information of the first media file.

[0035] In some possible implementations of the embodiments of the present application, the media files in the embodiments of the present application include but are not limited to: pictures, videos, text messages, etc. The pictures in the embodiments of the present application can be photos, screenshots, downloaded pictures, pictures shared by other users, video frames, etc.

[0036] In some possible implementations of the embodiments of the present application, for a picture, an object included in the picture can be extracted using image recognition technology; optical character recognition (OCR) can also be used to extract text in the picture, and then a large language model (LLM) is used to extract the object included in the picture from the extracted text. The large language model is a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text.

[0037] In some possible implementations of the embodiments of the present application, for text messages, LLM can be used to extract objects included in the text messages.

[0038] In some possible implementations of the embodiments of the present application, for a video, a multimodal model can be used to identify the objects included in the video. Among them, the multimodal model is a machine learning model that can process multiple different types of data inputs obtained by training multimodal information such as text, images, videos, and audio. The multimodal model can simultaneously accept and process information from multiple data sources, including multiple different types of data forms such as text, images, sounds, and videos.

[0039] In some possible implementations of the embodiments of the present application, after obtaining the objects included in each media file and the objects included in each video, a word vector model can be used to generate a word vector corresponding to each object, and then the similarity between the objects can be calculated using the word vector. The similarity between the objects can be measured using the word vector similarity, and the word vector similarity can be measured using the cosine value of the angle between the word vectors.

[0040] In some possible implementations of the embodiments of the present application, each element in the similarity matrix in the embodiments of the present application is the similarity between two objects.

[0041] Exemplarily, it is assumed that the extracted objects include a keyboard, a mouse, a wooden board, a battery, and a hand. The generated similarity matrix is ​​shown in Table 1 below.

[0042] Table 1

[0043] keyboard mouse board Battery hand keyboard 1 0.72 0.35 0.2 0.5 mouse 0.72 1 0.3 0.25 0.6 board 0.35 0.3 1 0.1 0.22 Battery 0.2 0.25 0.1 1 0.4 hand 0.5 0.6 0.22 0.4 1

[0044] In some possible implementations of the embodiments of the present application, when determining the second similarity between the second video and each media file respectively according to the similarity matrix, the third similarity between the objects included in the second video and the objects included in the second media file can be found from the similarity matrix, wherein the second media file is any one of the multiple media files; the similarity between the second video and the second media file is determined according to the third similarity, the first number of objects included in the second video, and the second number of objects included in the second media file.

[0045] In some possible implementations of the embodiments of the present application, when determining the similarity between the second video and the second media file based on the third similarity, the first number of objects included in the second video, and the second number of objects included in the second media file, the similarity can be calculated using the following formula (1):

[0046]

[0047] In formula (1), S V-T is the similarity between video V and media file T, S(V i ,T j ) is the similarity between the i-th object in video V and the j-th object in media file T, N V is the number of objects included in the video V, N T is the number of objects included in the media file T, and K is the quantity balance factor.

[0048] In some possible implementations of the embodiments of the present application, the quantity balance factor K can be calculated by the following formula (2):

[0049]

[0050] In formula (2), K is the quantity balance factor, N V is the number of objects included in the video V, N T is the number of objects included in the media file T.

[0051] It can be seen from formula (2) that when the number of objects included in the video V is equal to the number of objects included in the media file T, the value of K is 1; when the number of objects included in the video V is significantly different from the number of objects included in the media file T, K approaches 0.

[0052] Exemplarily, it is assumed that a video includes objects such as a mouse, a battery, and a hand, and a media file includes objects such as a keyboard, a mouse, and a wooden board.

[0053] From the above Table 1, we can see that S(mouse, keyboard) = 0.72, S(mouse, mouse) = 1, S(mouse, wooden board) = 0.3, S(battery, keyboard) = 0.2, S(battery, mouse) = 0.25, S(battery, wooden board) = 0.1, S(hand, keyboard) = 0.5, S(hand, mouse) = 0.6, S(hand, wooden board) = 0.22; K = 2min(3,3) / (3+3) = 1.

[0054] Then the similarity between the video and the media file is:

[0055] (0.72+1+0.3+0.2+0.25+0.1+0.5+0.6+0.22) / (3*3)*1=43%.

[0056] In some possible implementations of the embodiments of the present application, after determining a media file related to a video from multiple media files, a multimodal model can be used to combine description information of objects included in the video and objects related to the video to generate description information of the video.

[0057] In some possible implementations of the embodiments of the present application, prompt information for generating description information of a video, description information of objects included in the video, and media files related to the video may be input into a multimodal model, and the multimodal model outputs description information of the video. The prompt is used to guide the multimodal model to generate description information of the video.

[0058] Exemplarily, the prompt for generating the description information of the video may be:

[0059] “You are an expert in generating video description information. Please describe the video in concise natural language based on the description information of the objects included in the given video and the media files related to the video.

[0060] Require:

[0061] 1. The objects included in the video refer to the protagonists or important objects in the video, such as people, animals or specific objects, which are usually the focus of the video scene;

[0062] 2. The description information of the media files related to the video is some background information about the objects included in the video;

[0063] 3. Please describe the video in concise language.

[0064] Input: [objects included in the video], [description information of media files related to the video];

[0065] Output: Description of the video".

[0066] In some possible implementations of the embodiments of the present application, for photos, the shooting time and location of the photos can be extracted through the metadata of the photos; the objects included in the photos can be extracted by using image recognition; and then a multimodal model is used to generate description information of the photo in combination with the objects included in the photo.

[0067] In some possible implementations of the embodiments of the present application, a prompt for generating description information of a photo, a photo, and an object included in the photo may be input into a multimodal model, and the multimodal model outputs description information of the photo. The prompt is used to guide the multimodal model to generate description information of the photo.

[0068] Exemplarily, the prompt for generating description information of a photo may be:

[0069] “You are an expert in photo description. Please describe the photos in concise and natural language based on the objects in the photos.

[0070] Require:

[0071] 1. Please describe the photo in brief words;

[0072] 2. The description should focus on the objects included in the photo;

[0073] 3. If there are multiple objects, you can describe the relationship between multiple objects.

[0074] Input: [photo], [object included in the photo];

[0075] Output: Description of the photo".

[0076] In some possible implementations of the embodiments of the present application, for screenshots, downloaded pictures, and pictures shared by other users, the time information and location information corresponding to the pictures (for example, screenshot time, picture download time, picture reception time, screenshot location, picture download location, and picture reception location, etc.) can be extracted through the metadata of the pictures; OCR is used to extract the text in the picture, and LLM is used to extract the objects and events included in the text from the extracted text, and then it is combined with the time information and location information corresponding to the picture to generate description information of the picture.

[0077] In some possible implementations of the embodiments of the present application, a prompt for extracting objects and events from a text and text extracted from a picture can be input into the LLM, and the LLM outputs the objects and events in the text. The prompt is used to instruct the LLM to extract objects and events from the text. After the objects and events in the text are extracted, description information of the picture can be generated in combination with the time information and location information corresponding to the picture.

[0078] Exemplarily, the prompt for extracting objects and events from text may be:

[0079] "You are an expert in text information extraction. Please extract the objects and events in the text based on the following text data.

[0080] Require:

[0081] 1. The text comes from the text obtained by OCR recognition of the image. The text may be messy and have errors. Please try to find the important content;

[0082] 2. Objects are important and meaningful content in the text. Please try to find meaningful objects;

[0083] 3. Events describe the relationship between objects. Please describe the events based on the extracted objects.

[0084] Input: [Text obtained by OCR recognition of the image];

[0085] Output: [extracted objects], [extracted events]".

[0086] In some possible implementations of the embodiments of the present application, for SMS, since SMS are all text messages, LLM can be used to extract important event information from SMS, and then generate SMS description information. If time information can be extracted from the SMS, the time information is used to generate the SMS description information, and if time information cannot be extracted from the SMS, the time when the SMS was received is used to generate the SMS description information.

[0087] Step 103: The first video corresponding to the first similarity being greater than the first threshold is used as the video retrieval result.

[0088] Exemplarily, assume that multiple videos include video A, video B and video C, and the first threshold is 80%; determine that the text information input by the user for video retrieval has a similarity of 35% with the description information of video A, 55% with the description information of video B, and 85% with the description information of video C, and take video C as the video retrieval result.

[0089] In the embodiment of the present application, the text information input by the user for video retrieval is obtained; the first similarity between the text information and the description information of each video in multiple videos is determined; and the first video corresponding to the first similarity greater than the first threshold is used as the video retrieval result. In this way, the user only needs to input the text information for video retrieval, and the similarity between the text information input by the user and the description information of each video in multiple videos is determined, and then the first video corresponding to the similarity greater than the first threshold is used as the video retrieval result. The user does not need to browse the video list, which can improve the efficiency of video retrieval.

[0090] In some possible implementations of the embodiments of the present application, before step 102, the video retrieval method provided by the embodiments of the present application may also include: obtaining a third number of objects included in each video; obtaining a fourth number of objects included in each video that appear in the text information respectively; and taking the video corresponding to the first ratio being greater than the third threshold as the video retrieval result, wherein the first ratio is the ratio of the fourth number to the third number.

[0091] For example, assuming that the number of objects included in video A is 3, the number of objects included in video B is 3, the number of objects included in video A that appear in the text information input by the user for video retrieval is 1, the number of objects included in video B that appear in the text information is 2, and the third threshold is 50%, then video B will be used as the video retrieval result.

[0092] In the embodiment of the present application, searching by objects included in the video first can improve the efficiency of video retrieval.

[0093] In some possible implementations of the embodiments of the present application, after step 102, the video retrieval method provided by the embodiments of the present application may also include: inputting the third video and text information corresponding to the first similarity less than the first threshold and greater than the fourth threshold into the multimodal model, and the multimodal model outputs the video retrieval result, wherein the fourth threshold is less than the first threshold.

[0094] In some possible implementations of the embodiments of the present application, when the similarity between the description information of a video and the text information input by the user for video retrieval is less than a first threshold and greater than a fourth threshold, it indicates that the video may be a video retrieval result. At this time, the video and the text information are input into the multimodal model, and the multimodal model determines whether the video is a video retrieval result. If the multimodal model determines that the video is a video retrieval result, the video is output; if the multimodal model determines that the video is not a video retrieval result, the video is not output.

[0095] In the embodiment of the present application, the videos that the user may retrieve can be returned to the greatest extent possible, thereby improving the user experience.

[0096] In some possible implementations of the embodiments of the present application, the various thresholds in the embodiments of the present application can be set according to actual needs.

[0097] The process of the video retrieval method provided in the embodiment of the present application is explained below with reference to specific examples.

[0098] First, through Figure 2 The metadata of the photo shown is extracted Figure 2 The photo was taken at 20:18 on August 10, 2022 and at a location (Community C, City B, Province A). Figure 2 The objects included in the photo shown are: keyboard, mouse, and wooden board; the multimodal model is used to generate the description information of the image in combination with the objects included in the photo; the description information of the image can be "Time: 20:18, August 10, 2022, Location: Community C, City B, Province A, Objects: Keyboard, mouse, and wooden board, Event: The keyboard, mouse, and wooden board are placed on the table."

[0099] pass Figure 3 The metadata of the screenshot shown is extracted Figure 3 The screenshots were captured at 12:15 on August 8, 2022 and at a location (Community C, City B, Province A). Figure 3 The text of the screenshot shown is "Waiting for delivery Model D Mouse 269 yuan Quantity 1 Black Confirm delivery Application After-sales Order No. XXXXXXXXXX Payment method Bank card payment Payment time 2022-08-07 12:21 Delivery address A Province B City C Community Delivery information Mr. Li 138XXXXXXXX"; input the text into LLM, and LLM outputs the object in the text: mouse, event: Mr. Li bought a black mouse on a shopping platform at 2022-08-07 12:21. The description information of the screenshot can be "Time: 12:15, August 8, 2022, Location: A Province B City C Community, Object: Mouse, Event: 2022-08-07 12:21 After buying a black mouse on a shopping platform".

[0100] Using LLM Figure 4 The object: mouse, event: the mouse purchased on a shopping platform will arrive in about 3 days from the extracted text message. The text message was received at 15:30 on August 8, 2022. The description of the text message can be "Time: 15:30 on August 8, 2022, Object: mouse, event: the mouse purchased on a shopping platform will arrive in about 3 days from the extracted object: mouse purchased on a shopping platform will arrive in about 3 days from the extracted object".

[0101] Will Figure 2 Photos shown, Figure 3 The screenshots and Figure 4 The description information corresponding to the short messages is stored in the database, and the storage results are shown in Table 2:

[0102] Table 2

[0103]

[0104] for Figure 5 The video corresponding to the video frame shown in the figure is extracted from the metadata of the video: the shooting time of the video is 9:35 on August 11, 2022, and the shooting location is C community, B city, Province A; the multimodal model is used to identify the objects included in the video: mouse, battery and hand. The similarity between each object is calculated to construct a similarity matrix as shown in Table 1 above. Through the above similarity matrix and the above formula (1), it is calculated Figure 5 The video frame shown corresponds to the video Figure 2 Similarity of the photos shown is: 43%; Figure 5 The video frame shown corresponds to the video Figure 3 The similarity of the screenshots shown is: 30%; Figure 5 The video frame shown corresponds to the video Figure 4 The similarity of the text messages shown is: 30%. Assuming the second threshold is 40%, then Figure 5 The video-related media file corresponding to the video frame shown is Figure 2 Photos shown.

[0105] Combining multimodal models Figure 5 The video frame shown corresponds to the video and Figure 2 Descriptive information of the photo shown, generated Figure 5 The description information of the video corresponding to the video frame shown is, for example: a person is about to replace the battery of a mouse by hand.

[0106] Will Figure 5 The description information of the video corresponding to the video frame is stored in the database, and the storage result is shown in Table 3:

[0107] Table 3

[0108]

[0109] When performing video retrieval, it is assumed that the text information input by the user for video retrieval is: video of mouse battery replacement; the ratio of the number of objects included in each video in the multiple videos that appear in the text information to the number of objects included in the corresponding video is determined respectively, and when there is a ratio greater than a third threshold, the video corresponding to the ratio is used as the video retrieval result; if there is no video with a ratio greater than the third threshold, the similarity between the text information "video of mouse battery replacement" and the description information of each video in the multiple videos is determined respectively, and if a certain similarity is greater than the first threshold, the video corresponding to the similarity is used as the video retrieval result; if there is no similarity greater than the first threshold, but there is a similarity less than the first threshold and greater than the fourth threshold, the video corresponding to the similarity and the text information "video of mouse battery replacement" are input into the multimodal model, and the multimodal model outputs the video retrieval result.

[0110] The video retrieval method provided in the embodiment of the present application can be executed by a video retrieval device. In the embodiment of the present application, the video retrieval device executing the video retrieval method is taken as an example to illustrate the video retrieval device provided in the embodiment of the present application.

[0111] Figure 6 600 is a schematic diagram of the structure of a video retrieval device provided in an embodiment of the present application. The video retrieval device 600 may include:

[0112] A first acquisition module 601 is used to acquire text information input by a user for video retrieval;

[0113] A first determination module 602, configured to determine a first similarity between the text information and the description information of each of the plurality of videos;

[0114] The second determination module 603 is configured to take the first video corresponding to the first similarity being greater than the first threshold as the video retrieval result.

[0115] In the embodiment of the present application, the text information input by the user for video retrieval is obtained; the first similarity between the text information and the description information of each video in multiple videos is determined; and the first video corresponding to the first similarity greater than the first threshold is used as the video retrieval result. In this way, the user only needs to input the text information for video retrieval, and the similarity between the text information input by the user and the description information of each video in multiple videos is determined, and then the first video corresponding to the similarity greater than the first threshold is used as the video retrieval result. The user does not need to browse the video list, which can improve the efficiency of video retrieval.

[0116] In some possible implementations of the embodiment of the present application, the video retrieval device 600 provided in the embodiment of the present application further includes:

[0117] a third determination module, configured to determine a first media file related to a second video from a plurality of media files before obtaining text information for video retrieval input by a user, wherein the second video is any one of the plurality of videos;

[0118] The generating module is used to generate description information of the first video according to the objects included in the second video and the description information of the first media file.

[0119] In some possible implementations of the embodiments of the present application, the third determining module includes:

[0120] A generating submodule, for generating a similarity matrix according to the objects included in each media file and the objects included in each video in the plurality of media files;

[0121] A first determination submodule, used to determine a second similarity between the second video and each media file according to the similarity matrix;

[0122] The second determination submodule is configured to determine the media file image corresponding to the second similarity being greater than the second threshold as the first media file related to the second video.

[0123] In some possible implementations of the embodiments of the present application, the first determining submodule is specifically used to:

[0124] searching, from the similarity matrix, a third similarity between an object included in the second video and an object included in a second media file, wherein the second media file is any one object in the plurality of media files;

[0125] The similarity between the second video and the second media file is determined according to the third similarity, the first number of objects included in the second video, and the second number of objects included in the second media file.

[0126] In some possible implementations of the embodiment of the present application, the video retrieval device 600 provided in the embodiment of the present application further includes:

[0127] A second acquisition module, configured to acquire a third number of objects included in each video before determining a first similarity between the text information and the description information of each of the multiple videos;

[0128] A third acquisition module is used to acquire a fourth number of objects included in each video that appear in the text information;

[0129] The fourth determination module is used to take the video corresponding to the first ratio being greater than the third threshold as the video retrieval result, wherein the first ratio is the ratio of the fourth number to the third number.

[0130] In the embodiment of the present application, searching by objects included in the video first can improve the efficiency of video retrieval.

[0131] In some possible implementations of the embodiment of the present application, the video retrieval device 600 provided in the embodiment of the present application further includes:

[0132] The fifth determination module is used to input the third video and text information corresponding to the first similarity less than the first threshold and greater than the fourth threshold into the multimodal model, and the multimodal model outputs the video retrieval result, wherein the fourth threshold is less than the first threshold.

[0133] In the embodiment of the present application, the videos that the user may retrieve can be returned to the greatest extent possible, thereby improving the user experience.

[0134] The video retrieval device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0135] The video retrieval device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0136] The video retrieval device provided in the embodiment of the present application can achieve Figures 1 to 5 To avoid repetition, the various processes implemented in the video retrieval method embodiment will not be described here.

[0137] Alternatively, if Figure 7As shown, an embodiment of the present application further provides an electronic device 700, including a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can be executed on the processor 701, and when the program or instructions are executed by the processor 701, the various steps of the video retrieval method embodiment provided in the embodiment of the present application are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0138] Figure 8 It is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0139] The electronic device 800 includes but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810 and other components.

[0140] Those skilled in the art will appreciate that the electronic device 800 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 810 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0141] The processor 810 is used to: obtain text information input by a user for video retrieval; determine a first similarity between the text information and description information of each of a plurality of videos; and use a first video corresponding to a first similarity greater than a first threshold as a video retrieval result.

[0142] In the embodiment of the present application, the text information input by the user for video retrieval is obtained; the first similarity between the text information and the description information of each video in multiple videos is determined; and the first video corresponding to the first similarity greater than the first threshold is used as the video retrieval result. In this way, the user only needs to input the text information for video retrieval, and the similarity between the text information input by the user and the description information of each video in multiple videos is determined, and then the first video corresponding to the similarity greater than the first threshold is used as the video retrieval result. The user does not need to browse the video list, which can improve the efficiency of video retrieval.

[0143] In some possible implementations of the embodiments of the present application, the processor 810 is further configured to:

[0144] Before obtaining the text information input by the user for video retrieval, a first media file related to a second video is determined from multiple media files, wherein the second video is any one of the multiple videos; and description information of the second video is generated based on the objects included in the second video and the description information of the first media file.

[0145] In some possible implementations of the embodiments of the present application, the processor 810 is specifically configured to:

[0146] Generate a similarity matrix based on the objects included in each media file and the objects included in each video in the plurality of media files;

[0147] Calculate the second similarity between the second video and each object according to the similarity matrix;

[0148] The media file corresponding to the second similarity being greater than the second threshold is determined as the first media file related to the second video.

[0149] In some possible implementations of the embodiments of the present application, the processor 810 is specifically configured to:

[0150] searching, from the similarity matrix, a third similarity between an object included in the second video and an object included in a second media file, wherein the second media file is any one object in the plurality of media files;

[0151] The similarity between the second video and the second media file is determined according to the third similarity, the first number of objects included in the second video, and the second number of objects included in the second media file.

[0152] In some possible implementations of the embodiments of the present application, the processor 810 is further configured to:

[0153] Before determining the first similarity between the text information and the description information of each of the multiple videos, obtaining a third number of objects included in each video;

[0154] Obtaining a fourth number of objects included in each video that appear in the text information;

[0155] The video corresponding to the first ratio being greater than the third threshold is taken as the video retrieval result, wherein the first ratio is the ratio of the fourth number to the third number.

[0156] In the embodiment of the present application, video retrieval is first performed based on the objects included in the video, which can improve the efficiency of video retrieval.

[0157] In some possible implementations of the embodiments of the present application, the processor 810 is further configured to:

[0158] The third video and text information corresponding to the first similarity less than the first threshold and greater than the fourth threshold are input into the multimodal model, and the multimodal model outputs the video retrieval result, wherein the fourth threshold is less than the first threshold.

[0159] In the embodiment of the present application, the videos that the user may retrieve can be returned to the greatest extent possible, thereby improving the user experience.

[0160] It should be understood that in the embodiment of the present application, the input unit 804 may include a graphics processor (Graphics Processing Unit, GPU) 8041 and a microphone 8042, and the graphics processor 8041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 807 includes a touch panel 8071 and at least one of other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include two parts: a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0161] The memory 809 can be used to store software programs and various data. The memory 809 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 809 may include a volatile memory or a non-volatile memory, or the memory 809 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 809 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0162] The processor 810 may include one or more processing units; optionally, the processor 810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 810.

[0163] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the video retrieval method embodiment provided in the embodiment of the present application are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0164] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, and examples of computer-readable storage media include non-transitory computer-readable media, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0165] An embodiment of the present application also provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the video retrieval method embodiment provided in the embodiment of the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0166] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0167] The embodiment of the present application also provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the video retrieval method embodiment provided in the embodiment of the present application, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0168] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0169] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0170] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video retrieval method, characterized in that: The method comprises: Obtaining text information input by the user for video retrieval; Determine a first similarity between the text information and description information of each of the plurality of videos; The first video corresponding to the first similarity being greater than the first threshold is used as the video retrieval result.

2. The method according to claim 1, characterized in that Before obtaining the text information input by the user for video retrieval, the method further includes: Determining a first media file related to a second video from a plurality of media files, wherein the second video is any one of the plurality of videos; Generate description information of the second video according to the objects included in the second video and the description information of the first media file.

3. The method according to claim 2, characterized in that The determining of a first media file related to the second video from a plurality of media files includes: Generate a similarity matrix according to the objects included in each media file and the objects included in each video in the plurality of media files; Determining, according to the similarity matrix, a second similarity between the second video and each of the media files; The media file corresponding to the second similarity being greater than the second threshold is determined as the first media file related to the second video.

4. The method according to claim 3, characterized in that: Determining, according to the similarity matrix, a second similarity between the second video and each of the media files, includes: searching, from the similarity matrix, a third similarity between an object included in the second video and an object included in a second media file, wherein the second media file is any one of the plurality of media files; The similarity between the second video and the second media file is determined according to the third similarity, a first number of objects included in the second video, and a second number of objects included in the second media file.

5. The method according to claim 1, characterized in that Before determining the first similarity between the text information and the description information of each of the multiple videos, the method further includes: Obtaining a third number of objects included in each of the videos; Obtain a fourth number of objects included in each video that appear in the text information respectively; The video corresponding to the first ratio being greater than the third threshold is taken as the video retrieval result, wherein the first ratio is the ratio of the fourth number to the third number.

6. A video retrieval device, characterized in that: The device comprises: A first acquisition module, used to acquire text information input by a user for video retrieval; A first determination module, used to determine a first similarity between the text information and the description information of each of the multiple videos; The second determining module is used to take the first video corresponding to the first similarity being greater than the first threshold as the video retrieval result.

7. The device according to claim 6, characterized in that The device also includes: a third determination module, configured to determine, before acquiring the text information for video retrieval input by the user, a first media file related to a second video from a plurality of media files, wherein the second video is any one of the plurality of videos; A generation module is used to generate description information of the second video according to the objects included in the second video and the description information of the first media file.

8. The device according to claim 7, characterized in that The third determination module comprises: A generating submodule, configured to generate a similarity matrix according to the objects included in each media file and the objects included in each video in the plurality of media files; A first determination submodule, configured to determine, according to the similarity matrix, a second similarity between the second video and each of the media files; The second determining submodule is configured to determine the media file corresponding to the second similarity being greater than a second threshold as a first media file related to the second video.

9. The device according to claim 8, characterized in that The first determining submodule is specifically used for: searching, from the similarity matrix, a third similarity between an object included in the second video and an object included in a second media file, wherein the second media file is any object in the plurality of media files; The similarity between the second video and the second media file is determined according to the third similarity, a first number of objects included in the second video, and a second number of objects included in the second media file.

10. The device according to claim 6, characterized in that The device also includes: A second acquisition module, configured to acquire a third number of objects included in each video before determining a first similarity between the text information and the description information of each video in the plurality of videos; A third acquisition module, used for acquiring a fourth number of objects included in each video that appear in the text information; The fourth determination module is used to take the video corresponding to the first ratio being greater than the third threshold as the video retrieval result, wherein the first ratio is the ratio of the fourth number to the third number.

Images