Method, system and terminal device for stimulating search based on real-time video content

Abstract

The present disclosure provides a method for stimulating search based on real-time video content, the method comprising: obtaining a video clip of interest; according to the video clip of interest, querying from a cloud offline recognition database or extracting the video entity contained in the video clip of interest by using a local online recognition method; performing an associated search based on the obtained video entity to obtain a search result; and feeding back the search result to a user. The present disclosure also provides a system and a terminal device for stimulating search based on real-time video content.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of Internet technology, and in particular to a method, system and terminal device for stimulating search based on real-time video content. Background Art

[0002] In recent years, the form of Internet content has shown a clear trend towards video, that is, more and more content is presented in the form of video, especially the explosion of short video platforms, which makes users spend more and more time watching videos. Based on this, users' inherent information search needs, shopping needs, etc. are gradually converging to video player platforms, that is: in the video consumption link, there are more and more demands based on the video content watched, such as "Who is this / where is this / background music (BGM) / where to buy this thing / what car is this..."

[0003] In the prior art, the user's search method based on the above video content is generally that the user first locates the key frame by pausing the playback or dragging the progress bar, and then the system extracts the video entity from the acquired key frame according to a preset algorithm, and then obtains the search result by searching the video entity. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, system, terminal device, computer-readable medium, and computer program product for stimulating search based on real-time video content.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for stimulating search based on real-time video content, the method comprising:

[0006] Get interesting video clips;

[0007] According to the video segment of interest, query the video entity contained in the video segment of interest from a cloud offline recognition database or extract the video entity contained in the video segment of interest by using a local online recognition method;

[0008] Performing an associated search based on the acquired video entity to obtain search results;

[0009] The search results are fed back to the user.

[0010] In some embodiments, querying from a cloud offline recognition database or extracting the video entity included in the video segment of interest by using a local online recognition method according to the video segment of interest includes:

[0011] Determine whether the cloud offline recognition database stores relevant information of the video clip of interest;

[0012] When it is determined that the cloud offline recognition database stores the relevant information of the video segment of interest, querying the video entity included in the video segment of interest from the cloud offline recognition database;

[0013] When it is determined that the cloud offline recognition database does not store relevant information of the video segment of interest, a preset entity extraction algorithm is used to perform local online recognition on the video segment of interest to extract the video entities contained in the video segment of interest.

[0014] In some embodiments, the cloud offline recognition database stores video entities contained in each of the pre-selected popular videos.

[0015] In some embodiments, the cloud offline recognition database stores video identifiers of each of the popular videos;

[0016] After the step of obtaining the video segment of interest and before the step of querying the video entity contained in the video segment of interest from a cloud offline recognition database or extracting the video entity contained in the video segment of interest by using a local online recognition method according to the video segment of interest, the method further includes:

[0017] Determine the video identifier of the complete video to which the video clip of interest belongs;

[0018] The determining whether the cloud offline recognition database stores the relevant information of the video clip of interest specifically includes:

[0019] Querying whether the cloud offline recognition database stores a video identifier of the complete video to which the video clip of interest belongs;

[0020] If the video identifier of the complete video to which the video clip of interest belongs is queried in the cloud offline recognition database, it is determined that the cloud offline recognition database stores relevant information of the video clip of interest;

[0021] If the video identifier of the complete video to which the video segment of interest belongs is not found in the cloud offline recognition database, it is determined that the relevant information of the video segment of interest is not stored in the cloud offline recognition database.

[0022] In some embodiments, the step of obtaining the video segment of interest specifically includes:

[0023] Determine in real time whether the user performs the scheduled operation;

[0024] When it is determined that the user performs the predetermined operation, the video collection time period is determined to determine the video content watched by the user in the video collection time period as the video segment of interest; wherein the end time of the video collection segment is the time t2 corresponding to when the user performs the predetermined operation, and the start time of the video collection time period is t1, t1=t2-t0, t0 is a preset duration value.

[0025] In some embodiments, after the step of acquiring the video entity and before the step of searching according to the acquired video entity, the method further includes:

[0026] Determining whether the number of acquired video entities is greater than a predetermined threshold;

[0027] When it is determined that the number of acquired video entities is greater than a predetermined threshold, video entities associated with the user portrait acquired in advance are screened out, and the subsequent step of performing an associated search based on the acquired video entities is continued;

[0028] When it is determined that the number of acquired video entities is less than or equal to a predetermined threshold, it is further determined whether the number of acquired video entities is zero. When it is determined that the number of video entities is zero, relevant text prompts are fed back to the user and the search process ends.

[0029] In some embodiments, the user portrait includes: a preset vertical category recognition priority and the user's preference for a specific vertical category.

[0030] In some embodiments, the method of feeding back the search results to the user includes: a half-screen web page pop-up layer and a semi-transparent floating layer.

[0031] In a second aspect, the present disclosure also provides a system for stimulating search based on real-time video content, including:

[0032] A first acquisition module is used to acquire a video clip of interest;

[0033] A second acquisition module is used to query the video entity contained in the video segment of interest from a cloud offline recognition database or extract the video entity contained in the video segment of interest by using a local online recognition method according to the video segment of interest;

[0034] A search module, configured to perform an associated search based on the video entity extracted by the second acquisition module to obtain search results;

[0035] The feedback module is used to feed back the search results to the user.

[0036] In some embodiments, the second acquisition module includes:

[0037] A first determination unit is used to determine whether the cloud offline recognition database stores the relevant information of the video clip of interest;

[0038] a query unit, configured to query the video entity included in the video segment of interest from the cloud offline recognition database when the first judgment unit determines that the cloud offline recognition database stores relevant information of the video segment of interest;

[0039] The computing unit is used to perform local online recognition on the video segment of interest by using a preset entity extraction algorithm when the first judging unit determines that the cloud offline recognition database does not store relevant information of the video segment of interest, so as to extract the video entity contained in the video segment of interest.

[0040] In some embodiments, the cloud offline recognition database stores video entities contained in each of the pre-selected popular videos.

[0041] In some embodiments, the cloud offline recognition database also stores video identifiers of each of the popular videos;

[0042] The system further comprises:

[0043] an identification determination module, configured to determine, after the second acquisition module acquires the video segment of interest, a video identification of the complete video to which the video segment of interest belongs;

[0044] The first judgment unit is specifically used to query whether the cloud-based offline recognition database stores the video identifier of the complete video to which the video clip of interest belongs, as determined by the identifier determination module; and when the video identifier of the complete video to which the video clip of interest belongs is queried in the cloud-based offline recognition database, it is judged that relevant information of the video clip of interest is stored in the cloud-based offline recognition database; when the video identifier of the complete video to which the video clip of interest belongs is not queried in the cloud-based offline recognition database, it is judged that relevant information of the video clip of interest is not stored in the cloud-based offline recognition database.

[0045] In some embodiments, the acquisition module includes:

[0046] A monitoring unit, used to determine in real time whether the user performs a predetermined operation;

[0047] A determination unit is used to determine a video acquisition time period when the monitoring unit determines that the user performs the predetermined operation, and to determine the video content watched by the user in the video acquisition time period as the video segment of interest; wherein the end time of the video acquisition segment is the time t2 corresponding to when the user performs the predetermined operation, the start time of the video acquisition time period is t1, and t1=t2-t0, t0 is a preset duration value.

[0048] In some embodiments, the second acquisition module further includes:

[0049] A second judgment unit, configured to judge whether the number of the acquired video entities is greater than a predetermined threshold after the query unit or the calculation unit extracts the video entity;

[0050] a screening module, configured to screen out video entities associated with a user portrait obtained in advance when the second judging unit judges that the number of video entities is greater than a predetermined threshold;

[0051] The third judgment unit is used to further judge whether the number of acquired video entities is zero when the second judgment unit judges that the number of video entities is less than or equal to a predetermined threshold, and to instruct the feedback module to feed back relevant text prompts to the user when it is judged that the number of video entities is zero.

[0052] In some embodiments, the user portrait includes: a preset vertical category priority and the user's preference for a specific vertical category.

[0053] In some embodiments, the feedback module feeds back the search results to the user in the following ways: half-screen web page pop-up layer and semi-transparent floating layer.

[0054] In a third aspect, the embodiment of the present disclosure further provides a terminal device for stimulating search based on real-time video content, including:

[0055] one or more processors;

[0056] A storage device stores one or more programs thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0057] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable medium having a computer program stored thereon, wherein the program implements the above-mentioned method when executed by a processor.

[0058] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0059] The method, system, terminal device, computer-readable medium and computer program product for stimulating search based on real-time video content provided by the embodiments of the present disclosure can trigger the search process by determining in real time whether the user performs a predetermined operation, which can greatly shorten the user's operation path, thereby making the search scenario based on video content stimulation easier to implement and improving the user experience; and the preset cloud offline recognition database stores the video entities contained in the pre-selected popular videos with high views, and when extracting video entities from the video clips of interest to the user, the cloud offline query and local online recognition are combined, so that when the user is watching a popular video with high views, the video entity of interest to the user can be quickly extracted through the cloud query, and when the user is watching an unpopular video, the video entity of interest to the user can also be extracted through the local online recognition method, thereby maximally satisfying the user's video entity extraction needs for popular and unpopular videos, and at the same time improving the extraction rate, and then further improving the user experience when the search results are fed back to the user based on the extracted video entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings, in which:

[0061] Figure 1 A flowchart of a method for stimulating search based on real-time video content provided by an embodiment of the present disclosure;

[0062] Figure 2 A flowchart of another method for stimulating search based on real-time video content provided by an embodiment of the present disclosure;

[0063] Figure 3 A schematic diagram of the structure of a system for stimulating search based on real-time video content provided by an embodiment of the present disclosure;

[0064] Figure 4 A schematic diagram of the structure of another system for stimulating search based on real-time video content provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the technical solution of the present invention, the method, system and terminal device for stimulating search based on real-time video content provided by the present invention are described in detail below with reference to the accompanying drawings.

[0066] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0067] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0068] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0069] The embodiments described herein may be described with reference to plan views and / or cross-sectional views by means of idealized schematic diagrams of the present disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to the embodiments shown in the accompanying drawings, but include modifications of the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings have schematic properties, and the shapes of the regions shown in the figures illustrate the specific shapes of the regions of the elements, but are not intended to be limiting.

[0070] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0071] The method or system for stimulating search based on real-time video content provided by the present disclosure is used to realize the search based on certain video entities in the video content when the user is watching a certain video. The video entity refers to the things involved in the video content. Specifically, the video entity includes various types of entities such as visual entities, text entities and sound entities. Typical visual entities include, for example, celebrity faces, small commodities, cars, logos, animals and plants, famous attractions / landmark buildings, etc.; typical text entities include, for example, celebrity names, commodity names, place names, dish names, animal and plant species names, etc.; typical sound entities include, for example, background music tracks, people’s speech content, humming, bird calls, door closing sounds, car starting sounds, etc.

[0072] like Figure 1 As shown, the method for stimulating search based on real-time video content provided by the embodiment of the present disclosure includes:

[0073] Step S101: Obtain a video clip of interest.

[0074] In the present disclosure, an interesting video segment is a video segment that a user is interested in, specifically refers to a video segment that contains a video entity that the user is interested in.

[0075] Step S102: According to the video segment of interest, the video entity included in the video segment of interest is queried from a cloud offline recognition database or extracted using a local online recognition method.

[0076] Step S103: Search according to the acquired video entity to obtain search results.

[0077] Step S104: Feedback the search results to the user.

[0078] In the present disclosure, a cloud server is preset with a cloud offline recognition database, which stores pre-selected videos and frame processing results corresponding to each video, and the frame processing results include: video entities contained in each frame of the popular video. According to the acquired video clips of interest to the user, the cloud offline recognition database is queried to quickly extract the video entities contained in the video clips of interest.

[0079] The method for stimulating search based on real-time video content provided by the embodiments of the present disclosure can, after obtaining the video clip of interest of the user, extract the video entities in the video clip of interest by combining cloud online query and local online recognition, so as to quickly extract the video entities of interest for the user, thereby improving the subsequent search speed and improving the user experience.

[0080] The present disclosure also provides another method for stimulating search based on real-time video content, such as Figure 2 As shown, the method includes:

[0081] Step S201: Determine in real time whether the user performs a predetermined operation.

[0082] Step S202: When it is determined that the user performs a predetermined operation, a video collection time period is determined to determine the video content watched by the user in the video collection time period as an interest video segment.

[0083] In step S201, the predetermined operation refers to an action instruction that can trigger the acquisition of the video clip of interest to the user. Optionally, the predetermined operation in the present disclosure is a "shake" action. By acquiring and judging in real time whether the action instruction issued by the user is the predetermined operation, it can be determined whether the user wants to obtain the video clip of interest. Moreover, by setting the predetermined operation to stimulate the search process, compared with the method of manual pause by the user in the prior art, the user's operation path can be greatly shortened, thereby making the search scenario stimulated by the video content easier to achieve, further improving the user experience.

[0084] In step S202, when it is determined that the user has performed a predetermined operation, it is determined that the user wants to obtain an interest video segment of interest, and then the video content watched by the user in the video collection time period is used as the interest video segment, wherein the video collection time period is determined as follows:

[0085] Determine the time t2 corresponding to when the user performs a predetermined operation, and use t2 as the end time of the video acquisition time period; determine the start time of the video acquisition time period as t1, and t1=t2-t0, wherein t0 is a preset duration value, and the selection of the duration value t0 can be based on factors such as user habits, average human brain reaction time, and device characteristics, and can be determined through offline or online fine-tuning and learning, so that the determined video clip of interest can more accurately include the video entity of interest to the user.

[0086] Step S203: Determine the video identifier of the complete video to which the video segment of interest belongs.

[0087] Step S204: query whether the cloud offline recognition database stores the video identifier of the complete video to which the video clip of interest belongs.

[0088] In the present disclosure, each video has a specific video identifier; in step S203, by extracting relevant information of the video being watched by the user, the video identifier corresponding to the video can be obtained, and the video identifier is the video identifier of the complete video to which the video clip of interest to the user belongs.

[0089] In the present disclosure, the cloud offline recognition database stores pre-selected videos and the frame processing results corresponding to each video, and the frame processing results include: the video entities contained in each frame of the popular video, and optionally, the pre-selected videos are popular videos with high views. In addition, the cloud offline recognition database also stores the video identifiers corresponding to the pre-selected videos. Therefore, in step S204, based on the obtained video identifier, a query is made from the cloud offline recognition database to determine whether the video clip of interest to the user is stored in the cloud offline recognition database.

[0090] Specifically, when the query result is yes, it is determined that the relevant information of the video clip of interest is stored in the cloud offline recognition database, and step S205 is executed accordingly to query the video entity contained in each frame of the video clip of interest from the cloud offline recognition database; when the query result is no, it is determined that the relevant information of the video clip of interest is not stored in the cloud offline recognition database, and step S205' is executed accordingly to obtain the video entity in the video clip of interest through local online recognition.

[0091] Step S205: Query the cloud offline recognition database for video entities included in the video clip of interest.

[0092] Step S205 ′: perform local online recognition on the video clip of interest using a preset entity extraction algorithm to extract video entities contained in the video clip of interest.

[0093] In the present disclosure, video entities include various types of entities such as visual entities, text entities, and sound entities. Typical visual entities include, for example, celebrity faces, small commodities, cars, logos, animals and plants, famous attractions / landmark buildings, etc. When identifying this type of entity, the objects in the picture can be identified based on computer vision algorithms; typical text entities include, for example, celebrity names, commodity names, place names, dish names, animal and plant species names, etc. When identifying this type of entity, they can be obtained based on OCR technology; typical sound entities include, for example, background music tracks, people's speech content, humming, bird calls, door closing sounds, car starting sounds, etc. When identifying this type of entity, the audio track data of the video can be processed using audio matching / speech recognition algorithms.

[0094] In the present disclosure, in order to further control the computational cost, when extracting video entities from video clips of interest based on a pre-set entity extraction algorithm, the extraction can be performed in combination with the user portrait. Specifically, the user portrait includes a preset vertical category recognition priority and a user preference, wherein the preset vertical category recognition priority can be obtained by statistically sorting the conversion rates of the user's historical vertical category search results, and the user preference can be obtained by estimating the user's preference degree and / or conversion rate for a specific vertical category based on statistics. By combining user portraits for local online recognition, one or more vertical categories can be identified preferentially, thereby further controlling the computational cost, improving the efficiency of entity extraction, and better meeting user needs.

[0095] Step S206: Determine whether the number of acquired video entities is greater than a predetermined threshold.

[0096] In the present disclosure, the video entities extracted from the video clip of interest often include multiple video entities, and the multiple video entities are not necessarily all video entities that the user is interested in. If all video entities are searched, the computational cost will inevitably be too high. In step S206, by setting a predetermined threshold for judgment, the number of video entities obtained can be controlled, thereby further reducing the computational cost.

[0097] Specifically, when step S206 determines that the number of acquired video entities is greater than a predetermined threshold, it means that the calculation cost is too high, and step S207 is executed; when step S206 determines that the number of acquired video entities is less than the predetermined threshold, it means that the calculation cost is within a controllable range, and step S207' is executed.

[0098] Step S207: based on the user portrait of the user acquired in advance, filter out video entities associated with the user portrait.

[0099] In the present disclosure, the user portrait includes a preset vertical category recognition priority and the user's preference for a specific vertical category, wherein the preset vertical category recognition priority can be obtained by statistically ranking the conversion rates of the user's historical vertical category search results, and the user preference can be estimated based on the statistics of the user's preference for a specific vertical category and / or conversion rate. That is, the user portrait identifies the entity category that the user is interested in, and by filtering the video entities in combination with the user portrait, the video entities associated with the user portrait can be further filtered out, thereby obtaining video entities that are more in line with the user's interests and hobbies, and can reduce computing costs, wherein the video entity is associated with the user portrait specifically means that the category to which the video entity belongs is the entity category contained in the user portrait.

[0100] Step S207': determine whether the number of acquired video entities is zero.

[0101] In the present disclosure, when the number of video entities is within a predetermined threshold range, it is further determined whether the number of video entities is zero. Specifically, when the determination result of step S207' is zero, it means that no entity has been extracted, and step S208' is executed at this time; when the determination result of step S207' is not zero, it means that relevant entities have been extracted, and step S208 is executed at this time.

[0102] Step S208: Search according to the acquired video entity to obtain search results.

[0103] Step S208': provide relevant copywriting tips to the user and end the search process.

[0104] In step S208, after the video entity is obtained, the video entity is sent to a search server for associated search, such as a Baidu search server, to obtain search results related to the video entity. The search results may include, for example, Baidu Encyclopedia page links of celebrities, e-commerce shopping web links of products, music track audition / purchase / download web links, and tourist attraction website homepage links.

[0105] In step S208', when no video entity is extracted, relevant text is fed back to the user to inform the user, and the entire search process ends.

[0106] Step S209: Feedback the search results to the user.

[0107] In the present disclosure, the way to feed back search results to users includes half-screen web page pop-up layer and semi-transparent floating layer, and the specific feedback method can be set by the user according to personal preferences and habits. The information is displayed without affecting the normal playback of the video, and the user is allowed to jump to the corresponding link when actively clicking on a search result.

[0108] The method for stimulating search based on real-time video content provided by the embodiment of the present disclosure can trigger the search process by determining in real time whether the user performs a predetermined operation, which can greatly shorten the user's operation path, thereby making the search scenario based on video content stimulation easier to implement and improving the user experience; and the preset cloud offline recognition database stores the video entities contained in the pre-selected popular videos with high views. When extracting video entities from the video clips of interest to the user, the cloud offline query and local online recognition are combined, so that when the user is watching a popular video with high views, the video entities of interest to the user can be quickly extracted through the cloud query, and when the user is watching an unpopular video, the video entities of interest to the user can also be extracted through the local online recognition method, thereby maximally satisfying the user's video entity extraction needs for popular and unpopular videos, and at the same time improving the extraction rate, and further improving the user experience when the search results are fed back to the user based on the extracted video entities in the subsequent period.

[0109] The present disclosure also provides a system for stimulating search based on real-time video content, which is used to execute the method for stimulating search based on real-time video content provided in the embodiments of the present disclosure.

[0110] like Figure 3 As shown, the system includes: a first acquisition module 11, a second acquisition module 12, a search module 13 and a feedback module 14.

[0111] Among them, the first acquisition module 11 is used to obtain the interest video clips that the user is interested in; the second acquisition module 12 is used to query the video entities contained in the interest video clips from the cloud offline recognition database or use the local online recognition method according to the interest video clips; the search module 13 is used to search according to the video entities extracted by the second acquisition module 12 to obtain search results; the feedback module 14 is used to feed back the search results to the user.

[0112] In the present disclosure, a cloud server is preset with a cloud offline recognition database, which stores pre-selected videos and frame processing results corresponding to each video, and the frame processing results include: video entities contained in each frame of the popular video. According to the acquired video clips of interest to the user, the cloud offline recognition database is queried to quickly extract the video entities contained in the video clips of interest.

[0113] The system based on real-time video content-stimulated search provided by the embodiments of the present disclosure can, after acquiring the video clips of interest to the user, extract the video entities in the video clips of interest by combining cloud online query and local online recognition, so as to quickly extract the video entities of interest for the user, thereby improving the subsequent search speed and improving the user experience.

[0114] Furthermore, if Figure 4 As shown, in the system for stimulating search based on real-time video content provided by the embodiment of the present disclosure, the second acquisition module 12 further includes: a first judgment unit 121, a query unit 122 and a calculation unit 123.

[0115] The first judgment unit 121 is used to judge whether the cloud offline recognition database stores relevant information of the video segment of interest; the query unit 122 is used to query the cloud offline recognition database for the video entity contained in each frame of the video segment of interest when the first judgment unit 121 judges that the cloud offline recognition database stores relevant information of the video segment of interest; the calculation unit 123 is used to use a preset entity extraction algorithm to perform local online recognition on the video segment of interest to extract the video entity contained in the video segment of interest when the first judgment unit 121 judges that the cloud offline recognition database does not store relevant information of the video segment of interest;

[0116] In the present disclosure, each video has a specific video identifier; the cloud offline recognition database stores pre-selected videos and the frame processing results corresponding to each video, and the frame processing results include: the video entities contained in each frame of the popular video, and optionally, the pre-selected videos are popular videos with high views. In addition, the cloud offline recognition database also stores the video identifiers corresponding to the pre-selected videos. The system also includes: an identifier determination module 15, which is used to determine the video identifier of the complete video to which the video clip of interest belongs after the first acquisition module 11 acquires the video clip of interest that the user is interested in.

[0117] Further, the first judgment unit 121 determines whether the video segment of interest that the user is interested in is stored in the cloud offline recognition database by querying whether the cloud offline recognition database stores the video identifier of the complete video to which the video segment of interest determined by the determination module belongs.

[0118] Specifically, when the video identifier of the complete video to which the video clip of interest belongs is queried in the cloud offline recognition database, it is determined that the relevant information of the video clip of interest is stored in the cloud offline recognition database, and the query unit 122 is instructed to perform a query to query the video entity contained in each frame of the video clip of interest from the cloud offline recognition database; if the video identifier of the complete video to which the video clip of interest belongs is not queried in the cloud offline recognition database, it is determined that the relevant information of the video clip of interest is not stored in the cloud offline recognition database, and the calculation unit 123 is instructed to perform a calculation to obtain the video entity in the video clip of interest through a local online recognition method.

[0119] In the disclosed embodiment, the first acquisition module 11 specifically includes: a monitoring unit 111 and a determination unit 112, wherein the monitoring unit 111 is used to determine in real time whether the user performs a predetermined operation; the determination unit 112 is used to determine the video collection time period when the monitoring unit determines that the user performs a predetermined operation, and determine the video content watched by the user in the video collection time period as the video segment of interest.

[0120] Specifically, the video collection time period is determined as follows: the end time of the video collection period is the time t2 corresponding to when the user performs the predetermined operation, and the start time t1 of the video collection time period is t1=t2-t0, where t0 is a preset time value.

[0121] In the disclosed embodiment, the second acquisition module 12 further includes: a second judgment unit 124 , a screening unit 125 and a third judgment unit 126 .

[0122] Among them, the second judgment unit 124 is used to determine whether the number of video entities obtained is greater than a predetermined threshold after the query unit 122 or the calculation unit 123 completes the extraction operation of the video entity; the screening unit 125 is used to, when the second judgment unit 124 determines that the number of video entities is greater than the predetermined threshold, screen out video entities associated with the user portrait based on the user portrait of the user obtained in advance; the third judgment unit 126 is used to, when the second judgment unit 124 determines that the number of video entities is less than or equal to the predetermined threshold, further determine whether the number of video entities obtained is zero, and when it is determined that the number of video entities is zero, instruct the feedback module 14 to feedback relevant text prompts to the user, and when it is determined that the number of video entities is not zero, instruct the search module 13 to perform an associated search based on the extracted video entities.

[0123] In the present disclosure, the user portrait includes: a preset vertical category recognition priority and the user's preference for a specific vertical category. The preset vertical category recognition priority can be obtained by statistically sorting the conversion rates of the user's historical vertical category search results, and the user preference can be estimated based on the statistical user's preference for a specific vertical category and / or conversion rate.

[0124] In the present disclosure, the feedback module 14 provides users with feedback on search results in the following ways: half-screen web page pop-up layer and semi-transparent floating layer. The information is displayed without affecting the normal playback of the video, and the user can jump to the corresponding link when actively clicking on a search result.

[0125] The present disclosure also provides a terminal device for stimulating search based on real-time video content, comprising: one or more processors and a storage device; wherein the storage device stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for stimulating search based on real-time video content provided by the present disclosure.

[0126] The present disclosure also provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for stimulating search based on real-time video content provided by the present disclosure is implemented.

[0127] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, which implements any one of the above-mentioned methods for stimulating search based on real-time video content when executed by a processor.

[0128] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0129] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A method for stimulating search based on real-time video content, comprising: Get interesting video clips; According to the video clip of interest, the video entity contained in the video clip of interest is queried from the cloud offline recognition database or extracted by local online recognition; wherein the cloud offline recognition database stores pre-selected videos and frame processing results corresponding to each video; the pre-selected video is a popular video whose browsing volume exceeds a preset browsing threshold; the frame processing result includes: the video entity contained in each frame of the pre-selected video; wherein the video entity refers to the things involved in the video content, including visual entities, text entities and sound entities, the visual entity is obtained by identifying objects in the picture based on a computer vision algorithm, the text entity is obtained based on OCR technology, and the sound entity is obtained by processing the audio track data of the video using an audio matching / speech recognition algorithm; Performing an associated search based on the acquired video entity to obtain search results; Feeding back the search results to the user; The step of obtaining the video clip of interest specifically includes: Determine in real time whether the user performs the scheduled operation; When it is determined that the user performs the predetermined operation, a video collection time period is determined to determine the video content watched by the user in the video collection time period as the video segment of interest; wherein the end time of the video collection time period is the time t2 corresponding to when the user performs the predetermined operation, and the start time of the video collection time period is t1, t1=t2-t0, t0 is a preset duration value; wherein the preset duration value t0 is determined based on at least one of the following characteristics: user habits, average human brain reaction time, and device characteristics; Wherein, after the step of acquiring the video entity and before the step of searching according to the acquired video entity, the method further includes: Determining whether the number of acquired video entities is greater than a predetermined threshold; When it is determined that the number of acquired video entities is greater than a predetermined threshold, video entities associated with the user portrait acquired in advance are screened out, and the subsequent step of performing an associated search based on the acquired video entities is continued; When it is determined that the number of acquired video entities is less than or equal to a predetermined threshold, it is further determined whether the number of acquired video entities is zero. When it is determined that the number of video entities is zero, relevant text prompts are fed back to the user and the search process ends.

2. The method according to claim 1, wherein: The step of querying from a cloud offline recognition database or extracting the video entity contained in the video segment of interest by using a local online recognition method according to the video segment of interest includes: Determine whether the cloud offline recognition database stores relevant information of the video clip of interest; When it is determined that the cloud offline recognition database stores the relevant information of the video segment of interest, querying the video entity included in the video segment of interest from the cloud offline recognition database; When it is determined that the cloud offline recognition database does not store relevant information of the video segment of interest, a preset entity extraction algorithm is used to perform local online recognition on the video segment of interest to extract the video entities contained in the video segment of interest.

3. The method according to claim 2, wherein: The cloud offline recognition database stores video entities contained in each of the pre-selected popular videos.

4. The method according to claim 3, wherein: The cloud offline recognition database stores video identifiers of the popular videos; After the step of obtaining the video segment of interest and before the step of querying the video entity contained in the video segment of interest from a cloud offline recognition database or extracting the video entity contained in the video segment of interest by using a local online recognition method according to the video segment of interest, the method further includes: Determine the video identifier of the complete video to which the video clip of interest belongs; The determining whether the cloud offline recognition database stores the relevant information of the video clip of interest specifically includes: Querying whether the cloud offline recognition database stores a video identifier of the complete video to which the video clip of interest belongs; If the video identifier of the complete video to which the video clip of interest belongs is queried in the cloud offline recognition database, it is determined that the cloud offline recognition database stores relevant information of the video clip of interest; If the video identifier of the complete video to which the video segment of interest belongs is not found in the cloud offline recognition database, it is determined that the relevant information of the video segment of interest is not stored in the cloud offline recognition database.

5. The method according to claim 1, wherein: The user portrait includes: a preset vertical category recognition priority and the user's preference for a specific vertical category.

6. The method according to any one of claims 1 to 5, wherein: The method of feeding back the search results to the user includes: a half-screen web page pop-up layer and a semi-transparent floating layer.

7. A system for stimulating search based on real-time video content, comprising: A first acquisition module is used to acquire a video clip of interest; The second acquisition module is used to query the video entity contained in the video segment of interest from the cloud offline recognition database or extract the video entity contained in the video segment of interest by using the local online recognition method according to the video segment of interest; wherein the cloud offline recognition database stores pre-selected videos and the frame processing results corresponding to each video; the pre-selected video is a popular video whose browsing volume exceeds a preset browsing threshold; the frame processing results include: the video entity contained in each frame of the pre-selected video; wherein the video entity refers to the things involved in the video content, including visual entities, text entities and sound entities, the visual entity is obtained by identifying objects in the picture based on a computer vision algorithm, the text entity is obtained based on OCR technology, and the sound entity is obtained by processing the audio track data of the video using an audio matching / speech recognition algorithm; A search module, configured to perform an associated search based on the video entity extracted by the second acquisition module to obtain search results; A feedback module, used for feeding back the search results to the user; Wherein, the first acquisition module includes: A monitoring unit, used to determine in real time whether the user performs a predetermined operation; A determination unit is used to determine a video collection time period when the monitoring unit determines that the user performs the predetermined operation, and determine the video content watched by the user in the video collection time period as the video segment of interest; wherein the end time of the video collection time period is the time t2 corresponding to when the user performs the predetermined operation, the start time of the video collection time period is t1, and t1=t2-t0, t0 is a preset duration value; wherein the preset duration value t0 is determined based on at least one of the following characteristics: user habits, average human brain reaction time, and device characteristics; Wherein, the second acquisition module further includes: A second judgment unit is used to judge whether the number of the obtained video entities is greater than a predetermined threshold after the video entities included in the video clip of interest are queried from the cloud offline recognition database or extracted by a local online recognition method; a screening module, configured to screen out video entities associated with a user portrait obtained in advance when the second judging unit judges that the number of video entities is greater than a predetermined threshold; The third judgment unit is used to further judge whether the number of acquired video entities is zero when the second judgment unit judges that the number of video entities is less than or equal to a predetermined threshold, and to instruct the feedback module to feed back relevant text prompts to the user when it is judged that the number of video entities is zero.

8. The system according to claim 7, wherein: The second acquisition module includes: A first determination unit is used to determine whether the cloud offline recognition database stores the relevant information of the video clip of interest; a query unit, configured to query the video entity included in the video segment of interest from the cloud offline recognition database when the first judgment unit determines that the cloud offline recognition database stores relevant information of the video segment of interest; The computing unit is used to perform local online recognition on the video segment of interest by using a preset entity extraction algorithm when the first judging unit determines that the cloud offline recognition database does not store relevant information of the video segment of interest, so as to extract the video entity contained in the video segment of interest.

9. The system according to claim 8, wherein: The cloud offline recognition database stores video entities contained in each of the pre-selected popular videos.

10. The system according to claim 9, wherein: The cloud offline recognition database also stores the video identification of each of the popular videos; The system further comprises: an identification determination module, configured to determine, after the second acquisition module acquires the video segment of interest, a video identification of the complete video to which the video segment of interest belongs; The first judgment unit is specifically used to query whether the cloud-based offline recognition database stores the video identifier of the complete video to which the video clip of interest belongs, as determined by the identifier determination module; and when the video identifier of the complete video to which the video clip of interest belongs is queried in the cloud-based offline recognition database, it is judged that relevant information of the video clip of interest is stored in the cloud-based offline recognition database; when the video identifier of the complete video to which the video clip of interest belongs is not queried in the cloud-based offline recognition database, it is judged that relevant information of the video clip of interest is not stored in the cloud-based offline recognition database.

11. The system according to claim 7, wherein: The user portrait includes: a preset vertical category priority and the user's preference for a specific vertical category.

12. The system according to any one of claims 7 to 11, wherein: The feedback module feeds back the search results to the user in the following ways: half-screen web page pop-up layer and semi-transparent floating layer.

13. A terminal device for stimulating search based on real-time video content, comprising: one or more processors; A storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

14. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images