Video resource searching method and related equipment

Abstract

The embodiment of the invention provides a video resource searching method and related equipment, which can support a user to accurately inquire a desired video resource through a spoken problem by utilizing abundant UGC (User Generated Content) on the Internet, high-quality PGC in a video resource library and semantic comprehension capability and content comprehension capability of a large text model. Moreover, after the video resource segment conforming to the intention of the user is determined by using the Internet search result and the PGC search result, a more fine-grained segment conforming to the intention of the user can be further positioned in the video resource segment by using the multi-modal large model, so that the requirement of the user for fragmented film watching can be met.

Description

[0001] This application claims priority to Chinese Patent Application No. 202411600681.5, filed on November 8, 2024, entitled “Resource Search Method and Related Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic technology, and in particular to video resource search methods and related equipment. Background Technology

[0003] With the increasing popularity of voice-interactive products such as in-vehicle systems, smart screens, and smart speakers, users are increasingly inclined to use voice commands to search for and control video resources. Moreover, due to the rise of short videos and the increasing pressure of modern life, more and more users are accustomed to watching videos during fragmented time, leading to a growing demand for precise searches of video clips.

[0004] Traditional video resource searches rely more on explicit tags such as movie titles, actor names, narrator names, and movie genres. These tags require users to input them to find relevant video resources. They do not support users using more colloquial natural language for searching and cannot accurately find the specific segment a user wants within the video resources. Summary of the Invention

[0005] This application provides a video resource search method and related equipment, which can support users to search for video resources, audio resources and other resources using any natural language, and support users to accurately search for specific resource segments, thus meeting users' needs for precise resource search.

[0006] In a first aspect, embodiments of this application provide a video resource search method. This method can be applied to a terminal device and specifically includes the following steps: The terminal device receives first dialogue content input by a first user, the first dialogue content expressing a resource search intent in natural language. Based on the first dialogue content, the terminal device replies with a first resource fragment that matches the resource search intent. The first resource fragment is a video resource fragment, which is one or more episodes of a film or television work, or one or more film or television clips from an episode of a film or television work. Replying with the first resource fragment that matches the resource search intent includes displaying a first reply text and / or displaying a first service card. The first reply text describes the first resource fragment, and the first service card is used by the user to view the first resource fragment.

[0007] In the first aspect, the user can input the first dialogue content through a smart voice assistant, such as by speaking the first dialogue content aloud. However, the user can also input the first dialogue content via text input; this application embodiment does not limit the input method for the first dialogue content.

[0008] Implementing the method provided in the first aspect allows users to search for video resources using any natural language and to precisely search for specific resource segments, thus meeting users' needs for accurate resource searches.

[0009] In conjunction with the first aspect, in some embodiments, the rendered content on the first service card can also serve to describe the first resource fragment, and its rendered content may include one or more of the following: the name, type, release time, creator, cover image, etc. of the first resource fragment or its associated video resource.

[0010] In conjunction with the first aspect, in some embodiments, when the terminal device detects a user operation applied to the first service card, the terminal device can jump to the playback interface of the first resource segment and play the first resource segment. Besides clicking the first service card as user input, the terminal device can also receive user input to trigger the playback of the first resource segment through a continuation dialogue, and then jump to the playback interface of the first resource segment to play it. The continuation dialogue may include: first asking whether to play the first resource segment; then receiving the user's response to the inquiry. This continuation dialogue can be a voice dialogue or a text dialogue; this application embodiment does not limit the dialogue format. The implementation details of triggering the playback of the first resource segment can be found in the relevant content of the foregoing embodiments, and will not be repeated here.

[0011] In conjunction with the first aspect, in some embodiments, the terminal device may also send first dialogue content to the server and receive indication information of a first resource segment from the server. The indication information of the first resource segment includes the identifier information of the resource to which the first resource segment belongs, and the location information of the first resource segment within its resource. The location information includes the start time of the first resource segment. The identifier information of the resource may be, for example, the resource name, and the location information of the first resource segment within its resource may be the resource segment number, time range, etc. For example, the indication information of the first resource segment may consist of the resource name "TV Series 1" and the resource segment number "Episode 63". Another example is that the indication information of the first resource segment may consist of the resource name "Movie 2" and the time range "30:00-40:00". When using a time range as the location information of the first resource segment within its resource, the start time of the first resource segment can be used as a key parameter indicating the time range, and the end time of the first resource segment, if not explicitly indicated, can be assumed to be the end time of the resource to which the first resource segment belongs.

[0012] In conjunction with the first aspect, in some embodiments, the terminal device may also receive a first reply text from the server.

[0013] In conjunction with the first aspect, in some embodiments, the terminal device may also receive a first image from the server, the first image being generated based on a first resource fragment. Specifically, the first image may originate from the first resource fragment, such as the first frame image of the first resource fragment. In this way, the terminal device can utilize the first image as the cover image of the first service card when rendering and generating the first service card, thereby enhancing the user appeal of the first service card.

[0014] In conjunction with the first aspect, in some embodiments, the terminal device may also receive first dialogue content input by a second user, and based on the first dialogue content input by the second user, the terminal device may reply to the second user with a second resource fragment that matches the resource search intent. The second resource fragment may be one or more episodes of a film or television work, or one or more film / television clips from one episode of a film or television work. The second user is different from the first user, and the second resource fragment is different from the first resource fragment. The second resource fragment may also be one or more episodes of a film or television work, or one or more film / television clips from one episode of a film or television work. Replying to the second user with a second resource fragment that matches the resource search intent may include one or more of the following methods: displaying a second reply text, displaying a second service card. The second reply text may be used to describe the second resource fragment, and may include reasons for recommending the second resource fragment, a plot summary, etc. The second service card may be used by the user to view the second resource fragment.

[0015] In conjunction with the first aspect, in some embodiments, the memory information or personal profiles of the first user and the second user on the terminal device may differ. Memory information may include usage traces that indicate the user's video browsing history, video collection history, etc., which can represent the user's preferences for film and television resources. The personal profile may be a user profile determined by the cloud side based on user data uploaded by the terminal (such as gender, age, occupation, etc.) and / or the aforementioned memory information, used to indicate the user's film and television preferences. User film and television preferences can be broad, including user preferences for film and television content, user preferences for film and television creators (such as actors), user preferences for film and television sources, user preferences for playback resolution, etc.

[0016] In conjunction with the first aspect, in some embodiments, the second resource segment may differ from the first resource segment in, but is not limited to, the following ways: the second resource segment has different video content from the first resource segment, or the second resource segment has the same content as the first resource segment but belongs to different resources, or the second resource segment has the same content as the first resource segment but has different clarity.

[0017] In conjunction with the first aspect, in some embodiments, there may be multiple first resource fragments that match the resource search intent. In this case, after replying to the first user with the first resource fragments that match the resource search intent, the terminal device may further receive second dialogue content input by the first user. The second dialogue content expresses, in natural language, the resource selection intent to select a third resource fragment from the multiple first resource fragments. Then, the terminal device can select the third resource fragment from the multiple first resource fragments according to the second dialogue content and jump to the playback interface of the third resource fragment to play the third resource fragment.

[0018] In conjunction with the first aspect, in some embodiments, the first resource fragment conforming to the resource search intent can be determined by the terminal device based on the content of the first dialogue, especially when the terminal device has strong computing power. Alternatively, the first resource fragment conforming to the resource search intent can also be determined by the cloud-side server based on the content of the first dialogue and fed back to the terminal device, which then replies with the first resource fragment to the user.

[0019] Secondly, embodiments of this application provide a video resource search method, which can be applied to a server and specifically includes the following steps: the server receives first dialogue content from a terminal device, the first dialogue content expressing a resource search intent in natural language; the server determines a first resource segment that matches the resource search intent based on the first dialogue content and returns indication information of the first resource segment to the terminal device. The first resource segment is a video resource segment, which is one or more episodes of a film or television work, or one or more film / television segments from one episode of a film or television work; the indication information of the first resource segment includes the identification information of the resource to which the first resource segment belongs, and the location information of the first resource segment within its resource, wherein the location information includes the start time of the first resource segment.

[0020] In the second aspect, the indication information of the first resource segment includes the identification information of the resource to which the first resource segment belongs, and the location information of the first resource segment within its respective resource. The location information includes the start time of the first resource segment. The resource identification information can be, for example, the resource name, and the location information of the first resource segment within its respective resource can be the resource segment number, time range, etc. For example, the indication information of the first resource segment might consist of the resource name "TV Series 1" and the resource segment number "Episode 63". Another example is that the indication information of the first resource segment might consist of the resource name "Movie 2" and the time range "30:00-40:00". When using a time range as the location information of the first resource segment within its respective resource, the start time of the first resource segment can serve as a key parameter indicating that time range, and the end time of the first resource segment, if not explicitly indicated, can be assumed to be the end time of the resource to which the first resource segment belongs.

[0021] By implementing the method provided in the second aspect, the inference operations based on the large text model and the multimodal large model can be executed by the cloud-side server with stronger computing power, which can reduce the computing power requirements of the terminal device, reduce the power consumption of the terminal device, and reduce the waiting time for users.

[0022] In conjunction with the second aspect, in some embodiments, the server may also return a first response text to the terminal device, the first response text being used to describe the first resource fragment.

[0023] In conjunction with the second aspect, in some embodiments, the server may also return a first image to the terminal device. This first image may be generated based on a first resource fragment. Specifically, the first image may originate from the first resource fragment, such as the first frame of the first resource fragment. In this way, the terminal device can use the first image as the cover image of the first service card when rendering and generating it, thereby increasing the user appeal of the first service card.

[0024] In conjunction with the second aspect, in some embodiments, the server determines a first resource fragment that matches the resource search intent based on the first dialogue content. Specifically, this may include: the server using PGC search results and Internet search results associated with the first dialogue content to determine a first resource fragment that matches the resource search intent based on the first dialogue content.

[0025] Specifically, the internet search results associated with the first dialogue content are obtained by searching the internet based on the first dialogue content. The professionally produced content (PGC) search results associated with the first dialogue content are obtained by searching the PGC resource library based on the first resource identifier. The first resource identifier is extracted from the internet search results, and the PGC resource library records the resource fragments included in different resources. In the specific implementation, the server can use the first customized prompt guiding text model to extract resource identifiers such as video resource names from the internet search results.

[0026] In conjunction with the second aspect, in some embodiments, compared to the resource fragments recorded in the PGC resource library, the first resource fragment can be a shorter resource fragment, that is, a more granular resource fragment. To this end, the server can first determine a third resource fragment that matches the resource search intent based on internet search results and PGC search results. The duration of the third resource fragment can be consistent with the duration of the resource fragments recorded in the PGC resource library. Then, the server can further process the third resource fragment, internet search results, and PGC search results using a multimodal large model to further locate the first resource fragment from the third resource fragment. In this way, the first resource fragment can meet the user's fragmented viewing needs.

[0027] In conjunction with the second aspect, in some embodiments, the server determines a first resource fragment that matches the resource search intent based on the first dialogue content. Specifically, this may include: the server analyzing internet search results and PGC search results based on the first user's user profile to determine the first resource fragment. In this way, different users can receive different recommended resource fragments, even if different users input the same dialogue content.

[0028] In conjunction with the second aspect, in some embodiments, before the server determines a first resource fragment that matches the resource search intent based on the first dialogue content, the server also uses PGC search results to filter out inaccurate content in Internet search results to improve the accuracy of resource recommendations and responses.

[0029] In conjunction with the second aspect, in some embodiments, there may be multiple first resource fragments that match the resource search intent. After the server returns indication information for the first resource fragment to the terminal device, the server may also receive second dialogue content from the terminal device, which expresses the resource selection intent in natural language, and determine a third resource fragment that matches the resource selection intent from the multiple first resource fragments based on the second dialogue content, and then return indication information for the third resource fragment to the terminal device. The indication information for the third resource fragment is used to indicate the third resource fragment among the multiple first resource fragments.

[0030] Thirdly, this application provides a terminal device, which includes one or more processors and one or more memories; wherein the memories are coupled to the processors, and the one or more memories are used to store computer programs. When the processor executes the computer programs, it can implement the steps performed by the terminal device in the video resource search method described in the first aspect or any possible implementation of the first aspect.

[0031] Fourthly, this application provides a server comprising one or more processors and one or more memories; wherein the memories are coupled to the processors, and the one or more memories are used to store a computer program that, when executed by the processor, can implement the steps performed by the server in the video resource search method described in the second aspect or any possible implementation of the second aspect.

[0032] Fifthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to implement the steps performed by the terminal device in the video resource search method described in the first aspect or any possible implementation of the first aspect.

[0033] In a sixth aspect, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to implement the steps performed by the server in the video resource search method described in the second aspect or any possible implementation of the second aspect.

[0034] In a seventh aspect, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the steps performed by the terminal device in the video resource search method described in the first aspect or any possible implementation of the first aspect.

[0035] Eighthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the steps performed by the server in the video resource search method described in the second aspect or any possible implementation of the second aspect.

[0036] Ninthly, this application provides a computer program product containing instructions, which, when executed by a processor, can implement the steps performed by the terminal device in the video resource search method described in the first aspect or any possible implementation of the first aspect.

[0037] In a tenth aspect, this application provides a computer program product containing instructions that, when executed by a processor, can implement the steps performed by the server in the video resource search method described in the second aspect or any possible implementation of the second aspect.

[0038] In the eleventh aspect, this application provides an end-to-cloud collaborative system, which may include the terminal device described in the third aspect and the server described in the fourth aspect. Attached Figure Description

[0039] Figures 1A-1B The present application of the embodiments of this application in a voice dialogue scenario is illustrated by example;

[0040] Figure 2 This application illustrates an intelligent system 20 for resource searching, selection, and online playback based on a large model, as provided in an embodiment of this application. Figure 3 The overall flow of the resource search method provided in Embodiment 1 is shown; Figure 4 An example of the resource search method provided in Embodiment 1 is shown; Figure 5 This shows the first custom prompt used to guide the large text model; Figure 6The overall flow of the resource search method provided in Embodiment 2 is shown; Figure 7 An example of the resource search method provided in Embodiment 2 is shown; Figure 8 This demonstrates a second custom prompt used to guide the large text model; Figure 9 The overall flow of the resource search method provided in Embodiment 3 is shown; Figure 10 An example of the resource search method provided in Embodiment 3 is shown; Figure 11 A third custom prompt is shown for guiding the large text model; Figure 12 The overall flow of the resource search method provided in the embodiments of this application is illustrated; Figure 13 The flowchart illustrates the process of implementing the resource search method provided in the embodiments of this application through edge-cloud collaboration; Figure 14 The terminal device provided in the embodiments of this application is shown; Figure 15 The server provided in an embodiment of this application is shown. Detailed Implementation

[0041] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be a limitation of this application.

[0042] Supporting users to search for video resources using natural language can improve the user experience.

[0043] Currently, a technology for searching video resources via voice interaction uses a large model to identify user intent in resource queries, generate movie / TV show titles as search terms, and return these search terms to a video resource database via a JSON structure. The final search results are then displayed and presented via voice. However, this technology relies entirely on a static knowledge base provided by the large model to retrieve video resource names as search results. Updates to this knowledge base are delayed due to training limitations, affecting the timeliness of search results and the accuracy of finding emerging and popular videos. Furthermore, this technology may lead to the large model misinterpreting certain plot details due to the lack of such details in the knowledge base, potentially resulting in fabricated video resources or storylines.

[0044] Another technique based on natural language video segment search uses a knowledge graph and a BERT model to obtain natural language vector representations of the query, and encodes specific videos online into vector representations of video segments. It then calculates the Gumbel-Softmax score of these two vectors to obtain the corresponding video segments and presents them to the user. However, this technique implicitly calculates the most similar video segment by encoding the question and the specific video into vector representations of the same dimension. This makes it difficult to intuitively provide reasons for recommending video segments or explicit query logic, which can easily mislead users. Furthermore, since multiple vector calculations inevitably have a maximum value, and the maximum value does not necessarily represent similar segments, it may still provide a "most relevant" segment even when the user queries irrelevant information (i.e., no results), significantly impacting accuracy. Moreover, this technique relies on an offline knowledge graph database built on Wikipedia, and is affected by Wikipedia's timeliness and coverage, failing to meet users' broad resource query needs and achieving the same effect as internet resource searches. Additionally, this technique requires online frame-by-frame analysis, filtering, and selection of video resources, resulting in slow response times and high latency.

[0045] This application provides a resource search method that allows users to search for resources such as video and audio resources using any natural language, and also allows users to search for specific resource fragments, thus meeting users' needs for precise resource search.

[0046] Figures 1A-1B This example illustrates a typical application of the embodiments of this application in a voice dialogue scenario. In this voice dialogue scenario, users can search for video resources they want to learn about or watch by conversing with a voice assistant (such as "Xiaoyi"), and continue the conversation to select and control the playback of video resources. Wherein: (1) Figure 1A Example: The first step is for the user to open the voice assistant and speak their question. This question can be used to search for a single video resource and can be any natural language voice input, such as "Which episode features the blood test?" The question can be displayed in the voice assistant's dialogue interface 11. In response to the question, the voice assistant can display a reply text 15 in its dialogue interface 11, which describes which episode the user wants to search for, such as "TV series 1 is a TV series, not a movie, and the blood test scene occurs in episode 63..." The voice assistant can also read the reply text 15 aloud. This voice reading can be triggered automatically, such as displaying and reading the reply text 15 line by line; or it can be triggered by the user, such as by clicking the voice reading control in the dialogue interface 11.

[0047] Furthermore, in response to this issue, the voice assistant can display a service card 16 in the dialogue interface 11. Service card 16 is the service card for the retrieved target video resource. The service card is an interface display format that places important application information or operations on a card to achieve direct service access and reduce user experience layers. Clicking service card 16 will trigger playback of the target video resource. The target video resource is the video resource that the user intends to learn about and / or view based on the identified user intent, such as episode 63 of TV series 1.

[0048] The second step is for users to continue their conversation with the voice assistant to control the terminal device to play the target video resource.

[0049] For example, the voice assistant can continue to ask the user whether to play the target video resource. This inquiry can be made via voice, and / or by displaying a question text after the response text 15, such as "Found a relevant video, would you like to play it?". To this inquiry, the user can give an affirmative reply, such as "Yes, please play," which will redirect to the playback interface 17 of the target video resource, triggering its automatic playback. The playback interface 17 can be the video playback interface of a video application that provides the target video resource.

[0050] (2) Figure 1B Example: The first step is for the user to open the voice assistant and speak their question. This question can be used for fuzzy searching or recommendations of resources, and can be any natural language voice input, such as "Recommend three movies suitable for family reunions." This question can be displayed in the voice assistant's dialogue interface 11. In response to the question, the voice assistant can display a reply text 18 in the dialogue interface 11, describing several video resources that match the user's intent, including three movies suitable for family reunions, such as "Movie A: This is an animated film produced by director X, through... Movie B: This is an inspirational film starring actor Y, telling the story of... Movie C: This is a French film, telling the story of a naughty boy and..." The voice assistant can also read the reply text 18 aloud.

[0051] In addition, in response to this question, the voice assistant can display multiple service cards 19 in the dialogue interface 11. These service cards 19 represent multiple target video resources that have been retrieved. These target video resources are multiple video resources that the user intends to learn about and / or play based on the question, such as movie A, movie B, and movie C. The user can choose to click on a specific service card 19 to trigger the playback of the selected video resource.

[0052] The second step is for users to continue their conversation with the voice assistant to select and control video resources.

[0053] For example, the voice assistant can continue to ask the user which video resource they want to play. This question can be asked verbally, and / or by appending the question text after the response text 15, such as "Find these videos, just tell me which one you want to watch." The user can respond verbally to this question, such as saying "Play the one starring actor Y," to complete the video resource selection and be redirected to the playback interface 21 of that video resource, triggering its automatic playback.

[0054] from Figure 1A , Figure 1B As can be seen from the examples, the embodiments of this application support users to conduct multi-turn dialogues with the voice intelligent assistant based on any natural language, so as to search for the video resources or video resource segments they want, further select the video resources they want to watch, and control the playback of the video resources they want to watch.

[0055] Moreover, this application embodiment combines the intent recognition and semantic understanding capabilities of the text big model with the high-quality PGC from the rich resource library of user-generated content (UGC) and professionally-generated content (PGC) on the Internet. While expanding the query scope and improving the timeliness of content through UGC, it ensures the professionalism and accuracy of query results by combining PGC.

[0056] Figure 2 This application illustrates an intelligent system 20 for resource searching, selection, and online playback based on a large model, as provided in an embodiment of this application. Figure 2 The resources mentioned above take video resources as an example.

[0057] like Figure 2 As shown, the intelligent system 20 mainly includes three modules: an intent recognition module, a resource search module, and a resource selection module. Among them: 1. Functions of the intent recognition module: 1.1 Obtain user profiles from the user profiling platform based on user identifier (ID) and application scenario identifier (ID).

[0058] A user profiling platform can be a cloud server that stores and manages user profiles of many users across different application scenarios. A user profile in a specific application scenario reflects that user's preferences and behaviors within that scenario. User profiles help intelligent systems understand user intent and predict user needs.

[0059] 1.2 Obtain the context of the multi-turn dialogue between the user and the voice assistant, that is, obtain the content of the multi-turn dialogue.

[0060] 1.3 The intent recognition capability based on the large text model can identify the user intent expressed by the user's statements in the dialogue scenario. The user intent may include: resource search intent and resource selection intent.

[0061] If the user's intent is resource search, then the resource search module will perform the resource search; if the user's intent is resource selection, then the resource selection module will perform the resource selection. Resource selection generally follows the resource search, and the resource search provides the user with multiple resources that match the user's search intent. The user then needs to further select the resource to play from these multiple resources. In multi-turn dialogue scenarios, the resource search dialogue is usually the preceding dialogue to the resource selection dialogue.

[0062] 2. Functions of the resource search module: 2.1 Based on the question, request the search engine to obtain the Internet search results returned by the search engine, such as "search result 1", "search result 2", ..., "search result n".

[0063] Search engines can search the internet for content to obtain search results related to a question. The internet contains a lot of user-generated content (UGC) and some professionally generated content (PGC). The former has broad coverage and good timeliness; the latter provides more professional and high-quality content.

[0064] 2.2 Combining search results and customized prompts from the Internet, the content understanding capabilities of the text big data model are used to extract the names of video resources related to the problem from the search results, such as "movie name 1", "movie name 2", ..., "movie name n".

[0065] A custom prompt is used to guide the large text model to understand search results from the internet and output video resource names. This custom prompt will be described in detail in subsequent examples; it will not be elaborated upon here.

[0066] 2.3 Using the video resource names obtained in 2.2, query the PGC resource library to obtain the PGC search results for the video resource entities represented by the video resource names (such as "Video Resource Entity 1", "Video Resource Entity 2", ..., "Video Resource Entity n") in the PGC resource library. The PGC search results for a video resource entity may include some descriptive information (or feature information) of the video resource entity, such as film and television work introduction, actors, film and television genre, episode plot, time-sharing plot, etc.

[0067] Figure 2The example illustrates the content of a video resource entity in a PGC resource library, such as "Film Introduction: This film tells the story of...", "Actor Information: Lead Actor Y", "Film Type: Inspirational / Drama", "Episode Synopsis: Episode 1 synopsis is... Episode 2 synopsis is...", "Time-Based Synopsis: Episode 1 01:00-10:00 tells the story of... Episode 2 10:00-20:00 tells the story of...". Time-based synopsis is a more granular segment of the story than episode synopsis, and can be a segment of video resources lasting several minutes or even several seconds within a single episode video resource.

[0068] 2.4 Based on a text-based large-scale model, the system first analyzes internet search results and PGC search results to filter out inaccurate and low-quality content. Then, it identifies video resources that match the user's intent based on user dialogue content. The filtered search results are then used to summarize the text, resulting in film and television introductions and recommendation reasons, as shown in the aforementioned reply texts 15 and 18. Simultaneously, based on a multimodal large-scale model, inference operations are performed on the filtered search results and video resources that match the user's intent to locate video resource segments that match the intent. When locating video resource segments that match the user's intent, the multimodal large-scale model can also consider user profiles, ensuring that the located video resource segments are also closely aligned with user preferences.

[0069] After identifying video resources and video resource segments that match the user's intent, a service card for the video resource can be rendered on the client side. When the user clicks on the service card, they can be redirected to the video resource playback interface to watch the automatically playing video resource segment.

[0070] 3. Function of the resource selection module: Combining the aforementioned search results, the context of the previous dialogue, and the position of the video resource's service card in the previous dialogue interface, the module uses the semantic understanding capability of the text big data model to understand which video resource the user has selected, and finally jumps to the playback interface of that video resource to automatically play the video resource.

[0071] In the intelligent system 20, the intent recognition module, resource search module, and resource selection module can be deployed on a cloud-side server. The powerful computing capabilities of the cloud-side server can better ensure their smooth operation and performance, and also shorten inference computation time, saving users waiting time. Of course, the intent recognition module, resource search module, and resource selection module can also be deployed on edge devices such as mobile phones, tablets, in-vehicle devices, and smart screens, especially when these edge devices have strong computing capabilities.

[0072] Based on the foregoing, the resource search method provided by this application is described in detail below through several embodiments. The first embodiment focuses on a solution for accurately supplying video resource segments in response to natural language questions in a dialogue scenario; the second embodiment focuses on a solution for targeted supply of video resource segments by combining user profiles; and the third embodiment focuses on a solution for supporting users to query resources, select resources, and control resource playback in multi-turn dialogue scenarios.

[0073] In this application's embodiments, the term "question" or "user question" refers to a query in the field of natural language processing. Its meaning is not limited to a question; broadly speaking, it can be a task description. In the following text, "question" or "user question" can be replaced with user dialogue or user dialogue content.

[0074] All steps in the resource search method described in the following embodiments can be executed by terminal devices such as mobile phones, in-vehicle devices, and smart screens, especially when these terminal devices have strong computing capabilities. This resource search method can also be executed through edge-cloud collaboration. For example, human-computer interaction steps such as receiving queries through a voice assistant, outputting reply text, displaying resource service cards, and playback navigation can be executed by terminal devices such as mobile phones, in-vehicle devices, and smart screens. Meanwhile, the inference operations based on text-based large models and multimodal large models can be executed by cloud-based servers with stronger computing power. This reduces the computing power requirements of the terminal devices, lowers power consumption, and reduces user waiting time. In the edge-cloud collaboration approach, communication is required between the terminal devices and the cloud servers. The terminal needs to transmit the data that the text-based large model and multimodal large model need to process to the cloud, and the cloud needs to return the inference operation results to the terminal, such as which video resources match the user's intent, which video resource the user intends to play, etc. The following embodiments illustrate the resource search method using an edge-cloud collaboration approach.

[0075] Example 1 Figure 3 The overall flow of the resource search method provided in Embodiment 1 is illustrated. Further details are provided below.

[0076] S11. The terminal device can receive a question uttered by the user and transmit the question to the server.

[0077] This question is expressed in natural language and is used to query resources. The question may include a spoken description of the resource to be queried, which may describe one or more of the following information: the creator of the resource (including actors, directors, singers, composers, authors, etc.), creation time, release time, resource type, and resource content, etc.

[0078] Resource types may include: video resources, audio resources, text resources, image resources, etc. Video resources may include, for example, movies, TV series, variety shows, and other film and television works; audio resources may include, for example, audiobooks, operas, radio broadcasts, etc.; text resources may include, for example, e-novels, e-journals, etc.; image resources may include, for example, e-magazines, etc. These examples are merely for explaining embodiments of this application and should not be construed as limiting the scope of the application.

[0079] In this embodiment, resource content can refer to the data characteristics of a resource, such as voice features, image features, text features, etc., or it can refer to the meaning expressed by the resource, such as plot, paragraph summary, lyrics, etc. For video resources, resource content can refer to the plot of the video resource; for audio resources, resource content can refer to the audio features (such as song melody), lyrics, lines, etc.; for text resources, resource content refers to the text content; for image resources, resource content can refer to the image content, that is, the meaning expressed by the image. The resource content of other resources can be defined similarly, and will not be listed here.

[0080] After receiving the user's input question, the terminal device can perform text preprocessing, query segmentation, query error correction, query alignment, query expansion, and other processing on the question to obtain the processed question, and then transmit the processed question to the server for processing.

[0081] S12. Based on the question, the server identifies the user's intent as a resource search intent, and searches the Internet based on the question, obtaining multiple Internet search results.

[0082] Specifically, you can request internet search engines to obtain these multiple internet search results. Internet search results can be in webpage format and may include text results, image results, etc.

[0083] The internet is characterized by the timeliness and breadth of its content, and as a constantly evolving and open system, it is not static but continuously expanding in scale. This allows users to search for newer and more content. However, much of the content on the internet is user-generated content (UGC), the professionalism and accuracy of which are not guaranteed. To address this, this application's embodiments will incorporate a professionally generated content (PGC) resource library to ensure the accuracy of resource searches, as will be explained later.

[0084] S13. The server can use a large text model to extract one or more resource entities from multiple Internet search results.

[0085] In this embodiment of the application, the essence of the resource entity is information, which mainly refers to resource identifiers such as resource name, and may also include auxiliary information such as publication time and creator.

[0086] For video resources, the resource name can refer to the title of a film or television work (such as a movie or TV series), the release date can refer to the release date of the film or television work, and the creator can include the director, actors, etc. For audio resources, the resource name can refer to the title of an audio work (such as an audiobook), the release date can refer to the release date of the audio work, and the creator can include narrators, voice actors, etc. For image resources, the resource name can refer to the title of an image work (such as the title of a painting or pictorial), the release date can refer to the publication date of the image work, and the creator can include painters, etc. For text resources, the resource name can refer to the title of a text work (such as a novel), the release date can refer to the publication date of the text work, and the creator can include authors, etc.

[0087] In its implementation, the terminal device can execute S13 using a first customized prompt and a large text model. The first customized prompt can be used to guide the large text model to extract resource entities that match the user's intent from the aforementioned multiple internet search results. The implementation of the first customized prompt will be explained in detail below, and will not be elaborated on here.

[0088] S14. The server can use the resource entity obtained in S13 to search in the PGC resource library and obtain the PGC search results for that resource entity.

[0089] A PGC search result for a resource entity primarily includes: descriptions of each resource segment within that entity, such as episode summaries or time-based synopses for a film or television work, or chapter content for an audiobook. A PGC search result for a resource entity may also include: an introduction to the resource entity, the creator, the resource type, etc.

[0090] The PGC resource library provides more professional and high-quality content. Compared to UGC on the internet, PGC is more accurate and professional. Furthermore, the PGC resource library in this embodiment records descriptions of resource fragments (such as time-sharing plots of film and television works) for each different resource entity, supporting the text-based large model in locating resource fragments that match the user's intent. This avoids the large model experiencing "illusion" due to its inability to identify resource fragments. Here, "illusion" refers to the large model fabricating non-existent resource fragments, such as fabricating plot segments.

[0091] The PGC resource library can be a PGC resource library on a cloud-based PGC server. When a user queries resources, the resource search module can retrieve PGCs from the PGC resource library online, such as resource descriptions, descriptions of resource fragments, resource types, publication times, creators, etc.

[0092] S15. The server can utilize the content understanding capabilities of the large text model to filter the aforementioned internet search results and PGC search results, removing low-quality and inaccurate content, and determine the resource fragment A that matches the user's intent based on the filtered internet search results and PGC search results.

[0093] Correspondingly, the server can also transmit indication information for resource segment A to the terminal device, such as the name of the film / TV work, the number of episodes, and the time range of the video segment (e.g., "30:00-40:00"). In this way, the terminal device can render a resource service card based on this indication information to display the resource service card in the dialogue interface of the voice assistant. Furthermore, when the user triggers playback of resource segment A, the terminal device uses the indication information of resource segment A to send a request to the video application's server to request the acquisition of resource segment A and play resource segment A on the terminal side.

[0094] Furthermore, the server can also transmit an introductory image for resource fragment A to the terminal device. This allows the terminal device to use the introductory image as the cover image for the corresponding resource service card, thereby enhancing the card's appeal to users. The introductory image for resource fragment A can be extracted by the server from resource fragment A. For example, the introductory image for resource fragment A could be the first frame of resource fragment A, such as the first frame of the 30:00-40:00 segment in episode 9 of TV series 2.

[0095] In S15, internet search results can help cover more and newer content, while PGC search results can help ensure content accuracy.

[0096] S16. The server can utilize the semantic and content understanding capabilities of the large text model to comprehensively analyze the aforementioned internet search results and PGC search results, generating a response text tailored to the question. The server can then transmit this response text to the terminal device, allowing the terminal device to display the response text to the user within the voice assistant's dialogue interface. The terminal device can also read the response text aloud.

[0097] Here, the internet search results used to generate the response text can be either the internet search results obtained in S12 directly, or internet search results filtered by PGC search results. This filtering can remove inaccurate content from the internet search results, thereby improving the accuracy of the response.

[0098] This response text can be used to describe resource A or resource fragment A. Specifically, the response text may include a brief introduction to resource A or resource fragment A, reasons for recommendation, etc., so that users can better understand resource A, resource fragment A, and the reasons for the response.

[0099] For example, in response to the question "Which episode features the blood test?", the reply could be "TV series 1 is a TV series, not a movie. The blood test scene takes place in episode 63. In this episode, the protagonist..."

[0100] When generating recommendation reasons and summary introductions, internet search results help to broaden the scope of content, while PGC search results help to improve the accuracy of content.

[0101] S17. Based on the instruction information of resource fragment A, the terminal device can also display a resource service card for resource fragment A in the dialogue interface of the voice assistant. The rendered content of the resource service card can be generated based on the instruction information of resource fragment A. The user can click on the resource service card to view the automatic playback of resource fragment A.

[0102] Furthermore, by combining the filtered internet search results, PGC search results, and resource fragment A (i.e., continuous image frames), the server can leverage the multimodal large model's ability to process multimodal information to parse resource fragment A. This allows for the further localization of resource fragments within fragment A that better match the user's intent, particularly identifying the start time of the resource fragment. Compared to the resource fragments recorded in the PGC resource library, this further localized fragment is a finer-grained fragment identified by the multimodal large model, containing little or no redundant fragments related to other irrelevant issues. This enables users to more precisely query smaller resource fragments, meeting their needs for fragmented resource acquisition.

[0103] Accordingly, the server can transmit instruction information for resource fragment B to the terminal device, enabling the terminal device to render and generate a resource service card for resource fragment B. Users can then click on this resource service card to view the autoplay of resource fragment B. Furthermore, the server can transmit the first frame image of resource fragment B to the terminal device, allowing the terminal device to use this first frame image as the cover image for the resource service card, thereby increasing the card's appeal to users.

[0104] Resource service cards for resource fragments A and B can also be displayed in other locations, such as the message center, the negative one screen, the homepage of certain applications, the desktop, etc., and the resource service cards can even be transformed into other user interface forms. This application embodiment does not limit this.

[0105] More refined resource clip editing can also occur offline to reduce user waiting time online. First, large models can be used to identify video resources, generating finer-grained (minute-level, even second-level) resource clips along with descriptions. This is a time-consuming process; completing it offline reduces resource retrieval latency. Second, during offline resource clip editing, user history questions and user profiles can be combined to generate resource clips that users might be interested in. These clips can then be pushed to users in the message center, the negative one screen, and the application homepage, increasing user engagement and improving user experience.

[0106] The following will use video resources as an example to illustrate... Figure 4 The example further illustrates the resource search method provided in Implementation Example 1.

[0107] Figure 4 In the example, the question is "In which episode does Character 1 buy a watermelon?" A solution for providing precise video clips to address this question could include the following steps: Steps 1 to 4 primarily introduce the process of resource searching based on the intent recognition and content understanding capabilities of internet UGC and PGC resource libraries, as well as the large text model. This will be elaborated below: Step 1. For the question "In which episode does character 1 buy a watermelon?", based on the intent recognition capability of the text big data model, the user intent is identified as a resource search intent. Then, based on the question, the search engine interface is called to request the search engine to search for relevant search results on the Internet.

[0108] Internet search results are presented in web page format and may include text results and / or image results.

[0109] Figure 4 The example shows the following internet search results (text results): Search result 1: "[Source: Application 1] Character 1, male, is a gang leader in the 2003 Chinese police and gangster television series..." Search result 2. "[Source: Application 2] In which episode does character 1 buy watermelon? In the TV series 2, character 1 buys watermelon in episode 8. Actor S plays a gangster boss. Character 1 goes to a stall to buy watermelon..." Search result 3. "[Source: Application 3 2024-04-19 13:49:15] In TV series 2, the scene where character 1 buys a watermelon appears in episode 9. Character 1 discovers something wrong with the scale while buying the watermelon..." from Figure 4As the examples of search results illustrate, while the internet can cover a wider and more up-to-date range of content, the content on the internet is not necessarily accurate or professional. For instance, search result 2 indicates that the scene of "Character 1 buying a watermelon" is in episode 8, while search result 3 indicates that the same scene is in episode 9. Therefore, this application embodiment will combine professional, high-quality PGC (Professionally Generated Content) from the PGC resource library to perform resource searches, providing users with accurate resource search results.

[0110] Step 2. Combine the question and internet search results into a first customized prompt. Use the first customized prompt to guide the text model to extract one or more video resource entities from the internet search results.

[0111] The essence of a video resource entity is information, which mainly refers to resource identification information such as the video resource name, and may also include other information such as the resource creator, resource type, and resource release time.

[0112] Figure 4 In the example, the video resource entities extracted from internet search results include "episode 8 of TV series 2" and "episode 9 of TV series 2", among others.

[0113] Step 2 utilizes the text-based large-scale model's ability to understand the intent and content of internet search results. The first customized prompt guides the text-based large-scale model to extract resource entities from internet search results.

[0114] The first customized prompt can Figure 5 As shown, it may include, but is not limited to, the following parts: task description, entity notes, entity examples, search engine results, and questions. Among these, the task description guides the text model on what task to perform. Figure 4 The example specifically guides the large text model to output video resource entities according to entity considerations and entity examples; entity considerations constrain the format in which the large text model outputs video resource entities; entity examples provide examples to achieve better guidance; search engine results need to be concatenated with internet search results; and the question needs to be concatenated with the question "In which episode does character 1 buy a watermelon?".

[0115] Figure 5 This is just one example; the first custom prompt can include different or more bootstrap items, and there are no restrictions on their format.

[0116] Step 3. Based on the video resource entity obtained in Step 2, search in the PGC resource library to obtain the PGC search results for the video resource entity.

[0117] The PGC search results for this video resource entity mainly include episode summaries, time-based summaries, and other film and television clips, and may also include film and television introductions, cast information, resource type, etc.

[0118] Figure 4 In the example, the search results for the video entities "Episode 8 of TV Series 2" and "Episode 9 of TV Series 2" in the PGC resource library include: the time-sharing plot of Episode 8 of TV Series 2 and the time-sharing plot of Episode 9 of TV Series 2. Among them: The time-sharing plot of episode 8 of TV series 2 can be summarized as follows: "01:00-10:00 After learning that the police have preliminarily identified Character 1 as a key suspect in the '4.28' case, Character 2 immediately calls Character 1 and tells him to leave immediately..." ... "40:00-45:00 The cunning Character 1 has already realized the danger approaching..." The time-sharing plot of episode 9 of TV series 2 is as follows: "01:00-10:00 The police discovered the residence of Character 1 and others after investigation, but the cunning Character 1 had already slipped away before the police found them...", "30:00-40:00 Character 1 is buying watermelons...", "40:00-46:37 The police launched a large-scale manhunt for Character 1's subordinates, and a group of former members of Character 1's gang were successively brought to justice."

[0119] Steps 4 to 7 mainly introduce the semantic understanding capabilities of text-based large-scale models and the image understanding capabilities of multimodal large-scale models, detailing the processes of filtering, summarizing, and providing resources for the search results obtained in steps 1 and 3. This will be elaborated below: Step 4. Utilize the content understanding capabilities of the large text model to filter the search results (UGC) from the Internet search results (UGC) and the search results (PGC) in the PGC resource library, filtering out low-quality and inaccurate content, and determining the first video resource and the first video resource segment that match the user's intent.

[0120] For example, Figure 4 In the example, the internet search results for "episode 8 of TV series 2" indicated that the scene of "character 1 buying watermelon" in episode 8 was inaccurate and was filtered out. The first video resource is episode 9 of TV series 2, and the first video resource clip is the 30:00-40:00 segment in episode 9 of TV series 2.

[0121] After identifying the first video resource, the terminal device can reply to the user with the video resource in the dialogue interface of the voice assistant, such as "In TV series 2, the classic scene of character 1 buying melons appears in episode 9". Furthermore, in step 4, based on the content of the filtered internet search results and PGC search results, the dialogue capabilities of the text-based large model can be used to generate response text tailored to the question, so as to reply to the user with the first video resource and the first video resource clip that match the user's intent. This response text may specifically include a clip introduction and reasons for recommendation, such as... Figure 4 The example reply text reads: "In TV series 2, the classic scene of character 1 buying a watermelon appears in episode nine. This episode is a major turning point in the series, where character 1 (played by actor S) clashes with the watermelon vendor after discovering a magnet under his scale, and ultimately... This scene is memorable for its tense and exciting dramatic conflict and actor S's superb performance." Step 5. Utilize the multimodal information processing capabilities of the multimodal big data model to analyze the filtered search results and the first video resource segment, and further locate the second video resource segment that better matches the user's intent from the first video resource segment, so as to obtain a more granular video segment that matches the user's intent.

[0122] exist Figure 4 In the example, the 30:00-40:00 segment of episode 9 of TV series 2, along with images related to episode 9 of TV series 2 from internet search results, can be input into a multimodal large model. After inference and calculation, the video resource segment that matches the user's intent is located around 32:51, so subsequent playback can start from 32:51. The 32:51-40:00 segment of episode 9 of TV series 2 is a more granular second video resource segment.

[0123] In the specific execution of step 5, to improve the accuracy of locating the second video resource segment, image results can be extracted from the filtered internet search results. These image results, along with the first video resource segment, are then input into the multimodal large model to obtain the second video resource segment. This is because text has a certain degree of ambiguity, while images are more explicit and specific. Images can help the multimodal large model more accurately locate the second video resource segment from the first video resource segment, such as using the position of the image within the first video resource segment as the starting frame position of the second video resource segment.

[0124] After the second video resource segment is determined, the server can transmit the indication information and the first frame image of the second video resource segment to the terminal device. In this way, the terminal device can use the indication information and the first frame image to render a resource service card, wherein the first frame image is used as the cover image of the resource service card.

[0125] Step 5 is a further improvement of the embodiments of this application. Its purpose is to more accurately locate video resource segments that better match the user's intentions and meet the user's fragmented viewing needs.

[0126] Step 6. Receive the response text from the server regarding the question. The terminal device can display the response text in the dialogue interface of the voice assistant.

[0127] Step 7. Receive the instruction information and first frame image of the second video resource segment transmitted by the server. The terminal device can display the video resource service card in the dialogue interface of the voice smart assistant, such as the service card for the segment from 32:51 to 40:00 in episode 9 of TV series 2. The first frame image can be used as the cover image of the resource service card.

[0128] Upon detecting that a user has clicked on the resource service card, the terminal device can open the video playback interface and automatically play the second video resource clip. Furthermore, the user can also trigger the playback of the second video resource clip by continuing the conversation (e.g., saying "Thank you, please play").

[0129] The technical effects of Example 1 include at least the following: by leveraging the abundant UGC and high-quality PGC in the video resource library on the internet, as well as the semantic and content understanding capabilities of the text big data model, users can accurately search for the video resources they want through conversational questions. Furthermore, after identifying video resource segments that match the user's intent using internet search results and PGC search results, the multimodal big data model can further locate more granular segments within those video resource segments that match the user's intent, thus meeting the user's fragmented viewing needs.

[0130] Example 2 Example 2 is mainly used to handle users' fuzzy search or recommendation problems. It combines user profiles, Internet search results and PGC search results to automatically mine resources that match user intent using a large text model.

[0131] Figure 6 The overall flow of the resource search method provided in Embodiment 2 is shown. The details are as follows.

[0132] S21. The terminal device can receive a question uttered by the user and transmit the question to the server.

[0133] S22. Based on the question, the server identifies the user's intent as a resource search intent, and searches the Internet based on the question, obtaining multiple Internet search results.

[0134] S23. The server can extract one or more resource entities from multiple Internet search results.

[0135] S24. The server can use the resource entity obtained in S23 to search in the PGC resource library and obtain the PGC search results for that resource entity.

[0136] For specific explanations of S21-S24, please refer to S11-S14 in Implementation 1, which will not be repeated here.

[0137] After receiving the question, the terminal device can also send a request to the user profiling platform to obtain the user profile and transmit the user profile to the server.

[0138] S25. The server can analyze internet search results and PGC search results based on user profiles and large text models to obtain the optimal resource features that match the user profile.

[0139] The optimal resource feature describes which resources match the user's resource preferences. It can include, but is not limited to, one or more of the following parameters: optimal resource entity, optimal fragment feature, optimal person feature, etc. In this paper, "optimal" refers to the result output by the large text model, not the absolute optimal one. There may be resource features that match the user's resource preferences better than the optimal one, and the degree of optimization depends on the capabilities of the large model.

[0140] In the specific implementation, the server can execute S25 using a second customized prompt and a large text model. The second customized prompt can be used to guide the large text model to analyze the optimal resource features based on the question, user profile, the aforementioned internet search results, and the aforementioned PGC search results. The implementation of the second customized prompt will be explained in detail below, and will not be elaborated on here.

[0141] S26. Based on the optimal resource characteristics obtained in S25 and the aforementioned Internet search results and PGC search results, the server can further locate recommended resource fragments from the optimal resources.

[0142] The optimal resource refers to the resource represented by the optimal resource entity.

[0143] In specific implementation, the multimodal large model can be used to execute S26: input the aforementioned Internet search results and the aforementioned PGC search results, as well as the optimal resource features, into the multimodal large model, and obtain the resource fragments that match the optimal resource features, i.e., the recommended resource fragments, through the inference operation of the multimodal large model.

[0144] Thus, different users often receive different recommended resource snippets, even if they ask the same question. Assuming the first user and the second user are different, even if they both ask the voice assistant the same question, such as "What are some good variety shows lately?", the recommended resource snippets output for the first user and the second user will be different, and the reply text and resource service cards they receive in the dialogue interface will also be different. User identity can be determined based on the user's logged-in account, or it can be determined based on other user characteristic data such as the user's voice characteristics (the voice characteristics collected when the question is spoken).

[0145] S27. The server can utilize the semantic and content understanding capabilities of the large text model to comprehensively analyze the aforementioned internet search results, PGC search results, and optimal resource features to generate a response text addressing the question. The server can then transmit this response text to the terminal device, allowing the terminal device to output the response text to the user within the voice assistant's dialogue interface. The terminal device can also read the response text aloud.

[0146] The response text can be used to introduce the recommended resource snippets, which may include a brief introduction, reasons for recommendation, etc., so that users can better understand the best resources, recommended resource snippets, and reasons for recommendation.

[0147] For example, in response to the question "Which episode features the blood test?", the reply could be: "TV series 1 is a TV series, not a movie. The blood test scene takes place in episode 63. In this episode, character 1 is accused by character 2 of having an improper relationship with character 3, which leads to..."

[0148] When generating recommendation reasons and summary introductions, internet search results help to broaden the scope of content, while PGC search results help to improve the accuracy of content.

[0149] Furthermore, the server can also transmit indication information for recommended resource clips to the terminal device, such as the title of the film or television work, the number of episodes, and the time range of the video clip. The terminal device can then render a resource service card based on this indication information and display it in the dialogue interface of the voice assistant. Moreover, when a user triggers playback of a recommended resource clip, the terminal device can use the indication information to send a request to the video application's server to retrieve the recommended resource clip and play it on the device.

[0150] In addition, the server can transmit introductory images of recommended resource snippets to the terminal device. The terminal device can then use these images as the cover image for the resource service card of the recommended resource snippet, thereby increasing the attractiveness of the resource service card to the user. Recommended resource snippets can be extracted by the server from the best resources.

[0151] The following will use video resources as an example to illustrate... Figure 7 The example further illustrates the resource search method provided in Embodiment 2.

[0152] Figure 7 In the example, the question is "What are some good variety shows lately?" A personalized video clip supply solution for this question could include the following steps: Steps 1 to 3 primarily introduce the process of resource searching based on the intent recognition and content understanding capabilities of internet UGC and PGC resource libraries, as well as the large text model. For details on these steps, please refer to [link to relevant documentation]. Figure 4 The specific explanations of steps 1 to 3 in the example will not be repeated here.

[0153] Compare Figure 4 The example adds a step where, when identifying user intent based on questions, user profiles can also be obtained from the user profiling platform. Figure 7 The example illustrates user profiles based on factors such as gender, age, occupation, movie viewing preferences, and actor preferences.

[0154] Figure 7 In the example, the video resource entities extracted from internet search results include "Variety 1", "Variety 2", and "Variety 3". The PGC search results for each of these video resource entities from the PGC resource library can include: episode summaries, time-sharing segments, and other film and television clips, as well as: variety show type, guests, video platform, and other content.

[0155] Step 4. Concatenate the question, user profile, the aforementioned internet search results, and the aforementioned PGC search results into the second customized prompt. Use the second customized prompt to guide the text big model to output the optimal resource features based on the user profile, the aforementioned internet search results, and the aforementioned PGC search results.

[0156] Optimal resource features enable subsequent multimodal large models to locate video resource segments from optimal video resources that also match user preferences.

[0157] Figure 7 In the example, the optimal resource features include: optimal video resource, optimal segment feature, and optimal person feature. Among them: The optimal video resource is "Variety Show 3" because: the guest in this video resource is artist A, who matches the user's preferred actors "artist A, artist B..."; and the variety show type of this video resource is "educational reality puzzle interactive variety show", which also matches the user's viewing preferences "exciting, quirky, puzzle...".

[0158] The optimal segment characteristics are "exciting scenes, adventure sequences, puzzle-solving scenarios..." because these match the user's viewing preferences.

[0159] The optimal character traits are "Celebrity A, Singer, Comedian" because this matches the user's preference for actors.

[0160] The second customized prompt can be like... Figure 8 As shown, it may include, but is not limited to, the following parts: task description, answering rules, knowledge base, and questions. Among them, the task description guides the text model on what task to perform. Figure 7 In the example, the text-based large-scale model is specifically used to guide the comprehensive knowledge base and personalized user profile to output video resource entities and optimal video resource features that match the user's intent. The answer rules guide the text-based large-scale model on how to recommend video resources based on the knowledge base and can also guide it to avoid fabrication, thus preventing the "large-scale model illusion" problem. The knowledge base here needs to be concatenated with the aforementioned internet search results and PGC search results; the "search engine results are as follows" position can be used to concatenate internet search results, and the "video resource library" position can be used to concatenate PGC search results. The question here needs to be concatenated with the question "What are some good variety shows recently?". In addition, the user profile can also be concatenated into the second customized prompt.

[0161] Figure 8 This is just one example; a second custom prompt can include different or more guiding items, and its format is not limited.

[0162] Step 5. Input the optimal resource features obtained in Step 4, the aforementioned Internet search results, and the aforementioned PGC search results into the multimodal large model to locate the recommended film and television segments in the optimal video resources.

[0163] Recommended movie clips are more granular video resources that are matched with user profiles, satisfying users' fragmented viewing needs while also meeting their personalized viewing preferences.

[0164] Figure 7 In the example, recommended film clips include: "Variety 3 Episode 1 03:45-10:00", "Variety 3 Episode 2 25:12-31:28", and "Variety 3 Episode 3 42:36-49:01".

[0165] After locating the recommended movie clip from the optimal video resources, the server can transmit the instruction information and the first frame image of the recommended movie clip to the terminal device. In this way, the terminal device can use the instruction information and the first frame image to render a resource service card, where the first frame image is used as the cover image of the resource service card.

[0166] Step 6. Receive the response text for the question transmitted by the server. The terminal device can display the response text for the question in the dialogue interface of the voice smart assistant.

[0167] The response text can be generated based on the optimal resource features obtained in step 4 and the recommended video clips obtained in step 5, in order to recommend video resource clips that match the user's intent and preferences to the user. Specifically, the response text may include introductions to each recommended resource clip, reasons for recommendation, etc., such as: "In the first episode of 'Variety 3' from 03:45-10:00, Celebrity A's analysis of the tram puzzle is brilliant and insightful, earning praise from the other members..."; "In the second episode of 'Variety 3' from 25:12-31:28, Celebrity A faces his fear and challenges himself with a personal task; will he be able to pass the test..."; "In the third episode of 'Variety 3' from 42:36-49:01, facing the pursuit of zombies, Celebrity A bravely confronts them, protecting the team, and the team spirit erupts...".

[0168] In addition, upon receiving the instruction information and first frame image of recommended movie clips transmitted from the server, the terminal device can also display video resource service cards in the dialogue interface of the voice assistant. The first frame image can be used as the cover image of the resource service card. When the user clicks on the resource service card, the terminal device can open the movie playback interface and automatically play the recommended movie clip. If there are multiple recommended movie clips, the terminal device can display the service cards for each clip in the dialogue interface. The user can select and click on a specific resource service card to jump to the playback interface of the corresponding recommended movie clip, where that clip will play automatically.

[0169] The technical effects of Example 2 include at least the following: based on user profiles, large text models, and abundant internet content and high-quality PGC content, it supports users in finding optimal resources that match their intentions and preferences through fuzzy search or recommendation questions. Furthermore, based on the multimodal large model's ability to understand various types of content such as text, images, and videos, it can accurately locate the recommended resource segments that users are most interested in from the optimal resources, meeting users' needs for fragmented viewing and achieving the goal of accurate resource recommendation.

[0170] In Example 2, the user profile further includes real-time user characteristics, such as the user's current environment, surrounding people, specific holidays, and weather. This allows the large text model to further combine these real-time user characteristics to recommend resources or resource snippets, resulting in a more immersive and user-centric resource search experience. These real-time user characteristics can be obtained through the terminal device's perception of the physical world.

[0171] Example 3 Example 3 focuses on how to support a user in selecting resources by continuing the conversation, provided that the user retrieved multiple resources in the previous round of dialogue. Example 3's solution occurs in a multi-turn dialogue scenario. The user's fuzzy search question or recommendation question can trigger this multi-turn dialogue.

[0172] Figure 9 The overall flow of the resource search method provided in Embodiment 3 is illustrated. Further details are provided below.

[0173] First round of dialogue (S31-S36) S31. In the first round of dialogue, the terminal device can receive a question, such as "Recommend three movies suitable for family gatherings", and transmit the question to the server.

[0174] S32. Based on this question, the server identifies the user's intent as a resource search intent, and searches the Internet based on this question, obtaining multiple Internet search results.

[0175] S33. The server can extract resource entities from multiple internet search results.

[0176] S34. The server can use the resource entity obtained in S33 to search in the PGC resource library and obtain the PGC search results for that resource entity.

[0177] For a detailed explanation of S31-S34, please refer to S11-S14 in Implementation 1, which will not be repeated here.

[0178] S35. The server can utilize the content understanding capabilities of the large text model to first filter the aforementioned internet search results and the aforementioned PGC search results, filtering out low-quality and inaccurate content. Then, based on the question, the filtered internet search results, and the PGC search results, it can determine multiple resources that match the user's intent and / or multiple resource fragments from these multiple resources, and generate a response text.

[0179] Correspondingly, the server can also transmit indication information of these resources and / or resource fragments to the terminal device, such as the name of the film or television work, the number of episodes, and the time range of the video clip. In this way, the terminal device can render the corresponding resource service cards based on the indication information of these resources and / or resource fragments, and display these resource service cards in the dialogue interface of the voice intelligent assistant.

[0180] Furthermore, the server can also transmit the reply text to the terminal device, so that the terminal device can display the reply text in the dialogue interface of the voice assistant.

[0181] In addition, the server can also transmit introductory images of these resources and / or resource fragments to the terminal device. The terminal device can then use these introductory images as cover images for the corresponding resource service cards, thereby enhancing the appeal of the resource service cards to users. Specifically, the introductory image for a resource can come from the resource itself, such as the first frame image; the introductory image for a resource fragment can come from the resource fragment, such as the first frame image.

[0182] S36. Based on the indication information of the multiple resources and / or resource segments determined in S35, the terminal device can also display resource service cards for each of these multiple resources and / or resource segments that match the user's intent in the dialogue interface of the voice intelligent assistant. The user can click on a resource service card to view the automatic playback of the corresponding resource or resource segment.

[0183] In Example 3, the problem is a fuzzy search problem or a recommendation problem. Therefore, the large text model often identifies multiple resources that match the user's intent. Thus, the response text can include an introduction and reasons for recommendation for each resource, facilitating user reference for subsequent resource selection. There are also multiple resource service cards; users can click on a specific card to jump to the playback interface of the corresponding resource or resource segment and watch its automatic playback.

[0184] In addition to manually selecting resource service cards, users can also select resources by continuing the conversation. The technical implementation will be described in detail below.

[0185] Second round of dialogue (S37-S40) S37. After updating the dialogue interface via S35 and S36, the terminal device can ask the user which resource to select through the voice assistant.

[0186] For example, a question text, such as "Find these videos, which one do you want to watch?", can be appended to the end of the aforementioned reply text. The voice assistant can also read this question text aloud.

[0187] S38. In the second round of dialogue, the terminal device may receive a question, such as "Play the first one", and transmit the question to the server.

[0188] S39. The server can identify the user's intent as a resource selection intent based on this question, and use a large text model to analyze information such as the aforementioned internet search results, the aforementioned PGC search results, the previous round of dialogue text, the rendered content of the resource service cards, and the position of the resource service cards, to select the target resource and / or target resource fragment that matches the user's intent from multiple resources and / or resource fragments. Here, the position of the resource service cards refers to their arrangement order.

[0189] Correspondingly, the server can also transmit indication information of the target resource and / or target resource segment to the terminal device, such as the name of the film or television work, the number of episodes, and the time range of the video segment. In this way, the terminal device can request the target resource and / or target resource segment from the server of the video application based on the indication information of the target resource and / or target resource segment, and play the target resource and / or target resource segment on the terminal side.

[0190] The previous round of dialogue text may include the aforementioned response text, which describes the introduction and reasons for recommendation of each resource and / or each resource fragment. Users may use certain descriptions that appeared in the previous round of dialogue text to make resource selections. Users may also use other content such as plot, actors, and creation time to describe resource selections. Users may also utilize the location of resource service cards and rendered content in the dialogue interface to describe resource selections. This application embodiment fully utilizes the intent recognition and semantic understanding capabilities of the large text model to support users in describing resource selections in these ways, enabling users to express their resource selections more freely.

[0191] In practice, the terminal device can execute S40 using a third-party customized prompt and a large text model. The third-party customized prompt can be used to guide the large text model to select a target resource or target resource fragment that matches the user's intent from multiple resources and / or multiple resource fragments pushed to the user in the previous round of dialogue. The implementation of the third-party customized prompt will be explained in detail below, and will not be elaborated on here.

[0192] S40. Based on the indication information of the target resource and / or target resource segment transmitted by the server, the terminal device may request the server of the video application to obtain the target resource and / or target resource segment, and play the target resource and / or target resource segment on the terminal side.

[0193] The following will use video resources as an example to illustrate... Figure 10 The example further illustrates the resource search method provided in Example 3.

[0194] Figure 10 In the example, the question in the first round of dialogue is "Play the movie where actor Z plays a policeman," and the question in the second round is "Play the one with actor W." Solutions for supporting user selection of video resources in multi-turn dialogue scenarios may include the following steps: Steps 1 to 3 primarily introduce the process of resource searching based on the intent recognition and content understanding capabilities of internet UGC and PGC resource libraries, as well as the large text model. For details on these steps, please refer to [link to relevant documentation]. Figure 4 The specific explanations of steps 1 to 3 in the example will not be repeated here.

[0195] Step 4. The server can utilize the content understanding capabilities of the large text model to filter internet search results (UGC) and search results (PGC) in the PGC resource library for the question "Play the movie where actor Z plays a policeman". Low-quality and inaccurate content will be filtered out. Based on the question, the filtered internet search results and PGC search results, multiple video resources that match the user's intent and / or multiple video resource segments from these multiple video resources will be determined, and a response text will be generated.

[0196] Correspondingly, the server can also transmit indication information of these video resources and / or video resource segments to the terminal device, such as the name of the film or television work, the number of episodes, and the time range of the video segment. In this way, the terminal device can render the corresponding resource service cards based on the indication information of these video resources and / or video resource segments, and display these resource service cards in the dialogue interface of the voice intelligent assistant.

[0197] Furthermore, the server can also transmit the reply text to the terminal device, so that the terminal device can display the reply text in the dialogue interface of the voice assistant.

[0198] Figure 10 In the example, the multiple video resources that match the user's intent are: "Movie 1", "Movie 2", "Movie 3", "Movie 4", "Movie 5, Movie 6, Movie 7".

[0199] Figure 10 In the example, the reply text includes introductions and reasons for recommending each of these video resources, such as: "1. Movie 1: Actor Z plays a bomb disposal expert in this film...; 2. Movie 2: In this film, Actor Z plays an undercover police officer...; 3. Movie 3: In this film, Actor Z plays a detective...; 4. Movie 4: Actor Z plays the lead role in this film...; 5. Movie 5: Although Actor Z is not a main character in this film, but... In addition, Actor Z has also played police roles in other films, such as a senior inspector in movie 6, and a police officer from the future in movie 7,..." Step 5. Based on the indication information of the multiple video resources and / or video resource segments determined in Step 4, the terminal device can also display the resource service cards of the multiple video resources and / or video resource segments in the dialogue interface of the voice smart assistant.

[0200] After obtaining the multiple video resources and / or video resource clips, the user can select resources by continuing the conversation, such as saying "Play the one with actor W".

[0201] Step 6. The server can concatenate the question "Play the one with actor W" from the current dialogue, the text from the previous dialogue, and the rendered content of the resource service card into a third-party custom prompt. Using this third-party custom prompt, the server guides the text model to select the target video resource or video resource segment that matches the user's intent from multiple video resources and / or video resource segments pushed to the user in the previous dialogue. In this way, the user can use the content presented in the dialogue interface to describe the video resource or video resource segment they intend to select.

[0202] In addition, the terminal device can also splice the aforementioned Internet search results, PGC search results, and the location of resource service cards into a third-party customized prompt, so as to support users to describe the resources they intend to select by using content outside the dialog interface (such as a movie plot) and the location of video resources in the screen, providing more freedom of expression.

[0203] Step 7. After selecting the target video resource or target video resource segment, the server can also return indication information of the target video resource or target video resource segment to the terminal device to trigger the terminal device to play the target video resource or target video resource segment. Accordingly, after receiving the indication information, the terminal device can use the indication information to send a request to the server of the video application to request the acquisition of the target video resource or target video resource segment, and play the target video resource or target video resource segment on the terminal side.

[0204] Figure 10 In the example, the selected target video resource is "Movie 2". The text model comprehensively analyzes the response text of the first round of dialogue, "Movie 2: In this film, actor Z plays an undercover police officer who infiltrates a gang and swaps identities with a police academy student played by actor W, unfolding a dual narrative", the search results in the PGC resource library, "Movie 2 is a movie released in 2002, produced by film company H, with actors Z and W...", and the internet search results, "Movie 2 is a movie released in 2003, produced by film company H, with actors Z and W...", and combines this with the question "Play the one with actor W", to determine that the video resource the user intends to select is "Movie 2".

[0205] The third customized prompt can be like... Figure 11 As shown, it may include, but is not limited to, the following parts: task description, answering rules, constraints, knowledge base, and questions. Among these, the task description can guide the text model on what task to perform. Figure 11In this example, the text model is specifically used to guide the text-based learning platform to decide whether to play a video or which video resource to play based on the previous dialogue and the video resources pushed to the user in the previous round, according to the new question. Answer rules can be used to guide the text model on how to respond to different user intentions. Constraints can be used to guide the text model to correct repeated user expressions, such as repeated sequence numbers. The knowledge base needs to be concatenated with the previous dialogue text and card rendering content. Specifically, the "text reply content is as follows" position can be used to concatenate the previous dialogue text (including introductions and recommendation reasons for each video resource), and the "card rendering content is as follows" position can be used to concatenate the card rendering content for each video resource. The question position is used to concatenate the question from the current dialogue (e.g., "Play the one with actor W").

[0206] Figure 11 This is just one example; a third custom prompt can include different or more guiding items, and its format is not limited.

[0207] Figure 10 The example does not involve the supply of resource segments because the user's intent expressed in the multi-turn dialogue is to select a movie, not a segment from a movie. In practical applications, the user's intent expressed in multi-turn dialogue can also be the intent to select a video resource segment. For example, after a user says "Play the movie where actor Z plays a policeman" in the first round and receives several related movies, the user says "Play the rooftop showdown in movie 2" in the second round to express the intent to select a resource segment. As another example, after a user says "Recommend several videos of police catching thieves" in the first round and receives several related recommended videos, the user says "Play the first one" in the second round to express the intent to select a resource segment.

[0208] exist Figure 10 In the example, even if the user does not explicitly express the intention to select a video resource segment to play, this application embodiment can also locate a video resource segment that matches the user's preferences from the target video based on the user profile, and push the instruction information of the video resource segment to the terminal device. In this way, the terminal device can push the video resource segment to the user, such as rendering it as a video resource card and pushing it to the customer.

[0209] The technical effects of Embodiment 3 include at least the following: in scenarios involving fuzzy search or resource recommendation, multiple resources can be provided to the user, along with resource descriptions and reasons for recommendation, making it easy for the user to quickly understand each resource and its reasons for recommendation; moreover, users can select the resource they want to view from multiple resources using any natural language, without the need for manual selection; from resource query and selection to resource playback control, the entire process can be completed through spoken dialogue, which is simple and efficient.

[0210] Based on the above embodiments, the resource search method provided in this application is summarized below, and its overall process is as follows: Figure 12 As shown, it includes: S51. The terminal device receives the first dialogue content input by the first user, and the first dialogue content expresses the intention to search for resources in natural language.

[0211] Users can input the initial dialogue content through a smart voice assistant, such as by speaking the initial dialogue content aloud. However, users can also input the initial dialogue content via text input; this application embodiment does not limit the input method for the initial dialogue content.

[0212] S52. Based on the content of the first dialogue, the terminal device may reply to the first user with a first resource fragment that matches the resource search intent, wherein the first resource fragment is a video resource fragment.

[0213] The first resource segment can be one or more episodes of a film or television work, or one or more segments from one episode of a film or television work. The former could be, for example, episode 63 of TV series 1, and the latter could be, for example, the "blood test" segment in episode 63 of TV series 1.

[0214] The process of replying to the first user with a first resource fragment that matches their search intent may include one or more of the following methods: displaying the first reply text, displaying the first service card. The first reply text can describe the first resource fragment and may include reasons for recommending the fragment, a plot summary, etc. The first reply text may be, for example,... Figure 1A Reply text 15 Figure 1B The reply text in 18. The first service card can be used by the user to watch the first resource segment. The first service card can be, for example, Figure 1A Service card 16 Figure 1B Service card 19.

[0215] The rendered content on the first service card can also serve to describe the first resource fragment. Its rendered content may include one or more of the following: the name, type, release time, creator, cover image, etc. of the first resource fragment or its associated video resource.

[0216] As shown in S53, when the terminal device detects a user operation applied to the first service card, the terminal device can jump to the playback interface of the first resource segment and play the first resource segment. Besides clicking the first service card as user input, the terminal device can also receive user input to trigger the playback of the first resource segment through a continuation dialogue, and then jump to the playback interface of the first resource segment to play it. The continuation dialogue may include: first asking whether to play the first resource segment; then receiving the user's response to the inquiry. This continuation dialogue can be a voice dialogue or a text dialogue; this embodiment does not limit the dialogue format. The implementation details of triggering the playback of the first resource segment can be found in the relevant content of the foregoing embodiments, and will not be repeated here.

[0217] Furthermore, different users can receive different recommended resource fragments, even if different users input the same dialogue content, which may be a fuzzy search question or a recommendation question. Specifically, the terminal device can receive the first dialogue content input by the second user; based on the first dialogue content input by the second user, the terminal device can reply to the second user with a second resource fragment that matches the resource search intent. The second user is different from the first user, and the second resource fragment is different from the first resource fragment. The second resource fragment can be one or more episodes of a film or television work, or one or more clips from an episode of a film or television work. Replying to the second user with a second resource fragment that matches the resource search intent can include one or more of the following methods: displaying a second reply text, displaying a second service card. The second reply text can be used to describe the second resource fragment, including reasons for recommending the second resource fragment, plot summaries, etc. The second service card can be used by the user to view the second resource fragment.

[0218] The memory information or personal profiles of the first user and the second user on the terminal device may differ. Memory information may include the user's video browsing history, video collection history, and other usage traces that can indicate the user's preferences for film and television resources. The personal profile may be a user profile determined by the cloud side based on user data uploaded by the terminal (such as gender, age, occupation, etc.) and / or the aforementioned memory information, used to indicate the user's film and television preferences. User film and television preferences can be broad, including preferences for film and television content, preferences for film and television creators (such as actors), preferences for film and television sources, preferences for playback resolution, etc.

[0219] Based on different users' film and television resource preferences, this application embodiment recommends film and television resources that match user preferences in a targeted manner. The second resource segment differs from the first resource segment in the following ways, but is not limited to: the video content of the second resource segment is different from that of the first resource segment; or the content of the second resource segment is the same as that of the first resource segment but the resource to which they belong is different; or the content of the second resource segment is the same as that of the first resource segment but the clarity is different.

[0220] Furthermore, there may be multiple first resource fragments that match the resource search intent. In this case, after replying to the first user with the first resource fragments that match the resource search intent, the terminal device can further receive second dialogue content input by the first user. The second dialogue content expresses the resource selection intent of choosing a third resource fragment from the multiple first resource fragments in natural language. Then, the terminal device can select the third resource fragment from the multiple first resource fragments according to the second dialogue content and jump to the playback interface of the third resource fragment to play the third resource fragment.

[0221] The first resource fragment matching the resource search intent can be determined by the terminal device based on the content of the first dialogue, especially when the terminal device has strong computing power. Alternatively, the first resource fragment matching the resource search intent can also be determined by the cloud server based on the content of the first dialogue and fed back to the terminal device, which then replies with the first resource fragment to the user. The latter method is the end-to-cloud collaborative approach. The following section will combine... Figure 13 This explains one implementation method for edge-cloud collaboration: As shown in S61, S64, and S65, before the terminal device replies to the user with a first resource fragment that matches the resource search intent: the terminal device may send a first dialogue content to the server. Correspondingly, the server may receive the first dialogue content sent by the terminal device and determine the first resource fragment that matches the resource search intent based on the first dialogue content. Then, the server may return indication information of the first resource fragment to the terminal device. Thus, the terminal device can receive the indication information of the first resource fragment from the server, know which resource fragment matches the user's search intent, and execute S52 to provide the search result to the user.

[0222] The indication information for the first resource segment may include: the identifier information of the resource to which the first resource segment belongs, and the location information of the first resource segment within its resource. The identifier information of the resource may be, for example, the resource name, and the location information of the first resource segment within its resource may be the resource segment number, time range, etc. For example, the indication information for the first resource segment may consist of the resource name "TV Series 1" and the resource segment number "Episode 63". Another example is that the indication information for the first resource segment may consist of the resource name "Movie 2" and the time range "30:00-40:00". When using a time range as the location information of the first resource segment within its resource, the start time of the first resource segment can be used as a key parameter indicating the time range, and the end time of the first resource segment, if not explicitly indicated, can be assumed to be the end time of the resource to which the first resource segment belongs.

[0223] Specifically, the server can utilize the interconnected search results and PGC search results associated with the first dialogue content to determine the first resource fragment that matches the resource search intent. As shown in S62a and S62b, the internet search results can be obtained by searching the internet based on the first dialogue content; as shown in S63a and S63b, the PGC search results can be obtained by searching the PGC resource library based on the first resource identifier. The PGC resource library records the resource fragments included in different resources. The first resource identifier belongs to the video resource entity mentioned above, and it can be extracted from the interconnected search results. Specifically, the server can use a first customized prompt guiding text model to extract resource identifiers such as video resource names from the internet search results. Regarding the implementation of the first customized prompt, please refer to the relevant content in the preceding embodiments, which will not be repeated here.

[0224] Furthermore, compared to the resource fragments recorded in the PGC resource library, the first resource fragment can be a shorter fragment, meaning it can be a more granular fragment. To this end, the server can first determine a third resource fragment that matches the user's search intent based on internet search results and PGC search results. The duration of the third resource fragment can be the same as the resource fragments recorded in the PGC resource library. Then, the server can use a multimodal large model to process the third resource fragment, internet search results, and PGC search results to further locate the first resource fragment from the third resource fragment. In this way, the first resource fragment can meet the user's fragmented viewing needs.

[0225] As shown in S67, before the terminal device replies to the user with a first resource fragment that matches the resource search intent: in addition to the indication information of the first resource fragment, the server can also return a first reply text to the terminal device. Accordingly, the terminal device can receive the first reply text. In this way, the terminal device can display the first reply text to present the user with an introduction to the first resource fragment and the reasons for its recommendation, allowing the user to clearly understand the first resource fragment and the recommendation logic.

[0226] Specifically, as shown in S66, the server can use a large text model to analyze the internet search results and PGC search results related to the first dialogue content, and generate the first response text. This can be referred to in the relevant content of the preceding embodiments, and will not be repeated here.

[0227] As shown in S68, before the terminal device replies to the user with a first resource fragment that matches the resource search intent: in addition to the indication information of the first resource fragment, the server can also return a first image to the terminal device. The first image can be generated based on the first resource fragment. Specifically, the first image can come from the first resource fragment, such as the first frame image of the first resource fragment. In this way, the terminal device can use the first image as the cover image of the first service card when rendering and generating the first service card, thereby increasing the user appeal of the first service card.

[0228] As shown in S69, after S67-S68, the terminal device can use the first reply text and the first image to render and generate a first service card, and reply to the user with the first resource fragment by displaying the first service card and / or the first reply text.

[0229] In this embodiment of the application, in order to improve the accuracy of the response, before the server determines the first resource fragment that matches the resource search intent based on the first dialogue content, the server can first use PGC search results to filter out inaccurate content in Internet search results.

[0230] Furthermore, recommended resource snippets can also be based on user profiles. This way, different users can receive different recommended resource snippets, even if different users input the same question—whether it's a fuzzy search question or a recommendation question.

[0231] Specifically, when recommending the first resource fragment to the first user: the server can analyze internet search results and PGC search results related to the first dialogue content based on the first user's user profile to determine the first resource fragment. In specific implementation, the server can first use a second customized prompt guiding text model to output optimal resource features based on the user profile, internet search results, and PGC search results; then, based on the optimal resource features and the internet search results and PGC search results, the server can further locate the first resource fragment from the optimal resources to address the first user's question. The optimal resource features include the identification information of the optimal resource and the optimal fragment features.

[0232] For the implementation of the second customized prompt, please refer to the relevant content in the aforementioned embodiments, which will not be repeated here.

[0233] There may be multiple first resource fragments that match the resource search intent. In this case, after replying to the first user with the first resource fragments that match the resource search intent, the terminal device can further receive second dialogue content input by the first user. The second dialogue content expresses the resource selection intent of choosing a third resource fragment from the multiple first resource fragments in natural language. Then, the terminal device can select the third resource fragment from the multiple first resource fragments according to the second dialogue content and jump to the playback interface of the third resource fragment to play the third resource fragment.

[0234] There may be multiple first resource fragments that match the resource search intent. In this case, as shown in S70-S72, the server can also receive second dialogue content sent by the terminal device. The second dialogue content expresses, in natural language, the resource selection intent to select a third resource fragment from the multiple first resource fragments. Then, the server can determine the third resource fragment that matches the resource selection intent from the multiple first resource fragments based on the second dialogue content and return indication information of the third resource fragment to the terminal device. The indication information of the third resource fragment can be used to indicate the third resource fragment among the multiple first resource fragments.

[0235] Specifically, the server can use a third prompt to guide the text model to select a third resource fragment that matches the user's resource search intent from multiple first resource fragments pushed to the user in the previous round of dialogue. The implementation of the third customized prompt will be explained in detail below, and will not be elaborated upon here. For details on the implementation of the third customized prompt, please refer to the relevant content in the aforementioned embodiments; it will not be repeated here.

[0236] Figure 14 An exemplary embodiment of the terminal device 300 provided in this application is shown.

[0237] Terminal device 300 can possess both human-computer interaction capabilities and computing capabilities. The device type of terminal device 300 can be any of the following: mobile phone, tablet computer, in-vehicle device (also known as vehicle infotainment system), smart home device such as smart large screen, wearable device such as smartwatch and smart glasses, extended reality (XR) device such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), smart city device, etc.

[0238] like Figure 14As shown, the terminal device 300 may include: a processor 110, a memory 120, a display 130, a display driver integrated circuit (DDIC) 140, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180B, an accelerometer sensor 180E, and a touch sensor 180K, etc. The various components of the terminal device 300 can be connected via a bus.

[0239] The processor 110 provides computing power and can be used as the computing module of the terminal device 300. The display 130, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, and other input / output components provide human-computer interaction capabilities and can be used as the human-computer interaction module of the terminal device 300. When the computing module within the terminal device 300 has powerful computing capabilities, the terminal device 300 can independently execute the resource search method provided in this application embodiment. When the computing module within the terminal device 300 does not have powerful computing capabilities, the terminal device 300 can also only execute the human-computer interaction steps in the resource search provided in this application embodiment, such as receiving a query through a voice assistant, outputting a reply text, displaying resource service cards, and playing / jumping, etc., while the inference calculation steps based on the text-based large model and multimodal large model in this method can be executed by a cloud-based server with stronger computing power.

[0240] Processors 110 can be one or more, and they can be integrated into an integrated circuit of a system-on-a-chip (SOC). An SOC is a system-on-a-chip. Processors 110 may include a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU). The CPU may include an application processor (AP) and a baseband processor (BP). The AP is responsible for running the operating system, user interface, and applications on the terminal device 300; the BP is responsible for transmitting and receiving wireless signals and managing radio frequency services. The GPU is responsible for graphics rendering, performing tasks such as shading, material filling, rendering, and output based on rendering instructions and data from the CPU. The NPU, by referencing biological neural network structures, such as the transmission patterns between neurons in the human brain, can quickly process input information and continuously learn. The NPU can be used to run artificial intelligence algorithms, such as instruction recommendation algorithms, image processing algorithms, and image understanding algorithms. The CPU and GPU can be used to render and synthesize the image to be displayed on the monitor 130.

[0241] The memory 120 may include a program storage area and a user data storage area. The program storage area may store the operating system and one or more applications (such as game applications), while the data storage area may store data created by the user during use of the terminal device 300 (such as photos and contacts). The memory 120 may be a high-speed random access memory or a non-volatile memory, such as a hard disk, flash memory, or universal flash storage (UFS). The memory 120 may also be an external memory card, such as a Micro SD card.

[0242] The memory 120 may also store code instructions for the resource search method provided in the embodiments of this application. When the processor 110 reads the code instructions from the memory 120 and runs the code instructions, the terminal device 300 may execute the steps performed by the human-computer interaction module and / or the computing module in the resource search method provided in the embodiments of this application.

[0243] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0244] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the terminal device 300. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0245] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device 300, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0246] Figure 14 The illustrated structure does not constitute a specific limitation on the terminal device 300. The terminal device 300 may include more or fewer parts than illustrated, or combine some parts, or split some parts, or arrange different parts. The various parts illustrated may be implemented in hardware, software, or a combination of software and hardware.

[0247] Figure 15 A server 300 provided in an embodiment of this application is shown. Server 300 may be a cloud server mentioned in the foregoing embodiments. Figure 15As shown, server 300 may include: processor 210, memory 220, input / output device 230, communication module 240, etc., and these components can be coupled via a bus.

[0248] Server 300 may have powerful computing resources, and its processor 210 may include one or more powerful processors, such as central processing unit (CPU), neural network processing unit (NPU), graphics processing unit (GPU), etc.

[0249] The processor 210 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0250] The processor 210 may have a cache memory, which can be used to store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the cache memory, which can reduce the waiting time of the processor 210 and improve the program running efficiency.

[0251] The processor 210 can also connect to external memory. This memory can be high-speed random access memory or non-volatile memory, such as a hard disk, flash memory, universal flash memory (UFS), etc. The memory can also be an external memory card, such as a Micro SD card.

[0252] The processor 210 is the computing core of the server 300, possessing powerful computing capabilities. Coupled with the memory 220, it can read and execute computer-readable instructions stored in the memory 220, running the operating system and various programs. Specifically, the CPU 210 can call programs stored in the memory 220, such as the implementation program of the edge-cloud collaborative computing power scheduling method provided in this application embodiment, and execute the instructions contained in that program.

[0253] The memory 220 may include high-speed random access memory, non-volatile memory, such as disk, flash memory, or other non-volatile solid-state storage devices. The memory 220 can be used to store various software programs and multiple sets of instructions. The memory 220 can store an operating system, such as Linux. The memory 220 can also store one or more programs, such as programs involved in patch creation, such as compilers and linkers. The memory 220 can also store the implementation program of the edge-cloud collaborative computing power scheduling method provided in the embodiments of this application.

[0254] Input / output device 230 may include devices such as a display screen, keyboard, and mouse, and can be used to receive user input and output program execution results to the user.

[0255] The communication module 240 may include a wired communication module and a wireless communication module. The wired communication module supports wired communication protocols such as Universal Serial Bus (USB), serial port, and Ethernet, communicating with other devices via a physical communication cable. The wireless communication module may include 2G / 3G / 4G / 5G wireless communication modules, Wi-Fi communication modules, etc. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signal to the CPU 210. The wireless communication module can also receive signals to be transmitted from the CPU 210, modulate and amplify them, and then convert them into electromagnetic waves for radiation via the antenna.

[0256] Figure 15 The illustrated structure does not constitute a limitation on server 300. Server 300 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.

[0257] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0258] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the human-computer interaction steps in the above-described method embodiments, such as receiving a question (query) through a voice assistant, outputting a reply text, displaying a resource service card, and playing / jumping to a page.

[0259] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the reasoning operation steps based on the text big model and multimodal big model in the above-described method embodiments, such as user intent recognition, resource search, and resource selection.

[0260] This application also provides a computer program product that, when executed by a processor, can implement all the steps in the above-described method embodiments.

[0261] This application also provides a computer program product that, when executed by a processor, can implement the human-computer interaction steps described in the above-described method embodiments.

[0262] This application also provides a computer program product that, when executed by a processor, can implement the reasoning operation steps based on the text large model and the multimodal large model in the above-described method embodiments.

[0263] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory. When executed by the processor, the computer program can implement all the steps in the above-described method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0264] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory. When executed by the processor, the computer program can implement the human-computer interaction steps described in the various method embodiments above. The chip system can be a single chip or a chip module composed of multiple chips.

[0265] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory. When executed by the processor, the computer program can perform the inference operations based on the text-based large model and the multimodal large model described in the above-described method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0266] The term "user interface (UI)," used in the specification and accompanying drawings of this application, refers to the medium through which an application or operating system interacts and exchanges information with the user. It facilitates the conversion between the internal form of information and a form acceptable to the user. The user interface of an application is written in source code using specific computer languages ​​such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content, such as images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as XML tags. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a web page is also defined through tags or nodes in the web page source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>Used to define the elements and attributes of a webpage.

[0267] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an interface element such as an icon, window, or control displayed on the screen of a terminal device. The control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0268] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0269] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0270] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A method of searching for video resources, characterized by, The method comprises: The terminal device receives first dialogue content input by a first user, the first dialogue content expressing a resource search intention in natural language; According to the first dialogue content, the terminal device replies first content conforming to the resource search intention, the first content comprising a first resource segment and a first reply text; the first resource segment is a video resource segment, the first resource segment is one or more episodes of a film or television work, or one or more film or television segments in an episode of a film or television work; the first reply text is used to describe the first resource segment.

2. The method of claim 1, wherein, The first content conforming to the resource search intention comprises displaying the first reply text and a first service card, the first service card being used for the user to watch the first resource segment.

3. The method of claim 1 or 2, wherein, The method further comprises: The terminal device sends the first dialogue content to a server; The terminal device receives indication information of the first resource segment from the server, the indication information of the first resource segment comprising identification information of a resource to which the first resource segment belongs and positioning information of the first resource segment in the resource; wherein the positioning information comprises a start time of the first resource segment.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: the terminal device receives the first reply text from the server.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: The terminal device receives a first picture from the server, the first picture being generated according to the first resource segment; The terminal device uses the first picture as a cover picture of the first service card.

6. The method of claim 5, wherein, The first picture is the first frame picture of the first resource segment.

7. The method of any one of claims 1-6, wherein, The method further comprises: the terminal device detects a user operation acting on the first service card; and jumps to a play interface of the first resource segment to play the first resource segment.

8. The method of any one of claims 1-7, wherein, The rendered content on the first service card is used to describe the first resource segment, comprising one or more of the following: the name, type, release time, creator, and cover picture of the first resource segment or the video resource to which the first resource segment belongs.

9. The method of any one of claims 1-8, wherein, The method further comprises: The terminal device receives the first dialogue content input by a second user, the second user being different from the first user; According to the first dialogue content input by the second user, the terminal device replies second content conforming to the resource search intention to the second user, the second content comprising a second resource segment and a second reply text, the second resource segment being one or more episodes of a film or television work, or one or more film or television segments in an episode of a film or television work; the second reply text is used to describe the second resource segment, the video content of the second resource segment being different from that of the first resource segment, or the content of the second resource segment being the same as that of the first resource segment but belonging to different resources, or the content of the second resource segment being the same as that of the first resource segment but having different clarity.

10. The method of claim 9, wherein, The memory information or personal portrait of the first user and the second user on the terminal device are different.

11. The method of any one of claims 1-10, wherein, There are multiple first resource segments conforming to the resource search intention; After the reply matches the first content of the resource search intention, the method further includes: The terminal device receives second dialogue content of the first user input, and the second dialogue content expresses a resource selection intention of selecting a third resource segment from the plurality of first resource segments in a natural language; According to the second dialogue content, the terminal device selects a third resource segment from the plurality of first resource segments, jumps to a playing interface of the third resource segment, and plays the third resource segment.

12. A method of searching for video resources, characterized by, Comprise: The server receives first dialogue content from a terminal device, and the first dialogue content expresses a resource search intention in a natural language; The server determines a first resource segment that matches the resource search intention according to the first dialogue content; the first resource segment is a video resource segment, and the first resource segment is one or more episodes of a film and television work or one or more film and television segments in an episode of a film and television work; The server returns indication information of the first resource segment to the terminal device, and the indication information includes identification information of a resource to which the first resource segment belongs and positioning information of the first resource segment in the resource, wherein the positioning information includes a start time of the first resource segment.

13. The method of claim 12, wherein, Further comprise: The server also returns the first reply text to the terminal device, and the first reply text is used to describe the first resource segment.

14. The method of claim 12 or 13, wherein, Further comprise: The server also returns a first picture to the terminal device, and the first picture is generated according to the first resource segment.

15. The method of claim 14, wherein, The first picture is the first frame picture of the first resource segment.

16. The method of any one of claims 12-15, wherein, The server determines a first resource segment that matches the resource search intention according to the first dialogue content, specifically comprising: The server uses the PGC search result and the Internet search result associated with the first dialogue content to determine a first resource segment that matches the resource search intention according to the first dialogue content; The Internet search result associated with the first dialogue content is obtained by searching the Internet based on the first dialogue content, the professional production content (PGC) search result associated with the first dialogue content is obtained by searching the PGC resource library according to a first resource identifier, the first resource identifier is extracted from the Internet search result, and the PGC resource library records resource segments included in different resources.

17. The method of claim 16, wherein, The server determines a first resource segment that matches the resource search intention according to the first dialogue content, specifically comprising: The server first determines a third resource segment that matches the resource search intention based on the Internet search result and the PGC search result; The server further locates the first resource segment from the third resource segment by processing the third resource segment, the Internet search result and the PGC search result using a multi-modal large model.

18. The method of claim 16 or 17, wherein, The server determines a first resource segment that matches the resource search intention according to the first dialogue content, specifically comprising: The server analyzes the Internet search result and the PGC search result based on a user portrait of the first user, and determines the first resource segment.

19. The method of any one of claims 16-18, wherein, Before the server determines the first resource segment that meets the resource search intention based on the first dialogue content, the method further comprises: The server filters out inaccurate content in the Internet search result by using the PGC search result.

20. The method of any one of claims 12-19, wherein, There are multiple first resource segments that meet the resource search intention; after the server returns the indication information of the first resource segment to the terminal device, the method further comprises: The server receives second dialogue content from the terminal device, and the second dialogue content expresses a resource selection intention in natural language; The server determines a third resource segment that meets the resource selection intention from the multiple first resource segments based on the second dialogue content: The server returns indication information of the third resource segment to the terminal device, and the indication information of the third resource segment is used to indicate the third resource segment in the multiple first resource segments.

21. A terminal device, comprising: Comprise: A processor, a memory, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the method of any one of claims 1-11.

22. A server, comprising: Comprise: A processor, a memory, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the method of any one of claims 12-20.

23. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-20.

24. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-20.

Images