Conversation opening line generation method and device, electronic equipment, storage medium and product

Abstract

The disclosure provides a dialogue opening generation method and device, electronic equipment, storage medium and product, relates to the technical field of artificial intelligence, and in particular to the technical field of large language models or video interaction. The specific implementation scheme is: quality evaluation is performed on at least one frame of video picture displayed by a target video in a starting playing stage to obtain an evaluation result; based on the evaluation result, reference information of an opening of the target video is determined; and based on the opening reference information, first dialogue opening information for initiating an interactive dialogue is generated. In the scheme, whether to generate dialogue opening information based on the information of the video picture in the starting playing stage can be accurately controlled according to the evaluation result of the video picture, which helps to improve the quality of the dialogue opening information.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the field of large language model or video interaction technology. Specifically, this disclosure relates to a method, apparatus, electronic device, storage medium and product for generating dialogue opening remarks. Background Technology

[0002] In video interaction scenarios, proactively generating high-quality dialogue openings during the video playback phase is crucial for prompting users to engage in subsequent dialogue. Therefore, improving the quality of dialogue openings has become an important technical challenge. Summary of the Invention

[0003] To address at least one of the aforementioned deficiencies, this disclosure provides a method, apparatus, electronic device, storage medium, and product for generating dialogue openings.

[0004] According to a first aspect of this disclosure, a method for generating a dialogue opening is provided, the method comprising: The quality of at least one frame of video displayed during the startup playback phase of the target video is evaluated, and the evaluation result is obtained. Based on the evaluation results, the opening remarks reference information for the target video is determined; Based on the opening remarks reference information, first dialogue opening remarks information for initiating interactive dialogue is generated.

[0005] According to a second aspect of this disclosure, a dialogue opening statement generation apparatus is provided, the apparatus comprising: The video quality assessment module is used to assess the quality of at least one frame of video displayed during the startup playback phase of the target video and obtain the assessment result. The opening remarks reference information determination module is used to determine the opening remarks reference information of the target video based on the evaluation results. The opening statement generation module is used to generate first dialogue opening statement information for initiating interactive dialogue based on the opening statement reference information.

[0006] According to a third aspect of this disclosure, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to at least one of the aforementioned processors; wherein, The memory stores instructions that can be executed by at least one processor, which, when executed by at least one processor, enables the at least one processor to perform the dialogue opening generation method.

[0007] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to execute the above-described dialog opening generation method.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described dialogue opening generation method.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure.

[0011] Figure 1 This is a flowchart illustrating a method for generating a dialogue opening line according to an embodiment of this disclosure.

[0012] Figure 2 This is a schematic diagram of the structure of a dialogue opening line generation system provided in an embodiment of this disclosure.

[0013] Figure 3 This is a schematic diagram of a dialogue opening statement generation device provided in an embodiment of this disclosure.

[0014] Figure 4 This is a block diagram of an electronic device used to implement the dialogue opening generation method provided in the embodiments of this disclosure. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] In video interaction scenarios, one approach to opening remarks in related technologies is to provide fixed templates, such as simple opening remarks like "Good morning" or "Good afternoon" for different times of day, like morning and afternoon. However, the opening remarks provided in this way are too rigid and of low quality.

[0017] The inventors of this disclosure have discovered that, in order to improve the quality of dialogue opening remarks, visual information from the video frame during the video startup playback phase can be referenced to generate the opening remarks. However, the video frame during the startup playback phase may be of poor quality due to environmental or device factors, such as a black screen appearing at the start of video playback. This can render the visual information contained in the video frame invalid or misleading (e.g., misinterpreting "black screen" as "night"), thus affecting the quality of the generated dialogue opening remarks. Therefore, how to accurately determine whether to generate dialogue opening remarks based on the video frame during the startup playback phase becomes an important technical problem.

[0018] In addition, from the perspective of resource consumption, the processing of video images generally relies on multimodal large models. The computational resource consumption of multimodal large models is significantly higher than that of pure text models, and sending invalid video images into multimodal models is a clear waste of computational resources.

[0019] The dialogue opening line generation method, apparatus, electronic device, storage medium, and product provided in this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0020] Figure 1 This is a flowchart illustrating the method for generating first dialogue opening information provided in an embodiment of this disclosure, as shown below. Figure 1 As shown, the method may include the following steps: Step S110: Perform a quality assessment on at least one frame of video displayed during the startup playback phase of the target video to obtain the assessment result.

[0021] Step S120: Based on the evaluation results, determine the opening reference information for the target video.

[0022] Step S130: Based on the opening remarks reference information, generate the first dialogue opening remarks information for initiating the interactive dialogue.

[0023] As can be seen from the above process, this disclosure performs a quality assessment on at least one frame of video displayed during the startup playback phase of the target video, obtains an assessment result, determines the opening dialogue reference information of the target video based on the assessment result, and then generates the first dialogue opening dialogue information for initiating an interactive dialogue based on the opening dialogue reference information. This solution can precisely control whether to generate dialogue opening dialogue information based on the information of the video frame during the startup playback phase, based on the video frame assessment result, which helps to improve the quality of the dialogue opening dialogue information.

[0024] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments.

[0025] First, the above step S110, namely "to evaluate the quality of at least one frame of video displayed during the startup playback phase of the target video and obtain the evaluation result", will be described in detail with reference to the embodiments.

[0026] The target video is the video displayed to the user during the video interaction process and is the core of the video interaction. In this embodiment, the source of the target video can be diverse, such as a video stream captured in real-time by an image acquisition device (e.g., a camera) on the user terminal, a video sent from a server (e.g., a live webcast video stream), or a video stored on the user terminal.

[0027] The playback initiation phase is a crucial period immediately following the start of video interaction, used to collect the visual information needed to generate the opening dialogue. For example, the video footage during this phase can be a predetermined number of video frames after the target video starts playing, or video footage within a predetermined duration after the target video starts playing. For instance, it could be the first frame (first frame) or the first few frames of a real-time video stream captured after the user actively turns on their camera, or even just the first 0.01 seconds of video footage.

[0028] For example, quality assessment of at least one frame of video displayed during the start-up playback phase of the target video can specifically be performed on the first frame (first frame) or the preceding frames of the target video.

[0029] Quality assessment is the process of judging the effectiveness of visual information in video footage. A passing quality assessment indicates that the visual information in the video footage is valid and can be used for analysis and generating dialogue openings; in other words, the video footage is effective. For example, the video footage clearly shows certain physical objects (such as a coffee cup). A failing quality assessment indicates that the visual information in the video footage is invalid and cannot be used for analysis and generating dialogue opening information; in other words, the video footage is invalid. For example, a black screen in the video footage or poor lighting conditions that prevent the objects within it from being clearly displayed.

[0030] For example, video quality can be assessed based on quantitative metrics such as sharpness, brightness, and information entropy.

[0031] The following describes in detail, with reference to the embodiments, step S120, namely "determining the opening reference information of the target video based on the evaluation results" and step S130, namely "generating the first dialogue opening information for initiating interactive dialogue based on the opening reference information".

[0032] Opening remarks reference information refers to the collection of information used to analyze and generate dialogue opening remarks.

[0033] The evaluation results of the video footage can clearly reflect its effectiveness. Based on the effectiveness of the video footage, it is possible to determine whether the opening dialogue should be generated based on the video footage from the initial playback phase, thereby enabling precise control over the data-driven basis of the opening dialogue and ensuring its rationality.

[0034] For example, information from the video frame can be dynamically selected as the opening remarks reference information based on whether the video frame is valid, thereby avoiding the input of invalid or misleading visual information and effectively ensuring the quality of the generated dialogue opening remarks.

[0035] In summary, based on steps S110 to S130, this disclosure evaluates the quality of the video frame during the startup playback phase of the target video and selects appropriate opening dialogue reference information based on the evaluation results to generate the first dialogue opening dialogue information. This precisely controls whether to generate the dialogue opening dialogue information based on the video frame during the startup playback phase, avoids interference from invalid or low-quality visual information in the dialogue opening dialogue information generation process, and improves the stability and reliability of the generated dialogue opening dialogue information.

[0036] In one optional embodiment of this disclosure, the opening reference information of the target video is determined based on the evaluation results, including: In response to the assessment results indicating that the video footage is valid, the opening remarks reference information is determined based on the video footage.

[0037] In this process, after completing the quality assessment of the video footage during the startup playback phase and confirming that the video footage is valid, the opening remarks reference information can be determined based on the video footage. That is, the valid video footage is used as the core information source, and valuable visual information is extracted from it as the opening remarks reference information.

[0038] The initial dialogue opening, generated based on opening reference information containing visual information, can interact deeply with specific objects in the video frame, thereby effectively improving the quality of the initial dialogue opening. For example, visual information in the video frame includes a coffee cup on the table, and the generated initial dialogue opening could be, "Have you tried the XX coffee next to you?"

[0039] For example, the dialogue interaction can be in the form of text or voice. The opening dialogue information can be displayed directly in the video playback interface in the form of text, or it can be broadcast in the form of voice.

[0040] For example, when multiple video frames exist during the initial playback phase, each frame can be independently evaluated for quality. The evaluation results of all frames can then be aggregated, and a final evaluation result guiding subsequent path selection can be generated according to the following rules: If all video frames are deemed invalid, the system determines the overall video playback is invalid. If at least one valid video frame exists, the system determines the overall video playback is valid and extracts visual information based on that valid video frame to generate the initial dialogue opening information.

[0041] In the solution provided by the embodiments of this disclosure, by determining the opening remarks reference information based on the video frame after determining that the video frame is valid, high-quality visual information can be effectively used to generate dialogue opening remarks information, thereby improving the quality of dialogue opening remarks information.

[0042] In addition, by triggering video understanding processing, which has a relatively high computational cost, after the video image is determined to be valid, it is possible to ensure that computing power is invested in key processing steps, thereby optimizing system performance.

[0043] To provide a more specific construction method for providing opening remarks reference information when video footage is available, one optional method of this disclosure involves determining the opening remarks reference information based on the video footage, including: Alternatively, information about the entities in the video footage can be used as reference information for the opening remarks, or information about the entities in the video footage and relevant knowledge about the entities retrieved can be used as reference information for the opening remarks.

[0044] In this context, "entity objects" refers to visual targets with clear semantics that are extracted from video footage using computer vision technologies (such as object detection and image recognition). For example, entity objects identified from video footage might be "desk" and "coffee cup".

[0045] The information of an entity object can include its attributes, such as the attribute that a "coffee cup" is made of "paper cup". The information of an entity object can also include other information about it, such as the text and patterns on the surface of the "coffee cup".

[0046] For example, a set consisting of all entity objects contained in the video frame and related information including their attributes can be used as opening remarks reference information.

[0047] In addition to constructing opening remarks reference information based on the information of entity objects in the video footage, a knowledge enhancement step can be added, that is, querying relevant knowledge of entity objects based on entity objects, and using the queried relevant knowledge of entity objects and the information of entity objects together as opening remarks reference information.

[0048] For example, the names of identified entities can be used as search keywords. Based on Retrieval-Augmented Generation (RAG) technology, queries can be performed in internal or external knowledge bases (such as product databases, encyclopedias, and news repositories). The retrieved extended information closely related to the entity object can then be considered as entity-related knowledge. For instance, searching for "coffee cup" can yield information such as coffee cup types and usage tips, thus extending the generated initial dialogue to topics like coffee cup types and usage techniques.

[0049] The solution provided in this disclosure uses information about entities in a video frame as opening remarks reference information, ensuring that the generated first dialogue opening remarks are at least directly related to the entities in the video frame. This enables the generation of personalized first dialogue opening remarks and effectively improves the quality of the opening remarks. By querying relevant knowledge about the entities and using it as opening remarks reference information, external knowledge can be injected into the first dialogue opening remarks. This allows the first dialogue opening remarks to go beyond simple mentions of entities and extend to multiple, deeper dimensions, thereby significantly expanding the information hierarchy and density of the first dialogue opening remarks and providing better support for subsequent extended dialogue topics.

[0050] In order to construct more comprehensive and personalized opening remarks reference information, in one optional embodiment of this disclosure, the opening remarks reference information also includes at least one type of contextual information not derived from video footage.

[0051] Contextual information refers to auxiliary information that describes the environment, state, or user background when the video interaction occurs, and may include, but is not limited to, time information, location information, and user account identifiers.

[0052] Contextual information not derived from video footage means that the contextual information is not obtained through analysis of video footage, but is acquired independently through other means. This isolates the impact of the validity of video footage and ensures the diversity and complementarity of data sources.

[0053] For example, contextual information may also include real-time weather information, festival customs information, and user profile data. Real-time weather information can be obtained by calling a weather query tool, based on time and location information, such as "Current temperature 25 degrees Celsius, sunny." Festival customs information can be obtained by querying festival and custom knowledge bases for various regions based on time and location information. For example, first, based on the current date, it can be found that it is the Dragon Boat Festival; then, combined with the user's location "Region X," the custom knowledge base can be searched to find "Region X has the custom of eating zongzi (sticky rice dumplings) during the Dragon Boat Festival." User profile data can be obtained by querying based on user account identifiers, which can reflect the user's recent preferences or historical behavioral habits, such as "showing interest in classical music in historical conversations" or "frequently querying fitness recipes recently."

[0054] It should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of any type of information involved in the technical solution disclosed herein, such as user personal information (such as the aforementioned user account identifier, location information, and user profile data), comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0055] The solution provided in this disclosure integrates multi-source information into the opening remarks reference information, laying a solid foundation for generating personalized first dialogue opening remarks adapted to the actual scenario. Simultaneously, information from the video frame can be considered the core information source of the current video interaction, while various contextual information can be regarded as auxiliary information sources describing the user's or environment's state. This allows for the inference of first dialogue opening remarks related to the video frame and more closely aligned with the user's personal characteristics and the current spatiotemporal context within a richer and more hierarchical cognitive background.

[0056] When generating the opening remarks for the first dialogue in a valid video frame, one optional method of this disclosure involves generating the opening remarks for initiating an interactive dialogue based on opening remarks reference information, including: The first dialogue opening information is generated based on the opening reference information using a multimodal large model.

[0057] Among them, multimodal large models are a class of large-scale pre-trained artificial intelligence models that can simultaneously understand, process, and fuse multiple modal information (such as text, images, and audio). Multimodal large models have powerful cross-modal semantic alignment and generation capabilities, and can deeply understand multimodal data (especially video footage) in opening dialogue reference information to generate high-quality first dialogue opening dialogue information.

[0058] In the solution provided by the embodiments of this disclosure, by specifying the use of a multimodal large model to process the opening reference information that incorporates visual information, the quality of the generated first dialogue opening information can be improved under the guidance of effective video footage.

[0059] Meanwhile, by only activating the more computationally expensive but more powerful multimodal large model when the opening reference information contains valid visual information, a precise match between computing resources and content generation requirements is achieved.

[0060] For example, a user initiates video interaction in a supermarket, pointing their terminal's camera at a product on the shelf. A clear video image is captured during playback and deemed valid. The system uses an image recognition model to identify the main subject of the video image as the entity "XX beverage." It then queries related knowledge about the entity, using "XX beverage" as a keyword to search the product database via RAG, retrieving the external knowledge "There is currently a buy-one-get-one-half-price promotion." Simultaneously, the system searches the user profile, obtaining further contextual information: "The user's historical behavior indicates they are a price-sensitive consumer." At this point, "XX beverage," "currently a buy-one-get-one-half-price promotion," and "the user is price-sensitive" can be used as opening dialogue reference information. This information is input into a multimodal large-scale model, which infers based on this information and ultimately outputs the first dialogue opening message: "This beverage currently has a buy-one-get-one-half-price promotion, it's very cost-effective, would you like me to help you calculate it?"

[0061] To address the possibility of invalid video footage during the playback startup phase, and to ensure reliable generation of the opening sequence even when visual information in the video footage is unavailable, one optional method of this disclosure involves determining opening sequence reference information for the target video based on the evaluation results, including: In response to an evaluation result indicating that the video frame is invalid, at least one piece of contextual information not originating from the video frame is identified as the opening statement reference information.

[0062] In cases where visual information is unavailable, contextual information can be used as a substitute to construct opening remarks reference information, which can effectively meet the needs of generating the first dialogue opening remarks information.

[0063] The contextual information is not derived from video footage, thus avoiding the influence of invalid or misleading visual information on the effectiveness of the generated first dialogue opening information.

[0064] In this embodiment of the disclosure, by generating the first dialogue opening information based on contextual information when the video frame is invalid, it can serve as an alternative solution to ensure the timely output of reasonable first dialogue opening information, thereby effectively complementing the technical approach when the video frame is valid.

[0065] In generating the opening dialogue information for a situation where the video frame is invalid, one optional method of this disclosure involves generating the opening dialogue information for initiating an interactive dialogue based on the opening dialogue reference information, including: The first dialogue opening information is generated using a large text model based on contextual information.

[0066] Among them, the text big model is a type of large-scale language model that is specifically pre-trained and optimized for text data. Unlike the multimodal big model, its design focuses on understanding and generating natural language sequences and does not have the ability to natively process images or audio.

[0067] When the opening remarks serve as contextual information, the input to the text-based large model is plain text, and there is no need to process high-dimensional image data. Its inference speed is usually significantly faster than that of the multimodal large model, thus enabling a rapid response even when the video footage is invalid.

[0068] The solution provided in this disclosure uses a large text model to process the contextual information of plain text when the video frame is invalid, which can efficiently and reliably generate the opening dialogue information while avoiding excessive consumption of computing resources.

[0069] For example, a user initiates a video interaction in region X on the day of the Dragon Boat Festival. When the user turns on the camera, due to camera malfunction, extreme darkness, or severe obstruction, the video footage captured during the initial playback phase is invalid. In this case, the system can use the contextual information as reference information for the opening remarks to generate an opening message. The system can obtain the current time (the fifth day of the fifth lunar month) and geographical location (region X). Based on the "time + location" combination, the system queries the festival customs knowledge base and retrieves a personalized custom knowledge: "People in region X like to eat meat-filled rice dumplings during the Dragon Boat Festival." At the same time, based on the user's identifier, the system retrieves the user profile data, which includes the preference for "loving to eat delicious food." "Dragon Boat Festival," "region A likes to eat meat-filled rice dumplings," and "loves to eat delicious food" can be determined as the opening remarks reference information. This opening remarks reference information is input into the text model, which infers based on the opening remarks reference information and finally outputs the first dialogue opening message: "Happy Dragon Boat Festival! Have you already eaten hot meat-filled rice dumplings in region A?"

[0070] For example, a user initiates video interaction during a rainstorm. When they turn on their camera, the captured video is extremely blurry due to rain or lighting issues and is deemed invalid. In this case, the system can discard the blurry video and use the contextual information as opening dialogue reference information to generate an opening dialogue. The system uses a weather query tool to obtain weather information such as "Heavy rain warning currently in effect" based on the current time and location. It also uses the user account identifier to find that the user profile data includes information about commuting. "Heavy rain warning" and "commuting" are determined as opening dialogue reference information. This information is input into a text model, which infers based on this information and ultimately outputs the first dialogue opening dialogue: "Just a reminder, your city is currently under a heavy rain warning. It's windy and rainy outside, did you bring an umbrella?"

[0071] To ensure accurate and objective evaluation of video image quality, the quality evaluation in this disclosure can be based on several key quality indicators of the video image. Specifically, in one optional method of this disclosure, the video image quality is evaluated based on at least one of the video image's sharpness, brightness, and information entropy, resulting in an evaluation result including: If the score corresponding to at least one of the video image's sharpness, brightness, and information entropy is lower than a preset threshold, the video image is determined to be invalid; otherwise, the video image is determined to be valid.

[0072] Among them, the score refers to the numerical value obtained by quantifying the video image on the corresponding quality index dimension through a specific image processing algorithm.

[0073] For example, a sharpness score can be obtained by calculating the grayscale changes of pixels in a video frame. A higher sharpness score generally indicates sharper image edges and richer details. A brightness score can be obtained by calculating the average grayscale value of all pixels in a video frame, used to determine whether the video frame is within a reasonable exposure range. An information entropy score can be calculated based on the grayscale histogram distribution of the video frame. Its value reflects the randomness or richness of information contained in the image. Low information entropy often corresponds to large areas of solid color or monotonous texture.

[0074] For example, a video frame can be determined to be invalid if the score for at least one of the three parameters—clarity, brightness, and information entropy—is below a preset threshold; otherwise, the video frame is determined to be valid. In other words, if any one of these three parameters fails to meet the standard, the entire video frame will be deemed invalid, thus ensuring that the adopted visual information has high reliability.

[0075] The solution provided in this disclosure relies on key quality indicators such as sharpness, brightness, and information entropy to evaluate the quality of video images, which ensures the objectivity and accuracy of the evaluation results and provides a reliable basis for subsequent decision-making.

[0076] To construct a dialogue generation system optimized based on user feedback, in one optional embodiment of this disclosure, after generating first dialogue opening information for initiating an interactive dialogue based on opening reference information, the method further includes: Obtain feedback from multiple users regarding the opening remarks of the first conversation; Based on the feedback information, the second dialogue opening information is determined from the first dialogue opening information. The second dialogue opening information includes the dialogue opening information in the first dialogue opening information, and the corresponding feedback information indicates that the dialogue opening information meets the preset positive feedback conditions. The first training data is constructed based on the opening information of the second dialogue, and the first training data is used to fine-tune the large model that generates the opening information of the first dialogue.

[0077] Feedback information refers to the information about the user's subsequent dialogue interaction based on the initial dialogue opening information after it has been output. It can reflect the performance effect of the initial dialogue opening information in the actual scenario.

[0078] Feedback information can include positive and negative feedback. For example, if a user engages in further conversation based on the initial opening remarks, it indicates that the initial opening remarks received positive feedback. Conversely, if a user does not engage in further conversation within a preset time after the initial opening remarks are displayed, it indicates that the initial opening remarks received negative feedback.

[0079] When the opening dialogue information of the first dialogue meets the preset positive feedback conditions, it means that the opening dialogue information of the first dialogue performs well in the actual dialogue and is proven to be effective. The opening dialogue information of the first dialogue can be designated as the opening dialogue information of the second dialogue to construct the first training data.

[0080] For example, the preset positive feedback condition is that after the first dialogue opening information is displayed, the probability of the user replying to the first dialogue opening information is greater than a preset threshold. That is, after the first dialogue opening information is provided to multiple users, the proportion of users who reply based on the first dialogue opening information within a preset time period is greater than the preset threshold.

[0081] In the solution provided by the embodiments of this disclosure, first training data is constructed based on positive user feedback, and a large model used to generate the first dialogue opening information is fine-tuned based on the first training data. This enables the large model to learn and continuously improve itself based on training samples with good feedback, which helps to continuously improve the quality of the first dialogue opening information.

[0082] To generate a first dialogue opening message with a specific style, in one optional embodiment of this disclosure, the method further includes: Based on the opening dialogue information of samples with various styles and the corresponding sample opening dialogue reference information, a second training data is constructed. The second training data is used to fine-tune the large model that generates the opening dialogue information of the first dialogue.

[0083] The sample first-dialogue opening remarks, exhibiting various styles, show significant differences in tone, wording, and emotional nuance. For example, they can be categorized as "playful and cute," "warm and caring," or "concise and direct." By constructing a second training dataset using these diverse sample opening remarks, and fine-tuning the large model used to generate the first-dialogue opening remarks, the model can identify implicit needs within the opening remarks reference information, thereby automatically adjusting the language style of the generated first-dialogue opening remarks.

[0084] In the solution provided by this disclosure, a large model used to generate the opening dialogue information of the first dialogue is fine-tuned by using second training data with a clearly labeled style, so that the large model can generate the opening dialogue information of the first dialogue with different styles.

[0085] For example, the sample opening line reference information and the sample first dialogue opening line information can be constructed based on the first dialogue opening line information with a high response rate. Specifically, the first dialogue opening line information with a high response rate can be labeled with style tags, and first dialogue opening line information with high response rates and different style tags can be selected to construct the second training data. In this example, when constructing the second training data, the semantic consistency between the sample opening line reference information and the sample first dialogue opening line information can also be verified to ensure that the second training data has high quality.

[0086] In some optional implementations, to systematically balance generation quality and inference efficiency during model iteration and optimization, this disclosure also provides a two-stage model iteration strategy. Specifically, in the early stages of model application, a model with a large number of parameters is used to generate the opening dialogue information to ensure generation quality. As high-quality training data accumulates to the point where fine-tuning training of a model with a small number of parameters yields good results (e.g., high-quality training data accumulates to a preset amount), the fine-tuned model with a small number of parameters can be used to generate the opening dialogue information, thereby balancing generation quality and response speed while reducing computational costs. In this scheme, by utilizing a more capable model in the early stages of service to ensure generation quality and build a high-quality dataset, and by utilizing the accumulated data in the later stages of service to train a more efficient model to optimize service performance, a balance between generation quality and inference efficiency is effectively achieved.

[0087] For example, Figure 2 This is a schematic diagram of the structure of a dialogue opening line generation system provided in an embodiment of this disclosure.

[0088] like Figure 2 As shown, the system may specifically include input data, feature information acquisition, model routing, training modules, etc., as detailed below: The input data section may include the first frame of the video (the video frame during the start-up playback phase), the location (geographic information), the access time (time information), and the user account identifier.

[0089] For the first frame of the video, a quality assessment can be performed to determine whether it is a valid frame. If it is invalid, the first frame can be discarded, and the process can directly proceed to the "text branch" of the "model routing". If it is a valid frame, entity recognition is performed sequentially on the valid frame, such as identifying objects, people, and other entities in the frame. Then, entity knowledge is queried to obtain relevant knowledge about the entity objects.

[0090] Feature information acquisition is used to obtain multi-dimensional contextual information. Specifically, it can extract weather information (inferred from location and time), holiday information (inferred from time and location), and user profile data (inferred from user identifiers and behavioral data) based on location data, time data, and user account identifiers.

[0091] If the first frame of the video is valid, the opening remarks reference information can include the entity object in the first frame, related knowledge about the entity object, and contextual information. If the first frame of the video is invalid, the opening remarks reference information can include only contextual information.

[0092] The prompt instruction construction is used to build prompt instructions based on the opening remarks reference information to guide the large model in generating the first dialogue opening remarks information.

[0093] Model routing. Used to select different large models for the multimodal branch (when the first frame of the video is a valid image) and the text branch (when the first frame of the video is an invalid image).

[0094] For the multimodal branch, the multimodal large model can be called to generate the opening information of the first dialogue; for the text branch, the text large model can be called to generate the opening information of the first dialogue.

[0095] The training module is used to fine-tune the training of the large model that generates the opening dialogue information for the first dialogue.

[0096] This involves obtaining user feedback on the opening remarks of the first dialogue, filtering out opening remarks with high response rates, and constructing training data based on these opening remarks and their corresponding reference information.

[0097] Then, the samples with high response rates can be labeled with style tags, and samples with high semantic consistency with the opening reference information can be selected as the final training data.

[0098] Fine-tuning the training of large text models or multimodal models based on training data optimizes model performance and improves the quality of the opening dialogue information.

[0099] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0100] According to another embodiment, a first dialogue opening information generation device is provided. Figure 3 A schematic diagram of the structure of the first dialogue opening information generation apparatus according to one embodiment is shown. Figure 3 As shown, the first dialogue opening information generation device 300 includes: The video image quality assessment module 310 is used to assess the quality of at least one frame of video image displayed during the startup playback phase of the target video and obtain the assessment result. The opening reference information determination module 320 is used to determine the opening reference information of the target video based on the evaluation results; The opening statement generation module 330 is used to generate the first dialogue opening statement information for initiating an interactive dialogue based on the opening statement reference information.

[0101] As an optional approach, the opening remarks reference information determination module 320 is specifically used for: In response to the assessment results indicating that the video footage is valid, the opening remarks reference information is determined based on the video footage.

[0102] As an optional approach, when determining the opening remarks reference information based on the video frame, the opening remarks reference information determination module 320 is specifically used for: Alternatively, information about the entities in the video footage can be used as reference information for the opening remarks, or information about the entities in the video footage and relevant knowledge about the entities retrieved can be used as reference information for the opening remarks.

[0103] As an alternative, the opening remarks may also include at least one piece of contextual information not derived from the video footage.

[0104] As an optional approach, the opening statement generation module 330 is specifically used for: Based on the opening remarks reference information, a multimodal large model is used to generate the first dialogue opening remarks information for initiating an interactive dialogue.

[0105] As an optional approach, the opening remarks reference information determination module 320 is specifically used for: In response to an evaluation result indicating that the video frame is invalid, at least one piece of contextual information not originating from the video frame is identified as the opening statement reference information.

[0106] As an optional approach, the opening statement generation module 330 is specifically used for: The first dialogue opening information is generated based on contextual information using a large text model.

[0107] As an optional method, the video quality assessment module 310 is specifically used for: The quality of the video image is evaluated based on at least one of the following: sharpness, brightness, and information entropy, and the evaluation result is obtained.

[0108] As an alternative approach, the target video is a video stream captured by the image acquisition device on the user terminal after video interaction is initiated.

[0109] As an alternative, the above-mentioned device also includes a fine-tuning training module (not shown in the figure), used for: After generating the first dialogue opening information for initiating an interactive dialogue based on the opening reference information, obtain feedback information from multiple users regarding the first dialogue opening information. Based on the feedback information, the second dialogue opening information is determined from the first dialogue opening information. The second dialogue opening information includes the dialogue opening information in the first dialogue opening information, and the corresponding feedback information indicates that the dialogue opening information meets the preset positive feedback conditions. The first training data is constructed based on the opening information of the second dialogue. The first training data is used to fine-tune the large model that generates the opening information of the first dialogue.

[0110] As an alternative, the above-mentioned device also includes a fine-tuning training module (not shown in the figure), used for: Based on the opening dialogue information of samples with various styles and the corresponding sample opening dialogue reference information, a second training data is constructed. The second training data is used to fine-tune the large model that generates the opening dialogue information of the first dialogue.

[0111] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0112] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0113] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0114] like Figure 4As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0115] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0116] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 executes the dialogue opening line generation method described above. For example, in some embodiments, the dialogue opening line generation method described above can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the dialogue opening line generation method described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to execute the dialogue opening line generation method described above by any other suitable means (e.g., by means of firmware).

[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0121] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0122] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0123] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0124] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating a dialogue opening, comprising: The quality of at least one frame of video displayed during the startup playback phase of the target video is evaluated, and the evaluation result is obtained. Based on the evaluation results, the opening reference information of the target video is determined; Based on the opening remarks reference information, first dialogue opening remarks information for initiating interactive dialogue is generated.

2. The method according to claim 1, wherein, The determination of the opening reference information for the target video based on the evaluation results includes: In response to the evaluation result indicating that the video frame is valid, opening remarks reference information is determined based on the video frame.

3. The method according to claim 2, wherein, The determination of opening remarks reference information based on the video footage includes: The information of the entities in the video frame is used as reference information for the opening remarks, or... The information of the entity objects in the video footage and the relevant knowledge of the entity objects retrieved are used as reference information for the opening remarks.

4. The method according to claim 2 or 3, wherein, The opening remarks reference information also includes at least one type of contextual information not derived from the video footage.

5. The method according to any one of claims 2-4, wherein, The step of generating first dialogue opening information for initiating interactive dialogue based on the opening reference information includes: The first dialogue opening information is generated based on the opening reference information using a multimodal large model.

6. The method according to claim 1, wherein, The determination of the opening reference information for the target video based on the evaluation results includes: In response to the evaluation result indicating that the video frame is invalid, at least one piece of contextual information not originating from the video frame is determined as the opening remarks reference information.

7. The method according to claim 6, wherein, The step of generating first dialogue opening information for initiating interactive dialogue based on the opening reference information includes: Based on the context information, the first dialogue opening information is generated using a large text model.

8. The method according to any one of claims 1-7, wherein, The quality of the video footage is assessed, and the assessment results are as follows: The video image is evaluated for quality based on at least one of the video image's sharpness, brightness, and information entropy, and the evaluation result is obtained.

9. The method according to any one of claims 1-8, wherein, The target video is a video stream captured by the image acquisition device of the user terminal after video interaction is initiated.

10. The method according to any one of claims 1-9, wherein, After generating the first dialogue opening information for initiating an interactive dialogue based on the opening reference information, the method further includes: Obtain feedback from multiple users regarding the opening remarks of the first dialogue; Based on the feedback information, a second dialogue opening message is determined from the first dialogue opening message. The second dialogue opening message includes the dialogue opening message in the first dialogue opening message that corresponds to the feedback information indicating that the preset positive feedback conditions are met. First training data is constructed based on the second dialogue opening information, and the first training data is used to fine-tune the large model that generates the first dialogue opening information.

11. The method according to any one of claims 1-10, wherein, The method further includes: Based on sample dialogue opening information of various styles and corresponding sample opening reference information, a second training data is constructed. The second training data is used to fine-tune the large model for generating dialogue opening information.

12. A dialogue opening line generation device, comprising: The video quality assessment module is used to assess the quality of at least one frame of video displayed during the startup playback phase of the target video and obtain the assessment result. An opening remarks reference information determination module is used to determine the opening remarks reference information of the target video based on the evaluation results. The opening statement generation module is used to generate first dialogue opening statement information for initiating interactive dialogue based on the opening statement reference information.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images