Advertisement pushing method and device based on large model, electronic equipment and medium

Abstract

The invention provides an advertisement pushing method and device based on a large model, electronic equipment and a medium, and relates to the technical field of artificial intelligence, in particular to the technical field of large language models and personalized recommendation. According to the specific implementation scheme, in the playing process of a current video, the watching time information of the current video of a user is determined based on a first operation behavior executed by the user on the current video; and obtaining advertisement preference information of the user for the pushed advertisement, wherein the advertisement preference information is determined based on a second operation behavior executed by the user for the pushed advertisement. And acquiring emotion information of the user in the playing process of the pushed advertisement, and acquiring scene information of a plurality of video clips included in the current video. Then, based on the watching time information, the advertisement preference information, the emotion information and the scene information, utilizing a large language model to determine an advertisement to be pushed and to-be-inserted time of the advertisement to be pushed in the playing process of the current video; the more immersive video watching experience of the user is guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the field of large language model (LLM) and personalized recommendation technology. Background Art

[0002] Pushing ads to users while they are watching a video is a common ad push scenario. For example, after a user clicks on a video to play, an ad is played before the video is played. Another example is playing an ad during a fixed time period while the video is playing. This ad push method increases the amount of ads pushed and improves advertisers' revenue. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and medium for advertising push based on a large model.

[0004] According to a first aspect of the embodiments of the present disclosure, a method for pushing advertisements based on a large model is provided, comprising:

[0005] During playback of the current video, based on a first operation performed by the user on the current video, determining viewing time information of the user on the current video;

[0006] Acquiring advertisement preference information of the user for the pushed advertisement, where the advertisement preference information is determined based on a second operation behavior performed by the user on the pushed advertisement;

[0007] Acquiring emotional information of the user during the playing of the pushed advertisement;

[0008] Acquire scene information of multiple video clips included in the current video;

[0009] Based on the viewing time information, the advertisement preference information, the emotion information and the scene information, a large language model is used to determine the advertisement to be pushed and the time when the advertisement to be pushed is to be inserted during the playback of the current video.

[0010] A second aspect of the embodiments of the present disclosure provides an advertisement push device based on a large model, including:

[0011] A determination module, configured to determine, during playback of a current video, viewing time information of the current video by the user based on a first operation performed by the user on the current video;

[0012] An acquisition module, configured to acquire the user's advertisement preference information for the pushed advertisement, wherein the advertisement preference information is determined based on a second operation performed by the user on the pushed advertisement;

[0013] The acquisition module is further used to acquire the emotion information of the user during the playing process of the pushed advertisement;

[0014] The acquisition module is further used to acquire scene information of multiple video clips included in the current video;

[0015] The determination module is further used to determine the advertisement to be pushed and the time to insert the advertisement to be pushed during the playback of the current video based on the viewing time information, the advertisement preference information, the emotion information and the scene information using a large language model.

[0016] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method according to the first aspect.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute a method according to any one of the first aspects.

[0021] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method according to any one of the first aspects is implemented.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0024] Figure 1 is a flow chart of an advertisement push method based on a large model provided by an embodiment of the present disclosure;

[0025] Figure 2 is a flow chart of a method for determining advertisement preference information provided by an embodiment of the present disclosure;

[0026] Figure 3is a flow chart of a method for determining emotion information provided by an embodiment of the present disclosure;

[0027] Figure 4 is an exemplary schematic diagram of a process of determining emotion information provided by an embodiment of the present disclosure;

[0028] Figure 5 is a flow chart of a method for determining scene information provided by an embodiment of the present disclosure;

[0029] Figure 6 is a flow chart of another large model-based advertising push method provided by an embodiment of the present disclosure;

[0030] Figure 7 is an exemplary schematic diagram of an advertisement push process based on a large model provided by an embodiment of the present disclosure;

[0031] Figure 8 It is a structural schematic diagram of an advertisement push device based on a large model provided by an embodiment of the present disclosure;

[0032] Fig. 9 It is a block diagram of an electronic device used to implement the large model-based advertisement push method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] Traditional advertising strategies mostly use a fixed timeline insertion method, that is, playing ads within a fixed time period of video playback. However, with the development of online video platforms, traditional advertising methods are difficult to effectively attract users. A single advertising push time can easily interrupt users from watching videos, affecting their smooth viewing experience, and it is difficult to significantly increase the click-through rate and conversion rate of ads.

[0035] In order to solve the above problems, the embodiments of the present disclosure provide an advertisement push method based on a large model, which is applied to an electronic device, such as a server, a desktop computer, or a laptop computer, etc., which has data processing capabilities. Figure 1 As shown, the advertisement push method based on the big model provided by the embodiment of the present disclosure includes the following steps:

[0036] S101. During playback of a current video, determine, based on a first operation performed by a user on the current video, a viewing time information of the user on the current video.

[0037] For the convenience of description, the embodiment of the present disclosure refers to the operation behavior performed by the user on the current video as the first operation behavior, and the first operation behavior can be various operation behaviors supported by the video. Among them, the operation behavior has attributes such as operation type, operation time and operation user. For example, the operation type includes: play, close, skip, fast forward, pause and rewind, etc. The operation time represents the playback time of the video and the system time of the playback terminal when the user performs the operation behavior. The operation user is the user identifier (identifier, ID) of the user who performs the operation behavior. For example, the operation user can be the user ID of the logged-in user of the video playback terminal.

[0038] The viewing time information may include: cumulative viewing time, continuous viewing time period, skip time period, fast forward time period and rewind time period, etc. Each continuous viewing time period is the time period between a play action and a stop play action, wherein the stop play action includes closing, skipping, fast forwarding, pausing and rewinding, etc. The cumulative viewing time is the sum of each continuous viewing time period.

[0039] Optionally, the playback terminal of the current video may send the first operation behavior to the electronic device whenever receiving a first operation behavior during the playback of the current video, or periodically send the first operation behavior received in the previous cycle to the electronic device. Alternatively, the playback terminal may also send the first operation behavior to the electronic device in other ways, which is not specifically limited in the embodiments of the present disclosure.

[0040] Through the user's viewing time information, the user's interest distribution in the video content can be analyzed. For example, the user is more interested in video clips that are watched continuously for a long time, and is less interested in skipped video clips.

[0041] S102: Obtaining the user's advertisement preference information for the pushed advertisement, wherein the advertisement preference information is determined based on the second operation behavior performed by the user on the pushed advertisement.

[0042] Pushed ads include ads that have been inserted into the current video and ads that have been inserted into previous videos that the user has watched.

[0043] For the convenience of description, the embodiment of the present disclosure refers to the operation behavior performed by the user on the pushed advertisement as the second operation behavior, and the second operation behavior can be various operation behaviors supported by the advertisement.

[0044] The second operation performed by the user on the pushed advertisement can reflect the user's preference for the pushed advertisement. For example, if the user clicks pause or close during the advertisement playback, it can reflect that the user has low interest in the advertisement; if the user clicks the product link in the advertisement to jump to the product details page, or the user clicks the call to action (CTA) button in the advertisement to participate in the advertisement interaction, it can reflect that the user has high interest in the advertisement. Therefore, based on the second operation behavior, the user's advertising preference information for the pushed advertisement can be determined.

[0045] S103: Acquire the user's emotional information during the playback of the pushed advertisement.

[0046] The emotion information may include the emotion type, emotion intensity, and time period corresponding to the emotion type. The emotion information of the user may reflect the user's concentration when watching the video. For example, positive emotion types include excitement, pleasure, and happiness, while negative emotion types include anxiety, sadness, boredom, and fear. In addition, emotion types also include neutrality, doubt, and surprise, where neutrality means no obvious emotional fluctuations, such as relaxation and concentration.

[0047] S104: Obtain scene information of multiple video clips included in the current video.

[0048] The scene information may include: the video clip includes the beginning, development, climax or ending plot.

[0049] It should be noted that S101 to S104 can be executed in sequence or in parallel, and the execution order of S101 to S104 is not limited to Figure 1 in the order shown.

[0050] S105: Based on the viewing time information, the advertisement preference information, the emotion information and the scene information, the large language model is used to determine the advertisement to be pushed and the time to insert the advertisement to be pushed during the playback of the current video.

[0051] In the disclosed embodiment, the viewing time information can reflect the distribution of user interests at different playback times when watching a video, the scene information can represent the content of different video clips, and the advertising preference information and emotional information can reflect the user's interest in different advertisements. Therefore, by combining the viewing time information, advertising preference information, emotional information, and scene information, the user's preference for advertisements and video content can be obtained, and the advertisements to be pushed can be determined, which can increase the possibility that the advertisements to be pushed will be liked by the user. Moreover, by combining the viewing time information, advertising preference information, emotional information, and scene information to determine the time to be inserted, the possibility of the advertisement insertion time being at the playback time of a video clip with a high degree of user interest can be reduced, and the possibility of the advertisement insertion time being at the playback time of a video clip with a low degree of user interest can be increased, thereby reducing the interruption of advertisement playback to the user's video viewing and ensuring a more immersive video viewing experience for the user.

[0052] The following is a detailed description of the advertising push method based on the large model provided by the embodiment of the present disclosure:

[0053] In some embodiments of the present disclosure, the user's advertisement preference information for pushed advertisements in S102 includes the user's preference for various advertisement types. The advertisement types can be divided according to the advertisement content, for example, the advertisement types include: electronic products, cosmetics or sports equipment, etc.

[0054] See also Figure 2 The electronic device in S102 determines the user's advertisement preference information for the pushed advertisement based on the second operation performed by the user on the pushed advertisement, including:

[0055] S201: Determine a first preference score of a user for a first pushed advertisement based on a type of a first operation performed by a user on a first pushed advertisement, wherein the first pushed advertisement is a pushed advertisement that has been inserted during playback of a current video.

[0056] A data collection script may be pre-deployed in the playback terminal of the current video. Through the data collection script, the playback terminal may send the first operation behavior performed by the user on the first pushed advertisement to the electronic device through a communication protocol (web socket) during the playback of the first pushed advertisement. The first operation behavior has parameters such as advertisement ID, operation behavior ID, and operation time. In addition, the first operation behavior may also have other parameters, such as advertisement type, operation behavior score, and scene information of the video clip before the advertisement is inserted.

[0057] Each type of operation behavior is preset with a corresponding operation behavior score. For example, the operation behavior score of clicking an ad link is 5, the operation behavior score of skipping is 0, the operation behavior score of pausing or closing is -5, and the operation behavior score of re-watching is 3. For each first pushed advertisement, the sum or weighted sum of the operation behavior scores corresponding to the type of the first operation behavior performed on the first pushed advertisement can be used as the first preference score of the first pushed advertisement.

[0058] In addition, for each first pushed advertisement, if during the playback of the first pushed advertisement, no negative operation behavior is received from the user on the first pushed advertisement, for example, negative operation behaviors include pause, close and fast forward, etc., it means that the user has watched the first pushed advertisement in its entirety, and the first preference score of the first pushed advertisement can be increased by a default value, for example, a default value of 2.

[0059] S202: Determine the advertisement type to which the first pushed advertisement belongs.

[0060] The electronic device may pre-store the advertisement type to which each advertisement belongs, and thus may obtain the advertisement type to which the first pushed advertisement belongs.

[0061] S203: Obtain the user's preference for the advertisement type.

[0062] The preference is determined based on a second preference score of the user for a second pushed advertisement, where the second pushed advertisement is a pushed advertisement of the advertisement type inserted in a video that the user has watched in the past.

[0063] S204. Update the user's preference for the advertisement type according to the first preference score, and obtain updated user's preference for various advertisement types.

[0064] For this advertisement type, the weighted sum of the first preference score and the second preference score can be used as the user's preference for this advertisement type. The weight of each preference score is a time interval decay weight, that is, the farther the advertisement push time is from the current moment, the lower the weight of the preference score. For example, for an advertisement that has been pushed more than one month from the current moment, the weight corresponding to its preference score is 0.2; for an advertisement that has been pushed more than one week but less than one month from the current moment, the weight corresponding to its preference score is 0.5; for an advertisement that has been pushed less than one week from the current moment, the weight corresponding to its preference score is 1. Through this calculation method, the user's preference for the advertisement type can better reflect the user's recent advertisement preference.

[0065] Alternatively, for the advertisement type, the average of the first preference score and the second preference score may be used as the user's preference for the advertisement type. Alternatively, the user's preference for the advertisement type may be updated in other ways, which are not specifically limited in the embodiments of the present disclosure.

[0066] Through the above method, the embodiment of the present disclosure can update the user's preference for various types of advertisements based on the user's operation behavior on the advertisements inserted in the current video, so that the user's preference for various types of advertisements can be updated in real time, ensuring the accuracy of the user's advertising preferences reflected therein.

[0067] In some embodiments of the present disclosure, the emotion information in S103 includes the average emotion score of the user when watching the pushed advertisement. The electronic device can determine the emotion information of the user in real time during the playback of the pushed advertisement before the advertisement is recommended. The manner of determining the emotion information includes the following two methods.

[0068] See also Figure 3 ,The first way to determine emotional information includes the following steps:

[0069] S301. During the playback of the pushed advertisement, a user video shot of the user is obtained.

[0070] It should be noted that the electronic device obtains the emotional information of each pushed advertisement in the same way. Figure 3 The relevant steps are described using a pushed advertisement as an example.

[0071] The playback terminal to which the pushed advertisement has been placed has a built-in camera, or can be connected to an external camera, so that the playback terminal can shoot the user through the camera during the playback of the pushed advertisement to obtain the user video. If the user grants the user video transmission permission in advance, the playback terminal can send the user video to the electronic device.

[0072] For example, the playback terminal can play videos and advertisements in the form of applications, web pages, or mini-programs. The playback terminal can display a permission acquisition agreement when the application, web page, or mini-program is opened for the first time. The permission acquisition agreement is used to apply for multiple permissions, including user video transmission permissions. When the user agrees to the permission acquisition agreement, it indicates that the user has granted various permissions including user video transmission permissions. Alternatively, the playback terminal can display a pop-up window for applying for user video transmission permissions when the application, web page, or mini-program is opened for the first time. When the user clicks "Agree" in the pop-up window, it indicates that the user has granted user video transmission permissions. Alternatively, the user video transmission permission can also be obtained by other means, which is not specifically limited in the embodiments of the present disclosure.

[0073] Optionally, the playback terminal may send the captured user video to the electronic device every time a user video of a preset length is captured; or, after the playback of the pushed advertisement is completed, the playback terminal may send the user video captured during the time period from the start of the playback of the pushed advertisement to the completion of the playback to the electronic device.

[0074] S302: for video frames included in the user video, identify the facial expressions and body language of the user in the video frames through an emotion recognition model, and output the probability of the user in the video frames being in various emotion types based on the facial expressions and body language.

[0075] Among them, the emotion recognition model can be a visual algorithm model based on deep learning, such as Convolutional Neural Networks (CNN).

[0076] Each video frame can be input into an emotion recognition model, and the emotion recognition model can identify the facial area and limb area in the video frame, and extract features from the facial area, so as to identify facial expressions based on the extracted features. In addition, the emotion recognition model can also extract features from the limb area, so as to identify body language based on the extracted features, and then determine the probability of the user in the video frame being in various emotional types based on the facial expression and body language. For example, if the facial area includes a smile, it can reflect the user's excitement; if the facial area includes a frown, it can reflect the user's doubt. For another example, if the sitting posture included in the limb area is sitting upright, it can reflect the user's happiness; if the sitting posture included in the limb area is sitting with legs crossed, it can reflect the user's doubt.

[0077] S303: Determine the emotion score of the video frame based on the probability that the user is in various emotion types in the video frame.

[0078] For each video frame included in the user video, the weighted sum of the probabilities of the user being in various emotion types in the video frame can be used as the emotion score of the video frame. The weight of the probability of the emotion type can be preset.

[0079] For example, assuming that in a video frame, the probability of the user being excited is 0.6, the probability of being happy is 0.25, the probability of being confused is 0.1, and the probability of being bored is 0.05; and the weight of the probability of excitement is 1, the weight of the probability of happiness is 0.5, the weight of the probability of confusion is -0.3, and the weight of boredom is -1. The emotion score of the video frame is: 0.6×1+0.25×0.5+0.1×(-0.3)+0.05×(-1)=0.717.

[0080] The emotion score of each video frame ranges from [0 to 1]. The closer the emotion score is to 0, the less focused the user is on the ad, that is, the less interested the user is; the closer the emotion score is to 1, the more focused the user is on the ad, that is, the more interested the user is.

[0081] S304: Smoothing the emotion scores of consecutive video frames to obtain an average emotion score.

[0082] The emotion scores of each N consecutive video frames included in the user video may be smoothed to obtain an average emotion score, where N is a preset positive integer, for example, N=5.

[0083] The average emotion score can be calculated by using a time-weighted sliding average method. That is, for N consecutive video frames, the sum of the product of the emotion score of each video frame and the time decay factor is calculated, and the ratio of the sum to the sum of the time decay factors is used as the average emotion score. The time decay factor has a value range of [0,1]. The earlier the video frame is played, the smaller the time decay factor corresponding to its emotion score is.

[0084] The disclosed embodiment combines the user's facial expressions and body language to more comprehensively identify the user's emotions when watching videos, which can improve the accuracy of identifying emotion types. In addition, since the recognition results of the emotion recognition model may have errors, the emotion scores of consecutive video frames are smoothed to improve the accuracy of the average emotion score in reflecting the user's interest in the pushed advertisements, which can reduce the impact of the recognition errors of the emotion recognition model on advertisement recommendations and improve the accuracy of advertisement recommendations based on emotion information.

[0085] The second method of determining emotional information includes the following steps:

[0086] Step 1: During the playback of the pushed advertisement, multiple groups of probabilities sent by the playback terminal of the pushed advertisement are received.

[0087] The single group probability represents the probability of the user being in various emotion types at a time. The playback terminal can determine the probability of the user being in various emotion types in each video frame included in the user video in the manner of S302, and take the probability corresponding to each video frame as a group.

[0088] Optionally, the playback terminal that has pushed the advertisement can determine the probability of the user being in various emotion types in each video frame in real time during the playback of the pushed advertisement, and send a set of probabilities determined in real time to the electronic device. Alternatively, after obtaining a certain number of groups of probabilities, the playback terminal can send the obtained multiple groups of probabilities to the electronic device. Alternatively, the playback terminal can also send multiple groups of probabilities to the electronic device in other ways, which is not specifically limited in the embodiments of the present disclosure.

[0089] Step 2: Based on the same set of probabilities, determine the user's emotion score at a moment.

[0090] The method of determining the emotion score in step 2 is the same as that in the above S303 , and reference may be made to the description of the above S303 , which will not be repeated here.

[0091] Step 3: Smooth the continuous emotion scores to obtain the average emotion score.

[0092] The method of determining the average emotion score in step 3 is the same as that in the above S304 , and reference may be made to the description of the above S304 , which will not be repeated here.

[0093] In the disclosed embodiment, the playback terminal can collect and analyze user videos to obtain the probability of the user being in various emotions, and only send the probability of the user being in various emotion types to the electronic device, thereby avoiding the privacy leakage problem that may occur when transmitting user videos and ensuring security. In addition, since there may be errors in the emotion scores at the same time, the emotion scores of consecutive video frames are smoothed, which improves the accuracy of the average emotion score in reflecting the user's interest in the pushed advertisements, and improves the accuracy of advertisement recommendations based on emotion information.

[0094] The implementation process of the second method of determining emotional information can be found in Figure 4 , the playback terminal that has pushed the advertisement uses a camera to collect user videos during the playback of the pushed advertisement, and then inputs each video frame included in the user video collected by the camera into the emotion recognition model in the playback terminal, and obtains the probability of the user being in various emotion types in each video frame output by the emotion recognition model, and obtains multiple groups of probabilities. After that, the playback terminal sends each group of probabilities output by the recognition model to the server, and the server sends each group of probabilities received to the advertising system (Advertise System, AdvertiseSys) configured by itself, so that the advertising system determines the user's emotional information during the playback of the pushed advertisement based on each group of probabilities received. The advertising system is a software module that implements the large model-based advertising push method provided in the embodiment of the present disclosure.

[0095] It should be noted that there may be a situation where a user video includes multiple users. In this case, the facial areas and body areas of different users in the user video can be segmented, and the emotional information of the user can be determined based on the facial area and body area of ​​the same user.

[0096] Afterwards, the average value of the emotion information of each user may be used as the emotion information of each user during the playing process of the pushed advertisement.

[0097] Alternatively, the main viewers may be identified from the users, and the emotional information of the main viewers may be used as the emotional information of the users during the playback of the pushed advertisements. The main viewers may be the owner of the login account of the playback terminal, the user who mainly performs the operation, or the user with the highest concentration.

[0098] Alternatively, advertisements may be recommended for each user, and the advertisements pushed to the user may be displayed at a designated position of the current video, wherein the designated position may be determined based on the relative position between the user and the display screen of the playback terminal.

[0099] Alternatively, when it is detected that the user video includes multiple users, the playback terminal may display inquiry information and receive interest information selected by the user as the emotion information of each user.

[0100] In some embodiments of the present disclosure, the scene information in the above S104 includes: the emotion type and scene type of each video clip in the current video, the emotion transition time period and scene transition time period in the current video, and the playback time period of the specified object in the current video.

[0101] The scene information of the multiple video clips included in the current video can be predetermined, or determined in real time during the advertisement push process. Figure 5 , the scene information is determined by the following steps:

[0102] S501: Divide the current video into multiple video segments.

[0103] Optionally, the current video may be divided according to a preset fixed duration to obtain a plurality of video segments of fixed duration, for example, the fixed duration is 10 seconds or 30 seconds.

[0104] Alternatively, an image processing model may be used to identify the scene to which each video frame belongs, and consecutive video frames belonging to the same scene may be divided into a video segment. For example, a shot switching model may be used to identify whether each video frame is a shot switching video frame, and the video frames between adjacent shot switching video frames may be divided into a video segment.

[0105] Alternatively, an image recognition model may be used to identify whether each video frame is an emotion inflection point video frame, and the video frames between adjacent emotion inflection point video frames are taken as a video segment.

[0106] Alternatively, the current video may be divided in other ways, which are not specifically limited in the embodiments of the present disclosure.

[0107] S502: Using the emotion and content analysis model, perform image feature extraction on the video frames included in the video clip, perform text feature extraction on the text in the video frames, and perform audio feature extraction on the audio included in the video clip, and determine the emotion type of the video clip based on the feature extraction results.

[0108] The emotion and content analysis model is a multimodal model that has image processing capabilities, text processing capabilities, and audio processing capabilities, so it can recognize input images, text, and audio. The emotion and content analysis model is an artificial intelligence (AI) model based on natural language processing (NLP) and CNN, such as a large language model. The large language model used here can be the same model as the large language model used in S105 above, or they can be different models.

[0109] The video frames included in the video clip can reflect the visual tone of the video clip, various elements in the scene, changes in the expressions and body language of people in the scene, and changes in the relationship between people and objects. Therefore, the video frames can reflect the emotions expressed by the video clip.

[0110] The text in the video frame includes subtitles, such as character dialogues and narrations, etc. The text can express the emotions of the characters and the tone of the plot, so the text in the video frame can reflect the emotions expressed by the video clip.

[0111] The audio includes the voice of the characters and the background sound. The voice can reflect the tone of the user's speech, and the background sound can include the ambient sound and background music, which can reflect the emotional tone of the scene. Therefore, the audio included in the video clip can reflect the emotion expressed in the video clip.

[0112] For example, emotion types include: relaxed and cheerful, mildly tense, severely tense, conflicting, and peaceful and relaxed.

[0113] S503: Determine the emotion transition time period based on the emotion type of each video clip.

[0114] It can be determined whether the emotion types of each two adjacent video clips are the same. If they are different, the video frames within a preset time length before and after the division point between the adjacent video clips are used as the emotion transition time period, or the two video clips are used to form the emotion transition time period. Alternatively, the emotion transition time period can also be determined by other methods, which are not specifically limited in the embodiments of the present disclosure.

[0115] For example, assuming that video segments a and b are adjacent video segments, the emotion type of video segment a is relaxed and cheerful, and the emotion type of video segment b is slightly nervous, then the last 10 seconds of video segment a and the first 10 seconds of video segment b form an emotion transition time period.

[0116] S504: Use the object detection and scene recognition model to perform image recognition on the video frames included in the video clip, and output the object detection result of the video clip and the scene type to which it belongs.

[0117] The object detection result indicates whether the video clip includes a preset object and the playback time period of the preset object. For example, the preset objects include: landscape, food, cars, electronic products, pets or cosmetics.

[0118] For example, the scene types to which the video clip belongs include: climax, transition, and smooth, etc.

[0119] The object detection and scene recognition model may be a visual algorithm model based on deep learning. For example, the object detection and scene recognition model may be a CNN. Alternatively, the object detection and scene recognition model includes an object detection sub-model and a scene recognition sub-model, wherein the object detection sub-model identifies the object detection result, and the scene recognition sub-model identifies the scene type, wherein the object detection sub-model may be a target detection model (You Only Look Once, YOLO) or a fast regional convolutional neural network (faster R-CNN), etc., and the scene recognition sub-model may be a residual network (ResidualNetwork, ResNet), etc.

[0120] S505: Determine a scene transition time period based on the scene type to which each video clip belongs.

[0121] The method for determining the scene transition time period is the same as the method for determining the emotion transition time period described above. Please refer to the relevant description of S503 above, and will not be repeated here.

[0122] During the emotional transition period, the emotional atmosphere of the video changes, for example, from relaxed to tense, or from conflict to ease, etc. At this time, the content and plot of the video generally change, that is, the video content is generally not continuous. Similarly, during the scene transition period, the shooting scene of the video changes, so the video content is generally not continuous. Therefore, inserting advertisements during the emotional transition period or the scene transition period will generally not disturb the user. Therefore, the emotional transition period and the scene transition period have a certain reference value for the subsequent determination of the advertisements to be pushed and the time to be inserted.

[0123] Moreover, after obtaining the emotion type and scene type of the video clip, when recommending advertisements, it is possible to avoid recommending advertisements that do not match the emotion type or scene type of the video clip, such as inserting humorous advertisements into sad video clips, or inserting relaxing advertisements into tense video clips, or inserting interactive advertisements that may cause pauses into slow-paced video clips, etc., thereby improving the matching degree between the recommended advertisements and the emotion type or scene type of the video clips. For example, inserting travel advertisements into video clips of natural scenery such as mountains and rivers, inserting game advertisements or advertisements for game supporting hardware into video clips including e-sports, etc. Therefore, the emotion type and scene type of the video clip have a certain reference value for the subsequent determination of the advertisements to be pushed and the time to be inserted.

[0124] In addition, after obtaining the playback time period of the preset object included in the current video, when recommending advertisements, advertisements related to the preset object can be obtained as advertisements to be pushed. For example, pet product advertisements can be played during the playback time period of pets. Therefore, the playback time period of the preset object has a certain reference value for the subsequent determination of the advertisements to be pushed and the time to be inserted.

[0125] In some embodiments of the present disclosure, see Figure 6 The electronic device S105 determines the advertisement to be pushed and the time to insert the advertisement to be pushed during the playback of the current video based on the advertisement preference information, the emotion information and the scene information using the large language model, including the following steps:

[0126] S601: Determine a plurality of recommended insertion times based on scene information, wherein the recommended insertion times include the play times of video clips belonging to non-climax scenes.

[0127] For example, non-climax scenes include transition scenes and flat scenes, etc.

[0128] The emotional transition time period and the scene transition time period can be preset as the highest priority, the playback time period of the climax scene video clip can be preset as the lowest priority, and the playback time period of the video clip including the specified object in the remaining video clips can be preset as the middle priority. When executing S601, the video clips can be sorted in order from high to low priority according to the scene information, and multiple time periods can be selected as recommended insertion time. The number of selected time periods can be preset according to actual needs.

[0129] S602: Filter out an advertisement insertion trigger time from each recommended insertion time based on the user's current emotion type.

[0130] The user's emotion type within a preset time period from the current moment can be obtained as the current emotion type. If the concentration represented by the current emotion type is too high or too low, for example, the current emotion type is excitement or boredom, and the time difference between the start time of the recommended insertion time and the current moment is less than the preset time difference, the recommended insertion time is filtered out and the remaining recommended insertion times are retained.

[0131] S603: When the advertisement insertion trigger time is reached, the viewing time information, advertisement preference information, emotion information and scene information are input into the large language model, and the large language model is used to predict the click rate of each advertisement by the user, and when the click rate meets the preset conditions, the candidate advertisements, candidate advertisement versions and candidate insertion times are output based on the click rate, and when the click rate does not meet the preset conditions, the candidate advertisements, candidate advertisement versions and candidate insertion times are output based on the candidate advertisements. The advertisement version indicates the display form of the advertisement content.

[0132] In the disclosed embodiment, the viewing time information, advertisement preference information, emotion information and scene information can be updated in real time. When the advertisement insertion trigger time is reached, the latest viewing time information, advertisement preference information, emotion information and scene information can be input into the large language model. The large language model is used to predict the user's click rate for each advertisement and the degree of match between each advertisement and the current video, and based on the click rate and the degree of match, candidate advertisements, candidate advertisement versions and candidate insertion times are output.

[0133] In addition to viewing time information, advertising preference information, emotion information and scene information, other information can also be input into the large language model, such as the first operation behavior performed by the user on the current video, the second operation behavior performed by the user on the pushed advertisement, and the advertisement type and content information of each advertisement, such as whether the content information contains preset objects and the degree of match between the advertisement and the current video.

[0134] The ad version refers to the display format of the ad content. Different ad versions of the same ad have the same promotional objectives but different forms of expression or styles. For example, ad versions include: short version, long version, emotional, data-based, interactive, and passive viewing.

[0135] The large language model can be a multimodal pre-trained model based on a transformer structure. Through the attention mechanism inside the large language model, it can efficiently estimate the attractiveness of each advertisement to the user in the current video playback scenario, that is, the user's click-through rate (CTR) of the advertisement. Afterwards, it is determined whether the predicted CTR meets the preset conditions, for example, the preset conditions include: the number of CTRs exceeding the preset value exceeds the preset number. The large language model can output candidate advertisements, candidate advertisement versions, and candidate insertion times based on CTR when CTR meets the preset conditions, and output candidate advertisements, candidate advertisement versions, and candidate insertion times based on alternative advertisements when CTR does not meet the preset conditions.

[0136] S604: Determine the advertisement to be pushed based on the candidate advertisements and the candidate advertisement versions, and determine the insertion time based on the candidate insertion time.

[0137] The disclosed embodiment can filter out the time period containing the climax plot of the current video based on the scene information. Since the climax plot is more exciting than other plots, the user needs to be undisturbed more. Therefore, filtering out the time period containing the climax plot can reduce the disturbance to the user caused by playing advertisements in the video. The disclosed embodiment also filters the advertisement insertion trigger time based on the current emotion type, thereby avoiding inserting advertisements when the user's concentration is too high, causing disturbance to the user, or reducing the number of advertisements recommended to the user when the user's concentration is too low, avoiding aggravating the user's negative emotions. Moreover, the disclosed embodiment also recommends advertisements based on viewing time information, advertisement preference information, emotion information and scene information, thereby increasing the possibility that the determined candidate advertisements and candidate advertisement versions are liked by the user and match the current video.

[0138] In some embodiments of the present disclosure, the method in which the above-mentioned S602 electronic device filters out the advertising insertion trigger time from each recommended insertion time based on the user's current emotion type can be implemented as follows: based on the user's current emotion type and the scene information of the video clip included in the recommended insertion time, determine the predicted emotion type of the user at the recommended insertion time, and then filter out the advertising insertion trigger time from each recommended insertion time based on the predicted emotion type.

[0139] The deep learning emotion curve model can be used to predict the emotion type of the user at the recommended insertion time based on the current emotion type and the scene information of the video clips included in the recommended insertion time, and output the score suitable for inserting advertisements in the recommended insertion time based on the emotion type. Afterwards, multiple recommended insertion times can be selected in descending order of scores as the advertisement insertion trigger time. The number of selected recommended insertion times can be preset according to actual needs.

[0140] Through the above method, the embodiment of the present disclosure can predict the user's emotions during the recommended insertion time, reduce the insertion of advertisements when the user's concentration on the video is too high or too low, reduce the impact of advertising push on the user's video viewing, ensure the user's immersive viewing experience, and reduce the aggravation of the user's negative emotions caused by advertising push.

[0141] In the embodiment of the present disclosure, the above-mentioned S604 determines the method of the advertisement to be pushed based on the candidate advertisements and the candidate advertisement versions, which can be implemented as follows: select a target candidate advertisement from each candidate advertisement, and then filter out the version to be pushed from the candidate advertisement versions according to the total length of the current video, the user's current emotion type and the user's historical operation behaviors on each advertisement version, and use the target candidate advertisement of the push version as the advertisement to be pushed.

[0142] In one implementation, the candidate advertisements output by the large language model also have candidate scores, so the candidate advertisement with the highest candidate score can be selected as the target candidate advertisement.

[0143] Since the candidate score can reflect the possibility that the candidate ad is liked by the user and the degree of match between the candidate ad and the current video, selecting the candidate ad with the highest candidate score can increase the possibility that the ad recommended to the user is liked by the user, thereby increasing the ad click-through rate, increasing advertiser revenue, and reducing the interruption of the recommended ad to the user's video viewing.

[0144] In another implementation, videos that are associated with the current video and videos of interest to the user may be obtained as related videos, and then target candidate advertisements may be selected based on the correlation between the related videos and the candidate advertisements.

[0145] The videos associated with the current video include: other videos in the same series as the current video, and videos with the same theme or similar plot as the current video. For example, if the current video is an episode of a TV series, the videos associated with the current video include: other episodes of the TV series, and sequels and prequels of the TV series. For another example, if the current video is the first part of a movie, the videos associated with the current video include other parts of the movie. For another example, if the current video is a video about pet training, the videos associated with the current video include videos introducing pet training props.

[0146] The user's videos of interest include: videos that the user has watched more than a preset number of times within a unit historical time period, videos that are liked by the user, and videos that are collected by the user.

[0147] The correlation between each candidate advertisement and each related video may be calculated, and the candidate advertisement with the highest correlation may be selected as the target candidate advertisement.

[0148] For example, if the relevant video is a science fiction video, then science fiction-related advertisements can be selected as target candidate advertisements.

[0149] Since the current video has a content connection with other videos in the same series, selecting ads that are more relevant to related videos for recommendation can improve the coherence of ads played in the same series of videos and promote users' click-through rate on ads. Moreover, ads that are more relevant to videos that users are interested in are more likely to be liked by users, so they are more likely to be clicked by users after being pushed, increasing advertisers' revenue.

[0150] It is understandable that when the total duration of the current video is short, it is not suitable to insert an advertisement with a longer duration; conversely, when the total duration of the current video is long, an advertisement with a longer duration can be inserted.

[0151] Therefore, after determining the target candidate advertisement, the electronic device can determine the longest advertisement duration according to the total duration of the current video, for example, from a plurality of preset duration intervals, determine the target duration interval to which the total duration of the current video belongs, and then use the advertisement duration corresponding to the target duration interval as the longest advertisement duration. Then, from each advertisement version of the target candidate advertisement, filter out the advertisement version whose duration exceeds the longest advertisement duration, and retain the remaining advertisement versions.

[0152] Furthermore, from each advertisement version of the target candidate advertisement, the advertisement version that does not match the current emotion type of the user is filtered out, and the remaining advertisement versions are retained. For example, when the current emotion type is positive, the advertisement version with a negative emotion type is filtered out; when the current emotion type is negative, the advertisement version with a positive emotion type is filtered out.

[0153] And, based on the historical operation behaviors performed by the user on each advertisement version, the user's preference for each advertisement version is determined, and then the advertisement versions with a preference less than a preset threshold are filtered out, and the remaining advertisement versions are retained.

[0154] Through the above method, the embodiment of the present disclosure can filter out target candidate advertisements from the candidate advertisements output by the large language model, and select the advertisement version that best matches the current video, best matches the user's current emotion and is most likely to be liked by the user based on the total length of the current video, the user's current emotion type and the user's historical operation behaviors on each advertisement version.

[0155] In some embodiments of the present disclosure, after determining the push version, the electronic device may also determine the grade of the advertisement, wherein the grade indicates the degree of guidance on the click behavior. For example, the grades of advertisements include: elementary, intermediate and advanced.

[0156] Among them, primary refers to advertisements with weak guiding effect on click behavior, which are designed to attract users' initial attention, cultivate users' initial interest in the promotion object, and pave the way for subsequent levels of advertisements. Primary advertisements are short in duration, usually only 5 to 10 seconds, and the introduction of the promotion target is relatively simple. For example, primary advertisements are preview advertisements, brand appearance advertisements, and advertisements with simple attraction points.

[0157] Intermediate refers to ads that have a certain degree of guidance for click behavior, usually including multiple selling points of the promoted object and brand image details, aiming to deepen the user's impression of the promoted object and appropriately guide the user to interact. Intermediate ads have a medium duration, generally contain a relatively detailed introduction, and may have a small number of interactive buttons or product links.

[0158] Advanced refers to ads that are more likely to lead to clicks. The core goal is to drive users to click through strong calls to action, thereby generating purchases, registrations, or other events. Advanced ads are generally longer in duration, with a more comprehensive introduction to the promotion object and more complete logic, and usually include multiple product links and / or interactive buttons.

[0159] On this basis, the electronic device uses the push version of the target candidate advertisement as the advertisement to be pushed, including the following steps:

[0160] Step 1: When the user's current emotion type is a positive type, the user's historical operation behaviors on the target candidate advertisement are obtained as the target operation behaviors.

[0161] Before the user watches the current video, the user may have been pushed the target advertisement when watching other videos, so the historical operation behavior performed by the user on the target candidate advertisement can be obtained as the target operation behavior.

[0162] Step 2: Obtain the historical rankings of the target candidate ads that the user has watched.

[0163] For the convenience of description, the embodiment of the present disclosure refers to the level of the target candidate advertisements that the user has watched as the historical level.

[0164] In the disclosed embodiment, the grade of an advertisement may be preset according to actual conditions such as the duration of the advertisement, the type of content, and the number of product links and interactive buttons included in the advertisement.

[0165] Step 3: Based on the target operation behavior and the historical level, the push level is determined, and the target candidate advertisement of the push version of the push level is used as the advertisement to be pushed.

[0166] It can be determined whether the number of active target operation behaviors reaches a preset number. If yes, the previous level of the highest historical level can be used as the push level. If not, the highest historical level is used as the push level.

[0167] When the user's current emotion type is positive, it means that the user is in a good mood and may interact with the advertisement. Therefore, the push level is determined based on the target operation behavior and historical level. Higher-level advertisements can be pushed to the user. Through long-term preparation and guidance, the user will click on the advertisement, and then purchase the product or register the user, thereby maximizing the interests of the advertiser.

[0168] In some embodiments of the present disclosure, the manner in which the electronic device determines the time to be inserted based on the candidate insertion time in S604 may be implemented as follows: determining the time to be inserted based on preset restriction parameters and the candidate insertion time.

[0169] The restriction parameters include at least one of the following: a minimum time interval between adjacent advertisements, a maximum number of advertisement insertions in a single video, a maximum number of advertisement insertions per unit time, and a minimum time interval for pushing the same advertisement to the user.

[0170] That is, obtain the time of the last advertisement played during the current video playback process, and determine whether the time interval between this time and the candidate insertion time exceeds the minimum time interval between adjacent advertisements. If so, the candidate insertion time can be deleted. Also, calculate the sum of the number of advertisements inserted in the current video and the number of candidate insertion times, and determine whether the sum value exceeds the maximum number of advertisement insertions in a single video. If so, some candidate insertion times can be deleted so that the sum value does not exceed the maximum number of advertisement insertions in a single video. Also determine whether the time interval between the last time the advertisement to be pushed to the user and the candidate insertion time exceeds the minimum time interval for pushing the same advertisement to the user. If so, delete the candidate insertion time. Before deleting a candidate insertion time at any time, if there is only one candidate insertion time left, the candidate insertion time may not be deleted, and the candidate insertion time may be adjusted to meet the judgment condition.

[0171] Optionally, the restriction parameters may also include other parameters, such as a minimum time interval for pushing advertisements of the same type to the user, etc., which is not specifically limited in the embodiments of the present disclosure.

[0172] Through the above method, the embodiment of the present disclosure can control the maximum number of advertisement insertions in the same video, reducing excessive disturbance to users. Moreover, the time interval between pushed advertisements can be controlled to reduce the over-frequent push of advertisements to users. In addition, the push of repeated advertisements to users can be reduced to ensure the freshness of advertisements to users.

[0173] In the above S105, after the electronic device determines the advertisement to be pushed and the time to be inserted based on the viewing time information, advertising preference information, emotional information and scene information using the large language model, the electronic device can send the advertisement to be pushed and the time to be inserted to the playback terminal of the current video, so that the playback terminal plays the advertisement to be pushed within the time to be inserted.

[0174] Afterwards, the electronic device may also update the advertisement recommendation strategy based on the user's feedback on the pushed advertisement, including the following steps:

[0175] Step I: Obtain the third operation behavior performed by the user on the pushed advertisement.

[0176] Step II: based on the third operation behavior, adjust the ranking of the advertisement to be pushed among the alternative advertisements; and / or, based on the third operation behavior, adjust the minimum time interval for pushing the advertisement to be pushed to the user; and / or, based on the third operation behavior, adjust the ranking of related advertisements among the alternative advertisements, and the related advertisements include advertisements of the same type as the advertisement to be pushed.

[0177] Among them, the candidate ads are sorted according to their respective push weights, so the order of the ads to be pushed among the candidate ads can be adjusted by adjusting the push weights of the ads to be pushed. When the first operation behavior is of a positive type, the push weight of the ads to be pushed can be increased, and when the first operation behavior is of a negative type, the push weight of the ads to be pushed can be reduced. When predicting the user's click rate on the candidate ads, the large language model can also consider the push weights of the candidate ads.

[0178] When the first operation behavior is of a positive type, the minimum time interval for pushing the advertisement to be pushed to the user may be reduced; when the first operation behavior is of a negative type, the minimum time interval for pushing the advertisement to be pushed to the user may be increased.

[0179] When the first operation behavior is of a positive type, the push weight of the related advertisement may be increased, and when the first operation behavior is of a negative type, the push weight of the related advertisement may be reduced.

[0180] In addition to the above adjustments, other parameters used in the advertisement recommendation process may also be adjusted based on the third operation behavior, which is not specifically limited in the embodiments of the present disclosure.

[0181] Through the above method, the advertisement recommendation strategy can be continuously optimized based on the user's feedback behavior on the pushed advertisements, thereby increasing the possibility that the advertisements pushed to the users will be liked by the users and reducing the disturbance caused to the users by pushing advertisements to the users.

[0182] See also Figure 7 The overall process of the advertisement push method based on the large model provided by the embodiment of the present disclosure is described below:

[0183] Among them, the electronic device includes a user behavior monitoring module, an emotion recognition module, a video content analysis module, an advertisement push and strategy linkage module and an intelligent recommendation model module. These five modules are all software modules included in the electronic device.

[0184] During the playback of the current video, the playback terminal of the current video sends a first operation behavior performed by the user on the current video and a second operation behavior performed by the user on the pushed advertisement to the user behavior monitoring module.

[0185] The user behavior monitoring module determines the user's viewing time information for the current video based on the first operation behavior, and determines the user's advertising preference information for the pushed advertisement based on the second operation behavior, and sends the viewing time information and advertising preference information to the intelligent recommendation model module.

[0186] During the playback of the pushed advertisement, the playback terminal sends multiple groups of probabilities to the emotion recognition module.

[0187] The emotion recognition module determines the emotion information of the user during the playback of the pushed advertisement based on the received multiple groups of probabilities, and sends the emotion information to the intelligent recommendation model module.

[0188] The video content analysis module determines scene information of multiple video clips included in the current video, and sends the scene information to the intelligent recommendation model module.

[0189] The intelligent recommendation model module uses a large language model based on viewing time information, advertising preference information, emotional information and scene information to determine candidate advertisements, candidate advertising versions and candidate insertion times, and sends the candidate advertisements, candidate advertising versions and candidate insertion times to the advertising push and strategy linkage module.

[0190] The advertisement push and strategy linkage module determines the advertisement to be pushed based on the candidate advertisements and the candidate advertisement versions, and determines the insertion time based on the candidate insertion time.

[0191] The advertisement push and strategy linkage module sends the advertisement to be pushed and the time to be inserted to the playback terminal, so that the playback terminal plays the advertisement to be pushed within the time to be inserted.

[0192] The playback terminal sends the third operation behavior performed by the user on the advertisement to be pushed to the advertisement push and strategy linkage module.

[0193] The advertisement push and strategy linkage module adjusts the advertisement recommendation strategy based on the third operation behavior.

[0194] Figure 7 The specific implementation methods of each step in the above can refer to the above description and will not be repeated here.

[0195] Compared with the method of pushing advertisements with the same theme as the current video to users, this method cannot optimize the timing of advertisement push. Even if the pushed advertisements may belong to the user's interest category, the push timing may still disturb the user's viewing.

[0196] Compared with the method of pushing ads with high click-through rates to users, since different users have different interest distributions, using a unified method to push ads to all users will have low returns and it is difficult to significantly increase the click-through rate of ads.

[0197] The disclosed embodiment can optimize the pushed ads and the timing of ad push while the user is watching a video, thereby increasing the likelihood that the pushed ads will be liked by the user and reducing the interruption caused by ad push to the user's video viewing. Moreover, the disclosed embodiment can continuously learn user preferences during video playback, reduce the interruption caused by ads to the user's video viewing, and increase the click-through rate of ads. Moreover, no additional manual adjustments are required during the period, which can fundamentally improve the ad recommendation effect.

[0198] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of the videos, advertisements and emotional information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0199] It should be noted that the large language model in this embodiment is not a model for a specific user and cannot reflect the personal information of a specific user.

[0200] It should be noted that the video, advertisement and emotion information in this embodiment may come from a public dataset.

[0201] Based on the same inventive concept, corresponding to the above method embodiment, the embodiment of the present disclosure also provides an advertisement push device based on a large model, such as Figure 8 As shown, the device includes: a determination module 801 and an acquisition module 802;

[0202] The determination module 801 is used to determine the viewing time information of the user on the current video based on the first operation behavior performed by the user on the current video during the playback of the current video;

[0203] An acquisition module 802 is used to acquire the user's advertisement preference information for the pushed advertisement, where the advertisement preference information is determined based on a second operation performed by the user on the pushed advertisement;

[0204] The acquisition module 802 is also used to acquire the user's emotional information during the playback of the pushed advertisement;

[0205] The acquisition module 802 is further used to acquire scene information of multiple video clips included in the current video;

[0206] The determination module 801 is also used to determine the advertisements to be pushed and the time to insert the advertisements to be pushed during the playback of the current video based on the viewing time information, the advertisement preference information, the emotion information and the scene information using the large language model.

[0207] In some embodiments of the present disclosure, the determination module 801 is further configured to:

[0208] Determining a first preference score of the user for the first pushed advertisement based on a type of a first operation behavior performed by the user on the first pushed advertisement, where the first pushed advertisement is a pushed advertisement that has been inserted during playback of the current video;

[0209] Determining the advertisement type to which the first pushed advertisement belongs;

[0210] Obtaining a user's preference for an advertisement type, where the preference is determined based on a second preference score of the user for a second pushed advertisement, where the second pushed advertisement is a pushed advertisement of an advertisement type inserted in a video that the user has watched in the past;

[0211] The user's preference for the advertisement type is updated according to the first preference score to obtain updated user's preference for various advertisement types.

[0212] In some embodiments of the present disclosure, the determination module 801 is further configured to:

[0213] During the playback of the pushed advertisement, a user video shot of the user is obtained;

[0214] For the video frames included in the user video, the facial expressions and body language of the user in the video frames are identified by the emotion recognition model, and the probability of the user in the video frames being in various emotion types is output based on the facial expressions and body language;

[0215] Determine the emotion score of the video frame based on the probability of the user being in various emotion types in the video frame;

[0216] The emotion scores of consecutive video frames are smoothed to obtain the average emotion score.

[0217] In some embodiments of the present disclosure, the determination module 801 is further configured to:

[0218] During the playback of the pushed advertisement, multiple groups of probabilities sent by the playback terminal of the pushed advertisement are received, and each group of probabilities represents the probability of the user being in various emotion types at a moment;

[0219] Based on the same set of probabilities, determine the user's emotion score at a moment;

[0220] The continuous sentiment scores are smoothed to obtain the average sentiment score.

[0221] In some embodiments of the present disclosure, the determination module 801 is further configured to:

[0222] Divide the current video into multiple video segments;

[0223] Using the emotion and content analysis model, image features are extracted from the video frames included in the video clip, text features are extracted from the texts in the video frames, and audio features are extracted from the audio included in the video clip, and based on the feature extraction results, the emotion type to which the video clip belongs is determined;

[0224] Determine the emotional transition time period based on the emotional type of each video clip;

[0225] Using the object detection and scene recognition model, image recognition is performed on the video frames included in the video clip, and the object detection result of the video clip and the scene type to which it belongs are output. The object detection result indicates whether the video clip includes a preset object and the playback time period of the preset object;

[0226] Based on the scene type to which each video clip belongs, a scene transition time period is determined.

[0227] In some embodiments of the present disclosure, the determination module 801 is specifically configured to:

[0228] Based on the scene information, determining a plurality of recommended insertion times, the recommended insertion times including the playing time of the video clips belonging to the non-climax scene;

[0229] Based on the user's current emotion type, filter out the ad insertion trigger time from each recommended insertion time;

[0230] When the ad insertion trigger time is reached, the viewing time information, ad preference information, emotion information and scene information are input into the large language model, and the large language model is used to predict the user's click rate for each ad. When the click rate meets the preset conditions, the candidate ad, the candidate ad version and the candidate insertion time are output based on the click rate. When the click rate does not meet the preset conditions, the candidate ad, the candidate ad version and the candidate insertion time are output based on the alternative ad, where the ad version indicates the display form of the ad content.

[0231] The advertisement to be pushed is determined based on the candidate advertisements and the candidate advertisement versions, and the insertion time is determined based on the candidate insertion time.

[0232] In some embodiments of the present disclosure, the determination module 801 is specifically configured to:

[0233] Determining a predicted emotion type of the user at the recommended insertion time based on the user's current emotion type and scene information of the video clip included in the recommended insertion time;

[0234] Based on the predicted emotion type, the advertisement insertion trigger time is filtered out from each recommended insertion time.

[0235] In some embodiments of the present disclosure, the determination module 801 is specifically configured to:

[0236] Selecting a target candidate advertisement from among the candidate advertisements;

[0237] According to the total duration of the current video, the user's current emotion type and the user's historical operation behaviors on each advertising version, the version to be pushed is screened from the candidate advertising versions, and the target candidate advertisement of the push version is used as the advertisement to be pushed.

[0238] In some embodiments of the present disclosure, the determination module 801 is specifically configured to:

[0239] The candidate advertisement with the highest candidate score is selected as the target candidate advertisement.

[0240] In some embodiments of the present disclosure, the determination module 801 is specifically configured to:

[0241] Obtain videos that are associated with the current video and videos of interest to the user as related videos;

[0242] Based on the relevance between the relevant videos and the candidate ads, a target candidate ad is selected.

[0243] In some embodiments of the present disclosure, the determination module 801 is specifically configured to:

[0244] When the user's current emotion type is a positive type, the user's historical operation behavior on the target candidate advertisement is obtained as the target operation behavior;

[0245] Obtaining historical ratings of target candidate ads that have been viewed by the user, where the ratings represent the degree of guidance on click behavior;

[0246] Based on the target operation behavior and the historical level, the push level is determined, and the target candidate advertisement of the push version of the push level is used as the advertisement to be pushed.

[0247] In some embodiments of the present disclosure, the determination module 801 is specifically configured to:

[0248] Based on preset restriction parameters and candidate insertion times, the time to be inserted is determined, where the restriction parameters include at least one of the following: the minimum time interval between adjacent advertisements, the maximum number of advertisement insertions in a single video, the maximum number of advertisement insertions per unit time, and the minimum time interval for pushing the same advertisement to users.

[0249] In some embodiments of the present disclosure, the device further comprises:

[0250] The acquisition module 802 is further used to obtain a third operation behavior performed by the user on the advertisement to be pushed after determining the advertisement to be pushed and the time to be inserted into the advertisement during the playback of the current video based on the viewing time information, the advertisement preference information, the emotion information and the scene information using the large language model;

[0251] The adjustment module is used to adjust the order of the advertisement to be pushed among the alternative advertisements based on the third operation behavior; and / or, based on the third operation behavior, adjust the minimum time interval for pushing the advertisement to be pushed to the user; and / or, based on the third operation behavior, adjust the order of related advertisements among the alternative advertisements, and the related advertisements include advertisements of the same type as the advertisement to be pushed.

[0252] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0253] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0254] like Fig. 9As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0255] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0256] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a large model-based advertising push method. For example, in some embodiments, the large model-based advertising push method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the large model-based advertising push method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the large model-based advertisement push method in any other appropriate manner (eg, by means of firmware).

[0257] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0258] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0259] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0260] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0261] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0262] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0263] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0264] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An advertisement push method based on a large model, comprising: During playback of the current video, based on a first operation performed by the user on the current video, determining viewing time information of the user on the current video; Acquiring advertisement preference information of the user for the pushed advertisement, where the advertisement preference information is determined based on a second operation behavior performed by the user on the pushed advertisement; Acquiring emotional information of the user during the playing of the pushed advertisement; Acquire scene information of multiple video clips included in the current video; Based on the viewing time information, the advertisement preference information, the emotion information and the scene information, a large language model is used to determine the advertisement to be pushed and the time when the advertisement to be pushed is to be inserted during the playback of the current video.

2. The method according to claim 1, wherein the advertisement preference information is determined by: determining a first preference score of the user for the first pushed advertisement based on a type of a first operation performed by the user on the first pushed advertisement, wherein the first pushed advertisement is a pushed advertisement that has been inserted during playback of the current video; Determining the advertisement type to which the first pushed advertisement belongs; Acquire the user's preference for the advertisement type, where the preference is determined based on a second preference score of the user for a second pushed advertisement, where the second pushed advertisement is a pushed advertisement of the advertisement type inserted in a video that the user has watched in the past; The user's preference for the advertisement type is updated according to the first preference score to obtain updated preferences for various advertisement types of the user.

3. The method according to claim 1, wherein the emotion information is determined by: During the playback of the pushed advertisement, obtaining a user video shot of the user; For the video frames included in the user video, the facial expressions and body language of the user in the video frames are identified by the emotion recognition model, and the probabilities of the user in the video frames being in various emotion types are output based on the facial expressions and the body language; Determining an emotion score of the video frame based on the probability that the user is in various emotion types in the video frame; The emotion scores of consecutive video frames are smoothed to obtain the average emotion score.

4. The method according to claim 1, wherein the emotion information is determined by: During the playback of the pushed advertisement, receiving multiple groups of probabilities sent by a playback terminal of the pushed advertisement, wherein each group of probabilities represents the probability of the user being in various emotion types at a time; Based on the same set of probabilities, determining an emotion score of the user at a moment; The continuous sentiment scores are smoothed to obtain the average sentiment score.

5. The method according to claim 1, wherein the scene information is determined by the following steps: Dividing the current video into a plurality of video segments; Using the emotion and content analysis model, image feature extraction is performed on the video frames included in the video clip, text feature extraction is performed on the text in the video frames, and audio feature extraction is performed on the audio included in the video clip, and based on the feature extraction results, the emotion type to which the video clip belongs is determined; Determine the emotional transition time period based on the emotional type of each video clip; Using the object detection and scene recognition model, perform image recognition on the video frames included in the video clip, and output the object detection result of the video clip and the scene type to which it belongs, wherein the object detection result indicates whether the video clip includes a preset object and the playback time period of the preset object; Based on the scene type to which each video clip belongs, a scene transition time period is determined.

6. The method according to any one of claims 1 to 5, wherein: The determining, based on the viewing time information, the advertisement preference information, the emotion information and the scene information, using a large language model, of an advertisement to be pushed and a time at which the advertisement to be pushed is to be inserted during the playback of the current video includes: Based on the scene information, determining a plurality of recommended insertion times, the recommended insertion times including the play times of video clips belonging to non-climax scenes; Based on the current emotion type of the user, selecting an advertisement insertion trigger time from each recommended insertion time; When the advertisement insertion trigger time is reached, the viewing time information, the advertisement preference information, the emotion information and the scene information are input into the large language model, the large language model is used to predict the click rate of the user for each advertisement, and when the click rate meets a preset condition, candidate advertisements, candidate advertisement versions and candidate insertion times are output based on the click rate, and when the click rate does not meet the preset condition, candidate advertisements, candidate advertisement versions and candidate insertion times are output based on the alternative advertisements, wherein the advertisement version indicates the display form of the advertisement content; The advertisement to be pushed is determined based on the candidate advertisement and the candidate advertisement version, and the insertion time is determined based on the candidate insertion time.

7. The method according to claim 6, wherein: The step of selecting an advertisement insertion trigger time from among the recommended insertion times based on the current emotion type of the user includes: Determining a predicted emotion type of the user at the recommended insertion time based on the current emotion type of the user and scene information of the video clip included in the recommended insertion time; Based on the predicted emotion type, an advertisement insertion triggering time is filtered out from each recommended insertion time.

8. The method according to claim 6, wherein: The determining the advertisement to be pushed based on the candidate advertisement and the candidate advertisement version includes: Selecting a target candidate advertisement from among the candidate advertisements; According to the total duration of the current video, the current emotion type of the user and the historical operation behaviors of the user on each advertisement version, the version to be pushed is screened out from the candidate advertisement versions, and the target candidate advertisement of the push version is used as the advertisement to be pushed.

9. The method according to claim 8, wherein: The candidate advertisements have candidate scores, and selecting a target candidate advertisement from the candidate advertisements includes: The candidate advertisement with the highest candidate score is selected as the target candidate advertisement.

10. The method according to claim 8, wherein: The step of selecting a target candidate advertisement from the candidate advertisements includes: Acquire videos that are associated with the current video and videos of interest to the user as related videos; The target candidate advertisement is selected based on the relevance between the related video and the candidate advertisement.

11. The method according to any one of claims 8 to 10, wherein: The step of using the target candidate advertisement of the push version as the advertisement to be pushed includes: When the current emotion type of the user belongs to the positive type, obtaining the historical operation behavior performed by the user on the target candidate advertisement as the target operation behavior; Acquire a historical rating of the target candidate advertisement that the user has viewed, wherein the rating indicates a degree of guidance on click behavior; Based on the target operation behavior and the historical level, a push level is determined, and the target candidate advertisement of the push version of the push level is used as the advertisement to be pushed.

12. The method according to any one of claims 6 to 10, wherein: The determining the to-be-inserted time based on the candidate insertion time includes: The time to be inserted is determined based on preset restriction parameters and the candidate insertion time, wherein the restriction parameters include at least one of the following: a minimum time interval between adjacent advertisements, a maximum number of advertisement insertions in a single video, a maximum number of advertisement insertions per unit time, and a minimum time interval for pushing the same advertisement to the user.

13. The method according to claim 12, after determining the advertisement to be pushed and the time to insert the advertisement to be pushed during the playback of the current video using a large language model based on the viewing time information, the advertisement preference information, the emotion information and the scene information, further comprises: Acquiring a third operation behavior performed by the user on the advertisement to be pushed; According to the third operation behavior, adjusting the order of the advertisement to be pushed among the candidate advertisements; and / or, According to the third operation behavior, adjusting the minimum time interval for pushing the to-be-pushed advertisement to the user; and / or, According to the third operation behavior, the order of related advertisements in the candidate advertisements is adjusted, and the related advertisements include advertisements of the same advertisement type as the advertisement to be pushed.

14. An advertisement push device based on a large model, comprising: A determination module, configured to determine, during playback of a current video, viewing time information of the current video by the user based on a first operation performed by the user on the current video; An acquisition module, configured to acquire the user's advertisement preference information for the pushed advertisement, wherein the advertisement preference information is determined based on a second operation performed by the user on the pushed advertisement; The acquisition module is further used to acquire the emotion information of the user during the playing process of the pushed advertisement; The acquisition module is further used to acquire scene information of multiple video clips included in the current video; The determination module is further used to determine the advertisement to be pushed and the time to insert the advertisement to be pushed during the playback of the current video based on the viewing time information, the advertisement preference information, the emotion information and the scene information using a large language model.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images