Method, device and equipment for determining resource display time period, and storage medium

Abstract

The application provides a resource display time period determination method and device, equipment and a storage medium, and belongs to the technical field of computers. The method determines a coarse-grained time period based on text, so that the target time period can be determined by analyzing only the image frames in the time period, without analyzing all the image frames of the first multimedia resource. Obviously, the analysis range of the image frames is effectively shortened, high resource consumption is avoided, and efficiency is improved. Moreover, the target time period is determined based on the multidimensional information of the text and the image frames, so that the target time period determined is an accurate time period in which the relevant items appear, and the time period for displaying the second media resource is accurately determined. The method improves the determination efficiency of the display time period while ensuring the quality of the determined display time period.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for determining the time period of resource display. Background Technology

[0002] During video playback, multimedia resources can be displayed on the video playback interface. These resources are typically used to showcase a specific item. However, it's generally impossible to display multimedia resources for the entire duration of the video playback; therefore, it's necessary to determine the time segments during which these resources are displayed.

[0003] In related technologies, the time period for displaying multimedia resources in a video is usually determined manually. In this process, due to the large number of image frames in the video, it is necessary to manually analyze the video frame by frame in order to determine the appropriate time period for displaying multimedia resources, which is inefficient. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for determining the display time period of resources. This method improves the efficiency of determining the display time period. The technical solution provided by this application is as follows.

[0005] According to one aspect of the embodiments of this application, a method for determining a resource display time period is provided, the method comprising:

[0006] Based on the text of the first multimedia resource, a first time period is determined in the first multimedia resource, wherein the first time period is the time period in which the text segment containing the target keyword appears;

[0007] The target time period is determined from the second time period corresponding to the first time period. The second time period is the time period in the first multimedia resource based on the first time period. The target time period is the time period in which the image frame of the first item appears. The first item is the item indicated by the target keyword.

[0008] Based on the target time period, a time period for displaying the second multimedia resource in the first multimedia resource is determined, wherein the second multimedia resource is used to display a second item associated with the first item.

[0009] According to another aspect of the embodiments of this application, a device for determining a resource display time period is provided, the device comprising:

[0010] The first determining module is used to determine a first time period in the first multimedia resource based on the text of the first multimedia resource, wherein the first time period is the time period in which the text segment containing the target keyword appears;

[0011] The second determining module is used to determine a target time period from the second time period corresponding to the first time period, wherein the second time period is the time period in the first multimedia resource based on the first time period, the target time period is the time period in which the image frame of the first item appears, and the first item is the item indicated by the target keyword.

[0012] The third determining module is used to determine, based on the target time period, the time period in which the second multimedia resource is displayed in the first multimedia resource, wherein the second multimedia resource is used to display a second item associated with the first item.

[0013] In some embodiments, there are multiple target time periods, each target time period corresponding to a first item; the third determining module includes:

[0014] The first determining unit is used to determine, for each target time period, an effect evaluation value in multiple dimensions based on the location information of the target time period. The location information includes information on multimedia resource segments within the target time period and information on the second item. The effect evaluation value in each dimension is used to describe the display effect of the second multimedia resource displayed in the target time period in that dimension.

[0015] The second determining unit is used to select target time periods that meet the first condition from the multiple target time periods based on the effect evaluation values ​​of each of the multiple target time periods in the multiple dimensions, and to use the selected target time periods as the time periods for displaying the second multimedia resources in the first multimedia resources.

[0016] In some embodiments, the second determining unit is configured to:

[0017] For each target time period, the effect evaluation values ​​of the target time period in the multiple dimensions are weighted and summed to obtain the independent evaluation value of the target time period;

[0018] Select the target time period from the plurality of target time periods where the independent evaluation value reaches a preset value, or select the target time period from the plurality of target time periods where the independent evaluation value is ranked in the first preset position.

[0019] In some embodiments, the second determining unit is configured to:

[0020] Based on the multiple target time periods, various combinations of time periods are determined, and each combination of time periods includes a preset number of target time periods;

[0021] For each time period combination, based on the effect evaluation values ​​of each target time period in the time period combination in the multiple dimensions, the joint evaluation value of the time period combination in the multiple dimensions is determined. The joint evaluation value of each dimension is used to describe the distribution of the second multimedia resources displayed by each of the multiple target time periods in the time period combination in the dimension.

[0022] Based on the joint evaluation values ​​of the various time period combinations in the multiple dimensions, time period combinations that meet the second condition are selected from the various time period combinations, and the target time period in the selected time period combinations is taken as the selected target time period.

[0023] In some embodiments, the second determining unit is configured to:

[0024] For each target time period, the effect evaluation values ​​of the target time period in the multiple dimensions are weighted and summed to obtain the independent evaluation value of the target time period;

[0025] Based on the multiple target time periods, various combinations of time periods are determined, and each combination of time periods includes a preset number of target time periods;

[0026] For each time period combination, based on the effect evaluation values ​​of each target time period in the time period combination in the multiple dimensions, the joint evaluation value of the time period combination in the multiple dimensions is determined. The joint evaluation value of each dimension is used to describe the distribution of the second multimedia resources displayed by each of the multiple target time periods in the time period combination in the dimension.

[0027] Based on the joint evaluation value of each time period combination in the multiple dimensions and the independent evaluation value of each target time period in the time period combination, time period combinations that meet the third condition are selected from the multiple time period combinations, and the target time period in the selected time period combinations is taken as the selected target time period.

[0028] In some embodiments, the second determining unit is configured to:

[0029] For each time period combination, the effect evaluation value of the time period combination is determined based on the joint evaluation value of the time period combination in the multiple dimensions and the independent evaluation value of each target time period in the time period combination.

[0030] The time period combination with the highest effect evaluation value is selected from the various time period combinations.

[0031] In some embodiments, the apparatus further includes a fourth determining module, configured to:

[0032] For each image frame within the target time period, determine the display position of the first item within the image frame;

[0033] Based on the display position of the first item in the image frame, the position in the image frame where the second multimedia resource is displayed is determined.

[0034] In some embodiments, the fourth determining module is configured to perform at least one of the following:

[0035] The second multimedia resource is an introduction card for the second item. Based on the display position of the first item in the image frame, the adjacent positions of the display position of the first item in the image frame are determined as the positions for displaying the introduction card.

[0036] The second multimedia resource is a special effect displayed on the first item. The special effect is used to display the second item after being triggered. Based on the display position of the first item in the image frame, the display position of the first item is determined as the position for displaying the special effect.

[0037] In some embodiments, the apparatus further includes a fifth determining module, configured to:

[0038] Based on the display position of the first item in multiple image frames within the target time period, key image frames within the target time period are determined. The display position of the first item in the key image frames is used to restore the display position of the first item in image frames other than the key image frames within the target time period.

[0039] In some embodiments, the fifth determining module is configured to perform at least one of the following:

[0040] If the position of the first item remains unchanged in the plurality of image frames, any image frame within the target time period is determined as the key image frame;

[0041] If the trajectory of the first item in the plurality of image frames is a straight line, the first and last image frames in the target time period are both determined as the key image frames;

[0042] When the trajectory of the first item in the plurality of image frames is a parabola, the first image frame, the last image frame, and the image frame corresponding to the vertex of the parabola within the target time period are all determined as the key image frames.

[0043] In some embodiments, the apparatus further includes a sixth determining module, configured to:

[0044] The interpolation method corresponding to the key image frame is determined. The interpolation method is used to interpolate the display position of the first item in the key image frame to restore the display position of the first item in image frames other than the key image frame within the target time period.

[0045] In some embodiments, the sixth determining module is configured to perform at least one of the following:

[0046] When the key image frame includes the first and last image frames within the target time period, linear interpolation is determined as the interpolation method.

[0047] When the key image frame includes the first image frame, the last image frame, and the image frame corresponding to the vertex of the parabola within the target time period, quadratic interpolation is determined as the interpolation method.

[0048] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory being used to store a computer program, the computer program being loaded and executed by the processor to implement the method for determining the resource display time period in the embodiments of this application.

[0049] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the method for determining the resource display time period in the embodiments of this application.

[0050] On the other hand, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, a processor of a computer device reading the computer program from the computer-readable storage medium, the processor executing the computer program, causing the computer device to perform the method for determining the resource display time period as described in any of the above implementations.

[0051] This application provides a method for determining the display time period of a resource. When determining the time period for displaying a second multimedia resource within a first multimedia resource, the method first determines the first time period containing the text segment showing the target keyword based on the text of the first multimedia resource. Then, it analyzes image frames within the time period based on the first time period to determine the target time period containing the image frames showing the first item. This method first determines a coarse-grained time period based on the text, so only the image frames within this portion of the time period need to be analyzed to determine the target time period, without needing to analyze all image frames of the first multimedia resource. This effectively shortens the analysis scope of image frames, avoids high resource consumption, and improves efficiency. Furthermore, determining the target time period based on multi-dimensional information from text and image frames ensures that the determined target time period is the accurate time period when the relevant item appears, thus accurately determining the time period for displaying the second media resource. This method improves the efficiency of determining the display time period while ensuring its quality. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of a computer system provided in one embodiment of this application;

[0053] Figure 2 This is a flowchart of a method for determining the resource display time period provided in one embodiment of this application;

[0054] Figure 3 This is a flowchart of a method for determining the resource display time period provided in one embodiment of this application;

[0055] Figure 4 This is a flowchart of a method for determining the resource display time period provided in one embodiment of this application;

[0056] Figure 5 This is a schematic diagram of a resource display system provided in one embodiment of this application;

[0057] Figure 6 This is a block diagram of a device for determining a resource display time period according to an embodiment of this application;

[0058] Figure 7 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0060] Figure 1A structural block diagram of a computer system provided in an exemplary embodiment of this application is shown. The computer system includes a terminal 10 and a server 20.

[0061] Terminal 10 can be an electronic device such as a mobile phone, tablet computer, multimedia playback device, PC (Personal Computer), wearable device, in-vehicle terminal, VR (Virtual Reality) device, AR (Augmented Reality) device, MR (Mixed Reality) device, etc. Terminal 10 may have a client running the target application. The target application can provide the function of playing multimedia resources, which is not limited in this embodiment. This embodiment does not limit the implementation form of the target application; for example, it can be an application that requires download and installation, a mini-program that does not require installation, a web application, etc.

[0062] In this embodiment, server 20 provides background services for the target application, such as providing background services for playing multimedia resources in the target application, or providing background services for displaying other multimedia resources used to introduce a certain item in the multimedia resources played by the target application. Server 20 can be a single server or a server cluster composed of multiple servers. Terminal 10 can communicate with server 20 through a network, such as a wireless or wired network.

[0063] The above Figure 1 The embodiments described herein illustrate the entire implementation environment of this application, and based on the above implementation environment, the following can be provided: Figure 2 Please refer to the simplified flowchart shown below. Figure 2 , Figure 2 A flowchart illustrating a method for determining a resource display time period according to an embodiment of this application is shown. The method includes at least one of the following steps:

[0064] 1. Determine the first time period: Based on the text 201 of the first multimedia resource, determine the first time period 202 in which the text fragment containing the target keyword appears in the first multimedia resource. Since the target keyword is used to indicate the first item, and this application is to determine the display time period of the second multimedia resource of the second item associated with the first item, the determination of the first time period is also the determination of the coarse-grained time period of the display of the second multimedia resource.

[0065] 2. Determine the target time period: Determine the target time period 204 from the second time period 203 corresponding to the first time period 202. The target time period 204 is the time period during which the first item appears in the image frame. The second time period can be longer, shorter, or the same as the first time period. During the playback of the first multimedia resource, the first item may be displayed in the first multimedia resource before the text fragment containing the target keyword appears. Similarly, the first multimedia resource may continue to display the first item for a certain period after the text fragment appears. Therefore, taking the example that the second time period is longer than the first time period, the target time period is determined from the second time period, which includes the first time period but is longer than the first time period. This ensures that the target time period is as accurate and comprehensive as possible, and determining the target time period also determines the fine-grained time period during which the first item appears in the first multimedia resource.

[0066] 3. Determine the resource display time period: After determining the target time period 204, it can be directly used as the time period 205 for displaying the second multimedia resource. However, the target time period may also be filtered based on certain conditions, and only if the target time period meets certain conditions will it be used as the time period for displaying the second multimedia resource. For the first multimedia resource, multiple time periods can be determined for displaying the second multimedia resource.

[0067] In this embodiment, a coarse-grained time period is first determined based on the text. This means that only image frames within this specific time period need to be analyzed, rather than analyzing all image frames of the first multimedia resource. This effectively shortens the analysis scope, avoids high resource consumption, and improves efficiency. Furthermore, determining the target time period based on both text and image frame information ensures that the determined target time period accurately reflects the time when the relevant items appear, thus accurately determining the time period for displaying the second media resource. This approach improves both the quality and efficiency of determining the display time period.

[0068] In the embodiments of this application, based on Figure 2 The brief process described above briefly introduces the process of determining the resource display time period. The following examples will provide a detailed description of this process.

[0069] Please refer to Figure 3 , Figure 3 This is a flowchart of a method for determining a resource display time period according to an embodiment of this application. The executing entity for each step of this method can be a computer device, such as... Figure 1 The computer system shown is a server or terminal; this embodiment uses a server as an example for illustration. The method may include at least one of the following steps 301 to 303.

[0070] 301. Based on the text of the first multimedia resource, the server determines the first time period in the first multimedia resource, where the first time period is the time period in which the text fragment containing the target keyword appears.

[0071] In this embodiment, the first multimedia resource can be a video or graphic work. When the first multimedia resource includes audio, the text of the first multimedia resource includes the text obtained by speech recognition of the audio. When the first multimedia resource includes subtitles, the text of the first multimedia resource includes the subtitles. When the first multimedia resource includes both audio and subtitles, the time period of the text segment containing the target keyword is determined based on the text corresponding to each of the audio and subtitles; the first time period is the union of these two time periods. In this embodiment, for the same text segment, its playback time in the audio may not be perfectly aligned with its display time in the subtitles. For example, for a subtitle, when the subtitle is played in the audio, it may have already been displayed on the multimedia resource's playback interface. Therefore, taking the union of the time periods determined by the audio and subtitles respectively makes the first time period as comprehensive as possible. In other embodiments, the first time period can also be determined based on the text corresponding to one of the audio or subtitles; this is not specifically limited here.

[0072] In other embodiments, where the image frame of the first multimedia resource includes text other than subtitles, the text of the first multimedia resource also includes text obtained by text recognition of the image frame of the first multimedia resource.

[0073] The target keyword is used to indicate a specific item appearing in the first multimedia resource. This target keyword can be the item name or the item type name; no specific limitation is made here.

[0074] 302. The server determines the target time period from the second time period corresponding to the first time period. The second time period is the time period in the first multimedia resource based on the first time period. The target time period is the time period in which the image frame of the first item appears. The first item is the item indicated by the target keyword.

[0075] In this embodiment of the application, the second time period is based on the first time period, so the second time period can be longer than the first time period. That is, the second time period is a time period in the first multimedia resource that includes the first time period and is longer than the first time period.

[0076] The second time period can be pre-set to be longer than the first time period, and this pre-set duration can be set as needed. Furthermore, the second time period is centered on the first time period; that is, the second time period includes not only the first time period but also time periods of a certain duration before and after it. For example, the second time period includes the first time period, and includes a first-duration time period ending at the beginning of the first time period, and a second-duration time period starting at the end of the first time period. The sum of the first and second durations is the pre-set duration. The first and second durations can be the same or different; no specific limitation is made here.

[0077] In this embodiment, since the item indicated by the target keyword may appear in the image frame of the multimedia resource shortly before the target keyword appears in the audio or subtitle, and similarly, the item indicated by the target keyword may continue to appear in the image frame of the multimedia resource shortly after the target keyword appears in the audio or subtitle, the target time period is determined from a second time period that includes the first time period and is longer than the first time period, so that the target time period is determined more comprehensively and accurately.

[0078] In the above embodiments, the example of a second time period being longer than the first time period was used to illustrate the point, making the determination of the target time period more comprehensive and accurate. In other embodiments, the second time period is based on the first time period, so the second time period can also be the same as the first time period, or the second time period can be shorter than the first time period, such as the second time period being a portion of the first time period. Since determining the first time period from the first multimedia resource already determines a coarse-grained time period in which the first item may appear, and then determining the target time period from the first time period or a time period shorter than the first time period can further shorten the image frame analysis time period and improve processing efficiency.

[0079] 303. Based on the target time period, the server determines the time period in which the second multimedia resource is displayed in the first multimedia resource. The second multimedia resource is used to display the second item associated with the first item.

[0080] In this embodiment of the application, the target time period can be directly used as the time period for displaying the second multimedia resource in the first multimedia resource, or the target time period can be filtered. Only when the target time period meets certain conditions will it be used as the time period for displaying the second multimedia resource; otherwise, the target time period will not be used as the time period for displaying the second multimedia resource.

[0081] The second multimedia resource is used to display the second item, and the second multimedia resource includes at least one of images, text, and video.

[0082] In this embodiment of the application, an example is given where the second time period is longer than the first time period. For example, the first time period is the period from the 30th second to the 40th second in the first multimedia resource, the second time period is the period from the 28th second to the 42nd second in the first multimedia resource, and the target time period is the period from the 29th second to the 36th second in the first multimedia resource.

[0083] Let's take the example of a second time period being shorter than the first time period. For instance, the first time period is from the 30th to the 40th second of the first multimedia resource, the second time period is from the 32nd to the 38th second of the first multimedia resource, and the target time period is from the 32nd to the 36th second of the first multimedia resource.

[0084] If the second item is associated with the first item, then the second item can be an item of the same type as the first item, or the second item can be an item produced by the same producer as the first item.

[0085] In this embodiment of the application, an example is given where the second item is an object of the same type as the first item. For example, the first item is a cup, and the second item is also a cup.

[0086] Let's take the example of the first item and the second item belonging to the same producer. For instance, if the first item is a cup produced by a certain producer, then the second item can be any other item produced by that producer, such as a bowl or other cup produced by that producer.

[0087] In some embodiments, the second item may be the same as the first item, that is, the second multimedia resource is used to display the first item appearing in the first multimedia resource. In other embodiments, the second item may also be an item unrelated to the first item, which is not specifically limited here.

[0088] In this embodiment of the application, after the server determines the time period for displaying the second multimedia resource in the first multimedia resource, the time period is used to send to the terminal playing the first multimedia resource, so that the terminal displays the second multimedia resource during the time period while playing the first multimedia resource.

[0089] This application provides a method for determining the display time period of a resource. When determining the time period for displaying a second multimedia resource within a first multimedia resource, the method first determines the first time period containing the text segment showing the target keyword based on the text of the first multimedia resource. Then, it analyzes the image frames within the time period based on the first time period to determine the target time period containing the image frames showing the first item. This method first determines a coarse-grained time period based on the text, thus requiring analysis only of the image frames within this portion of the time period, without needing to analyze all image frames of the first multimedia resource. This effectively shortens the analysis scope of image frames, avoids high resource consumption, and improves efficiency. Furthermore, determining the target time period based on both text and image frame information ensures that the determined target time period is the accurate time period when the relevant item appears, thereby accurately determining the time period for displaying the second media resource. This approach improves the efficiency of determining the display time period while ensuring its quality.

[0090] In this embodiment of the application, through the above... Figure 3 The following is a brief introduction to the process of determining the resource display time period in the example. Figure 4 The embodiments described in detail illustrate this process. See also Figure 4 , Figure 4 This is a flowchart illustrating the determination of a resource display time period according to an embodiment of this application. The executing entity for each step of this method can be a computer device, such as... Figure 1 The computer system shown is a server or terminal; this embodiment uses a server as an example for illustration. The method may include at least one of the following steps 401 to 407.

[0091] 401. Based on the text of the first multimedia resource, the server determines the first time period in the first multimedia resource, where the first time period is the time period in which the text fragment containing the target keyword appears.

[0092] In this embodiment, since the target keyword has a short length, if it appears in audio, its playback time in the audio is short, and the time period it occupies is also short. It is generally impossible to display the second multimedia resource within such a short time period. Therefore, in this embodiment, the time period of the text segment containing the target keyword is determined. In this process, the target keyword in the text is identified and extracted, and the time period of the text segment containing the target keyword is obtained based on the text-aligned timeline.

[0093] In the process of obtaining the text of the first multimedia resource based on audio and then determining the first time segment, the audio is first preprocessed, including noise reduction and segmentation, to improve the accuracy of subsequent audio analysis. Then, speech recognition and natural language processing algorithms are used to process the audio to obtain text aligned with the audio's timeline. Text segments containing target keywords are identified, and the time segment containing these text segments is defined as the first time segment.

[0094] In the process of obtaining the text of the first multimedia resource based on the subtitles and then determining the first time period, the display time period of each subtitle in the first multimedia resource is obtained from the information of the first multimedia resource. Then, multiple subtitles are filtered one by one to determine the subtitles that include the target keywords, and the display time period of the subtitles is determined as the first time period.

[0095] In this embodiment, if the subtitles of the first multimedia resource are missing, the text can be determined based on the audio of the first multimedia resource; if the audio of the first multimedia resource is missing, the text can be determined based on the subtitles of the first multimedia resource. Optionally, after the server determines the first time period, it outputs the first time period together with the corresponding text segment to facilitate understanding which text segment corresponds to the first time period.

[0096] 402. The server determines the target time period from the second time period corresponding to the first time period. The second time period is the time period in the first multimedia resource based on the first time period. The target time period is the time period in which the image frame of the first item appears. The first item is the item indicated by the target keyword.

[0097] Specifically, the server extracts multimedia resource segments within a target time period from the first multimedia resource, extracts image frames from these segments, and performs image recognition on these image frames to determine the image frames containing the first item, thereby obtaining the time period in which these image frames are located. Optionally, the server inputs the image frames into a large-scale image recognition system, and uses techniques such as deep learning to recognize the image frames and determine whether the first item exists in the image frames.

[0098] It should be noted that due to various reasons such as camera transitions, the first item may not appear continuously within a certain time period in the second time period; that is, it may appear intermittently. For example, it may appear in multiple consecutive image frames, then after a certain interval, and then appear in subsequent image frames. Therefore, determining the target time period can include the following implementation methods:

[0099] (1) The time point when the first image frame of the first item appears in the second time period is determined as the starting point, the time point when the last image frame of the first item appears in the second time period is determined as the ending point, and the time period in the second time period with the starting point and the ending point as the starting and ending points respectively is taken as the target time period.

[0100] (2) Determine multiple sub-time periods in the second time period. If the first item appears in each image frame of these multiple sub-time periods, then the target time period is the set of time periods of these multiple sub-time periods.

[0101] 403. For each image frame within the target time period, the server determines the display position of the first item in the image frame.

[0102] In some embodiments, the display position of a first item is determined based on a preset shape, which is a region of a preset shape centered on the center of the first item on the image frame. The first item is displayed within this preset shape region. The preset shape can be set as needed, and the coordinate information used to indicate the display position of the first item varies depending on the shape of the region. For example, if the region is rectangular, the coordinate information includes the coordinates of the two opposite corners of the rectangular region on the image frame, such as the coordinates of the upper left and lower right corners, or the upper right and lower left corners. If the region is circular, the coordinate information includes the coordinates of the center of the circular region on the image frame and the radius of the circular region. If the region is elliptical, the coordinate information includes the coordinates of the four vertices of the elliptical region on the image frame.

[0103] In this embodiment, only rectangles, circles, and ellipses are used as examples of preset shapes. The preset shape can also be various shapes such as rhombuses and trapezoids, and is not specifically limited here. Optionally, the preset shape is determined based on the shape of the first object in the image frame. For example, if the first object is circular, the preset shape can be circular.

[0104] 404. The server determines the position of the second multimedia resource in the image frame based on the display position of the first item in the image frame.

[0105] In this embodiment, the second multimedia resource can be displayed at the display position of the first item or at an adjacent position, or at other positions related to the display position of the first item. Furthermore, the display position of the second multimedia resource varies depending on its type, and this process includes, but is not limited to, the following two implementation methods.

[0106] (1) The second multimedia resource is an introduction card for the second item. Based on the display position of the first item in the image frame, the server determines the adjacent position of the display position of the first item in the image frame as the position for displaying the introduction card.

[0107] The adjacent positions of the first item's display location can be set as needed. For example, the adjacent positions of the first item's display location include directly above, below, to the left, and to the right of the first item's display location, while the location for displaying the second multimedia resource can be at least one of these four adjacent positions. Furthermore, the adjacent positions belonging to the background area among the four adjacent positions are determined, and these adjacent positions belonging to the background area are selected as the locations for displaying the introduction card to avoid the introduction card obscuring other information on the image frame.

[0108] The display size of the second multimedia resource in the image frame can be set as needed, such as having a fixed display size in different image frames. Alternatively, the display size of the second multimedia resource in the image frame can be proportional to the size of the display area where the first item is located in the image frame.

[0109] The introduction card can display text, images, and other descriptive information about the second item. Furthermore, it can also display a video introducing the second item. In some embodiments, triggering the introduction card of the second item can lead to the item's details page or its redemption page; specific limitations are not specified here.

[0110] In this embodiment of the application, when the second multimedia resource is a card used to introduce the second item, it is displayed in an adjacent position to the display position of the first item. This establishes a visual association between the first item and the second item, allowing the user to know which content in the image frame the currently displayed second item is associated with, thereby improving the information transmission rate and thus improving the display effect of the second multimedia resource.

[0111] (2) The second multimedia resource is a special effect displayed on the first item. The special effect is used to display the second item after being triggered. The server determines the display position of the first item as the position for displaying the special effect based on the display position of the first item in the image frame.

[0112] The special effects displayed on the first item can be set as needed, such as outlining the first item, making the first item glow, enlarging the first item, or cartoonizing the first item. There are no specific restrictions on the type of special effects.

[0113] The display of the second item can be a description card, a redemption control, or a details page; no specific limitation is made here. In some embodiments, if no triggering operation on the special effect is detected during the effect display process, the second item can be displayed directly after the effect display is complete.

[0114] In this embodiment, the special effect is shown on the first item as an example. In other embodiments, a second item may also be displayed at the display position of the first item, thereby displaying the special effect of the second item.

[0115] In this embodiment, when the second multimedia resource is a special effect, the special effect is displayed on the first item in the image frame. This helps to attract attention and improve the interface display effect. On the other hand, displaying it on the first item allows the user to know which items in the image frame are being displayed with the relevant multimedia resources, thereby improving the information transmission rate and further enhancing the display effect of the second multimedia resource.

[0116] In this embodiment, the description uses the association between the display position of the second multimedia resource and the display position of the first item as an example. In other embodiments, the display position of the second multimedia resource can also be a fixed position on the image frame, such as the lower right corner of the image frame. Alternatively, when an introduction card and special effects are displayed simultaneously on the image frame, the special effects can be displayed at the display position of the first item, and the introduction card can be displayed at a fixed position on the image frame, such as the lower right corner. No specific limitation is made here.

[0117] In this embodiment of the application, the display position of the second multimedia resource in the image frame is determined based on the display position of the first item in the image frame, so that the display position of the second multimedia resource is associated with the display position of the first item. Thus, when the second multimedia resource is displayed, a visual association can be established with the first item in the image frame, thereby improving the display effect.

[0118] In this embodiment, one or more target keywords can be preset, each target keyword indicating a first item in the first multimedia resource. Correspondingly, one or more target time periods can be determined, each target time period corresponding to a first item. Further, since the same target keyword may appear multiple times in the first multimedia resource, and different target keywords can indicate the same first item, alternatively, different target time periods can also correspond to the same first item. Therefore, the process of determining the time period for displaying the second multimedia resource in the first multimedia resource based on the target time period includes the following steps 405-406.

[0119] It should be noted that steps 403-404 above are only one optional implementation step. Therefore, step 405 can also be executed directly after step 402, without specific limitations. Furthermore, when the location information of the target time period does not include the display position of the first item, the display position of the second multimedia resource, etc., steps 405-406 can be executed first after step 402, and then the display position of the second multimedia resource can be determined for the image frames within the selected time period for displaying the second multimedia resource, without having to determine the display position for all target time periods, thus reducing the amount of data processing and improving processing efficiency.

[0120] 405. For each target time period, the server determines the performance evaluation value of the target time period in multiple dimensions based on the location information of the target time period. The location information includes the information of multimedia resource segments and the information of the second item within the target time period. The performance evaluation value of each dimension is used to describe the display effect of the second multimedia resource displayed in the dimension during the target time period.

[0121] The multimedia resource segment within the target time period refers to a segment of the first multimedia resource within the target time period. The information of the multimedia resource segment includes at least one of the following: audio, video, and text; and includes the display position of the first item within the multimedia resource segment, the start and end times of the multimedia resource segment, and its duration. Furthermore, the information of the multimedia resource segment may also include the manner in which the first item appears, indicating whether the first item is an interactive prop, a foreground prop, or a background prop.

[0122] The information of the second item includes, but is not limited to, the name, type, size, and amount of resources required for redemption of the second item. Furthermore, it also includes information about the second multimedia resources used to display the second item, such as the display duration and type of the second multimedia resources. Furthermore, when the second multimedia resources are special effects, it also includes information such as the name and type of the special effects.

[0123] In some embodiments, the structured location information for each target time period can be represented as follows:

[0124]

[0125] The unique identifier of the location is used to identify the target time period. Since multiple target time periods may be identified, an identifier is set to distinguish between different target time periods.

[0126] The display position of the first item can be specifically represented as follows:

[0127]

[0128] Taking the display position of the first item as a rectangular area as an example, float64 left_up_x represents the horizontal coordinate of the upper left corner of the rectangular area recorded in double-precision floating-point data type, float64 left_up_y represents the vertical coordinate of the upper left corner of the rectangular area recorded in double-precision floating-point data type, float64 right_down_x represents the horizontal coordinate of the lower right corner of the rectangular area recorded in double-precision floating-point data type, float64 right_down_y represents the vertical coordinate of the lower right corner of the rectangular area recorded in double-precision floating-point data type, and int64 time represents the display time point of the image frame recorded in 64-bit integer data type, that is, indicating which image frame it is.

[0129] The details can be expressed as follows:

[0130]

[0131] In this embodiment, the effectiveness evaluation value of the target time period in multiple dimensions is determined based on these various pieces of information in the location information. The effectiveness evaluation value of each dimension is related to at least one piece of information in the location information. For example, taking the type dimension as an example, the effectiveness evaluation value of this dimension is determined based on the type of the second item and the type of the second multimedia resource. For instance, the similarity between the type of the second item and the type of the second multimedia resource is determined, and this similarity is used as the effectiveness evaluation value of the type dimension.

[0132] In this embodiment, the effect evaluation value for each dimension indicates whether displaying the second multimedia resource is effective within that dimension and during the target time period. The effect evaluation value is the estimated display effect, which in turn is the display feedback that displaying the second multimedia resource during the target time period will bring. A higher effect evaluation value indicates better feedback. Continuing with the type dimension as an example, if the type of the second item is defined to be more similar to the type of the second multimedia resource, it will bring better display feedback. Since the effect evaluation value is the similarity between the two, a higher effect evaluation value for this dimension indicates better display feedback in that dimension. Similarly, taking the resource quantity required to redeem the second item as an example, if a larger resource quantity is defined to bring better display feedback, and the effect evaluation value is positively correlated with the resource quantity, a higher effect evaluation value for this dimension indicates better display feedback in that dimension.

[0133] 406. Based on the performance evaluation values ​​of multiple target time periods in multiple dimensions, the server selects the target time period that meets the first condition from the multiple target time periods, and uses the selected target time period as the time period for displaying the second multimedia resource in the first multimedia resource.

[0134] In some embodiments, in order to facilitate the comparison and calculation of effect evaluation values ​​of multiple dimensions, the effect evaluation values ​​of each dimension are normalized so that the effect evaluation values ​​are normalized to the range of 1-10, thereby facilitating the selection of target time periods based on the normalized effect evaluation values.

[0135] For example, see the following formula (1), which is a normalization formula.

[0136]

[0137] Among them, X ′ X represents the normalized value, and X represents the effect evaluation value. max X represents the maximum effect evaluation value for this dimension. min This represents the minimum effect evaluation value for this dimension, where 'a' represents the minimum value of the normalized interval and 'b' represents the maximum value of the normalized interval.

[0138] For example, if the performance evaluation values ​​for a certain dimension are 5, 4, and 2 for multiple target time periods, then after normalizing the performance evaluation values ​​for that dimension, the normalized performance evaluation values ​​for each of the multiple target time periods are 10, 7, and 1, respectively.

[0139] In this embodiment, the first condition refers to: the independent evaluation value of the target time period reaches a preset value or the independent evaluation value of the target time period is ranked in the top preset position, and the independent evaluation value is determined based on the effect evaluation value of the target time period in multiple dimensions; or, the joint evaluation value of the combination of time periods in which the target time period is located satisfies the second condition in multiple dimensions, and the joint evaluation value is determined based on the effect evaluation value of each target time period included in the combination of time periods in multiple dimensions; or, the effect evaluation value of the combination of time periods in which the target time period is located is the largest, and the effect evaluation value is determined based on the joint evaluation value of the combination of time periods in multiple dimensions and the independent evaluation value of each target time period included. Accordingly, the process by which the server selects the target time period that satisfies the first condition from multiple target time periods based on the effect evaluation values ​​of each of the multiple target time periods in multiple dimensions includes the following implementation methods.

[0140] (1) For each target time period, the server performs a weighted summation of the effect evaluation values ​​of the target time period in multiple dimensions to obtain the independent evaluation value of the target time period; selects the target time period whose independent evaluation value reaches the preset value from multiple target time periods, or selects the target time period whose independent evaluation value is ranked in the first preset position from multiple target time periods.

[0141] The weight value for each dimension can be set as needed. Preset values ​​and the first preset position can also be set as needed; no specific limitations are made here.

[0142] In this embodiment, for each target time period, the effect evaluation values ​​of its multiple dimensions are weighted and summed to obtain a comprehensive value. This comprehensive value can represent the overall display effect of the target time period in multiple dimensions. Then, the target time periods with good overall display effect are selected, and the second multimedia resource is displayed in the selected target time periods. This further improves the accuracy and effectiveness of the display time periods and can improve the overall display effect of displaying the second multimedia resource in the first multimedia resource.

[0143] (2) The server determines multiple time period combinations based on multiple target time periods, each time period combination including a preset number of target time periods; for each time period combination, based on the effect evaluation values ​​of each target time period in the time period combination in multiple dimensions, the server determines the joint evaluation value of the time period combination in multiple dimensions, the joint evaluation value of each dimension is used to describe the distribution of the second multimedia resources displayed by each of the multiple target time periods in the time period combination in that dimension; based on the joint evaluation values ​​of each of the multiple time period combinations in multiple dimensions, the server selects the time period combination that meets the second condition from the multiple time period combinations, and uses the target time period in the selected time period combination as the selected target time period.

[0144] The preset quantity can be set as needed and is not specifically limited here. Optionally, the preset quantity is positively correlated with the duration of the first multimedia resource. That is, the longer the duration of the first multimedia resource, the more target time periods there are in each time period combination, and the more time periods the second multimedia resource is displayed in the first multimedia resource.

[0145] In some embodiments, for each dimension, based on the performance evaluation value of each target time period in the time period combination in that dimension, a joint evaluation value of the time period combination in that dimension is obtained through a joint scoring function.

[0146] The joint scoring function can be set as needed. To maximize the diversity of display effects across different dimensions for multiple target time periods within a time period combination, i.e., to distribute them as widely as possible, the joint scoring function should be one that increases the joint evaluation value based on the greater the difference. Conversely, to maximize the similarity of display effects across different dimensions for multiple target time periods, the joint scoring function should be one that increases the joint evaluation value based on the greater the similarity. For example, regarding the item type dimension, to maximize the variety of second items displayed across multiple target time periods, the joint scoring function should be one that increases the joint evaluation value based on the greater the difference. Similarly, to maximize the similarity of second item types displayed across multiple target time periods, the joint scoring function should be one that increases the joint evaluation value based on the greater the similarity. From the perspective of time clustering, multiple target time periods should be distributed as widely as possible; therefore, the joint scoring function should be one that increases the joint evaluation value based on the greater the clustering of multiple target time periods and increases the joint evaluation value based on the greater the dispersion of multiple target time periods.

[0147] For example, the joint scoring function can be a variance function, meaning the more dispersed the scoring, the larger the joint evaluation value. See Equation (2) below, which is an example of a variance function.

[0148]

[0149] Where Var(X) represents the joint evaluation value, n represents the number of target time periods in the time period combination, and x i This represents the effectiveness evaluation value for the i-th target time period. This represents the average of the effectiveness evaluation values ​​for multiple target time periods.

[0150] In this embodiment, the second condition refers to: the mean of the joint evaluation values ​​of the time period combination across multiple dimensions is the largest; or, the sum of the joint evaluation values ​​of the time period combination across multiple dimensions is the largest; or, the weighted sum of the joint evaluation values ​​of the time period combination across multiple dimensions is the largest. Accordingly, the above-mentioned method of selecting time period combinations that satisfy the second condition from multiple time period combinations based on their respective joint evaluation values ​​across multiple dimensions includes the following implementation methods:

[0151] For each time period combination, determine the mean of the joint evaluation values ​​across multiple dimensions, and select the time period combination with the largest mean from among multiple time period combinations. Alternatively, for each time period combination, determine the sum of the joint evaluation values ​​across multiple dimensions, and select the time period combination with the largest sum from among multiple time period combinations. Alternatively, perform a weighted sum of the joint evaluation values ​​across multiple dimensions to obtain the effect evaluation value for the time period combination, and select the time period combination with the largest effect evaluation value from among multiple time period combinations.

[0152] In this embodiment, multiple time period combinations are determined, and then based on the effect evaluation values ​​of each target time period in each time period combination in multiple dimensions, a joint evaluation value of the time period combination in multiple dimensions is obtained. This joint evaluation value indicates the display dispersion of multiple target time periods and represents the overall display effect. Then, the target time periods are filtered based on the joint evaluation value, so that the selected multiple time periods have the best overall display effect, thereby improving the filtering quality and improving the overall display effect of displaying the second multimedia resource in the first multimedia resource.

[0153] (3) For each target time period, the server weights and sums the effect evaluation values ​​of the target time period in multiple dimensions to obtain the independent evaluation value of the target time period; based on multiple target time periods, a variety of time period combinations are determined, each time period combination including a preset number of target time periods; for each time period combination, based on the effect evaluation values ​​of each target time period in the combination in multiple dimensions, the joint evaluation value of the time period combination in multiple dimensions is determined, and the joint evaluation value of each dimension is used to describe the distribution of the second multimedia resources displayed by each of the multiple target time periods in the time period combination in the dimension; based on the joint evaluation value of each time period combination in multiple dimensions and the independent evaluation value of each target time period in the time period combination, the time period combination that meets the third condition is selected from the multiple time period combinations, and the target time period in the selected time period combination is taken as the selected target time period.

[0154] In this embodiment, the selection is based not only on the joint evaluation value of multiple target time periods in the time period combination, but also on the independent evaluation value of each of the multiple target time periods in the time period combination. This takes into account both the overall display effect of multiple target time periods on the second multimedia resource and the independent display effect of each of the multiple target time periods on the second multimedia resource. In other words, it balances the overall display effect and the independent display effect, thereby ensuring both the overall display effect of multiple target time periods and the independent display effect, and improving the display effect of the second multimedia resource in the first multimedia resource.

[0155] In this embodiment, the third condition refers to the combination of time periods having the highest effectiveness evaluation value. The effectiveness evaluation value is determined based on the joint evaluation value of the time period combination across multiple dimensions and the independent evaluation value of each target time period included in the combination. Accordingly, the process of selecting time period combinations that satisfy the third condition from multiple time period combinations based on the joint evaluation value of each time period combination across multiple dimensions and the independent evaluation value of each target time period in the time period combination includes the following implementation: for each time period combination, determining the effectiveness evaluation value of the time period combination based on the joint evaluation value of the time period combination across multiple dimensions and the independent evaluation value of each target time period in the time period combination; selecting the time period combination with the highest effectiveness evaluation value from multiple time period combinations.

[0156] In this embodiment, the overall effect evaluation value of the time period combination is determined based on the independent evaluation value and the joint evaluation value, and then the time period combination with the largest effect evaluation value is selected. In this way, the multiple time period combinations in the selected time period combination can not only ensure the independent display effect, but also achieve a better overall display effect, thereby improving the display effect of the second multimedia resource in the first multimedia resource.

[0157] The process of determining the effectiveness evaluation value of a time period combination based on the joint evaluation value of the time period combination across multiple dimensions and the independent evaluation value of each target time period in the time period combination includes the following implementation: For each time period combination, determine the first sum and the second sum corresponding to that time period combination. The first sum refers to the sum of the joint evaluation values ​​of the time period combination across multiple dimensions, and the second sum refers to the sum of the independent evaluation values ​​of each of the multiple target time periods in the time period combination. The first sum and the second sum are weighted and summed to obtain the effectiveness evaluation value of the time period combination.

[0158] The weights of the first and second sums can be set and changed as needed. A higher weight for the first sum indicates a greater emphasis on the individual display effect of the second multimedia resources within each target time period. A higher weight for the second sum indicates a greater emphasis on the overall display effect of the second multimedia resources across multiple target time periods.

[0159] In this embodiment, the target time period in the combination of time periods with the highest effect evaluation value is used as the time period for displaying the second multimedia resource. In other embodiments, the target time period in the combination of time periods with effect evaluation values ​​ranked in other positions can also be used as the time period for displaying the second multimedia resource, such as the target time period in the combination of time periods with effect evaluation values ​​ranked in the second position. No specific limitation is made here.

[0160] In this embodiment, steps 405-406 above are used to determine the time period for displaying the second multimedia resource in the first multimedia resource based on the target time period. In this embodiment, for each target time period, its effect evaluation value in multiple dimensions is determined. Since the effect evaluation value in multiple dimensions is used to indicate its display effect in different dimensions, multiple target time periods that meet certain conditions are selected to display the second multimedia resource based on the effect evaluation values ​​in multiple dimensions of multiple target time periods. This comprehensively considers the display effect of displaying the second multimedia resource in different dimensions in each target time period, and can ensure that the display effect of displaying the second multimedia resource in the selected time period is achieved.

[0161] 407. The server sends the selected time period and the location of the second multimedia resource displayed in the image frame of that time period to the terminal used to play the first multimedia resource.

[0162] In this embodiment of the application, after the server determines the time period for displaying the second multimedia resource and the location for displaying the second multimedia resource, it sends this information to the terminal used to play the first multimedia resource, so that the terminal can display the second multimedia resource at the corresponding time period and location during the playback of the multimedia resource based on this information.

[0163] In some embodiments, the server directly sends the determined display time period and display location to the terminal. In other embodiments, the server first sends the determined display time period and display location to the management object, which then confirms the information before sending it to the terminal. Alternatively, the server determines the effect evaluation values ​​for multiple time period combinations and sends these values ​​to the management object, which then filters the time period combinations. The server then sends the target time period from the filtered combinations to the terminal.

[0164] In some embodiments, after the server determines the display position of the first item in multiple image frames within a target time period, the following implementation method is further included: the server determines key image frames within the target time period based on the display position of the first item in multiple image frames within the target time period, and the display position of the first item in the key image frames is used to restore the display position of the first item in image frames other than the key image frames within the target time period.

[0165] The display position of the first item in the key image frame is sent to the terminal for playing the first multimedia resource, so that the terminal can recover the display position of the first item in the image frames other than the key image frame within the target time period based on the display position of the first item in the key image frame. This can reduce the amount of data sent to the terminal and improve data transmission efficiency.

[0166] The process of determining key image frames within a target time period based on the display position of the first item in multiple image frames within the target time period includes at least one of the following implementation methods:

[0167] (1) If the position of the first item does not change in multiple image frames, the server will determine any image frame within the target time period as the key image frame.

[0168] In this embodiment, the display position of the first item does not change in multiple image frames, indicating that the display position of the first item is the same in multiple image frames. Therefore, the display position of the first item in other image frames can be recovered based on the display position of the first item in any image frame. In this way, the display position of the first item in other image frames can be recovered based on a small number of image frames, reducing the amount of data transmitted to the terminal and improving the terminal's ability to recover the display position of the first item in other image frames.

[0169] (2) If the trajectory of the first item is a straight line in multiple image frames, the server will determine the first and last image frames in the target time period as key image frames.

[0170] In this embodiment, the display position of the first item in multiple image frames is a straight line. Therefore, the display position of the first item in other image frames can be recovered based on the display position of the first item in the first and last image frames. In this way, only two image frames are needed to recover the display position of the first item in other image frames, which reduces the amount of data transmitted to the terminal and makes it easier for the terminal to recover the display position of the first item in other image frames based on these two image frames.

[0171] (3) If the trajectory of the first item in multiple image frames is a parabola, the server will determine the first image frame, the last image frame and the image frame corresponding to the vertex of the parabola in the target time period as key image frames.

[0172] In this embodiment, the display position of the first item changes parabolically across multiple image frames. Based on the display positions of the first item in the first image frame, the last image frame, and the image frame corresponding to the vertex of the parabola, the display positions in other image frames can be recovered. Since the parabola can be recovered based on these three key points, only these three image frames are needed to recover the display positions of the first item in other image frames, reducing the amount of data transmitted to the terminal and making it easier for the terminal to recover the display positions of the first item in other image frames based on these few image frames.

[0173] In some embodiments, the display position of the first item in other image frames is recovered by interpolating the display position of the first item in the key image frame. This further includes the following implementation: the server determines the interpolation method corresponding to the key image frame. The interpolation method is used to interpolate the display position of the first item in the key image frame to recover the display position of the first item in image frames other than the key image frame within the target time period.

[0174] In this embodiment, since the first item may be stationary or moving in the video, when it is moving, by interpolating the display position of the first item in the key image frame, the approaching movement route of the first item can be estimated, so that the non-key image frame can achieve the effect of the second multimedia resource following the first item.

[0175] Since the interpolation methods differ depending on the key image frame, the process of determining the interpolation method corresponding to the key image frame includes at least one of the following implementation methods:

[0176] (1) In cases where the key image frame includes the first and last image frames within the target time period, the server will determine linear interpolation as the interpolation method.

[0177] Specifically, interpolation of the display position is performed between the first and last image frames to restore the display position of the first item in the image frames between them. Taking the display position of the first item as the coordinate information in step 403 as an example, this process is based on linear interpolation of the coordinate information corresponding to the first and last image frames to obtain the coordinate information corresponding to each of the image frames between them, that is, to obtain the display position of the first item.

[0178] (2) In cases where the key image frames include the first image frame, the last image frame and the image frame corresponding to the vertex of the parabola within the target time period, the server will determine the quadratic interpolation as the interpolation method.

[0179] Specifically, interpolation of the display position is performed between the first image frame and the image frame corresponding to the vertex of the parabola, and interpolation of the display position is performed between the image frame corresponding to the vertex of the parabola and the last image frame, in order to restore the display position of the first item in the image frames other than the first image frame, the image frame corresponding to the vertex of the parabola, and the last image frame.

[0180] It should be noted that if the key image frame is any image frame within the target time period, the display position of the first item in that key image frame can be directly used as the display position of the first item in other image frames without further interpolation, and therefore there is no need to determine the interpolation method. Accordingly, the interpolation method field corresponding to the target time period in the data sent to the terminal will be empty.

[0181] In the above embodiments, the restoration of the display position of the first item is used as an example. If the position for displaying the second multimedia resource is the display position of the first item, then the second multimedia resource can be directly displayed at the restored display position of the first item. If the position for displaying the second multimedia resource is not the display position of the first item, but is an adjacent position to the display position of the first item, then the terminal also needs to determine the position for displaying the second multimedia resource based on the restored display position of the first item.

[0182] In other embodiments, since the location of the second multimedia resource is associated with the display location of the first item, after the server determines the key image frame, it can also determine the location of the second multimedia resource in the key image frame based on the display location of the first item in the key image frame, and send the location to the terminal so that the terminal can determine the location of the second multimedia resource in other image frames based on the location, such as determining the location based on interpolation. The process is the same as described above and will not be repeated here.

[0183] In the above embodiments, the example of sending the display position of the first item in a key image frame to the terminal was used for illustration. In other embodiments, the display positions of the first item in multiple image frames within a target time period can also be sent to the terminal, allowing the terminal to directly obtain the display positions of the first item in multiple image frames within the target time period. Accordingly, when sending the display positions of the first item in multiple image frames, an array can be used to record the display positions of the first item in these multiple image frames and the display time period of each image frame, and then the array can be sent to the terminal.

[0184] In some embodiments, the structured display information for each time period of the second multimedia resource is as follows, and this information is sent to the terminal.

[0185]

[0186] Taking the display of special effects at the display position of the first item as an example, the terminal is sent the start and end timestamps of the display time period, the display position of the first item in the key image frame, the interpolation method, the name of the special effect, and the extended parameters that the special effect depends on. The extended parameters are used to indicate how to display the special effect, such as the brightness and color of the special effect.

[0187] In some embodiments, the server can directly call the relevant interface to send the display information for each selected time period to the terminal along with the playback data of the first multimedia resource; alternatively, the server can send the display information for each time period to a management object, which will then confirm it before sending it to the terminal. When the client player on the terminal parses the playback data of the first multimedia resource, if an event to display the second multimedia resource occurs within the corresponding time period, the second multimedia resource is displayed based on the display information for the time period corresponding to that event. During this process, the display position of the first item can also be automatically interpolated to achieve the automatic following effect of the second multimedia resource across multiple image frames while sending less information.

[0188] For example, see Figure 5 , Figure 5 This is a schematic diagram of a resource display system provided in an embodiment of this application. The system uses video as the first multimedia resource, special effects as the second multimedia resource, and the first and second items being identical as an example. When determining the time period and location of the displayed special effects on the server side, the audio of the video is first analyzed to obtain text. Then, word segmentation analysis is performed on the audio text and the video subtitle text. Based on the analysis results, a coarse-grained time period is obtained, which is the time period in which the name of the first item appears in the text. This first time period is identified based on start and end timestamps. Then, keyframes are extracted from the video based on the start and end times of the first time period. Keyframes are image frames within the second time period corresponding to the first time period. Image recognition is performed on each keyframe, and then the image frames within the second time period are filtered based on the recognition results to obtain a fine-grained time period, which is the target time period and the coordinate sequence of the display position of the first item in multiple image frames within the target time period. Then, based on the pre-established strategy, multiple target time periods are evaluated in association, including independent evaluation of multiple target time periods and joint evaluation of multiple time period combinations. Based on the weighted evaluation ranking strategy, the independent evaluation values ​​and joint evaluation values ​​are weighted and summed to select the time period combination that ranks first. After the management object confirms the selected time period combination, it enters the usage stage. In this stage, the display information of the time period is automatically delivered through the resource placement and tracking system. Then, it is sent to the terminal along with the audio and video data of the video. When the terminal plays the audio and video, the corresponding special effect display event is triggered based on the timestamp, so that the special effects can be displayed in the video based on the tracking information.

[0189] In this embodiment, multiple audio and video elements are comprehensively used to ensure the accurate timing of the second multimedia resource display while significantly reducing the cost of confirming the display timing. First, text analysis is performed to quickly extract target keywords indicating the item and determine its time period. Based on the initially determined time period, audio and video materials within the preceding and following intervals are analyzed. For video materials, image frames within the time period are obtained based on the video encoding method for analysis, further reducing the number of image frames to be processed. In this process, because text analysis is relatively inexpensive, the text is shared first to narrow down the time period selection range. Fine-grained time periods are then determined based on image frames within the coarse-grained time period, without requiring frame-by-frame analysis of the multimedia resource, thus greatly reducing the cost of determining the display time period and improving efficiency. To balance display effect and user experience, multiple target time periods are evaluated in association, comprehensively quantifying the value of the time period for displaying the second multimedia resource. Evaluation factors such as clustering, correlation, and display value are introduced to evaluate the target time periods in the time period combination, prioritizing the time period with the highest effect evaluation value for displaying the second multimedia resource. In this process, based on the determined target time period, data such as its timestamp, duration, and secondary multimedia resources are acquired. While fully showcasing the secondary multimedia resources, the target time period is ensured to be distributed and differentiated to a certain extent, thus guaranteeing the display quality of the secondary multimedia resources and minimizing the impact on user experience. Furthermore, based on the image frame analysis results of the video, the display position of the primary item within the image frame can be accurately located. This allows for the display of special effects of the item during the playback of the primary multimedia resources, improving the interactivity and naturalness of the secondary multimedia resources and enhancing audience engagement and memorability.

[0190] This application provides a method for determining the display time period of a resource. When determining the time period for displaying a second multimedia resource within a first multimedia resource, the method first determines the first time period containing the text segment showing the target keyword based on the text of the first multimedia resource. Then, it analyzes the image frames within the time period based on the first time period to determine the target time period containing the image frames showing the first item. This method first determines a coarse-grained time period based on the text, thus requiring analysis only of the image frames within this portion of the time period, without needing to analyze all image frames of the first multimedia resource. This effectively shortens the analysis scope of image frames, avoids high resource consumption, and improves efficiency. Furthermore, determining the target time period based on both text and image frame information ensures that the determined target time period is the accurate time period when the relevant item appears, thereby accurately determining the time period for displaying the second media resource. This approach improves the efficiency of determining the display time period while ensuring its quality.

[0191] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0192] Please refer to Figure 6 , Figure 6 This is a block diagram of a device for determining a resource display time period according to an embodiment of this application. The device has the function of implementing the above-described method for determining a resource display time period; this function can be implemented in hardware or by hardware executing corresponding software. Figure 6 As shown, the device may include:

[0193] The first determining module 601 is used to determine a first time period in the first multimedia resource based on the text of the first multimedia resource. The first time period is the time period in which the text segment containing the target keyword appears.

[0194] The second determining module 602 is used to determine the target time period from the second time period corresponding to the first time period. The second time period is the time period in the first multimedia resource based on the first time period. The target time period is the time period in which the image frame of the first item appears. The first item is the item indicated by the target keyword.

[0195] The third determining module 603 is used to determine, based on the target time period, the time period in which the second multimedia resource is displayed in the first multimedia resource, and the second multimedia resource is used to display the second item associated with the first item.

[0196] In some embodiments, there are multiple target time periods, and each target time period corresponds to a first item; the third determining module 603 includes:

[0197] The first determining unit is used to determine the effect evaluation value of the target time period in multiple dimensions based on the location information of the target time period for each target time period. The location information includes the information of multimedia resource segments and the information of the second item within the target time period. The effect evaluation value of each dimension is used to describe the display effect of the second multimedia resource displayed in the dimension during the target time period.

[0198] The second determining unit is used to select target time periods that meet the first condition from multiple target time periods based on the effect evaluation values ​​of each of the multiple target time periods in multiple dimensions, and to use the selected target time periods as the time periods for displaying the second multimedia resources in the first multimedia resources.

[0199] In some embodiments, the second determining unit is configured to:

[0200] For each target time period, the performance evaluation values ​​of the target time period in multiple dimensions are weighted and summed to obtain the independent evaluation value of the target time period;

[0201] Select the target time period from multiple target time periods where the independent evaluation value reaches the preset value, or select the target time period from multiple target time periods where the independent evaluation value is ranked in the first preset position.

[0202] In some embodiments, the second determining unit is configured to:

[0203] Based on multiple target time periods, various combinations of time periods are determined, and each combination of time periods includes a preset number of target time periods.

[0204] For each time period combination, based on the effect evaluation values ​​of each target time period in the time period combination in multiple dimensions, the joint evaluation value of the time period combination in multiple dimensions is determined. The joint evaluation value of each dimension is used to describe the distribution of the second multimedia resources displayed by each of the multiple target time periods in the time period combination in the dimension.

[0205] Based on the joint evaluation values ​​of various time period combinations across multiple dimensions, time period combinations that meet the second condition are selected from the various time period combinations, and the target time period in the selected time period combinations is taken as the selected target time period.

[0206] In some embodiments, the second determining unit is configured to:

[0207] For each target time period, the performance evaluation values ​​of the target time period in multiple dimensions are weighted and summed to obtain the independent evaluation value of the target time period;

[0208] Based on multiple target time periods, various combinations of time periods are determined, and each combination of time periods includes a preset number of target time periods.

[0209] For each time period combination, based on the effect evaluation values ​​of each target time period in the time period combination in multiple dimensions, the joint evaluation value of the time period combination in multiple dimensions is determined. The joint evaluation value of each dimension is used to describe the distribution of the second multimedia resources displayed by each of the multiple target time periods in the time period combination in the dimension.

[0210] Based on the joint evaluation value of each time period combination in multiple dimensions and the independent evaluation value of each target time period in the time period combination, time period combinations that meet the third condition are selected from multiple time period combinations, and the target time period in the selected time period combinations is taken as the selected target time period.

[0211] In some embodiments, the second determining unit is configured to:

[0212] For each time period combination, the effectiveness evaluation value of the time period combination is determined based on the joint evaluation value of the time period combination in multiple dimensions and the independent evaluation value of each target time period in the time period combination.

[0213] Select the time period combination with the highest effect evaluation value from multiple time period combinations.

[0214] In some embodiments, the apparatus further includes a fourth determining module, configured to:

[0215] For each image frame within the target time period, determine the display position of the first item in the image frame;

[0216] Based on the display position of the first item in the image frame, determine the position in the image frame where the second multimedia resource is displayed.

[0217] In some embodiments, the fourth determining module is configured to perform at least one of the following:

[0218] The second multimedia resource is an introduction card for the second item. Based on the display position of the first item in the image frame, the adjacent positions of the display position of the first item in the image frame are determined as the positions for displaying the introduction card.

[0219] The second multimedia resource is a special effect displayed on the first item. The special effect is used to display the second item after being triggered. Based on the display position of the first item in the image frame, the display position of the first item is determined as the position for displaying the special effect.

[0220] In some embodiments, the apparatus further includes a fifth determining module, configured to:

[0221] Based on the display position of the first item in multiple image frames within the target time period, key image frames within the target time period are determined. The display position of the first item in the key image frames is used to restore the display position of the first item in image frames other than the key image frames within the target time period.

[0222] In some embodiments, the fifth determining module is configured to perform at least one of the following:

[0223] If the position of the first item remains unchanged across multiple image frames, any image frame within the target time period is identified as the key image frame.

[0224] If the trajectory of the first item is a straight line in multiple image frames, the first and last image frames in the target time period are both determined as key image frames.

[0225] When the trajectory of the first item in multiple image frames is a parabola, the first image frame, the last image frame, and the image frame corresponding to the vertex of the parabola within the target time period are all determined as key image frames.

[0226] In some embodiments, the apparatus further includes a sixth determining module, configured to:

[0227] The interpolation method corresponding to the key image frame is determined. The interpolation method is used to interpolate the display position of the first item in the key image frame in order to restore the display position of the first item in image frames other than the key image frame within the target time period.

[0228] In some embodiments, the sixth determining module is configured to perform at least one of the following:

[0229] When the key image frames include the first and last image frames within the target time period, linear interpolation is determined as the interpolation method.

[0230] When the key image frames include the first image frame, the last image frame, and the image frame corresponding to the vertex of the parabola within the target time period, quadratic interpolation is determined as the interpolation method.

[0231] This application provides a device for determining the display time period of a resource. When determining the time period for displaying a second multimedia resource within a first multimedia resource, the device first determines a first time period based on the text of the first multimedia resource, identifying the text segment containing the target keyword. Then, it analyzes image frames within a time period based on the first time period to determine the target time period containing the image frames of the first item. This device first determines a coarse-grained time period based on the text, thus requiring analysis only of the image frames within this specific time period, rather than analyzing all image frames of the first multimedia resource. This effectively shortens the analysis scope of image frames, avoids high resource consumption, and improves efficiency. Furthermore, determining the target time period based on both text and image frame information ensures that the determined target time period is the accurate time period in which the relevant item appears, thereby accurately determining the time period for displaying the second media resource. This approach improves the efficiency of determining the display time period while maintaining its quality.

[0232] Please refer to Figure 7 This diagram illustrates a structural block diagram of a computer device 700 provided in one embodiment of this application. The computer device 700 can be a terminal 10 or a server 20 in an implementation environment. The computer device 700 is used to implement the method for determining the resource display time period provided in the above embodiment. Specifically:

[0233] Typically, computer device 700 includes a processor 710 and a memory 720.

[0234] Processor 710 may include one or more processing cores, such as a quad-core processor or a seven-core processor. Processor 710 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 710 may also include a main processor and a coprocessor. The main processor, also known as the central processing unit, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 710 may include a GPU, which is responsible for executing the method steps provided in this application.

[0235] The memory 720 may include one or more computer-readable storage media, which may be non-transitory. The memory 720 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 720 are used to store a computer program configured to be executed by one or more processors (such as a GPU) to implement the method for determining the resource display time period described above.

[0236] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0237] This application also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the method for determining the resource display time period in any of the above implementations.

[0238] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to execute the method for determining the resource display time period of any of the above implementations.

[0239] In some embodiments, the computer program product involved in the present application may be deployed and executed on a computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected through a communication network.

[0240] It should be noted that the data collection and processing in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0241] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0242] All the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here. The above are only optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining the time period for resource display, characterized in that, The method includes: Based on the text of the first multimedia resource, a first time period is determined in the first multimedia resource, wherein the first time period is the time period in which the text segment containing the target keyword appears; The target time period is determined from the second time period corresponding to the first time period. The second time period is the time period in the first multimedia resource based on the first time period. The target time period is the time period in which the image frame of the first item appears. The first item is the item indicated by the target keyword. Based on the target time period, a time period for displaying the second multimedia resource in the first multimedia resource is determined, wherein the second multimedia resource is used to display a second item associated with the first item.

2. The method according to claim 1, characterized in that, There are multiple target time periods, each corresponding to a first item; determining the time period for displaying the second multimedia resource in the first multimedia resource based on the target time periods includes: For each target time period, based on the location information of the target time period, the effect evaluation value of the target time period in multiple dimensions is determined. The location information includes the information of multimedia resource segments within the target time period and the information of the second item. The effect evaluation value of each dimension is used to describe the display effect of the second multimedia resource displayed in the target time period in that dimension. Based on the effect evaluation values ​​of multiple target time periods in the multiple dimensions, target time periods that meet the first condition are selected from the multiple target time periods, and the selected target time periods are used as the time periods for displaying the second multimedia resource in the first multimedia resource.

3. The method according to claim 2, characterized in that, The step of selecting target time periods that meet the first condition from the multiple target time periods based on their respective effect evaluation values ​​across multiple dimensions includes: For each target time period, the effect evaluation values ​​of the target time period in the multiple dimensions are weighted and summed to obtain the independent evaluation value of the target time period; Select the target time period from the plurality of target time periods where the independent evaluation value reaches a preset value, or select the target time period from the plurality of target time periods where the independent evaluation value is ranked in the first preset position.

4. The method according to claim 2, characterized in that, The step of selecting target time periods that meet the first condition from the multiple target time periods based on their respective effect evaluation values ​​across multiple dimensions includes: Based on the multiple target time periods, various combinations of time periods are determined, and each combination of time periods includes a preset number of target time periods; For each time period combination, based on the effect evaluation values ​​of each target time period in the time period combination in the multiple dimensions, the joint evaluation value of the time period combination in the multiple dimensions is determined. The joint evaluation value of each dimension is used to describe the distribution of the second multimedia resources displayed by each of the multiple target time periods in the time period combination in the dimension. Based on the joint evaluation values ​​of the various time period combinations in the multiple dimensions, time period combinations that meet the second condition are selected from the various time period combinations, and the target time period in the selected time period combinations is taken as the selected target time period.

5. The method according to claim 2, characterized in that, The step of selecting target time periods that meet the first condition from the multiple target time periods based on their respective effect evaluation values ​​across multiple dimensions includes: For each target time period, the effect evaluation values ​​of the target time period in the multiple dimensions are weighted and summed to obtain the independent evaluation value of the target time period; Based on the multiple target time periods, various combinations of time periods are determined, and each combination of time periods includes a preset number of target time periods; For each time period combination, based on the effect evaluation values ​​of each target time period in the time period combination in the multiple dimensions, the joint evaluation value of the time period combination in the multiple dimensions is determined. The joint evaluation value of each dimension is used to describe the distribution of the second multimedia resources displayed by each of the multiple target time periods in the time period combination in the dimension. Based on the joint evaluation value of each time period combination in the multiple dimensions and the independent evaluation value of each target time period in the time period combination, time period combinations that meet the third condition are selected from the multiple time period combinations, and the target time period in the selected time period combinations is taken as the selected target time period.

6. The method according to claim 5, characterized in that, The step of selecting time period combinations that satisfy the third condition from the multiple time period combinations based on the joint evaluation value of each time period combination across the multiple dimensions and the independent evaluation value of each target time period in the time period combination includes: For each time period combination, the effect evaluation value of the time period combination is determined based on the joint evaluation value of the time period combination in the multiple dimensions and the independent evaluation value of each target time period in the time period combination. The time period combination with the highest effect evaluation value is selected from the various time period combinations.

7. The method according to claim 1, characterized in that, The method further includes: For each image frame within the target time period, determine the display position of the first item within the image frame; Based on the display position of the first item in the image frame, the position in the image frame where the second multimedia resource is displayed is determined.

8. The method according to claim 7, characterized in that, Determining the position of the second multimedia resource in the image frame based on the display position of the first item in the image frame includes: The second multimedia resource is an introduction card for the second item. Based on the display position of the first item in the image frame, the adjacent positions of the display position of the first item in the image frame are determined as the positions for displaying the introduction card. The second multimedia resource is a special effect displayed on the first item. The special effect is used to display the second item after being triggered. Based on the display position of the first item in the image frame, the display position of the first item is determined as the position for displaying the special effect.

9. The method according to claim 7, characterized in that, The method further includes: Based on the display position of the first item in multiple image frames within the target time period, key image frames within the target time period are determined. The display position of the first item in the key image frames is used to restore the display position of the first item in image frames other than the key image frames within the target time period.

10. The method according to claim 9, characterized in that, The step of determining key image frames within the target time period based on the display position of the first item in multiple image frames within the target time period includes at least one of the following: If the position of the first item remains unchanged in the plurality of image frames, any image frame within the target time period is determined as the key image frame; If the trajectory of the first item in the plurality of image frames is a straight line, the first and last image frames in the target time period are both determined as the key image frames; When the trajectory of the first item in the plurality of image frames is a parabola, the first image frame, the last image frame, and the image frame corresponding to the vertex of the parabola within the target time period are all determined as the key image frames.

11. The method according to claim 10, characterized in that, The method further includes: The interpolation method corresponding to the key image frame is determined. The interpolation method is used to interpolate the display position of the first item in the key image frame to restore the display position of the first item in image frames other than the key image frame within the target time period.

12. The method according to claim 11, characterized in that, The method for determining the interpolation method corresponding to the key image frame includes at least one of the following: When the key image frame includes the first and last image frames within the target time period, linear interpolation is determined as the interpolation method. When the key image frame includes the first image frame, the last image frame, and the image frame corresponding to the vertex of the parabola within the target time period, quadratic interpolation is determined as the interpolation method.

13. A device for determining the time period for resource display, characterized in that, The device includes: The first determining module is used to determine a first time period in the first multimedia resource based on the text of the first multimedia resource, wherein the first time period is the time period in which the text segment containing the target keyword appears; The second determining module is used to determine a target time period from the second time period corresponding to the first time period, wherein the second time period is the time period in the first multimedia resource based on the first time period, the target time period is the time period in which the image frame of the first item appears, and the first item is the item indicated by the target keyword. The third determining module is used to determine, based on the target time period, the time period in which the second multimedia resource is displayed in the first multimedia resource, and the second multimedia resource is used to display a second item associated with the first item.

14. A computer device, characterized in that, The computer device includes a processor and a memory, the memory being used to store a computer program, the computer program being loaded by the processor and executed as the method for determining the resource display time period according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for executing the method for determining the resource display time period as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method for determining the resource display time period as described in any one of claims 1 to 12.

Images