Emoji package generation method and device, electronic equipment, medium and program product

Abstract

The invention provides an expression package generation method and device, electronic equipment, a medium and a program product, and relates to the technical field of data processing, in particular to the fields of video processing, content generation, artificial intelligence and the like. According to the implementation scheme, video data used for generating an emoji package is obtained, wherein the video data comprises content associated with interaction between a first target object and a second target object; based on the video data, time-varying distribution data of interaction strength between the first target object and the second target object is determined, and the distribution data indicates a peak moment when the interaction strength meets a preset standard; performing sequential relationship fusion on the distribution data to determine at least one target moment associated with the peak moment; and extracting a video clip corresponding to the peak moment and the at least one target moment from the video data to generate an expression package.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to the fields of video processing, content generation, artificial intelligence, etc. Specifically, it relates to an expression package generation method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the rapid development of digital media and social media, expression packages have become an indispensable part of people's daily communication. Expression packages not only enrich language expression, but also quickly convey emotions and attitudes, enhance interactivity and interest. In recent years, the rise of video live streaming has further driven the demand for expression packages, and users can express different emotions about live streaming content by sending expression packages.

[0003] The methods described in this section can not necessarily be the methods previously conceived or adopted. Unless otherwise indicated, any method described in this section should not be considered prior art merely because it is included in this section. Similarly, unless otherwise indicated, the problems mentioned in this section should not be considered to have been recognized in any prior art. SUMMARY

[0004] The present disclosure provides an expression package generation method, device, electronic equipment, computer readable storage medium and computer program product.

[0005] According to an aspect of the present disclosure, an expression package generation method is provided, comprising: obtaining video data for generating an expression package, the video data comprising content associated with an interaction between a first target object and a second target object; determining distribution data of an interaction intensity between the first target object and the second target object changing with time based on the video data, the distribution data indicating a peak moment when the interaction intensity meets a preset standard; performing time sequence relationship fusion on the distribution data to determine at least one target moment associated with the peak moment; and extracting a video clip corresponding to the peak moment and the at least one target moment from the video data to generate an expression package.

[0006] According to an aspect of the present disclosure, there is provided an expression package generation apparatus, comprising: a data acquisition module configured to acquire video data for generating an expression package, the video data comprising content associated with an interaction between a first target object and a second target object; a data determination module configured to determine, based on the video data, distribution data of an interaction intensity between the first target object and the second target object varying with time, the distribution data indicating a peak moment when the interaction intensity meets a preset standard; a target determination module configured to perform time sequence relationship fusion on the distribution data to determine at least one target moment associated with the peak moment; and a result generation module configured to extract, from the video data, video clips corresponding to the peak moment and the at least one target moment to generate the expression package.

[0007] According to another aspect of the present disclosure, there is provided an electronic device, comprising at least one processor; and a memory communicatively connected with the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method as described above.

[0009] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method as described above.

[0010] According to one or more embodiments of the present disclosure, an expression package with emotional expression in interaction can be generated, which helps to improve user interaction effect.

[0011] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments and together with the description serve to explain exemplary implementations of the application. The illustrated embodiments are exemplary only and not limiting of the scope of the claims. In all the drawings, like reference numerals refer to like parts throughout the several views.

[0013] Figure 1 shows a schematic diagram of an exemplary system in which the various methods described herein can be implemented according to embodiments of the present disclosure;

[0014] Figure 2 A flowchart of an expression package generation method according to an embodiment of the present disclosure is shown;

[0015] Figure 3 A schematic diagram of first interaction information and second interaction information according to an embodiment of the present disclosure is shown;

[0016] Figure 4 A schematic diagram of a process of generating an expression package according to an embodiment of the present disclosure is shown;

[0017] Figure 5 A structural block diagram of an expression package generation apparatus according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A structural block diagram of an expression package generation apparatus according to another embodiment of the present disclosure is shown; and

[0019] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0021] In the present disclosure, the terms "first", "second", and the like are used to describe various elements only and do not intend to limit the positional relationship, the time sequence relationship, or the importance relationship of the elements, and such terms are only used to distinguish one element from another. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0022] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing the specific examples and are not intended to be limiting. Unless the number of elements is specifically limited, the element can be one or more than one, if the number of elements is not specifically limited. In addition, the term "and / or" used in the present disclosure encompasses any one of the listed items and all possible combinations thereof.

[0023] In the related art, an emoticon package can generally be generated based on a preset template, or manually generated from a video by extracting a segment. In the method based on the preset template, the preset template has a fixed layout and style pre-set for different scenes, and only needs to insert corresponding text in the preset template to generate an emoticon package. In the method of manually generating an emoticon package from a video by extracting a segment, it is usually limited to simply extracting interesting content from a video work such as a movie or a short video to generate an emoticon package. However, none of the current methods develops and utilizes the emotional communication and feedback generated in the interaction process between people, and it is difficult to well adapt to the rapid development of digital media and social media, especially in the video live streaming scenario with increasing demand.

[0024] To this end, embodiments of the present disclosure provide a more effective emoticon package generation method.

[0025] Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 A schematic diagram of an example system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of the present disclosure is shown. Referring to Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more application programs.

[0027] In embodiments of the present disclosure, the server 120 can run one or more services or software applications that enable the performance of one or more services or software applications described in embodiments of the present disclosure.

[0028] In certain embodiments, the server 120 can also provide other services or software applications, which can include non-virtual and virtual environments. In certain embodiments, these services can be provided as web-based services or cloud services, for example, to users of the client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0029] In Figure 1In the illustrated configuration, the server 120 can include one or more components that implement functionality performed by the server 120. These components can include software components that are executable by one or more processors, hardware components, or combinations thereof. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 can in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a wide variety of system configurations are possible, which can differ from system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0030] Users can participate in interactive scenarios, such as live video broadcasts, using the client devices 101, 102, 103, 104, 105, and / or 106. The client devices can provide interfaces that enable users of the client devices to interact with the client devices. The client devices can also output information to the users via the interfaces. Although Figure 1 Only six client devices are depicted, but one of skill in the art will appreciate that the present disclosure can support any number of client devices.

[0031] The client devices 101, 102, 103, 104, 105, and / or 106 can include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service kiosk devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices can run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices can include cellular telephones, smartphones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices can include head-mounted displays, such as smart glasses, and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices, and the like. The client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0032] Network 110 can be any type of network familiar to those skilled in the art, which can support data communications using any of a variety of available protocols, including without limitation TCP / IP, SNA, IPX, etc. As an example only, one or more of networks 110 can be a LAN, an Ethernet network, a Token Ring network, a WAN, the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a local area network (LAN), a wide area network (WAN), a wireless network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a Bluetooth network, a WIFI network), and / or any combination of these and / or other networks.

[0033] Server 120 can include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers), server clusters, or any other appropriate arrangement and / or combination. Server 120 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 can run one or more services or software applications that provide the functionality described below.

[0034] Computing units in server 120 can run one or more operating systems, including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 can also run any of a variety of additional server applications and / or mid-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0035] In some implementations, server 120 can include one or more applications to analyze and consolidate data feeds and / or event updates from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0036] In some implementations, server 120 can be a server of a distributed system, or a server that incorporates blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

[0037] The system 100 can also include one or more databases 130. In certain embodiments, these databases can be used to store data and other information. For example, one or more of the databases 130 can be used to store information such as audio files and video files. The databases 130 can reside in various locations. For example, databases used by the server 120 can be local to the server 120, or can be remote from the server 120 and can communicate with the server 120 via a network- based or dedicated connection. The databases 130 can be of different types. In certain embodiments, databases used by the server 120 can be, for example, relational databases. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

[0038] In certain embodiments, one or more of the databases 130 can also be used by applications to store application data. Databases used by applications can be databases of different types, such as key-value stores, object stores, or regular stores supported by file systems.

[0039] Figure 1 The system 100 can be configured and operated in various ways to enable the application of various methods and apparatuses described in accordance with the present disclosure.

[0040] Figure 2 A flowchart of an expression package generation method 200 according to an embodiment of the present disclosure is shown.

[0041] As shown in Figure 2 The method 200 includes a step S201, a step S202, a step S203, and a step S204.

[0042] In step S201, video data for generating an expression package is acquired, the video data including content associated with an interaction between a first target object and a second target object.

[0043] In an example, the first target object and the second target object can refer to any two participants in an interactive scenario, and thus the content included in the video data can be video material recording the interactive process of the two participants. The interaction between the first target object and the second target object can be carried out online or offline in a face-to-face live interaction, and thus the video data can be, for example, a real-time video stream in a video live broadcast, or a video file obtained by recording the live interactive process, etc.

[0044] In step S202, based on the video data, distribution data of an interaction intensity between the first target object and the second target object changing over time is determined, the distribution data indicating a peak moment when the interaction intensity meets a preset standard.

[0045] In an example, since the video data contains content associated with the interaction, by analyzing the video data, it can be determined how the intensity of the interaction between the two parties changes over time, i.e. distribution data of the intensity of the interaction over time. Therefore, the distribution data indicates which time point or time points are the peak moments when the intensity of the interaction meets the preset standard. The intensity of the interaction can be a quantification of emotional expression in the process of interaction between the two parties, which generally fluctuates over time, for example, when the interaction initiator (such as a live broadcast host, a host, etc.) mentions a specific topic or keyword, it can trigger a peak of interaction between the two parties, so the distribution data can indicate the peak moment when the intensity of the interaction meets the preset standard. When the intensity of the interaction is high, the atmosphere or atmosphere embodied by the content of the video data is more intense, and vice versa, when the intensity of the interaction is low, the atmosphere or atmosphere embodied by the content of the video data is more flat. In the scene using artificial intelligence technology, the video data can be understood by an artificial intelligence model to identify which dimensions can be used to analyze such atmosphere or atmosphere from its content.

[0046] In an example, whether the intensity of the interaction meets the preset standard can be determined based on the interaction information provided by the two parties in the interaction. Taking the interaction receiver as an example, one way of determining can be that when the number of comments or bullet screens with keywords sent by the interaction receiver per second reaches a certain number (such as 5 per second), it is considered that the intensity of the interaction meets the preset standard.

[0047] In an example, the intensity of the interaction can be intuitively represented in a binary quantization manner (for example, the intensity of the interaction when meeting and not meeting the preset standard is 1 and 0 respectively), or further quantization granularity can be set to obtain more refined quantization intervals (for example, further dividing several intervals between the intensity of the interaction 1 and 0).

[0048] In step S203, the distribution data is fused in time sequence to determine at least one target moment associated with the peak moment.

[0049] In an example, the time series relationship fusion is an operation for processing and analyzing time series data, aiming to capture the dynamic changes and mutual dependence of the data in the time dimension. As mentioned above, the distribution data is time series data of the interaction intensity changing over time. Since the distribution data indicates the peak moment when the interaction intensity meets the preset standard, by performing time series relationship fusion on the distribution data, a number of target moments associated with the peak moment can be found. That is, the interaction behavior of the target moment can jointly contribute to the interaction peak between the two parties at the peak moment. For example, by analyzing a certain video data, it is found that the interaction intensity meets the preset standard in the period of 3 minutes and 25 seconds to 3 minutes and 40 seconds, reaching an interaction peak, but it is determined via time series relationship fusion that this is because the live host mentioned the message of 50% off in the period of 3 minutes and 15 seconds to 3 minutes and 20 seconds, thereby forming the interaction peak. Therefore, the period of 3 minutes and 15 seconds to 3 minutes and 20 seconds is a target moment related to the peak moment of 3 minutes and 25 seconds to 3 minutes and 40 seconds.

[0050] In step S204, video clips corresponding to the peak moment and the at least one target moment are extracted from the video data to generate the sticker.

[0051] In an embodiment of the present disclosure, a method for generating a sticker based on the emotional expression between the two parties of the interaction is provided. In the method, after finding the peak moment when the interaction intensity meets the preset standard from the video data as the original material of the sticker, further time series relationship fusion is performed to find the target moment related to the peak moment, where the interaction behavior of the target moment can jointly contribute to the interaction peak between the two parties at the peak moment, thereby effectively extracting the associated moment from the video data that is most closely related to the emotional expression of the interaction between the two parties, thus improving the emotional expression of the sticker, enhancing the connection between the interaction receiver such as a user and the content, and further helping to improve the user interaction rate and the content conversion rate.

[0052] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution comply with the relevant legal regulations and do not violate public order and good customs.

[0053] In some embodiments, the content associated with the interaction between the first target object and the second target object can include first interaction information and second interaction information, the first interaction information being determined based on one of the first target object and the second target object as an interaction sender, and the second interaction information being determined based on one of the first target object and the second target object as an interaction receiver.

[0054] Correspondingly, as combined with the above description of the method for generating a sticker, the method for generating a sticker can be implemented by the sticker generation device. Figure 2The step S201 determines distribution data of the interaction intensity between the first target object and the second target object changing over time based on the video data, which can include: determining interaction intensity sequence data associated with the interaction intensity based on the first interaction information and the second interaction information; and mapping the interaction intensity sequence data to a time axis of the video data to obtain the distribution data.

[0055] In an example, the first interaction information being associated with the interaction sender can mean that the first interaction information can be used to reflect the scene semantics. The scene semantics refers to semantic information related to the current scene, which can help to understand the emotional atmosphere of the current scene. Taking a live promotion scene as an example, assuming that the host is introducing a limited-time purchase product, the host may mention keywords such as “limited-time purchase”, “discount”, “super value discount”, etc., and therefore semantic information related to these keywords can be at least part of the first interaction information. The semantic information related to the keywords can be determined by a FastText model. In addition, the rhythm and style of the background music at this time can also reflect the emotional atmosphere of the current scene, such as fast-paced music that is usually used in promotion scenes, and therefore semantic information related to these background music can also be at least part of the first interaction information. The semantic information related to the background music can be determined by MFCC (Mel-Frequency Cepstral Coefficients) spectrum analysis.

[0056] In an example, the second interaction information being associated with the interaction receiver can mean that the second interaction information can be used to reflect the emotional feedback of the interaction receiver in the corresponding scene. Still taking the above live promotion scene as an example, when the host mentions the above keywords, the user may send real-time bullet screen or comments such as “too good”, “I want to buy”, “too cost-effective” as feedback content, and therefore semantic information related to these feedback content can be at least part of the second interaction information. The semantic information related to the above bullet screen or comments can be determined by a BERT model. In addition, when the user actively participates in the live broadcast, he or she may actively request to connect with the host live, and therefore may also actively turn on the local camera equipment, at which time facial expressions such as micro-expressions of the user can be captured, thereby judging whether the emotional feedback reaches a certain degree, for example, when the mouth corner is raised to a certain angle, it is considered that the interaction has high attraction to the user, and therefore information obtained from these facial expressions can also be at least part of the second interaction information. Visual features can be extracted from the above facial expressions, and image features generated by a ResNet model can be analyzed.

[0057] Figure 3 A schematic diagram of the first interaction information and the second interaction information according to an embodiment of the present disclosure is shown.

[0058] AsFigure 3 As shown, still taking the live promotion scenario as an example, the first interaction information 310 associated with the interaction sending party such as the live host can include first sub-information 311 and second sub-information 312, where the first sub-information 311 can be related to the keywords mentioned by the live host, and the second sub-information 312 can be related to the emotional atmosphere of the current live promotion scenario.

[0059] In addition, the second interaction information 320 associated with the interaction receiving party such as the user can include third sub-information 321 and fourth sub-information 322, where the third sub-information 321 can be related to the real-time pop-up messages or comments sent by the user, and the fourth sub-information 322 can be related to the facial expression of the user.

[0060] It should be noted that the pop-up messages or comments and facial expressions provided by the user in the embodiments of the present disclosure are authorized by the user.

[0061] In an example, the interaction intensity sequence data determined based on the first interaction information and the second interaction information can be a series of sequential values, which are determined based on the time sequence of the video data, and thus are inherently related to time. After mapping it to the time axis of the video data, the resulting distribution data can quantitatively represent the emotional expression of both parties in the interaction process in a two-dimensional space of time and interaction intensity. As mentioned earlier, the interaction intensity can be represented in a binary quantization manner such as 0 and 1, or further divided into several intervals between 1 and 0 for more refined representation.

[0062] In an example, the distribution data can adopt the form of a heat map, such as a spatiotemporal heat map, one data dimension of which can represent time and the other data dimension can represent interaction intensity. Therefore, the distribution data can indicate the peak moment when the interaction intensity meets the preset standard.

[0063] In the above manner, a quantitative way of data characterization of emotional expression of both parties in the interaction is provided from the interaction behavior. By analyzing the multi-dimensional information provided by both the interaction sending party and the interaction receiving party in the interaction process, the interaction intensity of both parties can be more comprehensively measured to obtain the corresponding quantitative result. At the same time, since the distribution data of the interaction intensity contains time sequence information, it provides a data basis for the to-be-performed time sequence relationship fusion, which helps to find other target moments associated with the interaction peak moment through the fusion operation.

[0064] In some embodiments, the step of determining the interaction intensity sequence data associated with the interaction intensity based on the first interaction information and the second interaction information can include: fusing the first interaction information and the second interaction information based on a cross-attention mechanism to obtain the interaction intensity sequence data.

[0065] In an example, the first interaction information and the second interaction information can be represented in a vector form, for example, mapped to a unified high-dimensional space, for performing a fusion operation in the space.

[0066] In an example, the fusion of the first interaction information and the second interaction information can be implemented by a Transformer encoder. The Transformer encoder processes sequential data through a self-attention mechanism, capturing the relationships between elements in the sequence, thereby capturing long-range dependencies. The cross-attention mechanism is an extension of the self-attention mechanism, used to process information from different modalities or different sources. It dynamically adjusts the weight of information by calculating the correlation between the features of one modality and the features of another modality, thereby better integrating these information.

[0067] Since the cross-attention mechanism can be used to capture dependencies between various information. By dynamically adjusting the weight of various information (such as text features, image features, speech features, etc.), it is possible to more effectively integrate information from different modalities, thereby generating more accurate and richer feature representations.

[0068] In some embodiments, the first interaction information and the second interaction information can each include multi-modal information.

[0069] Since the emotional state of both parties participating in the interaction is to be analyzed, single modality information often can only provide partial information, and may not fully reflect the emotional state. For example, text may not be able to capture the speaker's tone and expression, and images may not be able to capture the subtle differences in language. Therefore, in embodiments of the present disclosure, by combining data from multiple modalities, the shortcomings of single modality in information expression can be overcome, thereby more comprehensively understanding emotional expression, facilitating the generation of more accurate and lively emotional representations, and enhancing the connection between users and content.

[0070] In some embodiments, as described in conjunction with Figure 2 The step S204 of generating the sticker pack can include applying a visual dynamic effect matching the scene in the sticker pack based on the scene indicated by the first interaction information.

[0071] In an example, taking the indication of a live promotion scene by the first interaction information as an example, a basic dynamic effect for the sticker pack can be generated, such as a particle light effect, flashing, etc., to match the interactive atmosphere that tends to be enthusiastic and excited in the live promotion scene. The basic dynamic effect can include basic dynamic elements in the sticker pack, so it can directly embody the visual dynamic effect, or it can obtain the final visual dynamic effect under further processing of the basic dynamic effect. The basic dynamic effect matching the scene can be generated by a generative adversarial network model such as StyleGAN3, and then the corresponding visual dynamic effect is obtained.

[0072] In this way, the emotion can be more accurately expressed in the sticker, and the emotional resonance is enhanced, so that the user is more likely to have emotional resonance when using the sticker matching the emotional state of the scene, thereby helping to improve the user interaction rate and content conversion rate.

[0073] In some embodiments, the step S204 of generating the sticker can include: in response to the first interaction information indicating the item recommendation scene, injecting content information associated with the recommended item into the sticker. Figure 2

[0074] In an example, taking the live promotion scene for a product as an example, the content information associated with the recommended item can include key description information, price information, discount information, etc. of the product. The content information can be additionally superimposed into the sticker through text, pattern, etc. visual elements. Such visual elements can be self-adaptive in transparency to avoid blocking the details of the product.

[0075] Therefore, in the item recommendation scene, by injecting the content information associated with the recommended item into the sticker, the visual appeal of the sticker can be facilitated to be increased, and the opportunity of the item being recommended can be increased through the sharing of the sticker by the user.

[0076] In some embodiments, the step S204 of generating the sticker can include: determining style information associated with one of the first target object and the second target object as an interaction sender, and injecting the style information into the sticker. Figure 2

[0077] In an example, the interaction sender can be a live host, a host, etc. Taking the live host as an example, it usually can have its own unique live style, such as liking to use a certain iconic action, liking to arrange the live room into a certain specific color, etc. In this case, such style information can be injected into the sticker. This process can use the AdaIN algorithm, which is a deep learning technology for style transfer, and can apply the style of one image to another image. In sticker generation, the features of the host (such as color preference, iconic action, etc.) can be injected into the generated sticker through the algorithm to obtain a more personalized and emotionally expressive effect.

[0078] Therefore, the sticker with further improved personalization and emotionalization can be obtained, thereby increasing the attraction of the content to the user, and facilitating to improve the user interaction rate and content conversion rate.

[0079] Figure 4 A schematic diagram of a process of generating a sticker according to an embodiment of the present disclosure is shown.

[0080] As​​Figure 4 As shown, the sticker pack 400 can be dynamic, and thus can include multiple frames to present dynamic effects, in Figure 4 Three frames 401, 402 and 403 are exemplarily shown in the sticker pack 400. During the generation of the sticker pack 400, additional dynamic effects and information can be added to the sticker pack 400, such as the visual dynamic effect 410, the content information 420 and the style information 430 as shown. The visual dynamic effect 410 can be matched with the scene, for example, in a scene with a relatively hot atmosphere, a relatively strong dynamic effect can be matched to be adopted visually. In the item recommendation scene, such as the live promotion scene of a commodity, the content information 420 can be associated with the recommended item, for example, the discount price information, the discount expiration information, etc. of the commodity. The style information 430 can be associated with the interaction sender, for example, can include the personalized features of the live host, such as the logo gesture, etc.

[0081] In some embodiments, the method 200 as described in connection with Figure 2 may further include determining a frequency of providing the sticker pack to one of the first target object and the second target object as the interaction receiver based on a time period in which the interaction between the first target object and the second target object occurs in a day.

[0082] In an example, the frequency of providing the sticker pack to the interaction receiver can involve different sticker push strategies. The push strategy can be determined based on a PPO (Proximal Policy Optimization) algorithm according to a double reward function including user interaction rate and commodity conversion rate. For example, a host has two live broadcasts, one of which is at 8 pm, which is the peak online time for users, for example, there are 100,000 users, so that the amount of bullet screen of a live segment reaches 10,000, and the bullet screen rate is 10%; the other live broadcast is at 2 am, at which time there are only 10,000 online users, but the amount of bullet screen of a live segment reaches 1,000, and the bullet screen rate is also 10%. However, since the users at 2 am are generally more tired and less active, increasing the frequency of providing stickers in this period of time can be more likely to arouse the interest of users. In comparison, the provision of excessively intensive stickers with special effects to users at 8 pm, the evening peak, can be reduced.

[0083] In this way, the regularity that the user behavior and interaction mode of users at different time periods differ significantly can be taken into account, and the frequency of providing stickers can be dynamically adjusted to enhance the interactivity achieved by users using stickers and avoid negative feedback from users due to inappropriate over-push.

[0084] In some embodiments, the method of the embodiments of the present disclosure can employ collaborative rendering of the cloud and the edge. That is, the method of generating the meme can be performed in at least one of a cloud device or an edge device.

[0085] The embodiments of the present disclosure also provide a meme generation apparatus.

[0086] Figure 5 A structural block diagram of a meme generation apparatus 500 according to an embodiment of the present disclosure is shown.

[0087] As shown in Figure 5 , the apparatus 500 includes a data acquisition module 501, a data determination module 502, a target determination module 503, and a result generation module 504.

[0088] The data acquisition module 501 is configured to acquire video data for generating a meme, the video data including content associated with an interaction between a first target object and a second target object.

[0089] The data determination module 502 is configured to determine, based on the video data, distribution data of an interaction intensity between the first target object and the second target object changing over time, the distribution data indicating a peak moment when the interaction intensity meets a preset standard.

[0090] The target determination module 503 is configured to perform time sequence relationship fusion on the distribution data to determine at least one target moment associated with the peak moment.

[0091] The result generation module 504 is configured to extract, from the video data, a video clip corresponding to the peak moment and the at least one target moment to generate a meme.

[0092] The above-mentioned data acquisition module 501, data determination module 502, target determination module 503, and result generation module 504 can correspond to the operations of steps S201, S202, S203, and S204 as shown in Figure 2 , respectively. Therefore, the details of each aspect will not be described here.

[0093] Figure 6 A structural block diagram of a meme generation apparatus 600 according to another embodiment of the present disclosure is shown.

[0094] As shown in Figure 6 , the apparatus 600 can include a data acquisition module 601, a data determination module 602, a target determination module 603, and a result generation module 604. The operations of these modules can be the same as the operations of the data acquisition module 501, the data determination module 502, the target determination module 503, and the result generation module 504 as shown in Figure 5 .

[0095] In some embodiments, the content associated with the interaction between the first target object and the second target object can include first interaction information and second interaction information, the first interaction information being determined based on one of the first target object and the second target object as an interaction sender, and the second interaction information being determined based on one of the first target object and the second target object as an interaction receiver. Accordingly, the data determination module 602 can include a sequence determination module 6021 configured to determine interaction intensity sequence data associated with the interaction intensity based on the first interaction information and the second interaction information, and a distribution determination module 6022 configured to map the interaction intensity sequence data to a time axis of the video data to obtain distribution data.

[0096] In some embodiments, the sequence determination module 6021 can include a fusion module 6021a configured to fuse the first interaction information and the second interaction information based on a cross-attention mechanism to obtain the interaction intensity sequence data.

[0097] In some embodiments, the result generation module 604 can include an effect adding module 6041 configured to apply a visual dynamic effect matching a scene indicated by the first interaction information in the sticker.

[0098] In some embodiments, the result generation module 604 can include a content adding module 6042 configured to inject content information associated with a recommended item in the sticker in response to the first interaction information indicating an item recommendation scene.

[0099] In some embodiments, the result generation module 604 can include a style adding module 6043 configured to determine style information associated with one of the first target object and the second target object as the interaction sender to inject the style information in the sticker.

[0100] In some embodiments, the apparatus 600 can further include a provision determination module 605 configured to determine a frequency of providing the sticker to one of the first target object and the second target object as the interaction receiver based on a time period in a day when the interaction between the first target object and the second target object occurs.

[0101] According to embodiments of the present disclosure, an electronic device is also provided, including at least one processor; and a memory communicatively connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0102] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to perform the method as described above.

[0103] According to an embodiment of the disclosure, a computer program product is also provided, comprising a computer program which, when executed by a processor, implements the method as described above.

[0104] Reference Figure 7 will now be described, which is an example of a hardware device that can be applied to aspects of the disclosure. The electronic device is intended to represent a wide variety of digital electronic computer devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computer devices. The electronic device can also represent a wide variety of mobile devices such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other like computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0105] As shown in Figure 7 , the electronic device 700 includes a computing unit 701 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0106] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the electronic device 700, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device, and can include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 707 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0107] The computing unit 701 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above. For example, in some embodiments, the method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the methods by any other appropriate means, such as by means of firmware.

[0108] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0109] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0110] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0111] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0112] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0113] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is generally established by computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0114] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0115] While embodiments or examples of this disclosure have been described with reference to the figures, it will be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the application is not limited to these embodiments or examples. Various elements of the embodiments or examples can be omitted or substituted by equivalents thereof. Furthermore, the steps can be performed in a different order than described in the disclosure. Further, various elements of the embodiments or examples can be combined in various ways. It is important that as technology evolves, many of the elements described herein can be substituted by equivalents which serve the same function.

Claims

1. A method for generating an emoticon package, comprising: Obtaining video data for generating an emoticon package, wherein the video data includes content associated with an interaction between a first target object and a second target object; determining, based on the video data, distribution data of interaction intensity between the first target object and the second target object over time, the distribution data indicating a peak moment when the interaction intensity meets a preset standard; Performing time series relationship fusion on the distribution data to determine at least one target moment associated with the peak moment; as well as A video segment corresponding to the peak moment and the at least one target moment is extracted from the video data to generate the emoticon package.

2. The method according to claim 1, wherein The content associated with the interaction between the first target object and the second target object includes first interaction information and second interaction information, the first interaction information being determined based on one of the first target object and the second target object being the interaction sender, and the second interaction information being determined based on one of the first target object and the second target object being the interaction receiver; Determining, based on the video data, distribution data of the interaction intensity between the first target object and the second target object over time includes: determining interaction intensity sequence data associated with the interaction intensity based on the first interaction information and the second interaction information; as well as The interaction intensity sequence data is mapped onto the time axis of the video data to obtain the distribution data.

3. The method according to claim 2, wherein: The determining, based on the first interaction information and the second interaction information, interaction intensity sequence data associated with the interaction intensity includes: The first interaction information and the second interaction information are fused based on a cross-attention mechanism to obtain the interaction intensity sequence data.

4. The method according to claim 2 or 3, wherein: The first interaction information and the second interaction information both include multimodal information.

5. The method according to any one of claims 2 to 4, wherein The generating of the expression package comprises: Based on the scene indicated by the first interactive information, a visual dynamic effect matching the scene is applied in the emoticon package.

6. The method according to claim 5, wherein: The generating of the expression package comprises: In response to the first interactive information indicating an item recommendation scenario, content information associated with recommending the item is injected into the emoticon package.

7. The method according to any one of claims 1 to 6, wherein The generating of the emoticon package comprises: Determine style information associated with one of the first target object and the second target object serving as an interaction issuer, so as to inject the style information into the emoticon package.

8. The method according to any one of claims 1 to 7, wherein The method further comprises: Based on a time period of a day when the interaction between the first target object and the second target object occurs, a frequency of providing the emoticon package to one of the first target object and the second target object as an interaction recipient is determined.

9. An emoticon package generating device, comprising: a data acquisition module configured to acquire video data for generating an emoticon package, wherein the video data includes content associated with an interaction between a first target object and a second target object; a data determination module configured to determine, based on the video data, distribution data of the interaction intensity between the first target object and the second target object over time, the distribution data indicating a peak moment when the interaction intensity meets a preset standard; a target determination module, configured to perform time series relationship fusion on the distribution data to determine at least one target moment associated with the peak moment; as well as The result generation module is configured to extract video segments corresponding to the peak moment and the at least one target moment from the video data to generate the emoticon package.

10. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method according to any one of claims 1-8.

12. A computer program product comprising a computer program, wherein The computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

Images