Method and device for generating emoticon package pictures

Abstract

The present invention relates to a method and device for generating an emoticon package image, and relates to the field of computer technology. The generation method includes: performing behavior recognition on a target in a video stream, determining multiple frames of related images for each behavior recognition result; determining a representation image of each behavior recognition result based on each related image; and generating an emoticon package image based on the representation image.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for generating an emoticon package image, a device for generating an emoticon package image, and a non-volatile computer-readable storage medium. Background Art

[0002] Currently, emoticons continue to gain popularity, becoming an indispensable element in chats. According to statistics, any conversation with more than 20 sentences will contain emoticons at least 90%. Applying emoticons to different fields can improve communication effectiveness and efficiency.

[0003] In the related art, users need to download and save emoticon packages made by others, or manually make emoticon packages. Summary of the Invention

[0004] The inventors of the present disclosure have discovered that the above-mentioned related technologies have the following problems: the production of emoticon packages is difficult and inefficient.

[0005] In view of this, the present disclosure proposes a technical solution for generating emoticon package images, which can automatically generate emoticon packages, thereby reducing the difficulty of producing emoticon packages and improving production efficiency.

[0006] According to some embodiments of the present disclosure, a method for generating emoticon package images is provided, including: performing behavior recognition on a target in a video stream, and determining multiple frames of related images for each behavior recognition result; determining a representation image for each behavior recognition result based on each related image; and generating an emoticon package image based on the representation image.

[0007] In some embodiments, determining the representation image of each behavior recognition result based on the related images includes: detecting the degree of difference between the related images of each behavior recognition result; and determining the representation image based on the detection result of the degree of difference.

[0008] In some embodiments, determining the key image based on the detection result of the degree of difference includes: when the degree of difference between two frames of related images of each behavior recognition result is less than a similarity threshold, determining the two frames of related images as a repeated image pair; performing clarity detection on the repeated image pair; and determining the representative image based on the clarity detection result.

[0009] In some embodiments, generating an emoticon package picture based on the representational image includes: determining a key frame image in each representational image based on at least one of the time information of the representational image of each behavior recognition result in the video stream, the degree of difference from the representational images of other behavior recognition results, and the degree of matching with the sample image; and generating the emoticon package picture based on the key frame image.

[0010] In some embodiments, determining a key frame image in each representation image includes: determining a representation image whose time information matches a time node condition as the key frame image.

[0011] In some embodiments, determining the key frame image in each representation image includes: detecting the degree of difference between the representation image of each behavior recognition result and the representation images of each other behavior recognition result; and selecting at least one representation image as the key frame image in descending order of the degree of difference.

[0012] In some embodiments, determining the key frame image in each representation image includes: matching image features of each representation image with image features of the sample image; and determining the representation image with a matching degree higher than a matching threshold as the key frame image.

[0013] In some embodiments, the behavior recognition of the target in the video stream and the determination of multiple frames of related images for each behavior recognition result include: determining the start frame image and the end frame image of each behavior recognition result in the video stream; and determining the multiple frames of related images among the start frame image, the end frame image, and at least one intermediate frame image between the start frame image and the end frame image.

[0014] In some embodiments, generating an emoticon package picture based on the representation image includes: generating a dynamic emoticon package picture based on the representation image and its adjacent frame images.

[0015] In some embodiments, the generation method further includes: saving the live multimedia stream captured by the live capture software as the video stream.

[0016] According to other embodiments of the present disclosure, a device for generating emoticon package images is provided, including: an identification unit for performing behavior recognition on a target in a video stream and determining multiple frames of related images for each behavior recognition result; a determination unit for determining a representation image for each behavior recognition result based on the related images; and a generation unit for generating emoticon package images based on the representation images.

[0017] In some embodiments, the recognition unit detects the degree of difference between the relevant images of each behavior recognition result, and determines the representative image based on the detection result of the degree of difference.

[0018] In some embodiments, when the difference between two frames of related images of each behavior recognition result is less than a similarity threshold, the recognition unit determines the two frames of related images as a repeated image pair, performs clarity detection on the repeated image pair, and determines the representation image based on the clarity detection result.

[0019] According to some further embodiments of the present disclosure, a device for generating emoticon package images is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the method for generating emoticon package images in any of the above embodiments based on instructions stored in the memory device.

[0020] According to some further embodiments of the present disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for generating an emoticon package picture in any of the above embodiments is implemented.

[0021] In the above embodiment, by identifying the target behavior in the video stream, an emoticon package can be automatically generated, which can reduce the difficulty of making the emoticon package and improve the production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0024] Figure 1 Flowcharts showing some embodiments of the method for generating emoticon pictures of the present disclosure;

[0025] Figure 2 Schematic diagrams showing some embodiments of the method for generating emoticon pictures of the present disclosure;

[0026] Figure 3a Schematic diagrams showing other embodiments of the method for generating emoticon pictures of the present disclosure;

[0027] Figure 3b Schematic diagrams showing other embodiments of the method for generating emoticon pictures of the present disclosure;

[0028] Figure 4a Schematic diagrams showing some further embodiments of the method for generating emoticon pictures of the present disclosure;

[0029] Figure 4b Schematic diagrams showing some further embodiments of the method for generating emoticon pictures of the present disclosure;

[0030] Figure 4c Schematic diagrams showing some further embodiments of the method for generating emoticon pictures of the present disclosure;

[0031] Figure 5 Schematic diagrams showing further embodiments of the method for generating emoticon images of the present disclosure;

[0032] Figure 6 Schematic diagrams showing further embodiments of the method for generating emoticon images of the present disclosure;

[0033] Figure 7 Schematic diagrams showing further embodiments of the method for generating emoticon images of the present disclosure;

[0034] Figure 8 A block diagram illustrating some embodiments of the apparatus for generating emoticon images disclosed herein;

[0035] Figure 9 A block diagram showing some other embodiments of the apparatus for generating emoticon pictures disclosed herein;

[0036] Figure 10 A block diagram showing some further embodiments of the apparatus for generating emoticon pictures of the present disclosure. DETAILED DESCRIPTION

[0037] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0038] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0040] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0041] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0042] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] As mentioned earlier, emojis can be applied to a variety of different fields. For example, combining emojis with video technology can quickly review video content, thereby facilitating the promotion and dissemination of videos.

[0044] However, the production of a set of emoticon packages requires many different emoticon pictures and emotional pictures. Compared with making a single emoticon picture, making a set of emoticon pictures is more difficult and less efficient.

[0045] In response to the above technical problems, the present invention proposes a technical solution for automatically generating emoticon package images, which can reduce the difficulty of making emoticon packages and improve production efficiency. The technical solution of the present invention can be applied to many fields and solve corresponding technical problems in various fields.

[0046] For example, live streaming faces inherent challenges such as low cohesion, one-off nature, and poor reach, resulting in poor live streaming results. Unlike note-taking and photography, once the host ends their broadcast, it's difficult to share their content with others, effectively losing its value.

[0047] In this case, by combining emoji generation technology with live streaming using the disclosed technical solution, emoji images can be generated without the user being aware of the entire process and without affecting the live broadcast effect. This allows for quick review of live broadcast content, expanding the promotion and dissemination of live video content, and thus increasing the popularity of live broadcasts.

[0048] For example, the technical solutions of the present disclosure can be implemented through the following embodiments.

[0049] Figure 1 A flowchart illustrating some embodiments of the method for generating emoticon pictures of the present disclosure.

[0050] like Figure 1 As shown, in step S110, behavior recognition is performed on the target in the video stream, and multiple frames of related images of each behavior recognition result are determined.

[0051] In some embodiments, the live multimedia stream collected by the live collection software is saved as a video stream. Figure 2 Get the live multimedia stream.

[0052] Figure 2 Schematic diagrams showing some embodiments of the method for generating emoticon pictures of the present disclosure.

[0053] like Figure 2 As shown, in step 101, the user registers an account on the live broadcast platform, applies for a live broadcast room and prepares for the live broadcast.

[0054] For example, live broadcast preparation may include setting up a live broadcast acquisition device (such as a camera, a microphone, a mobile phone, etc.) locally, and the user is in front of the live broadcast acquisition device.

[0055] Step 102 includes: step 102.1, when the user starts live broadcast, the live broadcast acquisition software is opened; step 102.2, the live broadcast acquisition software calls the local acquisition device in step 101 to acquire the live broadcast stream; step 102.3, uploading the acquisition result to the live broadcast server.

[0056] In step 103, if the authentication is successful, the live broadcast server receives and saves the live multimedia stream collected by the anchor collection software, and then encapsulates a live broadcast service layer in the outer layer for external service; if the authentication fails, the live broadcast server discards the live multimedia stream.

[0057] In step 104, when the host and the audience watch the live broadcast in the live broadcast room, the client software obtains the live multimedia stream by applying for the live broadcast service layer; after obtaining the live multimedia stream, the client software calls the local decoder for decoding and playback.

[0058] In some embodiments, in a video stream, the start frame image and the end frame image of each behavior recognition result are determined; and multiple frames of related images are determined among the start frame image, the end frame image, and at least one intermediate frame image between the start frame image and the end frame image.

[0059] After obtaining the video stream and identifying the relevant images of each behavior, you can continue to Figure 1 The other steps generate emoticon pictures.

[0060] In step S120 , a representative image of each behavior recognition result is determined based on the relevant images.

[0061] In some embodiments, the degree of difference between the images related to each behavior recognition result is detected; and a representative image is determined based on the detection result of the degree of difference.

[0062] For example, when the difference between two frames of related images of each behavior recognition result is less than a similarity threshold, the two frames of related images are determined as a repeated image pair; clarity detection is performed on the repeated image pair; and a representative image is determined based on the clarity detection result.

[0063] In some embodiments, the Figure 3a An embodiment determines a characterizing image.

[0064] Figure 3a Schematic diagrams showing other embodiments of the method for generating emoticon pictures of the present disclosure.

[0065] As shown in Figure 3, in step 201, behavior recognition is performed on the live stream. For example, behavior recognition methods such as the iDT (improved Dense Trajectories) algorithm and machine learning models can be used to process the live stream for behavior recognition.

[0066] In some embodiments, in step 201.1, based on the characteristics of live broadcasting, such as fixed camera position and simple human body trajectory, as well as the desired output content, it is determined to use a dense trajectory algorithm as the behavior recognition algorithm.

[0067] In some embodiments, in step 201.2, a set of rule lists may be formulated to record various behavior types identified by the behavior identification method.

[0068] In some embodiments, the Figure 3b Behavior recognition is performed using the embodiment in .

[0069] Figure 3b Schematic diagrams showing other embodiments of the method for generating emoticon pictures of the present disclosure.

[0070] like Figure 3b As shown, in step 201.3, after the live broadcast starts, the server collects the live stream and classifies the behaviors in the live stream into one of the behavior types specified in step 201.2 through a dense trajectory algorithm.

[0071] For example, multiple user behaviors such as user behaviors 1, 2, 3, etc. are identified in a live stream, and the behavior type to which each user behavior belongs is also identified.

[0072] In some embodiments, facial recognition technology can be used to identify user emotions in the live stream as user behavior according to an emotion classification method, and to determine related images. Thus, representative frame images during the live stream are determined as representative images.

[0073] In step 201.4, based on the end stage of each user behavior in the live stream, the server backtracks the relevant images of each frame from the start stage to the end stage of the user behavior, including the start frame image (first frame), the middle frame image, and the end frame image (last frame).

[0074] For example, frame images (such as middle A, middle B, etc.) between the start frame image and the end frame image in the live stream may be collected at preset time intervals as intermediate frame images.

[0075] In step 201.5, the collected frames of related images of the user behavior are saved as a live picture group of the user behavior.

[0076] For example, you can store related images in the following data structure: {positiveTime, imageUrl, actionType}. positiveTime is the offset from the live broadcast start time, imageUrl is the URL of the related image, and actionType is the user behavior type corresponding to the related image in the rule list above. This data structure is used to select keyframe images.

[0077] In some embodiments, if a dynamic expression package image (such as a picture in gif format) needs to be generated, multiple consecutive frames of images in the video stream are saved. The data in the data structure is saved according to the starting frame images (i.e., related images) of these consecutive frames.

[0078] After determining the relevant images of each frame, Figure 3a The remaining steps in determine the representation image.

[0079] In step 202, a decision is made on whether to keep or remove the relevant images of each frame to determine the representative image of each user behavior. For example, step 202 may include two steps: step 202.1, repeat determination, and step 202.2, clarity determination.

[0080] In step 202.1, a repeat determination is performed.

[0081] In step 202.1.1, the image is binarized to determine whether the related images are repeated.

[0082] In step 202.1.2, the relevant images of each frame can be divided into new images and old images according to the time sequence in the video stream; and the difference between the new image and all the old images is determined.

[0083] In some embodiments, an old image is considered duplicated if its difference from the new image is less than 10%. For example, for a 1920×1080 pixel related image, there are 2,073,600 pixels. If the number of binary pixels that differ from another related image is less than 207,360, the two images are considered duplicates.

[0084] In 202.1.3, if it is determined that the two frames of related images are not repeated, the two frames of related images are saved to the live broadcast image group of the corresponding user behavior; if it is determined that the two frames of related images are repeated, step 202.2 is executed.

[0085] In step 202.2, a clarity judgment is performed.

[0086] In step 202.2.1, considering the computational speed and effect, the Brenner gradient function is used to determine the clarity of the two related image frames. The Brenner gradient function is simple to implement and the results are close to those determined by the naked eye.

[0087] In step 202.2.2, the clarity of the two frames of related images obtained in the previous step is compared. If the old image is clearer, the new image is discarded and the old image is saved as the image representing the corresponding user behavior; if the new image is clearer, the old image is deleted and the new image is saved as the image representing the corresponding user behavior.

[0088] After determining the representation image of each user behavior, you can use Figure 1 The other steps in generate emoticon images.

[0089] In step S130, an emoticon package image is generated based on the representation image. For example, a key frame image can be determined from each representation image according to a preset strategy, and then the emoticon package image is generated based on the key frame image.

[0090] In some embodiments, a key frame image is determined in each representation image based on at least one of the time information of the representation image of each behavior recognition result in the video stream, the degree of difference from the representation images of other behavior recognition results, and the degree of matching with the sample image; and an emoticon package image is generated based on the key frame image.

[0091] In some embodiments, the image features of each representation image are matched with the image features of the sample image; the representation image with a matching degree higher than a matching threshold is determined as a key frame image. Figure 4a An embodiment determines a key frame image.

[0092] Figure 4a Schematic diagrams showing some further embodiments of the method for generating emoticon pictures of the present disclosure.

[0093] like Figure 4a As shown, in step 301, a matching strategy is used to determine a key frame image. For example, a set of sample images can be preset as template expressions or template actions, and the representative image of the anchor's expression or action that best matches them is found.

[0094] In step 301.1, template expressions and template actions can be created based on currently popular, standard, and other business standards.

[0095] For example, template creation methods include UI (user interface) hand-drawing, ranking list, etc. Get the original template image group (recorded as {template t}), then perform image feature extraction (denoted as {templateChar t}).

[0096] In step 301.2, the live picture group (denoted as {image t}) in each image feature representing the image (denoted as {imageChar t}).

[0097] In step 301.3, loop processing {templateChar t Each element templateChar in} n , in {imageChar t} to find the feature imageChar with the greatest similarity n , thus locating {image t} in the representation image image n , as the key frame image

[0098] In step 301.4, the located representation images are collected to generate an emoticon package image group.

[0099] In some embodiments, image feature extraction includes color feature extraction, texture feature extraction, shape feature extraction, spatial relationship feature extraction, etc. For example, spatial relationship feature extraction can be selected as the basis for determining image similarity.

[0100] For example, an image can be automatically segmented to identify the spatial positions or relative orientations of multiple objects. These relationships can be categorized as connection or adjacency, overlap or overlap, or containment or inclusion.

[0101] For example, based on the fixed nature of live broadcast footage, the spatial position can be expressed as a relative spatial position to express the relative position of the target.

[0102] For example, when dealing with the pose estimation problem of each target, monocular vision pose estimation is selected according to the characteristics of the live broadcast, and is implemented through a model-based pose estimation method.

[0103] In some embodiments, spatial relationship features of the image may be extracted based on the estimated pose and relative spatial position.

[0104] For example, spatial relationship features can include the relative position of a person's portrait and table decorations, the relative position of the head and body, or the relative position of the head and table. This type of spatial relationship feature is closest to everyday descriptions, such as a host supporting their head with their left hand, drinking from a cup with their right hand, tilting their head back slightly, and leaning back on a chair.

[0105] In some embodiments, the degree of difference between the representation image of each behavior recognition result and the representation images of other behavior recognition results is detected; at least one representation image is selected as a key frame image in descending order of the degree of difference. For example, Figure 4b An embodiment determines a key frame image.

[0106] Figure 4b Schematic diagrams showing some further embodiments of the method for generating emoticon pictures of the present disclosure.

[0107] like Figure 4b As shown, in step 302, according to the difference strategy, a group of pictures with the greatest mutual differences are selected as key frame images.

[0108] In step 302.1, the live picture group (denoted as {image t The binarization results of each representation image in {imageWB t}).{imageWB t Any element imageWB in} n The pixel can be expressed as a vector consisting of 0 and 1.

[0109] In step 302.2, calculate {imageWB t The number of pixel differences between any two elements in} is recorded as the following data structure {imageWB n1 ,imageWB n2 ,charCount}.

[0110] For example, if the element imageWB1 is [0,1,1,0,1,0,1] and the element imageWB2 is [0,0,1,1,0,0,0], then the number of differences between them is 4.

[0111] In step 302.3, after all calculations are completed, the data structure corresponding to the first n largest charCount (the n representation images with the largest difference number) is obtained, imageWB n Corresponding image n This is the selected keyframe image.

[0112] In step 302.4, the key frame images are collected to form an emoticon package image group.

[0113] For example, the representative images included in the live picture group include pictures 1 to 5; by judging the difference between pictures 1 to 5, the first two pictures with the largest difference (picture 2 and picture 5) are taken as key frame images.

[0114] In some embodiments, a representation image whose time information matches a time node condition is determined as a key frame image.

[0115] For example, the time node condition is the time interval between the time corresponding to the frame image in the preset video stream and the start time of the video stream, and the time information is the time interval between the image and the start time of the video stream.

[0116] For example, the time node condition is a preset time interval, and the time information is the time interval representing the start time of the image and the video stream. For example, a frame of image is selected as a key frame image every 10 minutes.

[0117] Figure 4c Schematic diagrams showing some further embodiments of the method for generating emoticon pictures of the present disclosure.

[0118] like Figure 4c As shown, in step 303, according to the time strategy, a group of representative images are selected as key frame images at fixed time intervals in the live broadcast.

[0119] In step 303.1, the time interval for selecting key frame images is specified, such as 10 minutes.

[0120] In step 303.2, each representation image saves its corresponding time information in the video stream. According to the above-specified time interval, the representation image closest to the time node corresponding to the time interval is selected as the key frame image.

[0121] For example: Figure 1 The time information is 3 minutes, Figure 2 The time information of Figure 3 is 3 minutes and 15 seconds, the time information of Figure 4 is 3 minutes and 30 seconds, and the set time interval is 3 minutes and 20 seconds. Figure 2 Determine the key frame image.

[0122] In step 303.3, the key frame images are collected to form an emoticon package image group.

[0123] In some embodiments, the representative images included in the live picture group include pictures with a time interval of 5 minutes, 10 minutes, 15 minutes, 20 minutes, and 25 minutes from the start of the video stream. The time node condition is that the time interval is 10 minutes.

[0124] In this case, the 10-minute picture and the 20-minute picture are selected as key frame images.

[0125] In step 304, regardless of the recognition method or preferred solution (strategy) adopted, the selected key frame images are stored physically or in memory. For example, the storage methods include cloud disks, physical servers, mobile storage devices, etc.

[0126] In some embodiments, a dynamic emoticon package image is generated based on the representative image and its adjacent frame images. For example, a key frame image can be used as a starting point, and multiple consecutive frames of images thereafter can be selected in the video stream as the material for generating the dynamic emoticon package image; the dynamic emoticon package image can be generated based on the key frame image and the consecutive multiple frames of images.

[0127] Figure 5 Schematic diagrams showing some further embodiments of the method for generating emoticon pictures of the present disclosure.

[0128] like Figure 5 As shown, in step 401, it is determined whether the generated emoticon package picture is in a static picture format (such as jpg, jpeg, png, etc.) or a dynamic picture format (such as gif, etc.). For example, the static picture format and the dynamic picture format can be packaged separately by judging by the file suffix.

[0129] In step 402, a target platform is preset (such as various social networks such as Ajian Emergency), and a suitable packaging format is selected according to different target platforms.

[0130] In step 403, different packaging tools are selected according to the packaging format selected in step 402. For example, the packaging formats include zip, gzip, rar, etc.

[0131] In step 404, dynamic, static, and multi-platform emoticon packages are generated.

[0132] In step 405, web technology (such as java-linux-tomcat-springboot, etc.) is used to provide services such as list display, detail query, modification, download, and deletion of emoticon packages.

[0133] Figure 6 Schematic diagrams showing some further embodiments of the method for generating emoticon pictures of the present disclosure.

[0134] like Figure 6 As shown, in step 501, through the service provided in step 405, the user can query the list of emoticons in the emoticon package and select the emoticon image that needs to be modified.

[0135] In step 502, after the user selects an emoticon package image, he or she can edit the text, overlay images, etc. on the front-end page and view the modified effect.

[0136] In step 502.1, the front end can use absolute positioning (with the upper left corner of the base image as the positioning 0,0 point, each horizontal movement of one pixel to the fourth quadrant is recorded as x+1, and each vertical movement of one pixel is recorded as y+1) to position the text and effect image to the specified position and preview the display.

[0137] In step 502.2, when the user clicks Save, if it is a rendering, proceed to step 502.2.1; if it is text, proceed to step 502.2.2.

[0138] In step 502.2.1, the image processing method is two-step: first, the image is uploaded, and the server returns the image address (imgUrl); the front end makes a second request to the server, and transmits it in the data format {imgUrl:xxxxxx, x:123, y:456}.

[0139] imgUrl is returned by the server when the image is uploaded. x is the x-coordinate of the absolute positioning of the effect image, and y is the y-coordinate of the absolute positioning of the effect image.

[0140] In step 502.2.2, the text processing method is one-step, and the request is made directly in the data format {text:xxxxx, x:123, y:456, size:18, color:1d2e3a, font:Microsoft YaHei}.

[0141] text is the text entered by the user, x is the x-coordinate of the absolute positioning of the text, y is the y-coordinate of the absolute positioning of the text, size is the text size, color is the RGB color of the text, and font is the text font.

[0142] Other descriptions of text attributes are also included in the rights, such as i:true / false - whether italicized, b:true / false - whether bolded.

[0143] In step 503, the backend server performs absolute positioning and processing based on the original expression image, effect image or text.

[0144] In some embodiments, there are many ways to process images, which can be accomplished using Java native ImageIO or the open source framework Ffmpeg. For example, a unique technical solution can be found using Java / Ffmpeg.

[0145] For example, the core of the ffmpeg synthesis command is: ffmpeg -i input -i logo-filter_complex' overlay=10:main_h-overlay_h-10' output.

[0146] ffmpeg is the operation command, -i input is the original expression picture path, -i logo is the effect picture -filter_complex is the overlay command, overlay is the overlay offset (corresponding to the above x and y), and output is the address of the generated new expression picture.

[0147] In step 504, the emoticon package image generated in step 503 is used to replace the corresponding emoticon image in the original emoticon package to complete the manual adjustment.

[0148] In some embodiments, users can download emoticon packs, import them into social media, and use them.

[0149] For example, in step 601, the live broadcast platform logo can be added when downloading the emoticon package.

[0150] In step 602, the live broadcast platform domain name address can be added during sharing.

[0151] In step 603, frequency restrictions, charging policies, etc. may be added during downloading.

[0152] In step 604, copyright, privacy confirmation, etc. may be added during downloading.

[0153] Steps 601 to 604 are optional steps and have no execution order.

[0154] Figure 7 Schematic diagrams showing some further embodiments of the method for generating emoticon pictures of the present disclosure.

[0155] like Figure 7 As shown, the system of the present disclosure includes a user live broadcast module 100, an identification module 200, an emoticon image generation module 300, an emoticon package generation module 400, a manual adjustment module 500, and a download and use module 600.

[0156] In some embodiments, the relevant processes of module 100 and module 600 are from the user perspective, including normal live broadcast and downloading and using emoticon packages after live broadcast; the relevant processes of module 100 and module 500 are external services of the server, including providing live broadcast services and providing emoticon package services; the relevant processes of module 200, module 300, and module 400 are the server back-end processing logic, including live broadcast recognition, emoticon image generation, and emoticon package generation.

[0157] For example, the user starts the live broadcast module 100 as an input unit, and the user starts the live broadcast module 100 and can execute steps 101 to 104 in any of the above embodiments depending on the relevant functions of the current live broadcast platform.

[0158] For example, the recognition module 200 is used to identify the host's behavior in real time during a live broadcast according to specified rules; generate images and make decisions on whether to keep or remove them, and save the remaining static / animated images. The recognition module 200 can perform steps 201 to 202 in any of the above embodiments.

[0159] For example, the expression image generation module 300 optimizes and saves the generated expression images. The expression image generation module 300 can perform steps 301 to 304 in any of the above-mentioned embodiments. The optimization strategy in any of the above-mentioned embodiments can be used to determine the key frame images. The expression images that have been optimized are stored in the expression image library for use by module 400.

[0160] For example, the emoticon package generation module 400 generates the generated emoticon package images into an emoticon package and provides a download function. The emoticon package generation module 400 can execute steps 401 to 405 in any of the above embodiments.

[0161] For example, the manual adjustment module 500 is an optional module for providing users with functions such as secondary processing of emoticon packages, adding text, effects, etc. The manual adjustment module 500 can execute steps 501 to 504 in any of the above embodiments.

[0162] For example, the download and use module 600 provides the user with functions such as downloading emoticon packages, importing them into social media, etc. The download and use module 600 can execute steps 601 to 604 in any of the above embodiments.

[0163] Figure 8 A block diagram illustrating some embodiments of the apparatus for generating emoticon images of the present disclosure.

[0164] like Figure 8 As shown, the emoticon package picture generation device 8 includes a recognition unit 81, a determination unit 82, and a generation unit 83.

[0165] The recognition unit 81 performs behavior recognition on the target in the video stream and determines multiple frames of related images for each behavior recognition result.

[0166] The determining unit 82 determines a representative image of each behavior recognition result based on the relevant images.

[0167] The generating unit 83 generates an emoticon package picture according to the representation image.

[0168] In some embodiments, the recognition unit 81 detects the degree of difference between the relevant images of each behavior recognition result, and determines the representative image according to the detection result of the degree of difference.

[0169] In some embodiments, when the difference between two frames of related images of each behavior recognition result is less than a similarity threshold, the recognition unit 81 determines the two frames of related images as a repeated image pair; performs clarity detection on the repeated image pair, and determines a representative image based on the clarity detection result.

[0170] In some embodiments, the determination unit 82 determines a key frame image in each representation image based on at least one of the time information of the representation image of each behavior recognition result in the video stream, the degree of difference from the representation images of other behavior recognition results, and the degree of matching with the sample image; the generation unit 83 generates an emoticon package image based on the key frame image.

[0171] For example, the determining unit 82 determines the representation image whose time information matches the time node condition as a key frame image.

[0172] For example, the determination unit 82 detects the degree of difference between the representation image of each behavior recognition result and the representation images of other behavior recognition results; and selects at least one representation image as a key frame image in descending order of the degree of difference.

[0173] For example, the determining unit 82 matches the image features of each representation image with the image features of the sample image; and determines the representation image with a matching degree higher than a matching threshold as a key frame image.

[0174] In some embodiments, the recognition unit 81 determines the start frame image and the end frame image of each behavior recognition result in the video stream, and determines multiple frames of related images among the start frame image, the end frame image, and at least one intermediate frame image between the start frame image and the end frame image.

[0175] In some embodiments, the generating unit 83 generates a dynamic emoticon package picture based on the representation image and its adjacent frame images.

[0176] In some embodiments, the identification unit 81 saves the live multimedia stream captured by the live capture software as a video stream.

[0177] Figure 9 A block diagram showing some other embodiments of the apparatus for generating emoticon pictures disclosed herein.

[0178] like Figure 9 As shown, the emoticon package picture generation device 9 of this embodiment includes: a memory 91 and a processor 92 coupled to the memory 91, and the processor 92 is configured to execute the emoticon package picture generation method in any embodiment of the present disclosure based on the instructions stored in the memory 91.

[0179] The memory 91 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0180] Figure 10 A block diagram showing some further embodiments of the apparatus for generating emoticon pictures of the present disclosure.

[0181] like Figure 10 As shown, the emoticon package image generation device 10 of this embodiment includes: a memory 1010 and a processor 1020 coupled to the memory 1010, and the processor 1020 is configured to execute the emoticon package image generation method in any of the aforementioned embodiments based on the instructions stored in the memory 1010.

[0182] The memory 1010 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0183] The emoticon image generation device 10 may further include an input / output interface 1030, a network interface 1040, a storage interface 1050, and the like. These interfaces 1030, 1040, and 1050, as well as the memory 1010 and the processor 1020, may be connected, for example, via a bus 1060. The input / output interface 1030 provides a connection interface for input / output devices such as a display, mouse, keyboard, touch screen, microphone, and speakers. The network interface 1040 provides a connection interface for various networked devices. The storage interface 1050 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0184] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] The method for generating emoticon images, the device for generating emoticon images, and the non-volatile computer-readable storage medium according to the present disclosure have been described in detail. To avoid obscuring the concepts of the present disclosure, some details known in the art have been omitted. Based on the above description, those skilled in the art will fully understand how to implement the technical solutions disclosed herein.

[0186] The methods and systems of the present disclosure may be implemented in many ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0187] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for generating an emoticon package picture, comprising: Perform behavior recognition on the target in the video stream and determine multiple frames of related images for each behavior recognition result; Determine a representation image of each behavior recognition result according to each relevant image; Generate emoticon pictures based on the representation image, The step of performing behavior recognition on a target in a video stream and determining multiple frames of related images of each behavior recognition result includes: In the video stream, determining a start frame image and an end frame image of each behavior recognition result; The plurality of frames of related images are determined among the start frame image, the end frame image, and at least one intermediate frame image between the start frame image and the end frame image.

2. The generation method according to claim 1, wherein: Determining the representation image of each behavior recognition result according to each related image includes: Detect the difference between the relevant images of each behavior recognition result; The representative image is determined according to the detection result of the difference degree.

3. The generation method according to claim 2, wherein: The determining the representation image according to the detection result of the difference degree comprises: When the difference between two frames of related images of each behavior recognition result is less than a similarity threshold, the two frames of related images are determined as a repeated image pair; Performing clarity detection on the repeated image pair; The representative image is determined according to the clarity detection result.

4. The generation method according to claim 1, wherein: The step of generating an emoticon package picture according to the representation image comprises: Determine a key frame image in each representation image according to at least one of time information of the representation image of each behavior recognition result in the video stream, a degree of difference from the representation images of other behavior recognition results, and a degree of matching with the sample image; The emoticon package image is generated according to the key frame image.

5. The generation method according to claim 4, wherein: Determining a key frame image in each representation image includes: The representation image whose time information matches the time node condition is determined as the key frame image.

6. The generation method according to claim 4, wherein: Determining a key frame image in each representation image includes: Detecting the degree of difference between the representation image of each behavior recognition result and the representation images of other behavior recognition results; At least one representation image is selected as a key frame image in descending order of difference degree.

7. The generation method according to claim 4, wherein: Determining a key frame image in each representation image includes: Matching the image features of each representation image with the image features of the sample image; The representation image with a matching degree higher than the matching threshold is determined as the key frame image.

8. The generation method according to claim 1, wherein: The step of generating an emoticon package picture according to the representation image comprises: Generate a dynamic emoticon package picture based on the representation image and its adjacent frame images.

9. The generation method according to any one of claims 1 to 8, further comprising: The live multimedia stream collected by the live collection software is saved as the video stream.

10. A device for generating an emoticon package picture, comprising: A recognition unit, used to perform behavior recognition on a target in a video stream and determine multiple frames of related images for each behavior recognition result; A determination unit, used to determine a representation image of each behavior recognition result according to each related image; A generation unit is used to generate an emoticon package image according to the representation image. Among them, the recognition unit determines the start frame image and the end frame image of each behavior recognition result in the video stream, and determines the multiple frames of related images among the start frame image, the end frame image, and at least one intermediate frame image between the start frame image and the end frame image.

11. The generating device according to claim 10, wherein: The recognition unit detects the difference between the relevant images of each behavior recognition result, and determines the representative image according to the detection result of the difference.

12. The generating device according to claim 11, wherein: When the difference between two frames of related images of each behavior recognition result is less than a similarity threshold, the recognition unit determines the two frames of related images as a repeated image pair, performs clarity detection on the repeated image pair, and determines the representation image according to the clarity detection result.

13. A device for generating an emoticon package picture, comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the method for generating an emoticon package image according to any one of claims 1 to 9 based on instructions stored in the memory.

14. A non-volatile computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for generating an emoticon package image according to any one of claims 1 to 9 is implemented.

Images