Video processing method, device and system

Abstract

The present application discloses a video processing method, device and system. The method includes: obtaining different types of objects displayed in a live video, wherein the objects include: recommended foreground objects, and alternative objects other than foreground objects displayed in the live video; based on screening conditions, identifying the foreground objects displayed in the live video from different types of objects, wherein the screening conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and outputting the foreground object. The present application solves the technical problem that the image recognition method in the related art has low recognition accuracy for the main promoted products in the live video.

Description

Technical Field

[0001] The present application relates to the field of the Internet, and more specifically, to a video processing method, device and system. Background Art

[0002] Currently, during live broadcasts, the host often displays multiple products to the audience in sequence, and only recommends one product at a time. Although the remaining products will be displayed in the live video, they are background products in the live broadcast room. That is, only one product is displayed as a display object in the live video at the same time.

[0003] In order to identify the products promoted by the anchor from the live video, it is necessary to perform object detection on the live video. The existing detection method is based on classifying candidate boxes, and all products of the same category can be detected, such as Figure 1 The boxes shown in the figure all belong to the clothing category. However, in live video, the above detection method can not only identify the product currently promoted by the anchor (such as Figure 1 1 products shown in ), and will identify other products that are background products (such as Figure 1 The second category of products shown in ), resulting in interference in the recognition results and low accuracy.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a video processing method, device and system to at least solve the technical problem that the image recognition method in the related art has low recognition accuracy for the main promoted products in the live video.

[0006] According to one aspect of an embodiment of the present application, a video processing method is provided, including: obtaining different types of objects displayed in a live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; based on filtering conditions, identifying the foreground object displayed in the live video from the different types of objects, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and outputting the foreground object.

[0007] According to another aspect of an embodiment of the present application, a video processing method is also provided, including: playing a live video in a display interface; displaying different types of objects played in the live video in the display interface, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; marking the foreground objects in the live video, wherein the foreground objects are objects that meet filtering conditions among the objects played in the live video, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground objects.

[0008] According to another aspect of an embodiment of the present application, a video processing device is also provided, including: an acquisition module, used to acquire different types of objects displayed in a live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; an identification module, used to identify the foreground object displayed in the live video from different types of objects based on a screening condition, wherein the screening condition is composed of at least one feature parameter for characterizing the characteristics of the foreground object; and an output module, used to output the foreground object.

[0009] According to another aspect of an embodiment of the present application, a video processing device is also provided, including: a playback module, used to play a live video in a display interface; a display module, used to display different types of objects played in the live video in the display interface, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; a labeling module, used to label foreground objects in the live video, wherein the foreground objects are objects that meet filtering conditions among the objects played in the live video, wherein the filtering conditions are composed of at least one characteristic parameter for characterizing the characteristics of the foreground object.

[0010] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned video processing method.

[0011] According to another aspect of an embodiment of the present application, a computer terminal is further provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the above-mentioned video processing method is executed when the program is run.

[0012] According to another aspect of an embodiment of the present application, a video processing system is also provided, including: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: obtaining different types of objects displayed in a live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; based on filtering conditions, identifying the foreground objects displayed in the live video from different types of objects, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and outputting the foreground object.

[0013] According to another aspect of an embodiment of the present application, a video processing method is also provided, including: obtaining live video by calling a first interface, wherein the first interface includes: a first parameter, and a parameter value of the first parameter is the live video; determining different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; based on filtering conditions, identifying the foreground object displayed in the live video from different types of objects, wherein the filtering conditions are composed of at least one characteristic parameter for characterizing the characteristics of the foreground object; outputting the foreground object by calling a second interface, wherein the second interface includes: a second parameter, and a parameter value of the second parameter is the foreground object.

[0014] According to another aspect of an embodiment of the present application, a video processing method is also provided, including: obtaining multiple objects displayed in the video, wherein the objects include: foreground objects and alternative objects; based on filtering conditions, identifying the foreground object of the video from the multiple objects, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and outputting the foreground object.

[0015] In an embodiment of the present application, after obtaining different types of objects displayed in a live video, different types of objects are filtered through filtering conditions, and the foreground objects displayed in the live video are identified and output, thereby achieving the purpose of detecting the products that the anchor mainly promotes. It is easy to notice that, since the filtering conditions are composed of at least one characteristic parameter for characterizing the characteristics of the foreground object, the objects filtered out by the filtering conditions are the products that the anchor mainly promotes, and do not include other background products, thereby avoiding interference from other background products, achieving the technical effect of improving the recognition accuracy of the main promoted products, and enhancing the user experience and favorability, thereby solving the technical problem that the image recognition method in the related art has low recognition accuracy for the main promoted products in the live video. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 is a schematic diagram of a target detection result according to the prior art;

[0018] Figure 2 It is a schematic diagram of locating a watch in an image according to the prior art;

[0019] Figure 3 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video processing method according to an embodiment of the present application;

[0020] Figure 4 is a flowchart of a video processing method according to an embodiment of the present application;

[0021] Figure 5a is a schematic diagram of an optional video frame according to an embodiment of the present application;

[0022] Figure 5b is a schematic diagram of a motion history graph corresponding to an optional video frame according to an embodiment of the present application;

[0023] Figure 5c is a schematic diagram of coordinates corresponding to each pixel in an optional video frame according to an embodiment of the present application;

[0024] Figure 6 is a flowchart of an optional video processing method according to an embodiment of the present application;

[0025] Figure 7 is a flowchart of another video processing method according to an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a video processing device according to an embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of another video processing device according to an embodiment of the present application;

[0028] Fig.10 is a flowchart of another video processing method according to an embodiment of the present application;

[0029] Fig.11 is a schematic diagram of another video processing device according to an embodiment of the present application;

[0030] Fig.12 It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0034] Detection: It can refer to locating objects of a specified category in an image and outputting coordinate frames, such as Figure 2 As shown, the object belonging to the watch in the image can be located.

[0035] YOLO v3: It can refer to a detection algorithm based on convolutional neural networks, which can detect targets using multi-scale features and classify targets through a logistic regression model.

[0036] Detection box regression model: It can be an algorithm model that fine-tunes and refines the detection position.

[0037] Motion History Image: Motion History Image, referred to as HMI, can be a method of representing the motion information of an object in a video.

[0038] Foreground object: can refer to the main product being introduced by the host in the live video;

[0039] Alternative products: These can refer to the main products that the host has not introduced yet but will introduce later in the host’s video, or they can refer to the products that will not be introduced by the host in the live video.

[0040] Existing target detection solutions are mainly implemented through deep learning, which logically includes the following three steps:

[0041] According to certain rules, hundreds of thousands or even more "candidate boxes" are defined in the image space. There may be overlaps between them. Since the number of candidate boxes is large enough, it can be guaranteed that some of them can be roughly located on the core objects that the user is concerned about.

[0042] The candidate boxes are filtered to remove the candidate boxes with a low degree of overlap with the object. This can be achieved through a neural network. The neural network trained with data can filter out candidate boxes of specified categories (such as clothing, luggage).

[0043] The coordinates of the selected candidate boxes are fine-tuned using the detection box regression model to ensure that their edges fit as closely as possible with the edges of the object of interest.

[0044] Since the existing target detection scheme is based on the classification of candidate frames, objects of the same category will be detected. In the live video scene, not only the products promoted by the anchor are included, but also other background products. Therefore, there is a lot of interference.

[0045] In order to solve the above problems, the present application provides a method for detecting the main products promoted in live videos, which mainly distinguishes the main products promoted by the anchor and the background products during the candidate frame classification process, ensuring that the main products promoted by the anchor in the live video can be detected in the end, and only the main products promoted by the anchor in the live video can be detected.

[0046] Example 1

[0047] According to an embodiment of the present application, a method for processing a video is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 3 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video processing method. Figure 3As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 3 More or fewer components as shown, or with Figure 3 Different configurations shown.

[0049] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0050] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the video processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned (video processing method) is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0051] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0052] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0053] It should be noted that, in some optional embodiments, the above Figure 3 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 3 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0054] Under the above operating environment, this application provides Figure 4 The processing method of the video shown. Figure 4 FIG. 1 is a flow chart of a video processing method according to an embodiment of the present application. Figure 4 As shown, the method may include the following steps:

[0055] Step S42, obtaining different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and candidate objects other than the foreground objects displayed in the live video.

[0056] The live video in the above steps can be a video clip captured by the audience during the live broadcast, or it can be a complete live video obtained after the live broadcast ends. This application does not make specific limitations on this.

[0057] In an optional embodiment, when a user needs to detect a product that is mainly promoted by the anchor in a live video, the user can directly select the live video on a mobile terminal (such as a smart phone, tablet computer, PDA, laptop computer, etc.) or a computer terminal, and the mobile terminal or computer terminal processes the live video, and marks and displays the detected main promoted products. In another optional embodiment, in order to avoid the detection method consuming a large amount of computing resources and storage resources of the mobile terminal or computer terminal, the detection method can be implemented by a server. When a user needs to detect a product that is mainly promoted by the anchor in a live video, the user can select the live video on a mobile terminal or a computer terminal, and send the selected live video or detection request to the server, which processes the relevant live video and returns the detected main promoted products to the mobile terminal, which is marked and displayed by the mobile terminal or the computer terminal.

[0058] The different types of objects in the above steps may refer to the products that the anchor promotes in the live video and other products displayed in the live broadcast room, where the products that the anchor promotes are foreground objects and other background products are candidate objects. The different types of objects have nothing to do with the category to which the objects themselves belong, that is, the different types of objects in this application can be objects of different categories, such as the foreground object is skin care products and the candidate object is a frying pan; different types of objects can also be objects of the same category, such as the foreground object is the clothes worn by the anchor and the candidate objects are the clothes hung on hangers in the live broadcast room or worn by other models.

[0059] In an optional embodiment, the live video can be processed by a pre-trained detection model to detect all products displayed in the live video, and different types of products can be detected, that is, different types of products can be located by candidate boxes. Since the candidate boxes at this time include both the candidate boxes of the main recommended products and the candidate boxes of other background products, it is necessary to further filter out the candidate boxes of the background products and only keep the candidate boxes of the main recommended products.

[0060] Step S44: identifying the foreground object displayed in the live video from different types of objects based on the screening condition, wherein the screening condition is composed of at least one characteristic parameter for characterizing the characteristics of the foreground object.

[0061] The characteristic parameters in the above steps may be the unique characteristics of the product promoted by the anchor in the live video. Optionally, the above characteristic parameters may include at least one of the following: the display position, display duration, motion information and matching results with the recommended information of the object in the live video, wherein the recommended information includes text information synchronously output in the live video and voice information output by the anchor. The above text information may be the recommended text displayed in the live video, which gives a detailed introduction to the product promoted by the anchor. The above voice information may be the detailed introduction of the product promoted by the anchor through language during the process of displaying the promoted product.

[0062] In an optional embodiment, for a product that the anchor promotes in a live video, in order to ensure that the live audience can clearly see the product, the product often appears in a relatively central position in the live video, while other background products often appear in relatively remote positions in the live video. Based on this, it can be seen that the difference between the product that the anchor promotes and other background products is the different display positions, so the characteristic parameters can include: the display positions of different objects in the live video.

[0063] In another optional embodiment, for the product that the anchor promotes in the live video, the anchor will often display and introduce the product during the recommended time period of the product; while other background products often appear for a short time, such as flashing in the live video, or other background products are always placed in the live broadcast room, and the appearance time can be the entire live video. On this basis, it can be known that the difference between the product that the anchor promotes and other background products is the different display duration, so the characteristic parameters can include: the display duration of different objects in the live video.

[0064] In another optional embodiment, for the product promoted by the anchor in the live video, in order to ensure that the live audience can clearly see the complete appearance of the product, the anchor often rotates the product continuously and brings it closer to the camera during the introduction of the product. For example, for clothing products, the model wearing the product (such as the anchor himself) often needs to rotate to the back or side; while other background products are often placed on shelves or displayed by other fake models, which are basically in a static state. Based on this, it can be seen that the difference between the product promoted by the anchor and other background products is the different motion information, so the feature parameters can include: the motion information of different objects in the live video.

[0065] In another optional embodiment, for the product that the anchor promotes in the live video, in order to ensure that the live audience can accurately know the detailed information of the product, the anchor often introduces the product in words or text during the display of the product; while other background products are often not introduced. Based on this, it can be seen that the difference between the product that the anchor promotes and other background products is that the product that the anchor promotes matches the recommended information, while other background products do not match the recommended information, so the characteristic parameters can include: the matching results of different objects and recommended information.

[0066] It should be noted that, in actual use, different characteristic parameters can be combined according to the detection accuracy and detection speed. In the embodiment of the present application, the characteristic parameters including display position and motion information are taken as an example for explanation.

[0067] The screening conditions in the above steps can be based on the unique features of the products promoted by the anchor, and can accurately screen out the candidate frames that frame the products promoted by the anchor, so as to accurately detect the products promoted by the anchor and avoid interference from other background products. Optionally, the above screening conditions can be composed of conditions determined by at least one of the following characteristic parameters: the display position in the live video is located at a key position, where the key position includes: the center position of the live video, the position with the smallest distance from the anchor, and a predetermined fixed position; the display time in the live video exceeds the predetermined time; the movement frequency generated within the predetermined time period in the live video is the highest; the movement time is the longest among all objects displayed in the live video; and the match between the object features displayed in the live video and the recommended information is the highest.

[0068] The above-mentioned predetermined duration can be a fixed recommended duration, or it can be an average display duration of the main product determined by statistics of all previous live videos of the anchor. It can be set according to actual needs, and this application does not make any specific limitations on this.

[0069] In an optional embodiment, after all products in the live video are detected by existing technology, all candidate frames can be classified and filtered according to pre-set filtering conditions, and the candidate frames that meet the filtering conditions are retained, and the remaining candidate frames are directly discarded, thereby ensuring that the retained candidate frames can locate the products promoted by the anchor, and thus the products promoted by the anchor that are finally detected can be obtained.

[0070] It should be noted that in order to simplify the screening operation, the screening process can be integrated into the training process of the detection model, so that the model can only identify the products mainly promoted by the anchor without identifying other background products.

[0071] Step S46, output the foreground object.

[0072] In an optional embodiment, in order to facilitate users to view the selected main products, after the mobile terminal or computer terminal detects the main products, it can mark the main products and display them on the display screen for users to view. In another optional embodiment, in order to facilitate users to view the selected main products, after the server detects the main products, it can return the detection results to the mobile terminal or computer terminal, and the mobile terminal or computer terminal can mark the main products and display them on the display screen for users to view.

[0073] Through the solution provided by the above-mentioned embodiment of the present application, after obtaining different types of objects displayed in the live video, the different types of objects are filtered through the filtering conditions, and the foreground objects displayed in the live video are identified and output, thereby achieving the purpose of detecting the products mainly promoted by the anchor. It is easy to notice that, since the filtering conditions are composed of at least one characteristic parameter for characterizing the characteristics of the foreground object, the objects filtered out by the filtering conditions are the products mainly promoted by the anchor, and do not include other background products, thereby avoiding the interference of other background products, achieving the technical effect of improving the recognition accuracy of the main promoted products, and enhancing the user experience and favorability, thereby solving the technical problem that the image recognition method in the related technology has low recognition accuracy for the main promoted products in the live video.

[0074] In the above embodiment of the present application, the method also includes the following steps: parsing the recommended information and generating key feature information, wherein the key feature information includes at least one of the following: key words and phrases parsed from text information and key words and phrases parsed from voice information; comparing the object features of each object displayed in the live video with the key feature information respectively, and obtaining a comparison result, wherein the comparison result is used to characterize the matching degree of each object with the recommended information respectively.

[0075] The object features in the above steps may be features of the product itself, such as the product's color, shape, parameters, material, specific functions, price, etc. The key feature information in the above steps may be words or sentences extracted from the object features of the product. For example, the key feature information may be the word "yellow", or may be the sentence "combing your hair with this comb can improve the smoothness and glossiness of your hair".

[0076] In an optional embodiment, when the recommendation information is text information, the key feature information in the recommendation information can be identified through a text recognition method, and the product features of each product in the live video can be identified through an image recognition method. Further, by calculating the similarity between the key feature information and the product features, the degree of match between each product and the key feature information can be determined, thereby obtaining the degree of match between each product and the recommendation information, that is, obtaining the above-mentioned comparison result.

[0077] In another optional embodiment, when the recommendation information is voice information, the key feature information in the recommendation information can be identified through a voice recognition method, and the product features of each product in the live video can be identified through an image recognition method. Further, by calculating the similarity between the key feature information and the product features, the degree of match between each product and the key feature information can be determined, thereby obtaining the degree of match between each product and the recommended information, that is, obtaining the above-mentioned comparison result.

[0078] In the above embodiment of the present application, obtaining different types of objects displayed in a live video includes the following steps: obtaining a live video to be detected; using a prediction model to identify the live video to be detected, and detecting different types of objects from the live screen of the live video, wherein a detection box is used to identify the identified objects.

[0079] The prediction model in the above steps may be a neural network model obtained through pre-training, for example, YOLO v3, but is not limited thereto, and may also be other detection models based on convolutional neural networks.

[0080] In an optional embodiment, a prediction model can be pre-trained to detect different types of products in a live video and label the different types of products by marking detection frames, wherein the detection frame includes the entire product and the edge of the detection frame is aligned with the edge of the product.

[0081] In the above embodiment of the present application, before using the prediction model to identify the live video to be detected, the method also includes the following steps: obtaining multiple frames of historical pictures in the historical live video; annotating each frame of the historical picture, marking the foreground objects and candidate objects in the historical picture, wherein the foreground objects and candidate objects in the historical picture are identified using different detection frames; training a neural network model based on the historical pictures with the foreground objects and candidate objects marked to generate a prediction model.

[0082] In an optional embodiment, before using the prediction model for detection, the prediction model needs to be trained first. The entire training process is as follows: multiple frames of historical pictures can be randomly extracted from the historical live video, or multiple frames of historical pictures can be extracted at predetermined time intervals. For example, Figure 1 For each frame of the historical image extracted, different types of products can be labeled by manual labeling, that is, the product in the image can be selected by the detection frame. It should be noted that in order to distinguish different types of products, different detection frames can be used to label different types of products, for example, by labeling with detection frames of different colors. For example, for example Figure 1In the image shown, the first type of detection box can be marked in red, and the second type of detection box can be marked in green. After multiple frames of historical pictures are marked, the multiple frames of historical pictures can be used as training samples to train the convolutional neural network to obtain the final prediction model.

[0083] It should be noted that if the prediction model needs to detect different types of products, the detection boxes corresponding to different types of products can be used as targets for training; if the preset model only detects the products mainly promoted by the anchor, the detection boxes corresponding to the products mainly promoted by the anchor can be used as targets for training.

[0084] In the above embodiment of the present application, obtaining multiple frames of historical pictures in the historical live video includes the following steps: obtaining the historical live video to be identified, obtaining the motion history graph corresponding to each frame of the video, and the coordinates corresponding to each pixel in the video frame; based on the coordinates corresponding to each pixel in the video frame, performing image stitching along the channel dimension to obtain multiple frames of historical pictures.

[0085] In an optional embodiment, for each historical video frame extracted, the pixel change at the same position in the time period of two adjacent frames can be calculated, and the target motion situation can be expressed in the form of image brightness, so as to obtain the motion history map MHI corresponding to the frame. For example, for Figure 5a The historical video frames shown in Figure 5b In addition, for each extracted historical video frame, a coordinate map corresponding to each pixel in the video frame can also be obtained. For example, for Figure 5a The historical video frames shown in Figure 5c After obtaining the historical video frames, motion history graphs, and coordinates corresponding to each pixel, they can be spliced ​​along the channel dimension to obtain historical images for model training.

[0086] It should be noted that the extracted historical video frames can be 3-channel color images, the historical motion map can be 1-channel images, and the image corresponding to the coordinates corresponding to each pixel can be a 2-channel image. Therefore, the final historical picture is a 6-channel image.

[0087] Combine the following Figure 5a-5c and Figure 6A preferred embodiment of the present application is described in detail. The method can be applied to a mobile terminal, a computer terminal or a server, and the present application does not make any specific limitation on this. Based on the special application scenarios of live video, it can be known that the products mainly promoted by the anchor often appear in a position relative to the center of the image, and often rotate continuously and approach the lens. On this basis, in order to solve the problem that the target detection method in the related technology has a low accuracy rate in identifying live video and there is interference, the method can consider the position of the candidate frame in the picture and the motion information of the product in the candidate frame during the candidate frame classification process, and obtain a prediction model that only detects the products mainly promoted by the anchor in the live video through model training.

[0088] This method mainly includes two processes: model training and actual testing. Figure 6 As shown in the figure, the specific process of the model training process is as follows:

[0089] Step S62, obtaining a 6-channel input image.

[0090] Optionally, the video frame to be detected can be extracted from the live video, and the motion history graph corresponding to the video frame and the coordinates corresponding to each pixel can be obtained, and then spliced ​​along the channel dimension to obtain a 6-channel input image.

[0091] Step S64, marking the detection frame of the product in the input image.

[0092] Optionally, the input image may be labeled with detection frames corresponding to the product mainly promoted by the anchor and other background products in the image.

[0093] Step S66, training the model to obtain a prediction model for predicting the detection frame of the main recommended product.

[0094] Optionally, the 6-channel input image can be used as input, and the annotated detection box can be used as the target for learning, and YOLO v3 can be trained to obtain the above prediction model.

[0095] The actual testing process can include the following two steps:

[0096] Step S68, obtaining a 6-channel test image.

[0097] Optionally, the implementation method of step S68 is the same as the implementation method of step S62, which is not repeated here.

[0098] Step S610: Use the trained prediction model to predict the 6-channel test image to obtain the detection frame of the anchor's main product.

[0099] Optionally, the prediction model can be used to predict the 6-channel test image, and the detection frames of different types of products in the image can be obtained. The detection frame of the main product can be further identified from the detection frames of different types of products through screening conditions.

[0100] The method provided by the above steps can accurately locate the product that the anchor is promoting in the live video, avoiding interference from other products.

[0101] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0103] Example 2

[0104] According to an embodiment of the present application, a video processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0105] Figure 7 FIG. 1 is a flowchart of another video processing method according to an embodiment of the present application. Figure 7 As shown, the method may include the following steps:

[0106] Step S72, playing the live video in the display interface.

[0107] The display interface in the above steps may be a playback interface of the live video. In an optional embodiment, the audience may watch the live broadcast through a mobile terminal (such as a smart phone, tablet computer, PDA, laptop computer, etc.) or a computer terminal, so the display interface may be displayed on a display screen of the mobile terminal or the computer terminal.

[0108] The live video in the above steps can be a video clip captured by the audience during the live broadcast, or it can be a complete live video obtained after the live broadcast ends. This application does not make specific limitations on this.

[0109] Step S74, displaying different types of objects played in the live video in the display interface, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video.

[0110] The different types of objects in the above steps may refer to the products that the anchor promotes in the live video and other products displayed in the live broadcast room, where the products that the anchor promotes are foreground objects and other background products are candidate objects. The different types of objects have nothing to do with the category to which the objects themselves belong, that is, the different types of objects in this application can be objects of different categories, such as the foreground object is skin care products and the candidate object is a frying pan; different types of objects can also be objects of the same category, such as the foreground object is the clothes worn by the anchor and the candidate objects are the clothes hung on hangers in the live broadcast room or worn by other models.

[0111] Step S76, marking the foreground object in the live video, wherein the foreground object is an object that meets the screening condition in the objects played in the live video, wherein the screening condition is composed of at least one characteristic parameter for characterizing the characteristics of the foreground object.

[0112] The characteristic parameters in the above steps may be the unique characteristics of the product promoted by the anchor in the live video. Optionally, the above characteristic parameters may include at least one of the following: the display position, display duration, motion information and matching results with the recommended information of the object in the live video, wherein the recommended information includes text information synchronously output in the live video and voice information output by the anchor. The above text information may be the recommended text displayed in the live video, which gives a detailed introduction to the product promoted by the anchor. The above voice information may be the detailed introduction of the product promoted by the anchor through language during the process of displaying the promoted product.

[0113] The screening conditions in the above steps can be based on the unique features of the products promoted by the anchor, and can accurately screen out the candidate frames that frame the products promoted by the anchor, so as to accurately detect the products promoted by the anchor and avoid interference from other background products. Optionally, the above screening conditions can be composed of conditions determined by at least one of the following characteristic parameters: the display position in the live video is located at a key position, where the key position includes: the center position of the live video, the position with the smallest distance from the anchor, and a predetermined fixed position; the display time in the live video exceeds the predetermined time; the movement frequency generated within the predetermined time period in the live video is the highest; the movement time is the longest among all objects displayed in the live video; and the match between the object features displayed in the live video and the recommended information is the highest.

[0114] The above-mentioned predetermined duration can be a fixed recommended duration, or it can be an average display duration of the main product determined by statistics of all previous live videos of the anchor. It can be set according to actual needs, and this application does not make any specific limitations on this.

[0115] In an optional embodiment, after the main recommended product is detected through the screening conditions, the main recommended product can be marked by marking the detection box and displayed on the display screen of the mobile terminal or computer terminal.

[0116] In the above embodiment of the present application, before marking the foreground object in the live video, the method also includes the following steps: parsing the recommended information and generating key feature information, wherein the key feature information includes at least one of the following: key words and phrases parsed from text information and key words and phrases parsed from voice information; comparing the object features of each object displayed in the live video with the key feature information respectively, and obtaining a comparison result, wherein the comparison result is used to characterize the degree of match between each object and the recommended information respectively.

[0117] In the above embodiment of the present application, before displaying different types of objects played in the live video in the display interface, the method also includes the following steps: obtaining the live video to be detected; using a prediction model to identify the live video to be detected, and detecting different types of objects from the live screen of the live video, wherein a detection box is used to identify the identified object.

[0118] In the above embodiment of the present application, before using the prediction model to identify the live video to be detected, the method also includes the following steps: obtaining multiple frames of historical pictures in the historical live video; annotating each frame of the historical picture, marking the foreground objects and candidate objects in the historical picture, wherein the foreground objects and candidate objects in the historical picture are identified using different detection frames; training a neural network model based on the historical pictures with the foreground objects and candidate objects marked to generate a prediction model.

[0119] In the above embodiment of the present application, obtaining multiple frames of historical pictures in the historical live video includes the following steps: obtaining the historical live video to be identified, obtaining the motion history graph corresponding to each frame of the video, and the coordinates corresponding to each pixel in the video frame; based on the coordinates corresponding to each pixel in the video frame, performing image stitching along the channel dimension to obtain multiple frames of historical pictures.

[0120] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0121] Example 3

[0122] According to an embodiment of the present application, a video processing device for implementing the above-mentioned video processing method is also provided. Figure 8 As shown, the device 800 includes: an acquisition module 802 , an identification module 804 and an output module 806 .

[0123] Among them, the acquisition module 802 is used to acquire different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; the identification module 804 is used to identify the foreground objects displayed in the live video from different types of objects based on filtering conditions, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground objects; the output module 806 is used to output the foreground objects.

[0124] It should be noted that the acquisition module 802, the identification module 804 and the output module 806 correspond to steps S42 to S46 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0125] In the above embodiments of the present application, the device further includes: a parsing module and a comparison module.

[0126] Among them, the parsing module is used to parse the recommendation information and generate key feature information, wherein the key feature information includes at least one of the following: key words and phrases parsed from text information and key words and phrases parsed from voice information; the comparison module is used to compare the object features of each object displayed in the live video with the key feature information respectively, and obtain the comparison results, wherein the comparison results are used to characterize the matching degree of each object with the recommendation information respectively.

[0127] In the above embodiments of the present application, the acquisition module includes: a first acquisition unit and a detection unit.

[0128] Among them, the acquisition unit is used to acquire the live video to be detected; the detection unit is used to use the prediction model to identify the live video to be detected, and detect different types of objects from the live screen of the live video, wherein a detection box is used to identify the identified object.

[0129] In the above embodiments of the present application, the device further includes: a labeling module and a training module.

[0130] Among them, the acquisition module is also used to obtain multiple frames of historical pictures in the historical live video; the labeling module is used to label each frame of the historical picture, marking the foreground objects and candidate objects in the historical picture, wherein the foreground objects and candidate objects in the historical picture are marked with different detection frames; the training module is used to train the neural network model based on the historical pictures with the foreground objects and candidate objects labeled to generate a prediction model.

[0131] In the above embodiment of the present application, the acquisition module includes: a second acquisition unit and a splicing unit.

[0132] Among them, the second acquisition unit is used to obtain the historical live video to be identified, obtain the motion history graph corresponding to each frame of the video, and the coordinates corresponding to each pixel in the video frame; the splicing unit is used to perform image splicing along the channel dimension based on the coordinates corresponding to each pixel in the video frame to obtain multiple frames of historical pictures.

[0133] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0134] Example 4

[0135] According to an embodiment of the present application, a video processing device for implementing the above-mentioned video processing method is also provided. Fig. 9 As shown, the device 900 includes: a playing module 902 , a display module 904 and a marking module 906 .

[0136] Among them, the playback module 902 is used to play the live video in the display interface; the display module 904 is used to display different types of objects played in the live video in the display interface, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; the marking module 906 is used to mark the foreground objects in the live video, wherein the foreground objects are objects that meet the screening conditions in the objects played in the live video, wherein the screening conditions are composed of at least one characteristic parameter used to characterize the characteristics of the foreground objects.

[0137] It should be noted that the above-mentioned playing module 902, display module 904 and marking module 906 correspond to steps S72 to S76 in Example 2, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0138] In the above embodiments of the present application, the device further includes: a parsing module and a comparison module.

[0139] Among them, the parsing module is used to parse the recommendation information and generate key feature information, wherein the key feature information includes at least one of the following: key words and phrases parsed from text information and key words and phrases parsed from voice information; the comparison module is used to compare the object features of each object displayed in the live video with the key feature information respectively, and obtain the comparison results, wherein the comparison results are used to characterize the matching degree of each object with the recommendation information respectively.

[0140] In the above embodiments of the present application, the acquisition module includes: a first acquisition unit and a detection unit.

[0141] Among them, the acquisition unit is used to acquire the live video to be detected; the detection unit is used to use the prediction model to identify the live video to be detected, and detect different types of objects from the live screen of the live video, wherein a detection box is used to identify the identified object.

[0142] In the above embodiments of the present application, the device further includes: a labeling module and a training module.

[0143] Among them, the acquisition module is also used to obtain multiple frames of historical pictures in the historical live video; the labeling module is used to label each frame of the historical picture, marking the foreground objects and candidate objects in the historical picture, wherein the foreground objects and candidate objects in the historical picture are marked with different detection frames; the training module is used to train the neural network model based on the historical pictures with the foreground objects and candidate objects labeled to generate a prediction model.

[0144] In the above embodiment of the present application, the acquisition module includes: a second acquisition unit and a splicing unit.

[0145] Among them, the second acquisition unit is used to obtain the historical live video to be identified, obtain the motion history graph corresponding to each frame of the video, and the coordinates corresponding to each pixel in the video frame; the splicing unit is used to perform image splicing along the channel dimension based on the coordinates corresponding to each pixel in the video frame to obtain multiple frames of historical pictures.

[0146] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0147] Example 5

[0148] According to an embodiment of the present application, a video processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0149] Fig.10 FIG. 1 is a flowchart of another video processing method according to an embodiment of the present application. Fig.10 As shown, the method may include the following steps:

[0150] Step S102, obtaining the live video by calling the first interface, wherein the first interface includes: a first parameter, and a parameter value of the first parameter is the live video.

[0151] The first interface in the above steps may be an interface for data interaction between the server and the client. The client may pass the live video into the interface function as a parameter of the interface function to achieve the purpose of uploading the live video to the server.

[0152] The live video in the above steps can be a video clip captured by the audience during the live broadcast, or it can be a complete live video obtained after the live broadcast ends. This application does not make specific limitations on this.

[0153] Step S104, determining different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and candidate objects other than the foreground objects displayed in the live video.

[0154] The different types of objects in the above steps may refer to the products that the anchor promotes in the live video and other products displayed in the live broadcast room, where the products that the anchor promotes are foreground objects and other background products are candidate objects. The different types of objects have nothing to do with the category to which the objects themselves belong, that is, the different types of objects in this application can be objects of different categories, such as the foreground object is skin care products and the candidate object is a frying pan; different types of objects can also be objects of the same category, such as the foreground object is the clothes worn by the anchor and the candidate objects are the clothes hung on hangers in the live broadcast room or worn by other models.

[0155] Step S106: identifying the foreground object displayed in the live video from different types of objects based on the screening condition, wherein the screening condition is composed of at least one characteristic parameter for characterizing the characteristics of the foreground object.

[0156] The characteristic parameters in the above steps may be the unique characteristics of the product promoted by the anchor in the live video. Optionally, the above characteristic parameters may include at least one of the following: the display position, display duration, motion information and matching results with the recommended information of the object in the live video, wherein the recommended information includes text information synchronously output in the live video and voice information output by the anchor. The above text information may be the recommended text displayed in the live video, which gives a detailed introduction to the product promoted by the anchor. The above voice information may be the detailed introduction of the product promoted by the anchor through language during the process of displaying the promoted product.

[0157] The screening conditions in the above steps can be based on the unique features of the products promoted by the anchor, and can accurately screen out the candidate frames that frame the products promoted by the anchor, so as to accurately detect the products promoted by the anchor and avoid interference from other background products. Optionally, the above screening conditions can be composed of conditions determined by at least one of the following characteristic parameters: the display position in the live video is located at a key position, where the key position includes: the center position of the live video, the position with the smallest distance from the anchor, and a predetermined fixed position; the display time in the live video exceeds the predetermined time; the movement frequency generated within the predetermined time period in the live video is the highest; the movement time is the longest among all objects displayed in the live video; and the match between the object features displayed in the live video and the recommended information is the highest.

[0158] The above-mentioned predetermined duration can be a fixed recommended duration, or it can be an average display duration of the main product determined by statistics of all previous live videos of the anchor. It can be set according to actual needs, and this application does not make any specific limitations on this.

[0159] Step S108: outputting the foreground object by calling the second interface, wherein the second interface includes: a second parameter, and the parameter value of the second parameter is the foreground object.

[0160] The second interface in the above step may be an interface for data interaction between the server and the client. The server may pass the foreground object into the interface function as a parameter of the interface function to achieve the purpose of sending the foreground object to the client.

[0161] In the above embodiment of the present application, the method also includes the following steps: parsing the recommended information and generating key feature information, wherein the key feature information includes at least one of the following: key words and phrases parsed from text information and key words and phrases parsed from voice information; comparing the object features of each object displayed in the live video with the key feature information respectively, and obtaining a comparison result, wherein the comparison result is used to characterize the matching degree of each object with the recommended information respectively.

[0162] The object features in the above steps may be features of the product itself, such as the product's color, shape, parameters, material, specific functions, price, etc. The key feature information in the above steps may be words or sentences extracted from the object features of the product. For example, the key feature information may be the word "yellow", or may be the sentence "combing your hair with this comb can improve the smoothness and glossiness of your hair".

[0163] In the above embodiment of the present application, determining different types of objects displayed in a live video includes the following steps: obtaining a live video to be detected; using a prediction model to identify the live video to be detected, and detecting different types of objects from the live screen of the live video, wherein a detection box is used to identify the identified objects.

[0164] The prediction model in the above steps may be a neural network model obtained through pre-training, for example, YOLO v3, but is not limited thereto, and may also be other detection models based on convolutional neural networks.

[0165] In the above embodiment of the present application, before using the prediction model to identify the live video to be detected, the method also includes the following steps: obtaining multiple frames of historical pictures in the historical live video; annotating each frame of the historical picture, marking the foreground objects and candidate objects in the historical picture, wherein the foreground objects and candidate objects in the historical picture are identified using different detection frames; training a neural network model based on the historical pictures with the foreground objects and candidate objects marked to generate a prediction model.

[0166] In the above embodiment of the present application, obtaining multiple frames of historical pictures in the historical live video includes the following steps: obtaining the historical live video to be identified, obtaining the motion history graph corresponding to each frame of the video, and the coordinates corresponding to each pixel in the video frame; based on the coordinates corresponding to each pixel in the video frame, performing image stitching along the channel dimension to obtain multiple frames of historical pictures.

[0167] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0168] Example 6

[0169] According to an embodiment of the present application, a video processing device for implementing the above-mentioned video processing method is also provided. Fig.11 As shown, the device 1100 includes: a first calling module 1102 , a determining module 1104 , an identifying module 1106 and a second calling module 1108 .

[0170] Among them, the first calling module 1102 is used to obtain live video by calling the first interface, wherein the first interface includes: a first parameter, and the parameter value of the first parameter is the live video; the determination module 1104 is used to determine different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; the identification module 1106 is used to identify the foreground object displayed in the live video from different types of objects based on filtering conditions, wherein the filtering conditions are composed of at least one characteristic parameter for characterizing the characteristics of the foreground object; the second calling module 1108 is used to output the foreground object by calling the second interface, wherein the second interface includes: a second parameter, and the parameter value of the second parameter is the foreground object.

[0171] It should be noted that the first calling module 1102, the determining module 1104, the identifying module 1106 and the second calling module 1108 correspond to steps S102 to S108 in Example 5, and the examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Example 1.

[0172] In the above embodiments of the present application, the device further includes: a parsing module and a comparison module.

[0173] Among them, the parsing module is used to parse the recommendation information and generate key feature information, wherein the key feature information includes at least one of the following: key words and phrases parsed from text information and key words and phrases parsed from voice information; the comparison module is used to compare the object features of each object displayed in the live video with the key feature information respectively, and obtain the comparison results, wherein the comparison results are used to characterize the matching degree of each object with the recommendation information respectively.

[0174] In the above embodiments of the present application, the acquisition module includes: a first acquisition unit and a detection unit.

[0175] Among them, the acquisition unit is used to acquire the live video to be detected; the detection unit is used to use the prediction model to identify the live video to be detected, and detect different types of objects from the live screen of the live video, wherein a detection box is used to identify the identified object.

[0176] In the above embodiments of the present application, the device further includes: a labeling module and a training module.

[0177] Among them, the acquisition module is also used to obtain multiple frames of historical pictures in the historical live video; the labeling module is used to label each frame of the historical picture, marking the foreground objects and candidate objects in the historical picture, wherein the foreground objects and candidate objects in the historical picture are marked with different detection frames; the training module is used to train the neural network model based on the historical pictures with the foreground objects and candidate objects labeled to generate a prediction model.

[0178] In the above embodiment of the present application, the acquisition module includes: a second acquisition unit and a splicing unit.

[0179] Among them, the second acquisition unit is used to obtain the historical live video to be identified, obtain the motion history graph corresponding to each frame of the video, and the coordinates corresponding to each pixel in the video frame; the splicing unit is used to perform image splicing along the channel dimension based on the coordinates corresponding to each pixel in the video frame to obtain multiple frames of historical pictures.

[0180] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0181] Example 7

[0182] According to an embodiment of the present application, a video processing system is also provided, including:

[0183] processor;

[0184] The memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: obtaining different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; based on screening conditions, identifying the foreground object displayed in the live video from different types of objects, wherein the screening conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and outputting the foreground object.

[0185] In the above embodiments of the present application, the memory is also used to provide the processor with instructions for processing the steps of the video processing method in the above embodiments 1 and 2.

[0186] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0187] Example 8

[0188] According to an embodiment of the present application, a video processing method is also provided, which includes the following steps: obtaining multiple objects displayed in the video, wherein the objects include: foreground objects and alternative objects; based on filtering conditions, identifying the foreground object of the video from the multiple objects, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and outputting the foreground object.

[0189] The video in the above steps may be a live video, specifically a video clip captured by viewers during the live broadcast, or a complete live video obtained after the live broadcast ends. This application does not make any specific limitations on this.

[0190] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0191] Example 9

[0192] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0193] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0194] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the video processing method: obtaining different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; based on filtering conditions, identifying the foreground object displayed in the live video from different types of objects, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and outputting the foreground object.

[0195] Optionally, Fig.12 is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig.12 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1202 and a memory 1204.

[0196] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned video processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0197] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; based on filtering conditions, identify the foreground object displayed in the live video from different types of objects, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and output the foreground object.

[0198] Optionally, the processor may also execute program code for the following steps: parsing the recommendation information and generating key feature information, wherein the key feature information includes at least one of the following: key words and phrases parsed from text information and key words and phrases parsed from voice information; comparing the object features of each object displayed in the live video with the key feature information to obtain a comparison result, wherein the comparison result is used to characterize the degree of match between each object and the recommendation information.

[0199] Optionally, the processor may also execute the program code of the following steps: obtaining a live video to be detected; using a prediction model to identify the live video to be detected, and detecting different types of objects from the live screen of the live video, wherein a detection box is used to identify the identified objects.

[0200] Optionally, the processor may also execute the program code of the following steps: obtaining multiple frames of historical pictures in the historical live video; annotating each frame of the historical picture, marking the foreground objects and candidate objects in the historical picture, wherein the foreground objects and candidate objects in the historical picture are marked with different detection frames; training a neural network model based on the historical pictures with the foreground objects and candidate objects marked to generate a prediction model.

[0201] Optionally, the processor may also execute the program code of the following steps: obtaining the historical live video to be identified, obtaining the motion history graph corresponding to each video frame, and the coordinates corresponding to each pixel in the video frame; based on the coordinates corresponding to each pixel in the video frame, performing image stitching along the channel dimension to obtain multiple frames of historical pictures.

[0202] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: playing the live video in the display interface; displaying different types of objects played in the live video in the display interface, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; marking the foreground objects in the live video, wherein the foreground objects are objects that meet the screening conditions among the objects played in the live video, wherein the screening conditions are composed of at least one characteristic parameter for characterizing the characteristics of the foreground objects.

[0203] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain the live video by calling the first interface, wherein the first interface includes: a first parameter, and the parameter value of the first parameter is the live video; determine the different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; based on the screening conditions, identify the foreground object displayed in the live video from the different types of objects, wherein the screening conditions are composed of at least one characteristic parameter for characterizing the characteristics of the foreground object; output the foreground object by calling the second interface, wherein the second interface includes: a second parameter, and the parameter value of the second parameter is the foreground object.

[0204] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain multiple objects displayed in the video, wherein the objects include: foreground objects and alternative objects; based on filtering conditions, identify the foreground object of the video from the multiple objects, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and output the foreground object.

[0205] By adopting the embodiment of the present application, a video processing solution is provided. By identifying different types of objects displayed in the live video and identifying the foreground object displayed in the live video through screening conditions, it is not necessary to identify the alternative objects displayed in the live video, thereby achieving the technical effect of improving the recognition accuracy of the main product and improving the user experience and favorability, thereby solving the technical problem of low recognition accuracy of the main product in the live video by the image recognition method in the related art.

[0206] It can be understood by those skilled in the art that Fig.12The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.12 It does not limit the structure of the above electronic device. For example, the computer terminal A may also include Fig.12 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.12 Different configurations are shown.

[0207] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0208] Example 10

[0209] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the video processing method provided in the above embodiment.

[0210] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0211] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining different types of objects displayed in a live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; based on filtering conditions, identifying the foreground object displayed in the live video from different types of objects, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; and outputting the foreground object.

[0212] Optionally, the storage medium is also configured to store program codes for executing the following steps: parsing the recommendation information and generating key feature information, wherein the key feature information includes at least one of the following: key words and phrases parsed from text information and key words and phrases parsed from voice information; comparing the object features of each object displayed in the live video with the key feature information respectively, and obtaining a comparison result, wherein the comparison result is used to characterize the degree of match between each object and the recommendation information respectively.

[0213] Optionally, the storage medium is also configured to store program codes for executing the following steps: obtaining a live video to be detected; using a prediction model to identify the live video to be detected, and detecting different types of objects from the live screen of the live video, wherein a detection box is used to identify the identified objects.

[0214] Optionally, the storage medium is also configured to store program codes for executing the following steps: obtaining multiple frames of historical pictures in historical live videos; annotating each frame of the historical picture, marking the foreground objects and candidate objects in the historical picture, wherein the foreground objects and candidate objects in the historical picture are identified using different detection frames; training a neural network model based on the historical pictures with the foreground objects and candidate objects annotated to generate a prediction model.

[0215] Optionally, the storage medium is also configured to store program codes for executing the following steps: obtaining the historical live video to be identified, obtaining the motion history graph corresponding to each video frame, and the coordinates corresponding to each pixel in the video frame; based on the coordinates corresponding to each pixel in the video frame, performing image stitching along the channel dimension to obtain multiple frames of historical pictures.

[0216] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: playing a live video in a display interface; displaying different types of objects played in the live video in the display interface, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; marking foreground objects in the live video, wherein the foreground objects are objects that meet filtering conditions among the objects played in the live video, wherein the filtering conditions are composed of at least one characteristic parameter for characterizing the characteristics of the foreground objects.

[0217] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining live video by calling a first interface, wherein the first interface includes: a first parameter, and a parameter value of the first parameter is the live video; determining different types of objects displayed in the live video, wherein the objects include: recommended foreground objects, and alternative objects other than the foreground objects displayed in the live video; based on filtering conditions, identifying the foreground object displayed in the live video from different types of objects, wherein the filtering conditions are composed of at least one feature parameter for characterizing the characteristics of the foreground object; outputting the foreground object by calling a second interface, wherein the second interface includes: a second parameter, and a parameter value of the second parameter is the foreground object.

[0218] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining multiple objects displayed in a video, wherein the objects include: a foreground object and an alternative object; based on a filtering condition, identifying a foreground object of the video from the multiple objects, wherein the filtering condition is composed of at least one feature parameter for characterizing the characteristics of the foreground object; and outputting the foreground object.

[0219] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0220] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0222] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0224] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0225] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video processing method, characterized in that: include: Obtaining different types of objects displayed in a live video, wherein the different types of objects are used to represent products mainly promoted by the anchor in the live video and other products displayed in the live broadcast room, and the objects include: recommended foreground objects, and candidate objects other than the foreground objects displayed in the live video, wherein the foreground objects are used to represent products mainly promoted by the anchor, and the candidate objects are used to represent other products displayed in the live broadcast room; Based on the screening conditions, the foreground object displayed in the live video is identified from the different types of objects, wherein the screening conditions are composed of characteristic parameters used to characterize the characteristics of the foreground object, and the screening conditions include at least one of the following: the highest movement frequency generated within a predetermined time period in the live video, and the longest movement time among all objects displayed in the live video, and the characteristic parameters are used to represent the unique characteristics of the product promoted by the anchor in the live video, and the characteristic parameters include at least: the display time and motion information of the object in the live video; The foreground object is output.

2. The method according to claim 1, characterized in that The characteristic parameters also include at least one of the following: the display position of the object in the live video, and the matching result with the recommended information, wherein the recommended information includes the text information synchronously output in the live video and the voice information output by the anchor.

3. The method according to claim 2, characterized in that The screening condition is also composed of at least one of the following conditions determined by the characteristic parameters: The display position in the live video is located at a key position, wherein the key position includes: the center position of the live video, the position with the smallest distance from the anchor, and a predetermined fixed position; The display time in the live video exceeds the predetermined time; The object features displayed in the live video have the highest matching degree with the recommended information.

4. The method according to claim 3, characterized in that The method further comprises: Parsing the recommendation information to generate key feature information, wherein the key feature information includes at least one of the following: key words and phrases parsed from the text information and key words and phrases parsed from the voice information; The object features of each object displayed in the live video are respectively compared with the key feature information to obtain a comparison result, wherein the comparison result is used to characterize the matching degree between each object and the recommended information.

5. The method according to any one of claims 1 to 4, characterized in that: Get different types of objects displayed in the live video, including: Obtaining the live video to be detected; The prediction model is used to identify the live video to be detected, and different types of objects are detected from the live screen of the live video, wherein a detection frame is used to mark the identified object.

6. The method according to claim 5, characterized in that Before using the prediction model to identify the live video to be detected, the method further includes: Get multiple frames of historical images from historical live videos; Annotating each frame of the historical image to identify the foreground object and the candidate object in the historical image, wherein the foreground object and the candidate object in the historical image are identified by using different detection frames; The prediction model is generated by training a neural network model based on historical images with foreground objects and candidate objects labeled.

7. The method according to claim 6, characterized in that Get multiple frames of historical images from historical live videos, including: Obtain the historical live video to be identified, obtain a motion history graph corresponding to each video frame, and the coordinates corresponding to each pixel in the video frame; Based on the coordinates corresponding to each pixel in the video frame, image stitching is performed along the channel dimension to obtain the multiple frames of historical pictures.

8. A video processing method, characterized in that: include: Play live video in the display interface; Displaying different types of objects played in the live video in the display interface, wherein the different types of objects are used to represent products mainly promoted by the anchor in the live video and other products displayed in the live broadcast room, and the objects include: recommended foreground objects, and candidate objects other than the foreground objects displayed in the live video, wherein the foreground objects are used to represent products mainly promoted by the anchor, and the candidate objects are used to represent other products displayed in the live broadcast room; Marking the foreground object in the live video, wherein the foreground object is an object that meets a screening condition among the objects played by the live video; Among them, the screening condition is composed of characteristic parameters used to characterize the characteristics of the foreground object, and the screening condition includes at least one of the following: the movement frequency generated within a predetermined time period in the live video is the highest, and the movement time is the longest among all objects displayed in the live video. The characteristic parameters are used to represent the unique characteristics of the product promoted by the anchor in the live video, and the characteristic parameters include at least: the display duration and motion information of the object in the live video.

9. The method according to claim 8, characterized in that The characteristic parameters also include at least one of the following: the display position of the object in the live video, and the matching result with the recommended information, wherein the recommended information includes the text information synchronously output in the live video and the voice information output by the anchor.

10. The method according to claim 9, characterized in that The screening condition is also composed of at least one of the following conditions determined by the characteristic parameters: The display position in the live video is located at a key position, wherein the key position includes: the center position of the live video, the position with the smallest distance from the anchor, and a predetermined fixed position; The display time in the live video exceeds the predetermined time; The object features displayed in the live video have the highest matching degree with the recommended information.

11. A video processing device, characterized in that: include: An acquisition module, used to acquire different types of objects displayed in a live video, wherein the different types of objects are used to represent products mainly promoted by the anchor in the live video and other products displayed in the live broadcast room, and the objects include: recommended foreground objects, and candidate objects other than the foreground objects displayed in the live video, wherein the foreground objects are used to represent products mainly promoted by the anchor, and the candidate objects are used to represent other products displayed in the live broadcast room; an identification module, configured to identify the foreground object displayed in the live video from the different types of objects based on a screening condition; An output module, used for outputting the foreground object; Among them, the screening condition is composed of characteristic parameters used to characterize the characteristics of the foreground object, and the screening condition includes at least one of the following: the movement frequency generated within a predetermined time period in the live video is the highest, and the movement time is the longest among all objects displayed in the live video. The characteristic parameters are used to represent the unique characteristics of the product promoted by the anchor in the live video, and the characteristic parameters include at least: the display duration and motion information of the object in the live video.

12. A video processing device, characterized in that: include: A playback module, used to play live video in the display interface; A display module, used for displaying different types of objects played in the live video in the display interface, wherein the different types of objects are used to represent products mainly promoted by the anchor in the live video and other products displayed in the live broadcast room, and the objects include: recommended foreground objects, and candidate objects other than the foreground objects displayed in the live video, wherein the foreground objects are used to represent products mainly promoted by the anchor, and the candidate objects are used to represent other products displayed in the live broadcast room; A marking module, used to mark the foreground object in the live video, wherein the foreground object is an object that meets a screening condition among the objects played by the live video; Among them, the screening condition is composed of characteristic parameters used to characterize the characteristics of the foreground object, and the screening condition includes at least one of the following: the movement frequency generated within a predetermined time period in the live video is the highest, and the movement time is the longest among all objects displayed in the live video. The characteristic parameters are used to represent the unique characteristics of the product promoted by the anchor in the live video, and the characteristic parameters include at least: the display duration and motion information of the object in the live video.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the video processing method according to any one of claims 1 to 10.

14. A computer terminal, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the video processing method described in any one of claims 1 to 10 when running.

15. A video processing system, characterized in that: include: processor; as well as The memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: obtaining different types of objects displayed in the live video, wherein the different types of objects are used to represent the products mainly promoted by the anchor in the live video and other products displayed in the live broadcast room, and the objects include: recommended foreground objects, and candidate objects other than the foreground objects displayed in the live video, wherein the foreground objects are used to represent the products mainly promoted by the anchor, and the candidate objects are used to represent other products displayed in the live broadcast room; based on the screening conditions, identifying the foreground objects displayed in the live video from the different types of objects; and outputting the foreground objects; Among them, the screening condition is composed of characteristic parameters used to characterize the characteristics of the foreground object, and the screening condition includes at least one of the following: the movement frequency generated within a predetermined time period in the live video is the highest, and the movement time is the longest among all objects displayed in the live video. The characteristic parameters are used to represent the unique characteristics of the product promoted by the anchor in the live video, and the characteristic parameters include at least: the display duration and motion information of the object in the live video.

16. A video processing method, comprising: Acquire the live video by calling a first interface, wherein the first interface includes: a first parameter, and a parameter value of the first parameter is the live video; Determine different types of objects displayed in the live video, wherein the different types of objects are used to represent products mainly promoted by the anchor in the live video and other products displayed in the live broadcast room, and the objects include: recommended foreground objects, and candidate objects other than the foreground objects displayed in the live video, wherein the foreground objects are used to represent products mainly promoted by the anchor, and the candidate objects are used to represent other products displayed in the live broadcast room; Based on the screening condition, identifying the foreground object displayed in the live video from the different types of objects; Outputting the foreground object by calling a second interface, wherein the second interface includes: a second parameter, a parameter value of the second parameter is the foreground object; Among them, the screening condition is composed of characteristic parameters used to characterize the characteristics of the foreground object, and the screening condition includes at least one of the following: the movement frequency generated within a predetermined time period in the live video is the highest, and the movement time is the longest among all objects displayed in the live video. The characteristic parameters are used to represent the unique characteristics of the product promoted by the anchor in the live video, and the characteristic parameters include at least: the display duration and motion information of the object in the live video.

Images