Methods, devices, computer equipment and media for detecting cover images in live streaming rooms

Abstract

This application relates to the field of live streaming technology, and proposes a method, apparatus, computer equipment, and storage medium for detecting cover images in live streaming rooms. The method includes: acquiring a cover image to be detected; performing face detection on the cover image to obtain face detection results; if one or more face regions exist in the face detection results, acquiring the region size of each face region and the size of the cover image; if the proportion of each face region in the cover image is less than a preset proportion threshold and the aspect ratio is within a preset aspect ratio threshold range, performing face contour key point detection on each face region to obtain face contour key point detection results; if the face contour key point detection results all meet preset conditions, the cover image is used as the target cover image, thereby improving the efficiency of cover image detection.

Description

Technical Field

[0001] This application relates to the field of live streaming technology, and in particular to a method, apparatus, computer equipment, and storage medium for detecting cover images in live streaming rooms. Background Technology

[0002] With the rapid development of internet technology, live streaming has gradually become a popular form of entertainment. During a live stream, hosts typically use live stream covers to showcase and recommend different content, attracting users to click and watch. Specifically, after a user opens a live streaming platform, such as a live streaming app, a list of live streams will appear on the app's interface. The content of each live stream in the list can be displayed using a pre-set live stream cover by the host. Users can click on the corresponding live stream cover to enter the live stream room and see the actual live content.

[0003] Currently, live stream covers are mainly composed of real-time screenshots of the live stream or the most relevant images from a candidate set of historical screenshots. This method cannot guarantee cover image quality, requiring manual review of each cover. However, with a massive volume of live stream covers, manual review is costly and inefficient. Summary of the Invention

[0004] This application provides a method, apparatus, computer device, and storage medium for detecting cover images in live streaming rooms, which can solve the technical problems of low quality live streaming cover images, high cost and low efficiency of manual review of live streaming cover images. The technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a method for detecting cover images in live streaming rooms, including the following steps:

[0006] Obtain the cover image to be detected;

[0007] Face detection is performed on the cover image to obtain the face detection results;

[0008] If one or more face regions are found in the face detection results, obtain the region size of each face region and the size of the cover image;

[0009] If, based on the area size of each face region and the size of the cover image, it is determined that the proportion of each face region in the cover image is less than a preset proportion threshold and the width-to-height ratio of each face region is within a preset proportion threshold range, then face contour key point detection is performed on each face region to obtain the face contour key point detection result for each face region.

[0010] If the detection results of the key points of the facial contour in each of the facial regions meet the preset conditions, the cover image is used as the target cover image.

[0011] Secondly, embodiments of this application provide a cover image detection device for live streaming rooms, comprising:

[0012] The cover image acquisition module is used to acquire the cover image to be detected.

[0013] A face detection module is used to perform face detection on the cover image and obtain face detection results;

[0014] The size acquisition module is used to acquire the size of each face region and the size of the cover image if one or more face regions exist in the face detection result;

[0015] The key point detection module is used to perform facial contour key point detection on each facial region if, based on the area size of each facial region and the size of the cover image, it is determined that the proportion of each facial region in the cover image is less than a preset proportion threshold and the width-to-height ratio of each facial region is within a preset proportion threshold range, and to obtain the facial contour key point detection result for each facial region.

[0016] The target cover image acquisition module is used to use the cover image as the target cover image if the detection results of the key points of the facial contour in each of the facial regions meet the preset conditions.

[0017] Thirdly, embodiments of this application provide a computer device, a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method of the first aspect.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect.

[0019] This application embodiment acquires a cover image to be detected; performs face detection on the cover image to obtain face detection results; if one or more face regions exist in the face detection results, obtains the region size of each face region and the size of the cover image; if, based on the region size of each face region and the size of the cover image, it is determined that the proportion of each face region in the cover image is less than a preset proportion threshold, and the aspect ratio of each face region is within a preset proportion threshold range, facial contour key point detection is performed on each face region to obtain the facial contour key point detection result for each face region; if the facial contour key point detection result for each face region meets preset conditions, the cover image is selected as the target cover image. This application embodiment automatically achieves high-quality cover image selection by sequentially performing face detection and facial contour key point detection on the cover image to be detected, filtering out cover images including those without faces, with excessively large face proportions, with disproportionate face proportions, and with incomplete faces, reducing manual costs and improving the efficiency of cover image detection.

[0020] To better understand and implement this application, the technical solution is described in detail below with reference to the accompanying drawings. Attached Figure Description

[0021] Figure 1 A schematic diagram illustrating an application scenario of the live stream cover image detection method provided in this application embodiment;

[0022] Figure 2 A flowchart illustrating the live stream cover image detection method provided in this application embodiment;

[0023] Figure 3 A schematic diagram illustrating facial contour key point detection provided in an embodiment of this application;

[0024] Figure 4 A flowchart illustrating steps S21 to S25 of the live stream cover image detection method provided in this application embodiment;

[0025] Figure 5 This is a flowchart illustrating steps S31 to S32 of the live stream cover image detection method provided in this application embodiment;

[0026] Figure 6 A flowchart illustrating steps S311 to S312 of the live stream cover image detection method provided in this application embodiment;

[0027] Figure 7 This is a flowchart illustrating steps S51 to S52 of the live stream cover image detection method provided in this application embodiment;

[0028] Figure 8A flowchart illustrating steps S61 to S62 of the live stream cover image detection method provided in this application embodiment;

[0029] Figure 9 A schematic diagram of the structure of the cover image detection device for a live streaming room provided in this application embodiment;

[0030] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0033] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0034] Please see Figure 1 , Figure 1 This is a schematic diagram of an application scenario for the live broadcast room cover image detection method provided in this application embodiment. The application scenario includes the anchor client 101, server 102 and audience client 103 provided in this application embodiment. The anchor client 101 and audience client 103 interact through server 102.

[0035] Among them, the broadcaster client 101 refers to the end that sends live video, which is usually the client used by the broadcaster (i.e., the live broadcaster user) in the live broadcast.

[0036] Viewer client 103 refers to the end that receives and watches live online videos. It is usually the client used by viewers (i.e., live stream viewers) watching videos in a live stream.

[0037] The hardware referred to by the broadcaster client 101 and the viewer client 103 essentially refers to computer equipment, specifically, such as... Figure 1 As shown, it can be a computer device such as a smartphone, a smart interactive whiteboard, or a personal computer. Both the broadcaster client 101 and the viewer client 103 can access the Internet through known network access methods and establish a data communication link with the server 102.

[0038] Server 102, acting as a business server, can further connect to related audio data servers, video streaming servers, and other servers providing related support, thus forming a logically interconnected service cluster to serve related terminal devices, such as… Figure 1 The broadcaster client 101 and the viewer client 103 shown provide services.

[0039] In this embodiment, the broadcaster client 101 and the viewer client 103 can join the same live broadcast room (i.e., live broadcast channel). The aforementioned live broadcast room refers to a chat room implemented using Internet technology, which typically has audio and video playback control functions. The broadcaster user conducts live broadcasts in the live broadcast room through the broadcaster client 101, and the viewers of the viewer client 103 can log in to the server 102 to enter the live broadcast room to watch the live broadcast.

[0040] Within a live streaming room, hosts and viewers can interact through well-known online communication methods such as voice, video, and text. Typically, the host performs programs for the audience in the form of audio and video streams, and economic transactions can occur during the interaction. Of course, the application of live streaming rooms is not limited to online entertainment; it can be extended to other related scenarios, such as user matching and interaction, video conferencing, product promotion and sales, and any other scenarios requiring similar interaction.

[0041] Specifically, the process of watching the live stream is as follows: Viewers can click to access the live streaming application (such as YY) installed on the viewer client 103 and select to enter any live room, which triggers the viewer client 103 to load the live room interface for the viewer. The live room interface includes several interactive components. By loading these interactive components, viewers can watch the live stream in the live room and engage in various online interactions.

[0042] To allow users to quickly access and choose content of interest during a live stream, hosts typically add a live stream cover image. For example, the live video content displayed by the host will be shown as a cover image on the platform; users can click on the cover image to enter the corresponding live stream and watch the video.

[0043] However, existing live stream covers are usually just a single frame taken from the live stream video content. The quality of the captured cover cannot be guaranteed, resulting in a poor visual experience for users. This makes it difficult to quickly attract users, leading to poor user click feedback and poor viewing performance of the live stream video content.

[0044] Therefore, this application provides a method for detecting the cover image of a live broadcast room, which can be executed by the broadcaster's client.

[0045] Please see Figure 2 , Figure 2 The flowchart of the live broadcast room cover image detection method provided in the first embodiment of this application is shown. The method includes the following steps:

[0046] S10: Obtain the cover image to be detected.

[0047] In this embodiment of the application, the cover image to be detected can be a cover image uploaded by the anchor on the live streaming platform, or a cover image automatically generated by the live streaming platform, such as a cover image obtained from a real-time screenshot or historical screenshot of the anchor's live stream by the live streaming platform.

[0048] S20: Perform face detection on the cover image to obtain the face detection result.

[0049] Face detection refers to searching a given image using a specific strategy to determine whether it contains a human face, and if so, returning the face's location, size, and pose. In this embodiment, the cover image to be detected is input into a face detection model to obtain face detection results. These results indicate the face regions included in the cover image, where each face region is a rectangular area. For example, inputting a cover image to be detected into the face detection model yields k face regions, where the coordinates of these k face regions are represented as follows: in, Let x be the x-coordinate of the top-left corner of the k-th face region. Let the coordinate of the top-left corner of the k-th face region be denoted as y. Let x be the x-coordinate of the top right corner of the k-th face region. This is represented by the ordinate of the top right corner of the k-th face region. If no face region is found in the face detection results (i.e., k = 0), the cover image to be detected is deemed unqualified. If one or more face regions are found in the face detection results (i.e., k > 0), subsequent detection is performed on each face region.

[0050] The face detection model in this embodiment is pre-trained. The training process is as follows: acquire several video frame images containing clear faces, use annotation tools to annotate the faces in the several video frame images, make training dataset and test dataset according to a certain ratio of the annotated images, input the training dataset into the face detection model for training, input the test dataset for testing after training, and if the face recognition rate of the test dataset reaches the preset requirement, the face detection model is obtained.

[0051] S30: If there are one or more face regions in the face detection results, obtain the region size of each face region and the size of the cover image.

[0052] In this embodiment, the dimensions of the face region include its width and height, and the dimensions of the cover image include its width and height. Specifically, this is based on the face region coordinates. That is, the width of the k-th face region. This is the height of the k-th face region. Similarly, the width and height of the cover image can also be obtained. S40: If, based on the area size of each face region and the size of the cover image, it is determined that the proportion of each face region in the cover image is less than a preset proportion threshold, and the width-to-height ratio of each face region is within a preset proportion threshold range, then facial contour key point detection is performed on each face region to obtain the facial contour key point detection result for each face region.

[0053] Facial contour landmark detection refers to locating the facial contour regions of a given facial image. For details, please refer to [link to relevant documentation]. Figure 3 This is a schematic diagram of facial contour key point detection. In this embodiment, after obtaining the region size of each face region and the size of the cover image, the proportion of each face region in the cover image is calculated, that is, the ratio of the width of each face region to the width of the cover image and the ratio of the height of each face region to the height of the cover image are calculated. Simultaneously, the width-to-height ratio of each face region is calculated, that is, the ratio of the width to the height of each face region itself is calculated.

[0054] The proportion of each face region in the cover image is compared with a preset proportion threshold, and the aspect ratio of each face region is compared with a preset aspect ratio threshold range. For example, the preset proportion threshold is 0.6. If the ratio of the width of each face region to the width of the cover image is greater than 0.6, and / or the ratio of the height of each face region to the height of the cover image is greater than 0.6, then the face region is determined to have an excessive proportion in the cover image. The preset aspect ratio threshold range is 0.5 to 1. If the aspect ratio of the width and height of each face region is not within 0.5 to 1, then the aspect ratio of the face region is determined to be unbalanced. If one or more face regions have an excessive proportion in the cover image, or if one or more face regions have an aspect ratio that is not within the preset aspect ratio threshold range, the cover image to be tested is deemed unqualified.

[0055] If the proportion of each face region in the cover image is less than a preset proportion threshold, and the aspect ratio of each face region is within a preset proportion threshold range, then face contour key point detection is performed on each face region to obtain the face contour key point detection result for each face region, so as to perform subsequent face integrity detection based on the face contour key point detection result.

[0056] S50: If the detection results of the key points of the facial contour in each of the facial regions meet the preset conditions, the cover image is used as the target cover image.

[0057] In this embodiment, the facial contour key point detection result for each face region includes the location information and confidence score of the facial contour key points corresponding to each face region. The preset conditions include the location of the facial contour key points being within the cover image to be detected and the confidence score of the facial contour key points being greater than a preset threshold. If the facial contour key point detection results for one or more face regions do not meet the preset conditions, the cover image to be detected is indicated as unqualified. If the facial contour key point detection results for each face region meet the preset conditions, the cover image is used as the target cover image. The target cover image is the cover image to be detected after passing face detection, virtual portrait detection, and face integrity detection, and can be used as a high-quality cover image when a broadcaster publishes a live video. In an optional embodiment, the live streaming platform takes real-time or historical screenshots of the broadcaster's live stream to obtain a screenshot candidate set, which includes multiple captured images. All the captured images are used as cover images to be detected, and the above detection process is performed one by one to obtain multiple target cover images.

[0058] This application embodiment acquires a cover image to be detected; performs face detection on the cover image to obtain face detection results; if one or more face regions exist in the face detection results, obtains the region size of each face region and the size of the cover image; if, based on the region size of each face region and the size of the cover image, it is determined that the proportion of each face region in the cover image is less than a preset proportion threshold, and the aspect ratio of each face region is within a preset proportion threshold range, facial contour key point detection is performed on each face region to obtain the facial contour key point detection result for each face region; if the facial contour key point detection result for each face region meets preset conditions, the cover image is selected as the target cover image. This application embodiment automatically achieves high-quality cover image selection by sequentially performing face detection and facial contour key point detection on the cover image to be detected, filtering out cover images including those without faces, with excessively large face proportions, with disproportionate face proportions, and with incomplete faces, reducing manual costs and improving the efficiency of cover image detection.

[0059] In an optional embodiment, please refer to Figure 4 Step S20 includes steps S21 to S25, as follows:

[0060] S21: Use the first convolutional neural network to perform face detection on the cover image to obtain multiple face detection boxes including faces in the cover image and corresponding confidence scores.

[0061] In this embodiment, a first convolutional neural network is used to perform face detection on the cover image, obtaining multiple face detection boxes and corresponding confidence scores. During face detection, a large number of face detection boxes are generated at the same face location, and these boxes may overlap. The first convolutional neural network can be an FPN18 convolutional neural network, which is a Feature Pyramid Network (FPN) that addresses the multi-scale problem in object detection. By simply changing the network connections, it significantly improves object detection performance without substantially increasing the computational load of the original model.

[0062] S22: Sort the multiple face detection boxes in descending order of confidence scores to obtain a face detection box list;

[0063] S23: Add the face detection box with the highest confidence score to the output list, and delete the face detection box with the highest confidence score from the face detection box list;

[0064] S24: Calculate the overlap between the face detection box with the highest confidence score and other face detection boxes, and delete face detection boxes with an overlap greater than the overlap threshold from the face detection box list;

[0065] S25: Calculate the overlap between the face detection box with the highest confidence score remaining in the face detection box list and other face detection boxes. Delete face detection boxes with an overlap greater than the overlap threshold from the face detection box list until the face detection box list is empty. Obtain the face detection result based on the output list. In this embodiment, multiple face detection boxes are filtered, and the filtered face detection boxes are used as the face detection result, removing redundant face detection boxes and thus improving the accuracy of face detection.

[0066] In an optional embodiment, please refer to Figure 5 Step S30 includes steps S31 to S32, as follows:

[0067] S31: If there are one or more face regions in the face detection results, perform virtual portrait detection on each face region to obtain the virtual portrait detection result for each face region;

[0068] S32: If the virtual human image detection result for each of the face regions does not contain a virtual human image, obtain the region size of each face region and the size of the cover image.

[0069] Virtual avatars refer to non-realistic human images, including cartoon characters and 3D virtual characters. In this embodiment, if one or more face regions are found in the face detection results, virtual avatar detection is performed on each face region to identify virtual avatars that the face detection model may misidentify, thus obtaining a virtual avatar detection result for each face region. The virtual avatar detection result indicates the virtual avatars included in the face region. If a virtual avatar is found in the virtual avatar detection results of one or more face regions, the cover image to be detected is deemed unqualified. If no virtual avatar is found in the virtual avatar detection results of any face region, a subsequent judgment is made regarding the reasonableness of the face region proportion and ratio.

[0070] In an optional embodiment, please refer to Figure 6 Step S31 includes steps S311 to S312, as follows:

[0071] S311: If there are one or more face regions in the face detection results, use the second convolutional neural network to perform virtual portrait detection on each face region to obtain the virtual portrait confidence score of each face region.

[0072] S312: If the confidence score of the virtual human image is less than the preset score, it is determined that there is no virtual human image in the corresponding face area.

[0073] In this embodiment, a second convolutional neural network is used to detect virtual human images in each face region, obtaining a virtual human image confidence score for each face region. This improves both the accuracy and efficiency of the convolutional neural network model. The second convolutional neural network can be an EfficientNet-B0 convolutional neural network. If the virtual human image confidence score is greater than a preset value, a virtual human image is found in the face region. If one or more face regions contain virtual human images, the cover image to be detected is deemed unqualified. If the virtual human image confidence score is less than a preset value, no virtual human image is found in the face region. If one or more face regions do not contain virtual human images, subsequent judgments are made regarding the proportion and rationality of the face region, thereby filtering out cover images containing virtual human images.

[0074] In an optional embodiment, please refer to Figure 7 Step S50 includes steps S51 to S52, as follows:

[0075] S51: If, based on the area size of each face region and the size of the cover image, it is determined that the proportion of each face region in the cover image is less than a preset proportion threshold and the width-to-height ratio of each face region is within a preset proportion threshold range, then each face region is expanded outward along its own diagonal direction by a preset proportion to obtain the corresponding target face region.

[0076] S52: Input each of the target face regions into the face contour key point detection model to obtain the face contour key point detection result for each of the target face regions; wherein, the face contour key point detection result is used to indicate the position and confidence score of the face contour key points in each of the target face regions.

[0077] In this embodiment, each face region is expanded outward along its diagonal by a preset ratio to obtain a corresponding target face region. For example, if the pixel size of the face region is 300*200 and the preset ratio is 1.5, then the pixel size of the expanded target face region is 450*300, meaning the width of the face region is expanded by 75 pixels in both the left and right directions, and the height of the face region is expanded by 50 pixels in both the up and down directions. Each target face region is input into a pre-trained face contour key point detection model to obtain the position and confidence score of the face contour key points in each target face region. For each target face region, there are n corresponding positions of face contour key points [(x0, y0), ..., (x...]. i y i ), …(x n-1 y n-1 [p1, ... p2], and the confidence scores corresponding to the key points of each facial contour [p1, ... p2]. i , ...p n-1 By expanding the face region to obtain the target face region, and obtaining the key points of the face contour based on the target face region, incomplete faces can be located for subsequent face integrity detection.

[0078] In an optional embodiment, please refer to Figure 8 Step S60 includes steps S61 to S62, as follows:

[0079] S61: If the position of the facial contour key point of each facial region is within the cover image, and the confidence score of the facial contour key point of each facial region is greater than a preset threshold, the facial contour key point is determined to be the target contour key point.

[0080] S62: If the number of key points of the target contour is greater than a preset value, the cover image is used as the target cover image.

[0081] In this embodiment, each face region has n facial contour key points, the cover image has a width of w and a height of h, and the preset threshold is alphal. For the i-th facial contour key point, the position is (x... i y i The confidence score is p. i If the location of the i-th facial contour key point is within the cover image, and the confidence score of the i-th facial contour key point is greater than a preset threshold, that is, the x-coordinate of the i-th facial contour key point is... i Satisfying 0 < x i <w, the ordinate y of the i-th face contour key point i Satisfying 0 < yi <h, and p i If the value is greater than alphal, then the i-th face contour key point is determined as the target contour key point. The number of target contour key points in each face region is counted. If the number of target contour key points in each face region is greater than a preset value, for example, if the number of target contour key points in each face region exceeds 20, then the cover image to be detected is determined to meet the face integrity detection, and the cover image to be detected is determined as the target cover image. By determining the target contour key points and counting the number of target contour points, cover images to be detected that include incomplete faces are filtered out.

[0082] Please see Figure 9 , Figure 9 A schematic diagram of the cover image detection device for live streaming rooms provided in this application. The cover image detection device 7 for live streaming rooms provided in this embodiment includes:

[0083] The cover image acquisition module 71 is used to acquire the cover image to be detected;

[0084] The face detection module 72 is used to perform face detection on the cover image and obtain face detection results;

[0085] The size acquisition module 73 is used to acquire the size of each face region and the size of the cover image if there are one or more face regions in the face detection result;

[0086] The key point detection module 74 is used to perform face contour key point detection on each face region if, based on the area size of each face region and the size of the cover image, it is determined that the proportion of each face region in the cover image is less than a preset proportion threshold and the width-to-height ratio of each face region is within a preset proportion threshold range, and to obtain the face contour key point detection result of each face region.

[0087] The target cover image acquisition module 75 is used to take the cover image as the target cover image if the detection results of the key points of the facial contour in each of the facial regions meet the preset conditions.

[0088] It should be noted that the cover image detection device for live streaming rooms provided in the above embodiments is only illustrated by the division of the above functional modules when executing the cover image detection method for live streaming rooms. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the cover image detection device for live streaming rooms and the cover image detection method for live streaming rooms provided in the above embodiments belong to the same concept, and its implementation process is detailed in the method embodiments, which will not be repeated here.

[0089] Please see Figure 10 This is a schematic diagram of the structure of the computer device provided in this application. Figure 10 As shown, the computer device 21 may include: a processor 210, a memory 211, and a computer program 212 stored in the memory 211 and capable of running on the processor 210, such as a live broadcast room cover image detection program; when the processor 210 executes the computer program 212, it implements the steps in the above embodiments.

[0090] The processor 210 may include one or more processing cores. The processor 210 connects to various parts within the computer device 21 using various interfaces and lines. It executes various functions of the computer device 21 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 211, and by accessing data in the memory 211. Optionally, the processor 210 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 210 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 210.

[0091] The memory 211 may include random access memory (RAM) or read-only memory. Optionally, the memory 211 may include a non-transitory computer-readable storage medium. The memory 211 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 211 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch instructions), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 211 may also be at least one storage device located remotely from the aforementioned processor 210.

[0092] This application also provides a computer storage medium that can store multiple instructions. These instructions are applicable to being loaded by a processor and executed by the method steps of the above embodiments. For details of the execution process, please refer to the specific description of the above embodiments, which will not be repeated here.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0094] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0095] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0096] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0098] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0099] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0100] This invention is not limited to the above-described embodiments. If any modifications or variations to this invention do not depart from the spirit and scope of this invention, and if such modifications and variations fall within the scope of the claims and equivalent technologies of this invention, then this invention also intends to include such modifications and variations.

Claims

1. A method for detecting cover images in live streaming rooms, characterized in that, The method includes the following steps: Obtain the cover image to be detected, wherein the cover image to be detected is a real-time screenshot or historical screenshot of the live broadcast by the live broadcast platform, and the cover image is obtained from the real-time screenshot or historical screenshot of the live broadcast. Face detection is performed on the cover image to obtain the face detection results; If one or more face regions exist in the face detection results, perform virtual portrait detection on each face region to obtain the virtual portrait detection result for each face region; If the virtual human image detection result for each of the face regions does not show a virtual human image, obtain the region size of each face region and the size of the cover image; If, based on the area size of each face region and the size of the cover image, the ratio of the width of each face region to the width of the cover image, the ratio of the height of each face region to the height of the cover image, and the ratio of the width to the height of each face region itself are calculated, and it is determined that the proportion of each face region in the cover image is less than a preset proportion threshold and the width-to-height ratio of each face region is within a preset proportion threshold range, then facial contour key point detection is performed on each face region to obtain the facial contour key point detection result for each face region; If the detection results of the key points of the facial contour in each of the facial regions meet the preset conditions, the cover image is used as the target cover image.

2. The method for detecting the cover image of a live streaming room according to claim 1, characterized in that: The step of performing face detection on the cover image and obtaining the face detection result includes: The first convolutional neural network is used to perform face detection on the cover image to obtain multiple face detection boxes including faces in the cover image and corresponding confidence scores. The multiple face detection boxes are sorted in descending order of confidence scores to obtain a list of face detection boxes; Add the face detection box with the highest confidence score to the output list, and then delete the face detection box with the highest confidence score from the face detection box list. Calculate the overlap between the face detection box with the highest confidence score and other face detection boxes, and delete face detection boxes with an overlap greater than the overlap threshold from the face detection box list; Calculate the overlap between the face detection box with the highest confidence score remaining in the face detection box list and other face detection boxes. Delete face detection boxes with an overlap greater than the overlap threshold from the face detection box list until the face detection box list is empty. Obtain the face detection result based on the output list.

3. The method for detecting the cover image of a live streaming room according to claim 1, characterized in that: The step of performing virtual portrait detection on each face region to obtain a virtual portrait detection result for each face region if one or more face regions exist in the face detection result includes: If one or more face regions exist in the face detection results, a second convolutional neural network is used to perform virtual portrait detection on each face region to obtain a virtual portrait confidence score for each face region. If the confidence score of the virtual avatar is less than a preset score, it is determined that there is no virtual avatar in the corresponding face area.

4. The method for detecting the cover image of a live streaming room according to claim 1, characterized in that: The step of determining, based on the area size of each face region and the size of the cover image, that the proportion of each face region in the cover image is less than a preset proportion threshold and the aspect ratio of each face region is within a preset aspect ratio threshold range, and then performing facial contour key point detection on each face region to obtain the facial contour key point detection result for each face region includes: If, based on the area size of each face region and the size of the cover image, it is determined that the proportion of each face region in the cover image is less than a preset proportion threshold and the width-to-height ratio of each face region is within a preset proportion threshold range, then each face region is expanded outward along its own diagonal direction by a preset proportion to obtain the corresponding target face region. Each target face region is input into a face contour key point detection model to obtain the face contour key point detection result for each target face region; wherein, the face contour key point detection result is used to indicate the position and confidence score of the face contour key points in each target face region.

5. The method for detecting the cover image of a live streaming room according to claim 4, characterized in that: The step of using the cover image as the target cover image if the detection results of the facial contour key points of each facial region meet the preset conditions includes: If the location of the facial contour key point of each facial region is within the cover image, and the confidence score of the facial contour key point of each facial region is greater than a preset threshold, the facial contour key point is determined to be a target contour key point. If the number of key points of the target contour is greater than a preset value, the cover image will be used as the target cover image.

6. A cover image detection device for live streaming rooms, characterized in that, include: The cover image acquisition module is used to acquire the cover image to be detected, wherein the cover image to be detected is a real-time screenshot or historical screenshot taken by the live streaming platform of the anchor's live stream, and the cover image is obtained from the real-time screenshot or historical screenshot of the live stream. A face detection module is used to perform face detection on the cover image and obtain face detection results; The size acquisition module is used to perform virtual portrait detection on each face region if one or more face regions exist in the face detection results, and obtain the virtual portrait detection result of each face region; and to obtain the region size of each face region and the size of the cover image if the virtual portrait detection results of each face region do not contain a virtual portrait. The key point detection module is used to calculate the ratio of the width of each face region to the width of the cover image, the ratio of the height of each face region to the height of the cover image, and the ratio of the width and height of each face region itself, based on the area size of each face region and the size of the cover image. It determines that the proportion of each face region in the cover image is less than a preset proportion threshold and the width-to-height ratio of each face region is within a preset proportion threshold range. Then, it performs face contour key point detection on each face region to obtain the face contour key point detection result for each face region. The target cover image acquisition module is used to use the cover image as the target cover image if the detection results of the key points of the facial contour in each of the facial regions meet the preset conditions.

7. The cover image detection device for live streaming rooms according to claim 6, characterized in that, The face detection module includes: The face detection box acquisition unit is used to perform face detection on the cover image using a first convolutional neural network, and obtain multiple face detection boxes including faces in the cover image and corresponding confidence scores. The face detection box list acquisition unit is used to sort the multiple face detection boxes in descending order of confidence scores to obtain a face detection box list. The output list addition unit is used to add the face detection box with the highest confidence score to the output list and delete the face detection box with the highest confidence score from the face detection box list. The overlap calculation unit is used to calculate the overlap between the face detection box with the highest confidence score and other face detection boxes, and to delete face detection boxes with an overlap greater than the overlap threshold from the face detection box list. The face detection result acquisition unit is used to calculate the overlap between the face detection box with the highest confidence score remaining in the face detection box list and other face detection boxes, delete face detection boxes with an overlap greater than the overlap threshold in the face detection box list until the face detection box list is empty, and obtain the face detection result according to the output list.

8. A computer device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method as claimed in any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

Images