Live broadcast multi-dimensional monitoring method and device, equipment and medium

Abstract

The invention provides a live broadcast multi-dimensional monitoring method and device, equipment and a medium. The method comprises the following steps: acquiring video stream information; based on the video stream information, analyzing to obtain face information, audio information and picture brightness information; analyzing the face information, the audio information and the picture brightness information, and respectively judging whether the face information, the audio information and the picture brightness information meet preset conditions or not; when at least one of the face information, the audio information and the picture brightness information does not meet a preset condition, performing early warning on the violation information according to a preset priority, and displaying related early warning information; wherein the violation information indicates information which does not meet a preset condition; processing the video stream information based on the early warning information to obtain a processed video stream; and pushing the processed video stream to an audience side for playing. According to the scheme, the live broadcast content is monitored and managed in real time by adopting a multi-dimensional front monitoring technology, the legality and health degree of the live broadcast content are guaranteed, and the live broadcast effect is improved.

Description

Technical Field

[0001] The present application relates to the technical field of e-commerce live broadcast monitoring, and in particular to a multi-dimensional live broadcast monitoring method, device, equipment and medium. Background Art

[0002] With the rapid expansion of the live broadcast industry, live broadcast content has become more diverse and complex, which has also brought a series of problems to the anchor management and live broadcast quality of live broadcast companies. For example, the live broadcast screen is not bright enough, the anchor broadcasts banned words that cause the platform to deduct points, the microphone is abnormal, the anchor walks out of the screen for a long time or stands in a different position, etc. These problems not only affect the audience's viewing experience, but also damage the reputation of the live broadcast platform.

[0003] At present, live broadcasts are mostly monitored by third-party platforms, that is, the platforms where viewers watch live broadcasts. However, this monitoring method has many shortcomings. First, its monitoring scope is narrow, mainly for reminders of banned words, indecent actions and other content; second, the monitoring content is relatively simple and cannot fully cover various problems that may arise during the live broadcast; finally, the monitoring method is relatively simple, usually using manual / system real-time monitoring or systematic spot checks of recorded content.

[0004] Specifically, although human / system real-time monitoring can detect and handle problems to a certain extent, it is post-monitoring, that is, the content involved in the monitoring has been broadcast and known to the audience, so it is often unable to avoid adverse effects. The systematic spot check of recorded content is even more delayed, and warnings will only be issued when the spot check contains banned words, indecent actions, etc., which undoubtedly increases the time window for risk exposure.

[0005] In order to ensure the legality and healthiness of live broadcast content and improve the live broadcast effect to meet the needs of live broadcast companies, it is necessary to develop effective multi-dimensional front-end monitoring technology to monitor and manage live broadcast content in real time. Summary of the invention

[0006] In order to overcome the defects of the above-mentioned prior art, the present application provides a multi-dimensional monitoring method, device, equipment and medium for live broadcast, which adopts effective multi-dimensional front-end monitoring technology to monitor and manage the live broadcast content in real time, can ensure the legality and health of the live broadcast content, improve the live broadcast effect and meet the needs of the live broadcast company.

[0007] In order to solve the above technical problems, this application provides the following technical solutions: According to a first aspect of an embodiment of the present application, a live broadcast multi-dimensional monitoring method is provided, comprising: Get video stream information; Based on the video stream information, facial information, audio information and picture brightness information are obtained by parsing; Analyze the face information, the audio information, and the picture brightness information to determine whether the face information, the audio information, and the picture brightness information meet preset conditions; When at least one of the face information, the audio information and the screen brightness information does not meet the preset condition, an early warning is issued for the violation information according to the preset priority, and relevant early warning information is displayed; wherein the violation information indicates the information that does not meet the preset condition; Processing the video stream information based on the warning information to obtain a processed video stream; The processed video stream is pushed to the audience end for playback.

[0008] In an exemplary embodiment, based on the video stream information, parsing to obtain face information and picture brightness information includes: Intercepting the video stream information to obtain a video screen; Performing grayscale processing on the video picture to obtain a grayscale picture; Traversing all pixels of the grayscale image, and calculating the average value thereof as the image brightness information; Adjusting the contrast of the grayscale image, enhancing the details of the grayscale image, and obtaining a balanced image; Extracting facial features from the equalized image and locating the facial area; Based on the face area, the face information corresponding to the face area is determined; the face information includes face coordinates.

[0009] In an exemplary embodiment, parsing and obtaining audio information based on the video stream information includes: Extracting audio based on the video stream information; The audio is analyzed to obtain volume information, and the voice content in the audio is converted into text information.

[0010] In an exemplary implementation, analyzing the facial information to determine whether the facial information meets a preset condition includes: Based on the equalized picture, obtaining the picture width; Based on the face coordinates, obtaining a rectangular frame corresponding to the face coordinates; Based on the live broadcast category identifier carried in the video stream information, analyze whether the ratio information between the rectangular box and the picture width meets the preset condition; the ratio information includes the relationship between the coordinates of the rectangular box and the picture width, and the proportion of the rectangular box in the equalized picture.

[0011] In an exemplary embodiment, the analyzing, based on the live broadcast category identifier carried in the video stream information, whether the ratio information between the rectangular frame and the screen width meets the preset condition includes: Determine whether the rectangular frame has any coordinate loss; When coordinates of the rectangular frame are missing, it is determined that the face is out of the picture; when coordinates of the rectangular frame are not missing, based on the relationship between the coordinates of the rectangular frame and the width of the picture, it is determined that the picture meets the preset condition, or the picture is biased to the left, or the picture is biased to the right, and based on the live broadcast category identifier, it is determined that the distance between the anchor and the screen meets the preset condition, or the distance is too close, or the distance is too far.

[0012] In an exemplary embodiment, when the audio information does not meet the preset condition, the video stream information is processed based on the warning information to obtain a processed video stream, including: When the warning information indicates that the text information does not meet the preset condition, intercepting abnormal audio corresponding to the abnormal text; the abnormal text indicates the part of the text information that does not meet the preset condition, and the abnormal audio carries time axis data; Based on the time axis data of the abnormal audio, the sample audio is processed to obtain a matching sample audio; Using the matching sample audio to replace the abnormal audio to obtain edited audio; The video stream information is processed based on the edited audio to obtain the processed video stream.

[0013] In an exemplary embodiment, the step of issuing an early warning for violation information according to a preset priority level includes: Warnings are issued in sequence according to the priorities of the audio information, the face information, and the screen brightness information.

[0014] According to a second aspect of an embodiment of the present application, a live broadcast multi-dimensional monitoring device is provided, which adopts any one of the above-mentioned live broadcast multi-dimensional monitoring methods, and the device includes: A video stream acquisition module is used to acquire video stream information; A video stream analysis module, used to analyze the video stream information to obtain face information, audio information and picture brightness information; An information analysis module, used to analyze the face information, the audio information and the picture brightness information, and respectively determine whether the face information, the audio information and the picture brightness information meet preset conditions; An information warning module, used to issue a warning for violation information according to a preset priority and display relevant warning information when at least one of the face information, the audio information and the screen brightness information does not meet the preset condition; wherein the violation information indicates information that does not meet the preset condition; A video stream processing module, used for processing the video stream information based on the warning information to obtain a processed video stream; The push module is used to push the processed video stream to the audience end for playback.

[0015] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a live multi-dimensional monitoring method as described in any one of the above.

[0016] According to the fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement a live multi-dimensional monitoring method as described in any one of the above.

[0017] By adopting the above technical solution, the present application has the following beneficial effects: (1) Improving user experience: By real-time monitoring of the host's facial information, audio information, and screen brightness information, we can effectively avoid the appearance of illegal content, thereby improving the audience's viewing experience. When watching live broadcasts, viewers can feel a more comfortable and safe viewing environment, which will help improve user retention and activity.

[0018] (2) Protecting brand image: Brand image is extremely important for live streaming companies. By implementing effective multi-dimensional front-end real-time monitoring, violations can be discovered and dealt with in a timely manner, preventing negative events from occurring, and thus protecting the company’s brand image and reputation.

[0019] (3) Enhanced supervision capabilities: Traditional content review methods often rely on manual review, which is inefficient and prone to errors. By introducing advanced technical means to conduct multi-dimensional front-end real-time monitoring, automated and intelligent supervision can be achieved, greatly improving the efficiency and accuracy of supervision.

[0020] (4) Promoting compliant operations: With the continuous improvement and strict enforcement of relevant laws and regulations, the live broadcast industry faces increasingly stringent compliance requirements. By implementing effective multi-dimensional front-end real-time monitoring, live broadcast companies can better comply with laws and regulations and achieve compliant operations.

[0021] (5) Reduce operating costs: In the long run, by implementing a multi-dimensional front-end real-time monitoring technology solution, additional expenses such as fines and compensation caused by violations can be reduced, thereby reducing overall operating costs.

[0022] (6) Improving competitiveness: In the fierce market competition, providing high-quality and healthy live broadcast content is the key to attracting users. By implementing effective multi-dimensional front-end real-time monitoring, live broadcast companies can provide better quality and safer live broadcast services, thereby standing out from the competition.

[0023] (7) Promote technological innovation: With the continuous development and application of technology, the live broadcast industry will also usher in more innovation opportunities. By introducing advanced technical means to conduct multi-dimensional front-end real-time monitoring, it can not only improve the quality and effect of live broadcasts, but also provide more possibilities for the future development of the industry.

[0024] To sum up, effective multi-dimensional front-end real-time monitoring of the anchor's live broadcast can be carried out through facial information, audio information, and screen brightness information, and corresponding management measures can be taken after monitoring illegal information. This can not only ensure the legality and health of the live broadcast content, and improve the live broadcast effect to meet the needs of the live broadcast company, but also bring a series of beneficial effects, including improving user experience, protecting brand image, enhancing regulatory capabilities, promoting compliant operations, reducing operating costs, enhancing competitiveness, and promoting technological innovation. In addition, this technical solution is applicable to all types of live broadcasts and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flowchart of a multi-dimensional live broadcast monitoring method provided in an embodiment of the present application; Figure 2 A schematic diagram of a practical application of a live broadcast multi-dimensional monitoring method provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a facial information warning provided in an embodiment of the present application; Figure 4 A schematic diagram of a practical application of a live audio monitoring method provided in an embodiment of the present application; Figure 5 A schematic diagram of a practical application of a large-screen video display algorithm provided in an embodiment of the present application; Figure 6A structural block diagram of a live broadcast multi-dimensional monitoring device provided in an embodiment of the present application; Figure 7 A hardware structure block diagram of an electronic device that runs a live multi-dimensional monitoring method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] As used herein, "one embodiment" or "embodiment" refers to a The specific features, structures or characteristics in at least one implementation of the present application. In the description of the embodiments of the present application, it is to be understood that the orientation or positional relationship indicated by the terms "upper", "lower", "top", "bottom", etc. is based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present application. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can include one or more of the features explicitly or implicitly. Moreover, the terms "first", "second", etc. are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here.

[0029] See also Figure 1 , which is a flow chart of a live broadcast multi-dimensional monitoring method provided by an embodiment of the present application, the live broadcast multi-dimensional monitoring method comprising: Step S1: Obtain video stream information; Step S2: Based on the video stream information, analyzing to obtain face information, audio information and picture brightness information; Step S3: analyzing the face information, the audio information and the picture brightness information, and determining whether the face information, the audio information and the picture brightness information meet preset conditions; Step S4: when at least one of the face information, the audio information and the screen brightness information does not meet the preset condition, an early warning is issued for the violation information according to a preset priority, and relevant early warning information is displayed; wherein the violation information indicates information that does not meet the preset condition; Step S5: Processing the video stream information based on the warning information to obtain a processed video stream; Step S6: Push the processed video stream to the viewer end for playback.

[0030] In a specific embodiment, the live broadcast multi-dimensional monitoring method may further include: When the facial information, the audio information and the picture brightness information all meet the preset conditions, the video stream information is pushed to the viewer end for playback.

[0031] In an optional embodiment, in the above step S2, the facial information, the audio information and the picture brightness information are obtained by synchronous analysis, and there is no priority relationship between them.

[0032] Specifically, based on the video stream information, parsing to obtain face information and picture brightness information may include: Intercepting the video stream information to obtain a video screen; Performing grayscale processing on the video picture to obtain a grayscale picture; Traversing all pixels of the grayscale image, and calculating the average value thereof as the image brightness information; Adjusting the contrast of the grayscale image, enhancing the details of the grayscale image, and obtaining a balanced image; Extracting facial features from the equalized image and locating the facial area; Based on the face area, the face information corresponding to the face area is determined; the face information includes face coordinates.

[0033] Specifically, parsing and obtaining audio information based on the video stream information may include: Extracting audio based on the video stream information; The audio is analyzed to obtain volume information, and the voice content in the audio is converted into text information.

[0034] In an optional embodiment, in the above step S3, the facial information, the audio information and the picture brightness information are analyzed synchronously without any priority relationship.

[0035] Specifically, analyzing the facial information to determine whether the facial information meets a preset condition may include: Based on the equalized picture, obtaining the picture width; Based on the face coordinates, obtaining a rectangular frame corresponding to the face coordinates; Based on the live broadcast category identifier carried in the video stream information, analyze whether the ratio information between the rectangular box and the picture width meets the preset condition; the ratio information includes the relationship between the coordinates of the rectangular box and the picture width, and the proportion of the rectangular box in the equalized picture.

[0036] The analyzing, based on the live broadcast category identifier carried in the video stream information, whether the ratio information between the rectangular frame and the screen width meets the preset condition may specifically include: Determine whether the rectangular frame has any coordinate loss; When coordinates of the rectangular frame are missing, it is determined that the face is out of the picture; when coordinates of the rectangular frame are not missing, based on the relationship between the coordinates of the rectangular frame and the width of the picture, it is determined that the picture meets the preset condition, or the picture is biased to the left, or the picture is biased to the right, and based on the live broadcast category identifier, it is determined that the distance between the anchor and the screen meets the preset condition, or the distance is too close, or the distance is too far.

[0037] In an optional embodiment, since abnormal sound will more directly affect the audience, followed by the face, and finally the brightness (brightness and face position can sometimes be noticed by the operation of the live broadcast room and the field, and the impact on the live broadcast is relatively small, and the sound operation of the audience cannot be observed), therefore, if there are multiple violation information at the same time, the warning of sound will be prioritized, followed by the face, and finally the brightness, that is, in the above step S4, the warning of violation information according to the preset priority can include: Warnings are issued in sequence according to the priorities of the audio information, the face information, and the screen brightness information.

[0038] Specifically, for different warning objects, warning detection will be carried out at the same time, and a warning record will be kept for each warning information. It can be displayed from top to bottom according to the preset priority only in the warning display link, or the warning information can be displayed in pop-up windows in sequence according to the preset priority. Finally, the warning information will be collected into an interface, and the information will be pushed to directly notify the managers of the corresponding live broadcast room.

[0039] Since live broadcast is a dynamic process, for example, the anchor may leave the screen to take a sample, and the face may only be out of the screen for a few seconds. At this time, the warning information should not be issued. Therefore, the warning is divided into different levels, and different levels need to last for different periods of time before the level is upgraded. Warnings will be issued after the highest level.

[0040] The live broadcast room can be divided into areas for display according to its corresponding live broadcast category logo. When displaying warning information, in addition to displaying specific warning information, the display frame of the live broadcast room can also show the corresponding warning color.

[0041] In an optional embodiment, in the above step S5, for the warning information generated when the face information, the audio information or the screen brightness information does not meet the preset condition, different processing measures correspond to the processing of the video stream information, as follows: When the facial information does not meet the preset conditions, the corresponding warning information is pushed to notify the management personnel following the corresponding live broadcast room; wherein, the warning information generated when the facial information does not meet the preset conditions may include: the face is out of the picture, the picture is biased to the left, the picture is biased to the right, the anchor is too close to the screen, the anchor is too far from the screen, etc.

[0042] When the audio information does not meet the preset condition, processing the video stream information based on the warning information to obtain a processed video stream may include: In the case where the warning information indicates that the text information does not meet the preset condition, the corresponding warning information is pushed to notify the management personnel of the corresponding live broadcast room to warn, and the abnormal audio corresponding to the abnormal text is intercepted; the abnormal text indicates the part of the text information that does not meet the preset condition, and the abnormal audio carries timeline data; Based on the time axis data of the abnormal audio, the sample audio is processed to obtain a matching sample audio; Using the matching sample audio to replace the abnormal audio to obtain edited audio; The video stream information is processed based on the edited audio to obtain the processed video stream.

[0043] When the warning information indicates that the volume information does not meet the preset condition, determining whether the volume of the left channel and the right channel is within a preset range, and determining whether the absolute value of the difference between the volume of the left channel and the volume of the right channel meets a preset threshold; When the volume of the left channel or / and the right channel is not within the preset range, adjust the volume of the left channel or / and the right channel; when the absolute value of the difference between the volume of the left channel and the volume of the right channel does not meet the preset threshold, push the corresponding warning information to notify the management personnel of the corresponding live broadcast room; wherein, the warning information generated when the absolute value of the difference between the volume of the left channel and the volume of the right channel does not meet the preset threshold may include that the host is too far away from the microphone and the position is to the left, or that the host is too far away from the microphone and the position is to the right.

[0044] When the screen brightness information does not meet the preset conditions, the corresponding warning information is pushed to notify the management personnel of the corresponding live broadcast room to adjust the brightness of the live broadcast room in time.

[0045] See also Figure 2 , which is a schematic diagram of a practical application of a live broadcast multi-dimensional monitoring method provided by an embodiment of the present application; as shown in the figure, in an actual application scenario, the live broadcast multi-dimensional monitoring method includes: Step 1: Video stream crawler: collect live video streams based on the basic information corresponding to the anchor's live broadcast room; Step 2: Key video stream data collection, including: Face recognition: capture a frame of the live video stream, and perform face recognition and coordinate identification based on TensorFlow and face-api.js; Sound volume recognition: Get the left and right channel data of the video stream based on the HTML5 bottom layer; Speech-to-text: Based on a third-party open source model, implemented by vosk; Brightness recognition: Get the brightness and darkness data of the video stream based on the underlying HTML5; Step 3: Key video stream data analysis, including: Face warning algorithm: Figure 3 As shown, according to the face recognition in step 2, the coordinates a and d of the anchor's face area during the live broadcast can be obtained; Through the a and d coordinates, a rectangular frame of the face in the video can be obtained. By determining the position of the rectangular frame, abnormal data can be obtained, as follows: (1) One of the coordinates a or d is missing -> the face is out of the picture; (2) Assuming the video width is w, if the x-axis coordinate of a (3) Assuming the video width is w, if the x-axis coordinate of d > ww*10, and the x-axis coordinate of a is not less than w*10%, the picture is biased to the right; (4) Based on the a and d coordinates, a rectangular frame can be drawn to define the range of the face. Based on the size of the video, the proportion of the face in the entire video can be determined. Through AI learning and daily accumulation, the safety values ​​of the range of faces in different categories during live broadcasts can be obtained. By comparing the values, it can be determined whether the host is too close or too far from the screen. Abnormal sound warning algorithm: Through AI learning and daily accumulation, the suitable range of volume n~m and the deviation value of left and right channels (left channel volume - right channel volume) x can be obtained; (1) If the volume of either the left or right channel is less than n or greater than m, the microphone volume is abnormal; (2) If the absolute value of the left channel volume minus the right channel volume is greater than x, the microphone volume is abnormal, usually because the host is too far away from the microphone and is positioned to the left or right; Banned word warning algorithm: like Figure 4 As shown, the live broadcast room is broadcasted through live broadcast hardware such as camera equipment and bees. The system receives data stream slices (i.e., video stream information) and separates the video from the voice. The voice will be uploaded to the server for banned word detection and returned to the system after the detection is completed. If there is banned word information, the system will mute and replace the original audio related words, and then synchronize the audio and video with the picture, and finally return to the live broadcast on the third-party platform. Because it is a live broadcast scene, the above processing of the system needs to be controlled within 5 seconds. Among them, in order to expand the banned word library, the crawler service regularly crawls banned word data on third-party websites or public cosmetics industry websites. Python+ffmpeg can be used for video stream slicing and audio and video separation. The open source voice processing model Whisper can be used to identify the time nodes when banned words and banned words appear. When the banned words are at the same time node, the recorded mute template (currently a blank sound) is used and superimposed on the original audio, so that a mute audio can be generated. ffmpeg can be used to synthesize the sliced ​​audio and video, and push the video to the third-party platform after integration. Brightness warning algorithm: Through AI learning and daily accumulation, the suitable range of volume y~z can be obtained, and if the brightness is not within the range, it is abnormal; Step 4: Early warning; Live broadcast is a dynamic process. For example, the anchor may go out of the screen to pick up a sample for only a few seconds. At this time, the warning information cannot be issued. Therefore, the system divides the warning into different levels, and different levels need to last for different periods of time before the level is upgraded. Warnings will be issued after the highest level. Step 5: Display the algorithm on a large screen video; Purpose: To achieve the best display effect on the large screen of the live broadcast center, the same categories should be placed in the same row as much as possible, and those not broadcasted should be placed on both sides.

[0046] like Figure 5 As shown, the large-screen video display algorithm specifically includes: Use AI algorithms to distinguish live broadcast categories; if the live broadcast is a product live broadcast, it will be distinguished according to the product category, for example, it can be divided into 4 categories: clothing, shoes, jewelry, and cosmetics; Use crawler programs to determine whether the live broadcast room is on; Create array 1, array 2, and array 3, using categories as key values.

[0047] Optionally, the specific process may include: Get live broadcast room data; Determine whether the number of the same category in the array in the live broadcast is greater than 8; If yes, determine whether the broadcast has started; if not, add it to array 1, and finally get array 1 with category as key value, and traverse to get the number of vacancies m (8-number of live broadcast rooms) in each category; If yes, that is, it has been broadcasted, then it is added to array 2, and finally array 2 with category as key value is obtained, and the number of category live broadcast rooms n in array 2 is obtained; if no, that is, it has not been broadcasted, then it is added to array 3, and finally array 3 with category as key value is obtained; Traverse array 1 and array 2, search array 1 for the category with the smallest mn for each category in array 2, and fill the live broadcast information in array 1 until 8 are full; For arrays that are not yet full of 8, use the data in array 3 to fill them.

[0048] It can be seen from the above technical solutions of the embodiments of the present application that in the embodiments of the present application, effective multi-dimensional front-end real-time monitoring of the anchor's live broadcast is carried out through three aspects: facial information, audio information, and picture brightness information, and corresponding management measures are taken after monitoring illegal information. This can not only ensure the legality and health of the live broadcast content, and improve the live broadcast effect to meet the needs of the live broadcast company, but also bring a series of beneficial effects, including improving user experience, protecting brand image, enhancing regulatory capabilities, promoting compliant operations, reducing operating costs, enhancing competitiveness, and promoting technological innovation, etc., and this technical solution is applicable to all types of live broadcasts and has a wide range of applications.

[0049] Corresponding to the live multi-dimensional monitoring method provided in the above embodiment, the embodiment of the present application also provides a live multi-dimensional monitoring device. Since the live multi-dimensional monitoring device provided in the embodiment of the present application corresponds to the live multi-dimensional monitoring method provided in the above embodiment, the implementation method of the aforementioned live multi-dimensional monitoring method is also applicable to the live multi-dimensional monitoring device provided in this embodiment, and will not be described in detail in this embodiment.

[0050] See also Figure 6 , which is a structural block diagram of a live multi-dimensional monitoring device provided in an embodiment of the present application; the device includes: 01: Video stream acquisition module, used to obtain video stream information; 02: A video stream analysis module, used to analyze the video stream information to obtain face information, audio information and picture brightness information; 03: an information analysis module, used to analyze the face information, the audio information and the picture brightness information, and respectively determine whether the face information, the audio information and the picture brightness information meet preset conditions; 04: an information warning module, used to issue a warning for violation information according to a preset priority and display relevant warning information when at least one of the face information, the audio information and the screen brightness information does not meet the preset condition; wherein the violation information indicates information that does not meet the preset condition; 05: a video stream processing module, used to process the video stream information based on the warning information to obtain a processed video stream; 06: Push module, used to push the processed video stream to the audience end for playback.

[0051] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0052] The live broadcast multi-dimensional monitoring device of the embodiment of the present application performs effective multi-dimensional front-end real-time monitoring of the anchor's live broadcast through three aspects: facial information, audio information, and picture brightness information, and takes corresponding management measures after monitoring illegal information. It can not only ensure the legality and healthiness of the live broadcast content, and improve the live broadcast effect to meet the needs of the live broadcast company, but also bring a series of beneficial effects, including improving user experience, protecting brand image, enhancing regulatory capabilities, promoting compliant operations, reducing operating costs, enhancing competitiveness and promoting technological innovation, etc., and this technical solution is applicable to all types of live broadcasts and has a wide range of applications.

[0053] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the live multi-dimensional monitoring method provided in the above method embodiment.

[0054] The memory can be used to store software programs and modules. The processor executes various functional applications and realizes high-level autonomous driving by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, application programs required for functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0055] The method embodiments provided in the embodiments of the present application may be executed in a computer terminal, a server or a similar computing device, that is, the above-mentioned electronic device may include a computer terminal, a server or a similar computing device. Figure 7 is a hardware structure block diagram of an electronic device for running a live multi-dimensional monitoring method provided in an embodiment of the present application, such as Figure 7 As shown, the internal structure of the electronic device may include but is not limited to: a processor, a network interface and a memory. The processor, network interface and memory in the electronic device may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0056] Among them, the processor (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device. The network interface may optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.). Memory is a memory device in an electronic device for storing programs and data. It is understandable that the memory here can be a high-speed RAM storage device or a non-volatile memory device, such as at least one disk storage device; optionally, it can also be at least one storage device located away from the aforementioned processor. The memory provides a storage space, which stores the operating system of the electronic device, including but not limited to: Windows system (an operating system), Linux (an operating system), Android (Android, a mobile operating system) system, IOS (a mobile operating system) system, etc., and this application does not limit this; and, in the storage space, one or more instructions suitable for being loaded and executed by the processor are also stored, and these instructions can be one or more computer programs (including program codes). In the embodiment of this specification, the processor loads and executes one or more instructions stored in the memory to implement the live multi-dimensional monitoring method provided in the above method embodiment.

[0057] An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the live multi-dimensional monitoring method provided by the method embodiment.

[0058] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0059] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. And the above describes a specific embodiment of this specification. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-small sample image classification and parallel processing are also possible or may be advantageous.

[0060] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0061] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0062] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multi-dimensional monitoring method for live broadcast, characterized in that: include: Get video stream information; Based on the video stream information, facial information, audio information and picture brightness information are obtained by parsing; Analyze the face information, the audio information, and the picture brightness information to determine whether the face information, the audio information, and the picture brightness information meet preset conditions; When at least one of the face information, the audio information and the screen brightness information does not meet the preset condition, an early warning is issued for the violation information according to the preset priority, and relevant early warning information is displayed; wherein the violation information indicates the information that does not meet the preset condition; Processing the video stream information based on the warning information to obtain a processed video stream; The processed video stream is pushed to the audience end for playback.

2. A live broadcast multi-dimensional monitoring method according to claim 1, characterized in that: Based on the video stream information, facial information and screen brightness information are obtained by parsing, including: Intercepting the video stream information to obtain a video screen; Performing grayscale processing on the video picture to obtain a grayscale picture; Traversing all pixels of the grayscale image, and calculating the average value thereof as the image brightness information; Adjusting the contrast of the grayscale image, enhancing the details of the grayscale image, and obtaining a balanced image; Extracting facial features from the equalized image and locating the facial area; Based on the face area, the face information corresponding to the face area is determined; the face information includes face coordinates.

3. A live broadcast multi-dimensional monitoring method according to claim 2, characterized in that: Based on the video stream information, the audio information is parsed and obtained, including: Extracting audio based on the video stream information; The audio is analyzed to obtain volume information, and the voice content in the audio is converted into text information.

4. A live broadcast multi-dimensional monitoring method according to claim 2, characterized in that: Analyzing the facial information to determine whether the facial information meets a preset condition includes: Based on the equalized picture, obtaining the picture width; Based on the face coordinates, obtaining a rectangular frame corresponding to the face coordinates; Based on the live broadcast category identifier carried in the video stream information, analyze whether the ratio information between the rectangular box and the picture width meets the preset condition; the ratio information includes the relationship between the coordinates of the rectangular box and the picture width, and the proportion of the rectangular box in the equalized picture.

5. A live broadcast multi-dimensional monitoring method according to claim 4, characterized in that: The analyzing, based on the live broadcast category identifier carried in the video stream information, whether the ratio information between the rectangular frame and the picture width meets the preset condition includes: Determine whether the rectangular frame has any coordinate loss; When coordinates of the rectangular frame are missing, it is determined that the face is out of the picture; when coordinates of the rectangular frame are not missing, based on the relationship between the coordinates of the rectangular frame and the width of the picture, it is determined that the picture meets the preset condition, or the picture is biased to the left, or the picture is biased to the right, and based on the live broadcast category identifier, it is determined that the distance between the anchor and the screen meets the preset condition, or the distance is too close, or the distance is too far.

6. A live broadcast multi-dimensional monitoring method according to claim 3, characterized in that: When the audio information does not meet the preset condition, the video stream information is processed based on the warning information to obtain a processed video stream, including: When the warning information indicates that the text information does not meet the preset condition, intercepting abnormal audio corresponding to the abnormal text; the abnormal text indicates the part of the text information that does not meet the preset condition, and the abnormal audio carries time axis data; Based on the time axis data of the abnormal audio, the sample audio is processed to obtain a matching sample audio; Using the matching sample audio to replace the abnormal audio to obtain edited audio; The video stream information is processed based on the edited audio to obtain the processed video stream.

7. A live broadcast multi-dimensional monitoring method according to any one of claims 1 to 6, characterized in that: The warning of violation information according to the preset priority includes: Warnings are issued in sequence according to the priorities of the audio information, the face information, and the screen brightness information.

8. A live broadcast multi-dimensional monitoring device, using a live broadcast multi-dimensional monitoring method according to any one of claims 1 to 7, characterized in that: The device comprises: A video stream acquisition module is used to acquire video stream information; A video stream analysis module, used to analyze the video stream information to obtain face information, audio information and picture brightness information; An information analysis module, used to analyze the face information, the audio information and the picture brightness information, and respectively determine whether the face information, the audio information and the picture brightness information meet preset conditions; An information warning module, used to issue a warning for violation information according to a preset priority and display relevant warning information when at least one of the face information, the audio information and the screen brightness information does not meet the preset condition; wherein the violation information indicates information that does not meet the preset condition; A video stream processing module, used for processing the video stream information based on the warning information to obtain a processed video stream; The push module is used to push the processed video stream to the audience end for playback.

9. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a live multi-dimensional monitoring method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded and executed by a processor to implement a live multi-dimensional monitoring method as described in any one of claims 1 to 7.

Images