Live monitoring method, device and electronic equipment

Abstract

The application provides a live broadcast monitoring method and device and electronic equipment, and relates to the technical field of live broadcast. The method comprises the following steps: acquiring a live broadcast video of a live broadcast room and interaction information corresponding to the live broadcast video; acquiring an image motion feature of the live broadcast video based on the live broadcast video; acquiring an interaction feature related to the live broadcast video based on the interaction information, wherein the interaction feature represents a change trend of the interaction frequency of a user in a time period corresponding to the live broadcast video; fusing the image motion feature and the interaction feature to generate a target fusion feature; and monitoring whether the live broadcast video has a violation risk according to the target fusion feature. The application can realize joint decision of the image sequence mode and the interaction information mode, effectively overcome the decision drawbacks of a single image, and thus realize more accurate and effective monitoring of violation behaviors.

Description

TECHNICAL FIELD

[0001] The present application relates to the live broadcast technical field, in particular to a live broadcast monitoring method and device and electronic equipment. BACKGROUND

[0002] At present, with the continuous development of network live broadcast technology, there are more and more online entertainment projects that users can participate in. Among them, network live broadcast is favored by more and more users due to its strong real-time and strong interaction. However, with the rapid development of the live broadcast industry, some undesirable live broadcast information may be spread on the network, thereby causing undesirable social influence. In order to ensure the benign development of the live broadcast industry, it is necessary to monitor the video information of the anchor live broadcast and seal the risky illegal live broadcast behavior.

[0003] Generally, for illegal behaviors with time sequence characteristics such as suggestive dance, vulgar and indecent behavior, the monitoring method adopted is to extract key frames from the live broadcast video stream, generate corresponding pictures, and then judge whether the pictures contain illegal behaviors according to the picture features. However, the live broadcast scene has rich picture information, and making illegal decision only from the single frame screenshot level cannot make the live broadcast monitoring accurate and effective, and cannot accurately identify the live broadcast room with risks, thereby causing the risk live broadcast video to be played illegally. SUMMARY

[0004] In order to at least overcome the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a live broadcast monitoring method, device and electronic equipment.

[0005] In a first aspect, the embodiments of the present application provide a live broadcast monitoring method, which comprises:

[0006] obtaining a live broadcast video of a live broadcast room and interaction information corresponding to the live broadcast video;

[0007] based on the live broadcast video, obtaining image motion features of the live broadcast video;

[0008] based on the interaction information, obtaining interaction features related to the live broadcast video, the interaction features representing the change trend of the user interaction frequency in the time period corresponding to the live broadcast video;

[0009] fusing the image motion features and the interaction features to generate target fusion features;

[0010] monitoring whether the live broadcast video has a risk of violation according to the target fusion features.

[0011] In a possible implementation manner, the step of obtaining the image motion features of the live broadcast video based on the live broadcast video comprises:

[0012] performing optical flow analysis on a plurality of live video images in the live video to obtain a plurality of optical flow information maps corresponding to the plurality of live video images;

[0013] obtaining image motion features of the live video according to the optical flow information maps.

[0014] In a possible implementation, the step of obtaining the image motion features of the live video according to the optical flow information maps includes:

[0015] performing feature extraction on each of the optical flow information maps to obtain optical flow feature information corresponding to each of the optical flow information maps;

[0016] performing time sequence feature extraction on each of the optical flow feature information in a time sequence to obtain optical flow time sequence features as the image motion features of the live video.

[0017] In a possible implementation, the step of performing time sequence feature extraction on each of the optical flow feature information in a time sequence to obtain optical flow time sequence features as the image motion features of the live video includes:

[0018] capturing the optical flow time sequence features of the live video based on the optical flow feature information by using a Bi-GRU;

[0019] obtaining weights of the optical flow time sequence features by using a self-attention mechanism, distributing the optical flow time sequence features, and obtaining the image motion features of the live video.

[0020] In a possible implementation, the interaction information includes text interaction information.

[0021] The step of obtaining the interaction features related to the live video based on the interaction information includes:

[0022] obtaining a text interaction frequency in the live room according to a set time interval, the text interaction frequency including a sending amount of barrage information;

[0023] generating an interaction frequency vector according to the text interaction frequency in a plurality of time intervals in a time sequence;

[0024] performing feature extraction on the interaction frequency vector to obtain the interaction features.

[0025] In a possible implementation, the method further includes:

[0026] performing feature extraction on audio information in the live video to obtain audio features;

[0027] The step of fusing the image motion features and the interaction features to generate target fusion features includes:

[0028] The image motion feature, the interaction feature and the audio feature are fused to generate a target fusion feature.

[0029] In a possible implementation, after the step of fusing the image motion feature and the interaction feature to generate a target fusion feature, the method further includes:

[0030] Based on the target fusion feature, a fusion decision is made through a Projector network structure to monitor whether the live video has a violation risk.

[0031] In a second aspect, the embodiments of the present application further provide a live monitoring apparatus, including:

[0032] A receiving module configured to acquire a live video of a live room and interaction information corresponding to the live video;

[0033] An image processing module configured to acquire an image motion feature of the live video based on the live video;

[0034] An interaction information processing module configured to acquire an interaction feature related to the live video based on the interaction information, the interaction feature representing a change trend of an interaction frequency of a viewer user in a time period corresponding to the live video;

[0035] A fusion module configured to fuse the image motion feature and the interaction feature to generate a target fusion feature;

[0036] A monitoring module configured to monitor whether the live video has a violation risk according to the target fusion feature.

[0037] In a third aspect, the embodiments of the present application further provide an electronic device, including:

[0038] A memory configured to store one or more programs;

[0039] A processor configured to implement the live screenshot processing method provided in the first aspect when the one or more programs are executed by the processor.

[0040] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium having a computer program stored thereon, and the computer program is configured to implement the live screenshot processing method provided in the first aspect when executed by a processor.

[0041] Based on any one of the above aspects, the embodiment of the present application provides a live monitoring method, device and electronic equipment, which obtains image motion features of a live video and interactive features of interactive information, and fuses the image motion features and the interactive features, so as to realize joint decision of image sequence modal and interactive information modal, effectively overcome the decision drawbacks of single frame image, and thus realize more accurate and effective monitoring of illegal action behaviors. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be called in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 A schematic application scenario diagram of the live monitoring method provided by the present embodiment is shown in FIG. 1.

[0044] Figure 2 A schematic flowchart of the live monitoring method provided by the present embodiment is shown in FIG. 2.

[0045] Figure 3 A sub-step schematic diagram of step S200 provided by the present embodiment is shown in FIG. 3.

[0046] Figure 4 A sub-step schematic diagram of step S220 provided by the present embodiment is shown in FIG. 4.

[0047] Figure 5 A sub-step schematic diagram of step S222 provided by the present embodiment is shown in FIG. 5.

[0048] Figure 6 A schematic diagram of network structure provided by the present embodiment is shown in FIG. 6.

[0049] Figure 7 A sub-step schematic diagram of step S300 provided by the present embodiment is shown in FIG. 7.

[0050] Figure 8 A schematic diagram of network structure provided by the present embodiment is shown in FIG. 8.

[0051] Figure 9 A schematic structural block diagram of the electronic equipment provided by the present embodiment is shown in FIG. 9.

[0052] Figure 10 A functional module schematic diagram of the live monitoring device provided by the present embodiment is shown in FIG. 10.

[0053] Icon: 100 - server; 200 - audience terminal; 300 - anchor terminal; 700 - electronic device; 710 - processor; 720 - computer readable storage medium; 730 - live broadcast monitoring device; 731 - receiving module; 732 - image processing module; 733 - interactive information processing module; 734 - fusion module; 735 - monitoring module. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts fall within the scope of protection of the present application.

[0056] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0057] In the description of the present application, it should be noted that the orientation or position relationship indicated by the terms "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the product of the present application is usually placed, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", etc. are only used for differentiation in description, and cannot be understood as indicating or implying relative importance.

[0058] In the description of the present application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set", "mount", "connected", "connected" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0059] It should be noted that the different features in the embodiments of the present application can be combined with each other in the case of no conflict.

[0060] The present embodiment provides a solution to the above problems. The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0061] Please refer to Figure 1 , Figure 1 The example live monitoring system provided by the present embodiment includes a server 100, a viewer terminal 200 and a host terminal 300. The server 100, the viewer terminal 200 and the host terminal 300 can communicate with each other through wired or wireless networks. The server 100 can be a stand-alone electronic device or a cluster composed of multiple electronic devices. The live monitoring method provided by the present embodiment can be executed by the server 100, which is configured to receive network live information transmitted by the host terminal 300 and receive interactive information uploaded by the viewer terminal 200.

[0062] In the embodiments of the present application, the viewer terminal 200 refers to the end receiving and watching network live video, which is usually the terminal used by the viewer user watching the video in the network live. The host terminal 300 refers to the end sending network live video, which is usually the terminal used by the host user in the network live. Specifically, the viewer terminal 200 and the host terminal 300 can adopt mobile terminal devices, such as smart phones, personal computers (PCs), tablet computers, etc. The viewer terminal 200 and the host terminal 300 can be the same device or different devices. For example, the viewer terminal 200 and the host terminal 300 can both be mobile phones, or the host terminal 300 can be a personal computer and the viewer terminal 200 can be a mobile phone.

[0063] It should be noted that the viewer terminal 200 and the host terminal 300 are only functionally divided. In actual application, the same terminal can execute the functions of the viewer terminal 200 and the host terminal 300 at different times. Therefore, the same client can act as the viewer terminal 200 when watching network live and act as the host terminal 300 when publishing live video.

[0064] Please refer to Figure 2 , Figure 2 The flowchart of the live monitoring method provided by the present embodiment is shown in Figure 1 The method can include the following steps.

[0065] Step S100, acquiring live video of a live room and interactive information corresponding to the live video.

[0066] In the embodiment, the live room can be a virtual live room belonging to a host user. The host user can broadcast a live video in the live room, and a viewer user entering the live room can watch the live video. Meanwhile, the viewer user and the host user in the live room can interact, for example, send comments, send bullet screen, like, and send gifts.

[0067] In step S100, the live video broadcasted in the live room in a time period can be acquired, and the interaction information executed in the live room in the time period can be acquired.

[0068] In step S200, based on the live video, an image motion feature of the live video is acquired.

[0069] In the embodiment, the image motion feature with time sequence can be acquired according to the live video acquired in step S100. Compared with the image feature determined based on a single live image, the image motion feature determined based on the whole live video can better represent the characteristics of the live video such as dance.

[0070] In step S300, based on the interaction information, an interaction feature related to the live video is acquired, and the interaction feature represents the change trend of the interaction frequency of the user in the time period corresponding to the live video.

[0071] The inventor found that in the live video with violation risk such as dance, the violation action usually does not always appear continuously in the whole live video, and when some violation actions appear, the frequency of the interaction operation of the viewer user in the live room usually changes, for example, the interaction frequency increases sharply. Therefore, in the embodiment, the interaction feature representing the change trend of the interaction frequency of the user in the time period corresponding to the live video can be extracted according to the interaction information.

[0072] In step S400, the image motion feature and the interaction feature are fused to generate a target fusion feature.

[0073] In the embodiment, the image motion feature and the interaction feature can be fused by splicing to obtain the target fusion feature. Because it is difficult to effectively capture some short-time violation actions by the image motion feature in the whole live video, and the interaction information can more obviously capture some short-time violation actions, capturing the mutual correlation information between multiple modalities, that is, fusing the image motion feature and the interaction feature to generate the target fusion feature, can make the monitoring result more accurate and effective.

[0074] Step S500, monitoring whether the live video has a risk of violation according to the target fusion feature.

[0075] In the embodiment, the target fusion feature fuses relevant data of different modalities, can make full use of the complementarity of different data, and monitors whether the live video has a risk of violation. Compared with single-modality fusion, a more reasonable and accurate judgment can be made.

[0076] It can be seen that, based on the above design, the live monitoring method provided in the embodiments of the present application can obtain the image motion feature of the live video and the interaction feature of the interaction information corresponding to the live video, fuse the image motion feature and the interaction feature, generate a target fusion feature, and monitor whether the live video has a risk of violation, so as to realize joint decision of the image sequence modality and the interaction information modality, effectively overcome the decision drawbacks of single-frame images, and thus realize more accurate and effective monitoring of the violation action behavior, and reduce the risk of live video violation.

[0077] In a possible implementation manner, referring to Figure 3 , step S200 can further include the following sub-steps.

[0078] Step S210, performing optical flow analysis on a plurality of live video images in the live video to obtain a plurality of optical flow information maps corresponding to the live video images.

[0079] In the embodiment, optical flow is a method for finding the correspondence between the previous frame and the current frame in the image sequence by using the change of pixels in the time domain and the correlation between adjacent frames, so as to calculate the motion information of objects between adjacent frames. Specifically, the essence of optical flow is a two-dimensional vector field, each vector represents the displacement of a point in the scene from the previous frame to the next frame, which can be decomposed into a horizontal vector and a vertical vector. The optical flow information can show the moving state of the object in the live video image at adjacent time, i.e., the motion state of the object in the live video image at t1 and t2, which can be used to determine the motion of the object. The optical flow information map can be obtained by the optical flow information of two adjacent time. The values on the optical flow information map can represent the vertical displacement value and / or the horizontal displacement value between each pixel point in the live video image at t1 and the corresponding pixel point in the live video image at t2. Therefore, the optical flow information map can focus on the moving object, identify the action, facilitate the analysis of the dynamic change of the live video image, and exclude background interference information.

[0080] Exemplarily, in a live video with a risk of violation, such as a dance, for two live video images with a size of HxWx3 at time t1 and time t2, after optical flow analysis, the optical flow information image with an output size of HxWx2 can be obtained, which can focus on the dance itself, identify the dance, and reduce the interference of background information.

[0081] Optionally, the PWC-Net optical flow neural network can be used to identify and analyze the dance in the live video, so as to quickly and efficiently extract the optical flow information image from the live video image.

[0082] In the above design, by performing optical flow analysis on multiple live video images in the live video to model the image motion state with optical flow, the problem of too much interference information in a live scene with rich pictures can be effectively solved.

[0083] Step S220: obtaining the image motion feature of the live video according to the optical flow information image.

[0084] In this embodiment, the optical flow information image representing the motion information of the object in the live video image obtained in step S210 can be used to reduce the interference of the background in the live video image, and determine the image motion feature of the live video.

[0085] In a possible implementation, please refer to Figure 4 Step S220 can further include the following sub-steps.

[0086] Step S221: performing feature extraction on each optical flow information image to obtain the optical flow feature information corresponding to each optical flow information image.

[0087] In this embodiment, the optical flow information image with background interference information removed in step S210 can be encoded to further compress the redundant information, so as to obtain the optical flow feature information with high general discriminability.

[0088] Specifically, the general neural network pre-trained in a large-scale image data set can be used for feature encoding. Exemplarily, Xception-Net can be selected to encode the optical flow information image. Since the network structure of Xception-Net is based on depth separable convolution, compared with the traditional convolution operation, the parameter amount and computational complexity in the network model can be reduced, the cross-channel correlation and spatial correlation can be effectively decoupled, the running time is low, and the optical flow feature information can be better extracted, thereby improving the accuracy.

[0089] Step S222, time sequence feature extraction is performed on each of the optical flow feature information according to time sequence to obtain optical flow time sequence features as image motion features of the live video.

[0090] In this embodiment, the time sequence feature extraction can refer to time sequence feature extraction processing performed on each of the optical flow feature information. The image motion features with time sequence can be further obtained according to the optical flow feature information obtained in step S221.

[0091] In a possible implementation, referring to Figure 5 , step S222 can further include the following sub-steps.

[0092] Step S223, based on the optical flow feature information, the optical flow time sequence features of the live video are captured by Bi-GRU.

[0093] In this embodiment, the optical flow feature information obtained in step S221 represents motion information of adjacent time of the live video image, and the judgment of the violation behavior in the live room is a long-time process, therefore, the motion information of the live video image for a long time, i.e., the optical flow time sequence features, need to be captured. Specifically, the optical flow time sequence features can be captured by Bi-GRU, which combines two GRU models and is a bidirectional gated recurrent unit model, which can process the input sequence from the forward and reverse directions. When processing the forward sequence, the model processes the input from left to right step by step according to the natural order of the sequence; when processing the reverse sequence, the input sequence is reversed, i.e., processed in the order from right to left.

[0094] In the above design, the optical flow time sequence features of the live video are captured by Bi-GRU, which can realize the long-time detection technology of the live room. In addition, since the live content for a short time is difficult to make accurate decisions on the violation behavior of the live video, the accuracy of live monitoring can also be improved.

[0095] Step S224, the weight of the optical flow time sequence features is obtained by using the self-attention mechanism, the optical flow time sequence features are distributed, and the image motion features of the live video are obtained.

[0096] In this embodiment, for the live video with violation risk such as dance, some violation behaviors usually depend on certain key actions, therefore, the features of these key actions should have greater weight values.

[0097] Specifically, by obtaining the weight of the optical flow time sequence feature and redistributing the optical flow time sequence feature by using the self-attention mechanism, the weighted image motion feature can be obtained. The self-attention mechanism can dynamically capture context-related information in the sequence and apply important context relationships to the representation of each position. Therefore, the image motion feature can better capture the key actions in some rule violation behaviors in the live video, thereby achieving more accurate behavior decision.

[0098] Specifically, please refer to Figure 6 After capturing the optical flow time sequence feature with long-time motion characteristics by Bi-GRU, the weight of the optical flow time sequence feature can be obtained by using a fully connected layer. According to the obtained weight relationship, the optical flow time sequence feature is effectively distributed by using the self-attention mechanism. Then, the optical flow time sequence feature is spliced by using the Concatenate layer, and then expanded by using the Flatten layer to convert the input data with multiple dimensions into a one-dimensional vector form, thereby obtaining the image motion feature that can better represent the characteristics of, for example, dance live videos.

[0099] In a possible implementation, the interaction information can include text interaction information.

[0100] For example, the text interaction information can be the bullet screen information sent by the audience. The bullet screen information can reflect the popularity of the live video, and the sending amount of the bullet screen information is different at different times.

[0101] In this embodiment, in a live video with rule violation risk such as dance, rule violation actions usually do not always persistently appear, and when some rule violation actions appear, the frequency of the audience users performing interaction operations in the live room usually changes. For example, the sending amount of the bullet screen information can increase sharply at the rule violation moment of the live video. Therefore, by obtaining the interaction information related to the bullet screen information and modeling the performance behavior of the audience when contacting the rule violation behavior, the rule violation action behavior can be effectively monitored.

[0102] Further, please refer to Figure 7 Step S300 can further include the following sub-steps.

[0103] Step S310: Obtain the text interaction frequency in the live room according to a set time interval, and the text interaction frequency includes the sending amount of the bullet screen information.

[0104] In the embodiment, when a violation occurs in the live video, the text interaction frequency in the live room can change obviously, for example, the sending amount of the live room's barrage information can increase obviously. Therefore, acquiring the sending amount of the barrage information in the live room according to the set time interval can better monitor the performance of the audience in different time periods, so as to monitor whether the live video has a violation risk.

[0105] In step S320, the interaction frequency vector is generated according to the text interaction frequency in the plurality of time intervals.

[0106] In the embodiment, for the text interaction information in the live video, the text interaction information needs to be processed into the input format required by the embodiment, that is, the text information converted from the barrage information is represented in the form of a vector, so that the text interaction information can be feature extracted to obtain the text interaction feature, and the text interaction feature and the image motion feature are fused.

[0107] For example, when the set time interval is 5 seconds, a 1*30-dimensional interaction frequency vector can be obtained, and the vector value of the interaction frequency vector represents the sending amount of the barrage information in 5 seconds.

[0108] In step S330, the interaction frequency vector is feature extracted to obtain the interaction feature.

[0109] In the embodiment, the operation of feature extracting the interaction frequency vector is simpler than the operation of feature extracting the optical flow feature information. Please refer to Figure 8 , Figure 8 For example, a structure block diagram of a neural network can be used to feature extract the interaction frequency vector obtained in step S320 to obtain the interaction feature satisfying the input format required by the embodiment, so that the interaction feature and the image motion feature are fused, and more accurate and effective violation action monitoring of the live room can be realized.

[0110] In other possible implementations, the interaction information can include text interaction information or voice interaction information, where the text interaction information can be the online number of the live room, the barrage information sent by the audience, the gift giving amount, the like amount, and the comment amount. Since the online number of the live room, the barrage information sent by the audience, the gift giving amount, the like amount, and the comment amount can surge when a violation occurs in the live video, a neural network can be constructed to obtain the feature of the interaction information.

[0111] In a possible implementation, the live monitoring method provided in the application can also perform feature extraction on audio information in the live video to obtain audio features. For example, the audio information can be voice information of the host or voice information of the audience in the live broadcast.

[0112] In this embodiment, for the audio information in the live video, the audio information needs to be processed into the input format required by the application before being fused, that is, the audio information is subjected to feature extraction to obtain the audio features.

[0113] When the image motion features and the interaction features are fused to generate target fusion features, the image motion features, the interaction features and the audio features can also be fused to generate target fusion features.

[0114] In this embodiment, the final decision is made through multi-modal fusion, which can make full use of the complementarity of different data and can more accurately and effectively monitor the live room. This fusion method can fuse the features of multiple modal signals in the video.

[0115] In the above design, by fusing the image motion features, the interaction features and the audio features, compared with single modal technology based on a single live image, the application can make the live monitoring result more accurate and has higher performance.

[0116] In a possible implementation, after the image motion features and the interaction features are fused to generate target fusion features, the target fusion features can also be used to make fusion decisions through a Projector network structure to monitor whether the live video has a violation risk.

[0117] In this embodiment, the Projector network structure is composed of an Encoder network and a Projection head. The Projector network structure extracts features of input data through the Encoder network and projects the features to a low-dimensional representation space through the Projection head, so as to monitor whether the live video has a violation risk.

[0118] It should be noted that the Projector network structure for making fusion decisions is only a preferred embodiment of this embodiment, and in other embodiments of this embodiment, other different network structures can also be used to make fusion decisions.

[0119] Based on the same inventive concept, this embodiment also provides an electronic device 700, which can be a server 100 as shown in Figure 1 , please refer to Figure 9 ,Figure 9 An example block diagram of the electronic device 700 is shown. The electronic device 700 includes a live monitoring apparatus 730, a computer readable storage medium 720, and a processor 710.

[0120] The computer readable storage medium 720 and the processor 710 are directly or indirectly connected to each other electrically to realize the transmission or interaction of data. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The live monitoring apparatus 730 includes a plurality of software function modules which can be stored in the computer readable storage medium 720 in the form of software or firmware or solidified in the operating system (OS) of the live monitoring apparatus 730. The processor 710 is used to execute the executable modules stored in the computer readable storage medium 720, such as the software function modules included in the live monitoring apparatus 730 and computer programs, etc.

[0121] The computer readable storage medium 720 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The computer readable storage medium 720 is used to store programs, and the processor 710 executes the programs after receiving execution instructions.

[0122] The processor 710 can be an integrated circuit chip having a processing capability. The processor 710 described above can be a general processor 710, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. It can realize or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor 710 can be a microprocessor or the processor 710 can also be any conventional processor or the like.

[0123] Please refer to Figure 10 The embodiment of the present application further provides a live monitoring device 730. The live monitoring device 730 comprises a plurality of function modules which can be stored in the computer readable storage medium 720 in the form of software. Functionally, the live monitoring device 730 can comprise a receiving module 731, an image processing module 732, an interactive information processing module 733, a fusion module 734 and a monitoring module 735.

[0124] Wherein:

[0125] The receiving module 731 is configured to acquire a live video of a live room and interactive information corresponding to the live video.

[0126] In the embodiment, the receiving module 731 can be configured to perform the step S100 shown in the figure, and the specific description of the receiving module 731 can refer to the description of the step S100. Figure 2

[0127] The image processing module 732 is configured to acquire an image motion feature of the live video based on the live video.

[0128] In the embodiment, the image processing module 732 can be configured to perform the step S200 shown in the figure, and the specific description of the image processing module 732 can refer to the description of the step S200. Figure 2

[0129] The interactive information processing module 733 is configured to acquire an interactive feature related to the live video based on the interactive information, the interactive feature representing a change trend of the interactive frequency of the audience user in a time period corresponding to the live video.

[0130] In the embodiment, the interactive information processing module 733 can be configured to perform the step S300 shown in the figure, and the specific description of the interactive information processing module 733 can refer to the description of the step S300. Figure 2

[0131] The fusion module 734 is configured to fuse the image motion feature and the interactive feature to generate a target fusion feature.

[0132] In the embodiment, the fusion module 734 can be configured to perform the step S400 shown in the figure, and the specific description of the fusion module 734 can refer to the description of the step S400. Figure 2

[0133] The monitoring module 735 is configured to monitor whether there is a violation risk in the live video according to the target fusion feature.

[0134] ​​​​In the embodiment, the monitoring module 735 can be configured to perform Figure 2 The specific description of the monitoring module 735 can refer to the description of step S500.

[0135] In summary, the live monitoring method, device and electronic equipment provided by the embodiments of the present application can obtain the image motion feature of the live video and the interaction feature of the interaction information corresponding to the live video, and fuse the image motion feature and the interaction feature, so as to realize the joint decision of the image sequence mode and the interaction information mode, effectively overcome the decision drawbacks of single frame image, and thus improve the accuracy of live monitoring and reduce the risk of live video violation playing.

[0136] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0137] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims.

Claims

1. A live monitoring method, characterized in that, The method comprises: acquiring a live video of a live room and interaction information corresponding to the live video; based on the live video, acquiring image motion features of the live video with time sequence; based on the interaction information, acquiring interaction features related to the live video, the interaction features representing the change trend of the user's interaction frequency on the time sequence within the time period corresponding to the live video; fusing the image motion features and the interaction features to generate target fusion features; according to the correlation of the image motion features and the interaction features in the target fusion features, monitoring whether the live video has a violation risk.

2. The live monitoring method of claim 1, wherein, The step of acquiring the image motion features of the live video based on the live video comprises: performing optical flow analysis on a plurality of live video images in the live video to obtain a plurality of optical flow information maps corresponding to the live video images; acquiring the image motion features of the live video according to the optical flow information maps.

3. The live monitoring method of claim 2, wherein, The step of acquiring the image motion features of the live video according to the optical flow information maps comprises: extracting features from each optical flow information map to obtain optical flow feature information corresponding to each optical flow information map; extracting time sequence features from each optical flow feature information according to the time sequence to obtain optical flow time sequence features as the image motion features of the live video.

4. The live monitoring method of claim 3, wherein, The step of extracting time sequence features from each optical flow feature information according to the time sequence to obtain optical flow time sequence features as the image motion features of the live video comprises: based on the optical flow feature information, capturing the optical flow time sequence features of the live video through Bi-GRU; using a self-attention mechanism to obtain the weight of the optical flow time sequence features, assigning the optical flow time sequence features, and acquiring the image motion features of the live video.

5. The live monitoring method of claim 1, wherein, The interaction information includes text interaction information; The step of acquiring the interaction features related to the live video based on the interaction information comprises: acquiring the text interaction frequency in the live room according to the set time interval, the text interaction frequency including the amount of barrage information sent; generating an interaction frequency vector according to the text interaction frequency in a plurality of time intervals in sequence; extracting features from the interaction frequency vector to obtain the interaction features.

6. The live monitoring method of claim 1, wherein, The method further comprises: extracting features from the audio information in the live video to obtain audio features; The step of fusing the image motion features and the interaction features to generate target fusion features comprises: fusing the image motion features, the interaction features, and the audio features to generate target fusion features.

7. The live monitoring method of claim 1, wherein, After the step of fusing the image motion features and the interaction features to generate target fusion features, comprising: based on the target fusion features, making a fusion decision through a Projector network structure to monitor whether the live video has a violation risk.

8. A live monitoring device, characterized in that, comprising: a receiving module for acquiring a live video of a live room and interaction information corresponding to the live video; An image processing module is configured to acquire image motion features with time sequences in the live video based on the live video. An interactive information processing module is configured to acquire interactive features related to the live video based on the interactive information, the interactive features representing a change trend of the interactive frequency of the audience users on the time sequence within a time period corresponding to the live video. A fusion module is configured to fuse the image motion features and the interactive features to generate target fusion features. A monitoring module is configured to monitor whether there is a violation risk in the live video according to the correlation between the image motion features and the interactive features in the target fusion features.

9. An electronic device, comprising: A memory is configured to store one or more programs. A processor is configured to implement the method in any one of claims 1-7 when the one or more programs are executed by the processor. A computer program is stored on a computer readable medium, and when the computer program is executed by a processor, the method in any one of claims 1-7 is implemented.

10. A computer-readable storage medium, characterized in that, ​

Images