Live broadcast data processing method and device, terminal and storage medium

Abstract

The invention relates to a live broadcast data processing method and device, a terminal and a storage medium, and relates to the technical field of video processing, and the method comprises the steps: firstly obtaining a video address list of a target live broadcast room, storing a plurality of video addresses in the video address list, and downloading a segment video based on each video address, each video segment is a video segment played in the target live broadcast room; combining the downloaded clip videos into a live video corresponding to the target live room; and calculating the similarity between the live video and a pre-stored sample video, wherein the similarity is used for determining whether a target live room corresponding to the live video is abnormal or not. Through application of the method and the device, the problem of low troubleshooting stability of infringement behaviors of live videos in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a live data processing method, device, terminal and storage medium. Background Art

[0002] With the popularization of the Internet, the improvement of bandwidth, and the development of streaming media technology, live video has also been rapidly improved. The HLS (Http live streaming) protocol is widely used in the field of live transcoding, among which m3u8 is a widely used streaming media playlist format in the HLS protocol. m3u8 supports segmented transmission and adaptive bitrate playback of video streams, providing efficient and flexible support for real-time live broadcast.

[0003] In order to ensure the content quality, stability and security of the m3u8 video stream, it is necessary to monitor the live content in real time, such as monitoring whether there is copyright infringement in the live content. Specifically, the live content is currently uploaded to the cloud service, and the monitoring tools and services provided by the cloud service are used to achieve real-time monitoring and management of the live content. In order to ensure the stable transmission, efficient processing and flexible management of the live content, this technology must rely on the powerful computing, storage and network resources of the cloud service. Once the cloud service fails or is interrupted, it will affect the continuity and stability of the live monitoring, so it is limited in the application scenario of real-time monitoring of live content.

[0004] In summary, the related technologies have the problem of low stability in detecting infringements in live videos. Summary of the invention

[0005] The embodiments of the present application provide a live data processing method, device, terminal and storage medium to solve the technical problem of low stability in detecting infringement of live videos in the prior art.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions: In a first aspect, an embodiment of the present application provides a live broadcast data processing method, the method comprising: Obtain a video address list of a target live broadcast room, wherein the video address list stores multiple video addresses; Downloading a video clip based on each of the video addresses, wherein each of the video clips is a video clip played in the target live broadcast room; Merging the downloaded video clips into a live video corresponding to the target live broadcast room; The similarity between the live video and the pre-stored sample video is calculated, wherein the similarity is used to determine whether the target live room corresponding to the live video is abnormal.

[0007] In combination with the first aspect, in a possible design manner, calculating the similarity between the live video and the pre-stored sample video includes: Extracting target feature information of the live video and sample feature information of the sample video; The similarity between the target feature information and the sample feature information is calculated.

[0008] In combination with the first aspect, in a possible design manner, the extracting the target feature information of the live video and the sample feature information of the sample video includes: Separating an audio stream from the live video, extracting audio content of the audio stream, and converting the audio content into a dense vector format to obtain audio feature information; Acquire an image frame from the live video, extract pixel content of the image frame, and obtain pixel feature information of the image frame; The audio feature information and the pixel feature information at the same timestamp are fused to obtain target feature information of the live video; Extract sample feature information of the sample video.

[0009] In combination with the first aspect, in a possible design manner, the fusion processing of the audio feature information and the pixel feature information at the same timestamp to obtain the target feature information of the live video includes: The audio feature information and the pixel feature information are respectively subjected to hash encryption processing to generate encrypted audio feature information and encrypted pixel feature information; The encrypted audio feature information and the encrypted pixel feature information at the same timestamp are concatenated to obtain the target feature information of the live video.

[0010] In combination with the first aspect, in a possible design, the method further includes: after merging the downloaded video clips into the live video corresponding to the target live room, the method further includes: performing hash encryption processing on the live video to generate an encrypted live video; The step of splicing the encrypted audio feature information and the encrypted pixel feature information at the same timestamp to obtain the target feature information of the live video includes: The target feature information at each time stamp is merged in chronological order and then spliced ​​with the encrypted live video to obtain the target feature information of the live video.

[0011] In combination with the first aspect, in a possible design, merging the downloaded video clips into a live video corresponding to the target live broadcast room includes: According to the chronological order of each of the video clips during the playback process in the target live broadcast room, the downloaded video clips are spliced ​​to obtain the live broadcast video corresponding to the target live broadcast room.

[0012] In combination with the first aspect, in a possible design, obtaining a list of video addresses of a target live broadcast room includes: Access the m3u8 file regularly and obtain the video address list of the target live broadcast room by parsing the m3u8 file.

[0013] In a second aspect, an embodiment of the present application provides a live broadcast data processing device, including: An acquisition module, used to acquire a video address list of a target live broadcast room, wherein the video address list stores a plurality of video addresses; A download module, used for downloading a video clip based on each of the video addresses, wherein each of the video clips is a video clip played in the target live broadcast room; A merging module, used for merging each of the video clips into a live video corresponding to the target live broadcast room; A calculation module is used to calculate the similarity between the live video and a pre-stored sample video, wherein the similarity is used to determine whether the target live room corresponding to the live video is abnormal.

[0014] In a third aspect, an embodiment of the present application provides a terminal, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method of the first aspect and possible design methods thereof.

[0015] In a fourth aspect, an embodiment of the present application provides a storage medium, in which a computer program is stored, wherein the computer program is configured to execute the method of the first aspect and its possible design manner when running.

[0016] Compared with the prior art, a live data processing method, device, terminal and storage medium provided in an embodiment of the present application first obtains a video address list of a target live broadcast room, wherein a plurality of video addresses are stored in the video address list, and a video segment is downloaded based on each video address, wherein each video segment is a segment of video played in the target live broadcast room; each video segment is merged into a live video corresponding to the target live broadcast room; and the similarity between the live video and the pre-stored sample video is calculated, and the similarity is used to determine whether the target live broadcast room corresponding to the live video is abnormal. The present application ensures the stable transmission of the live video by downloading the segmented videos broadcast live in the target live broadcast room and splicing them into a complete live video. Therefore, when the live video is subsequently monitored for abnormalities, there will be no problem of inaccurate troubleshooting results due to incomplete videos, thereby improving the stability of troubleshooting for infringements in live videos.

[0017] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A hardware structure block diagram of a terminal provided in an embodiment of the present application is shown; Figure 2 A flowchart of a live broadcast data processing method provided by an embodiment of the present application is shown; Figure 3 A system architecture diagram of a live broadcast data processing method provided by an embodiment of the present application is shown; Figure 4 A schematic diagram showing a live broadcast data processing method provided by an embodiment of the present application is shown; Figure 5 A structural block diagram of a live broadcast data processing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0020] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0021] In the related art, manual monitoring is used to monitor the activity content of each live broadcast room in the live broadcast platform. However, in the face of a large number of live broadcast rooms, the simple manual monitoring method seems to be inadequate. It is not only labor-intensive, but also has a narrow monitoring coverage and the monitored activity content is relatively lagging, resulting in the inability to comprehensively and timely monitor the activity content of each live broadcast room in the live broadcast platform.

[0022] As the monitoring of live broadcast platforms is pursued in real time, manual monitoring methods are gradually eliminated. At present, it is technically possible to use monitoring tools and services provided by cloud services to achieve real-time monitoring of live broadcast content, timely detection and resolution of problems. However, this method is highly dependent on cloud services. Once the cloud service fails or is interrupted, it may affect the continuity and stability of live broadcast monitoring, so the stability is poor.

[0023] In addition, related technologies also use AI and machine learning algorithms to intelligently analyze and identify live broadcast content to improve the accuracy and efficiency of monitoring. For example, face recognition technology is used to identify specific people in live broadcasts, and natural language processing technology is used to analyze speech in live broadcasts. However, the accuracy and stability of the algorithm model are greatly affected by training data, and it needs to be constantly updated and optimized to adapt to new live broadcast content and scenarios, which results in high monitoring costs.

[0024] As can be seen from the above, the previous cloud service real-time monitoring live broadcast solution is highly dependent on the network. Once the network is unstable or even interrupted, it will directly affect the monitoring effect of the live broadcast channel. AI and machine learning algorithms not only require technicians to have rich knowledge reserves, but also require a large amount of training data, which is too costly.

[0025] In addition, whether it is the cloud service solution for real-time monitoring of live broadcasts, or the AI ​​and machine learning algorithm solution for monitoring live broadcasts, data security and privacy protection are not perfect, which can easily lead to risks such as monitoring leaks.

[0026] In view of this, the embodiment of the present application provides a live broadcast data processing method, which is applied to monitoring live broadcast data. Specifically, during the process of the target user broadcasting live in the target live broadcast room, the server will continuously monitor whether the live broadcast data involves copyright infringement, and the monitoring will not be affected by network fluctuations. The method provided by the embodiment of the present application first obtains a video address list of the target live broadcast room, and multiple video addresses are stored in the video address list. A video clip is downloaded based on each video address, wherein each video clip is a video played in the target live broadcast room; each video clip is merged into a live video corresponding to the target live broadcast room; the similarity between the live video and the pre-stored sample video is calculated; based on the similarity, it is determined whether the target live broadcast room corresponding to the similarity is abnormal. The present application ensures the stable transmission of the live video by downloading the segmented video broadcast by the target live broadcast room and splicing it into a complete live video. Then, when the live video is subsequently monitored for abnormalities, there will be no problem of inaccurate investigation results caused by incomplete videos, thereby improving the stability of investigating infringements of live videos.

[0027] The method can be executed in a terminal, a computer or a similar system. Taking running on a terminal as an example, Figure 1 FIG. 1 shows a hardware structure block diagram of a terminal provided in an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102, and memory 104 for storing data. Processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, to implement the above method. The memory 104 can be used to store data, for example, video clips, sample feature information, target feature information, etc. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0030] The present application provides a method for processing live broadcast data. Figure 2 A flowchart of a live data processing method provided by an embodiment of the present application is shown. Figure 2 As shown, the method includes steps S201 to S205.

[0031] Step S201: Obtain a video address list of a target live broadcast room, wherein the video address list stores multiple video addresses.

[0032] When the target user broadcasts live in the target live room, a video address list corresponding to the target live room will be generated. The video address list is a file in m3u8 format, and the video address stored in the video address list is used to indicate the location of the clip video.

[0033] Using a client (such as Python's requests library), you can access the m3u8 file at a scheduled time, obtain the video address list of the target live broadcast room by parsing the m3u8 file, and save the video address list locally.

[0034] Step S202: Download a video clip based on each video address, wherein each video clip is a video played in the target live broadcast room.

[0035] Each video address in the video address list corresponds to a video clip, so the video clip can be downloaded from the specified location according to the location indicated by the video address. Downloading the video clip can be understood as transferring the video clip from the location indicated by the video address to a local (such as a PC or server) for storage. In some embodiments, when the server downloads the video clips, it is not necessary to distinguish the time sequence, but only needs to download and store each video clip in the corresponding storage location, thereby improving the download efficiency.

[0036] During the live broadcast process, the live video is saved in segments, and each video is a clip video. The clip video is not directly stored in the file, but stored in the text form of the video address. This step downloads the clip video based on the video address instead of directly monitoring the live video. The advantage of this is that it can avoid the problem of incomplete live video caused by unstable server network, which leads to interference with the continuity and stability of monitoring.

[0037] For this step, asynchronous downloading can be used to avoid lag. Specifically, each video clip is first saved in the memory, and then downloaded and written to the disk at one time. Since there is no network overhead when reading and writing to the disk, the user can continue to use other functions of the application during the download process without being affected by the download operation.

[0038] In addition, multi-threaded downloading can also be used to improve downloading efficiency. Specifically, a thread pool is created to manage downloading tasks. The thread pool includes n threads, where n represents the maximum number of downloading tasks that can be executed. Taking n=3 as an example, the first thread obtains the first video segment based on the first video address. Synchronously, the second thread obtains the second video segment based on the second video address, and the third thread obtains the third video segment based on the third video address. Multi-threaded downloading reduces the total downloading time of the three video segments, thereby improving downloading efficiency.

[0039] Step S203: merge the downloaded video clips into a live video corresponding to the target live broadcast room.

[0040] In this step, merging can be understood as splicing the video clips together in a way that the end point of the previous video clip is merged with the start point of the next video clip. After merging, several shorter video clips are processed into a longer live video. The purpose of merging is to facilitate subsequent processing of only one video (live video) and improve processing efficiency.

[0041] In some of the embodiments, the above step S203 includes: splicing the video clips according to the time sequence of each video clip during the playback process in the target live broadcast room to obtain the live video corresponding to the target live broadcast room.

[0042] It should be noted that the live video can be a complete video of the target live broadcast room from the start of broadcast to the stop of broadcast, or it can be a video during the live broadcast process of the target live broadcast room. For example, the target live broadcast room starts broadcasting at time t0, the client accesses the m3u8 file regularly, accesses the m3u8 file for the first time at time t1 to download multiple segments of video clips, merges them to obtain the first live video, accesses the m3u8 file for the second time at time t2 to download multiple segments of video clips, merges them to obtain the second live video, and the target live broadcast room stops broadcasting at time t2. A complete video from the start of broadcasting to the stop of broadcasting in the target live broadcast room (i.e., within the duration from t0 to t2) includes two segments of the first live video and the second live video. Then, the abnormal monitoring of the target live broadcast room can be regarded as two parts, one is to compare the similarity of the first live video to determine whether it is abnormal when the live broadcast room plays the first live video, and the second is to compare the similarity of the second live video to determine whether it is abnormal when the live broadcast room plays the second live video, wherein the similarity comparison method for any live video can refer to the description of step S204 below.

[0043] In some of the embodiments, step S203 includes: splicing the video clips according to the time sequence of each video clip during playback in the target live broadcast room to obtain the live video corresponding to the target live broadcast room.

[0044] Specifically, when using multi-threaded downloading to download each video clip, the order in which the video clips are downloaded is not necessarily the order in which the video clips are played in the target live broadcast room. Therefore, this step will splice the video clips according to the time sequence during the playback process, so that the live video obtained is continuous in the time dimension, and the features obtained by subsequent feature extraction of the live video will be more accurate.

[0045] As some other embodiments, step S203 includes: splicing the video segments according to the downloading order of each video segment to obtain the live video corresponding to the target live broadcast room.

[0046] Specifically, when the method of sequentially downloading the video segments is adopted, the video segments played first are downloaded and downloaded first, and the video segments played later are downloaded and downloaded later. Therefore, the video segments can be directly spliced ​​based on the download order to obtain a merged live video.

[0047] Step S204: Calculate the similarity between the live video and the pre-stored sample video, wherein the similarity is used to determine whether the target live broadcast room corresponding to the live video is abnormal.

[0048] The pre-stored sample videos include clips of copyrighted works, such as highlights or key scenes in movies, TV series, variety shows, documentaries, cartoons, short videos, etc. The sample videos may also include specific types of performances or audio and video recordings, such as recordings of live events such as concerts, sports events, press conferences, etc. The sample videos may also include elements or symbols with distinctive features, such as specific trademarks, patterns, music melodies, character images, etc.

[0049] In this step, the similarity can be calculated in a variety of ways. For example, a set of video frame sets are extracted from the live video and the pre-stored sample video respectively, each set of video frame sets includes multiple video frames, and image recognition is performed on the multiple video frames in the two sets of video frame sets respectively. By comparing the image similarity between the two sets of video frame sets, the similarity between the live video and the pre-stored sample video is determined.

[0050] For another example, the audio features in the live video and the pre-stored sample video are extracted respectively, and the audio features include the speech text generated by converting the audio into text. The similarity between the speech text corresponding to the live video and the speech text corresponding to the sample video is compared, so as to determine the similarity between the live video and the pre-stored sample video.

[0051] In some embodiments, a similarity threshold can be combined to determine whether the target live broadcast room corresponding to the similarity is abnormal. After step S204, the method further includes: if the similarity is greater than or equal to the similarity threshold, determining that the target live broadcast room corresponding to the similarity is an abnormal live broadcast room; if the similarity is less than the similarity threshold, determining that the target live broadcast room corresponding to the similarity is a normal live broadcast room.

[0052] This step monitors infringement of live videos pulled in real time by similarity comparison. During the comparison process, once abnormalities or violations are found, such as pirated live broadcasts, content violations, etc., the target live broadcast room can be determined to be abnormal, and an alarm can be issued to the system, and corresponding countermeasures can be taken, such as blocking live broadcasts, recording alarm logs, etc. This real-time feedback mechanism helps protect copyrights, maintain live broadcast order, and improve user experience.

[0053] In the embodiment recorded in the above steps S201 to S204, the problem of low stability in troubleshooting live broadcast risks is solved by downloading each video clip and splicing it into a live video, and then monitoring the live video for infringement. The low stability mainly comes from the following factors. First, due to the direct use of cloud services for online monitoring, once the cloud service fails or is interrupted, the stability of the monitoring work will be directly affected. The subsequent processing of each video clip after downloading can be performed locally or on a backup server, which reduces the dependence on a single cloud service provider and the risk of failure. Second, the use of cloud services to monitor live streams in real time may be affected by factors such as network delays and packet loss, resulting in unstable monitoring results. After downloading and splicing the video clips into videos and then monitoring them, the problem of unstable monitoring results can be avoided, and the continuity and stability of monitoring can be improved. In addition to improving the stability of the monitoring process, this embodiment can also improve the efficiency and accuracy of infringement monitoring by preprocessing the video clips (such as denoising, enhancement, etc.), and these preprocessing operations are more convenient and flexible to perform locally. In summary, the method provided in the embodiments of the present application can reduce dependence on cloud services, improve the continuity and stability of monitoring, optimize the efficiency and accuracy of infringement monitoring, and reduce operating costs.

[0054] In some of the embodiments, step S204 includes: extracting target feature information of the live video and sample feature information of the sample video; and calculating the similarity between the target feature information and the sample feature information.

[0055] In this embodiment, the content recognition of the video is achieved by training the residual network model. Specifically, a CNN neural network, such as ResNet V2, can be used to perform content recognition on the video frames in the live video and the video frames in the sample video, and respectively generate representation vectors corresponding to the video frames, and use the representation vectors of each video frame of the live video as target feature information, and use the representation vectors of each video frame of the sample video as sample feature information.

[0056] In some of the embodiments, the above-mentioned extraction of target feature information of live video and sample feature information of sample video includes separating an audio stream from the live video, extracting audio content of the audio stream, and converting the audio content into a dense vector format to obtain audio feature information; obtaining image frames from the live video, extracting pixel content of the image frames, and obtaining pixel feature information of the image frames; fusing the audio feature information and pixel feature information at the same timestamp to obtain target feature information of the live video; and extracting sample feature information of the sample video.

[0057] In this embodiment, the live video is separated into WAV audio (or audio stream) and video frame data, and the WAV audio is converted into audio feature information in a dense vector format. After the video frames are unified into the image size, the pixel feature information is obtained by extracting the pixel content, and then the audio feature information and the pixel feature information are aligned on the time axis and fused into the multi-mode features of the real-time live video. For the sample video, the above method of extracting pixel features and voice features separately and fusing them can also be used to extract the multi-mode features of the sample video. The feature information obtained by fusing the features of the two modes of audio and pixels has richer information expression capabilities, so the similarity calculation based on the fused feature information is performed, and the calculation result obtained is more accurate.

[0058] As an example, the multi-modal features of the extracted sample video can be stored in a feature library to achieve the feature of once-in-library reuse.

[0059] In this embodiment, a frame extraction process may be used to extract some video frames from the live video. Specifically, the frame extraction process may include: extracting video frames from the live video according to a sampling number that is positively correlated with the duration of the live video; or extracting video frames from the live video according to a preset sampling number.

[0060] Frame extraction reduces the total data volume of the video by reducing the number of frames in the live video, thereby saving storage space. This is particularly important for scenarios that require storage of a large amount of video data, such as surveillance video storage, online video platforms, and the like. In addition, during video transmission, reducing the number of frames can reduce the amount of data required for transmission, thereby speeding up transmission and reducing transmission time. This is particularly beneficial for real-time video transmission or bandwidth-constrained environments such as mobile networks. The number of samples can be positively correlated with the length of the live video, meaning that the longer the live video, the more samples are taken. This is to ensure that the number of samples extracted is reasonable, and to avoid losing key information in the live video due to too few samples, resulting in a low subsequent recognition rate. Therefore, through this embodiment, while ensuring the content integrity of the live video, the resources consumed by calculation, transmission, and storage are reduced by extracting frames.

[0061] In this embodiment, the fusion method can be to splice the audio feature information and pixel feature information at the same timestamp. Alternatively, the fusion method can also be to extract the correlation coefficient between the audio feature information and the pixel feature information, determine the fusion weight based on the correlation coefficient, and then fuse the audio feature information and the pixel feature information according to the fusion weight to obtain multimodal enhanced features, and then splice the multimodal enhanced features with the audio feature information and the pixel feature information to obtain the target feature information. This fusion method extracts the correlation coefficient between the multimodal features, making the fusion method more reasonable, and the obtained enhanced features have richer information expression capabilities.

[0062] In some embodiments, in order to improve the security of information transmission and storage, the feature information is encrypted. Specifically, the above-mentioned fusion processing of the audio feature information and the pixel feature information at the same timestamp to obtain the target feature information of the live video includes: respectively performing hash encryption processing on the audio feature information and the pixel feature information to generate encrypted audio feature information and encrypted pixel feature information; splicing the encrypted audio feature information and the encrypted pixel feature information at the same timestamp to obtain the target feature information of the live video.

[0063] The method of hashing and encrypting the feature information not only hides the original content of the data and protects privacy, but also directly compares the hash value of the target feature information and the hash value of the sample feature information when comparing a large number of data values, which can significantly reduce the amount of calculation. Therefore, it saves more computing time than comparing the original feature information.

[0064] In some of the embodiments, the method further includes: performing hash encryption processing on the live video to generate an encrypted live video; the encrypted audio feature information and the encrypted pixel feature information under the same timestamp are spliced ​​to obtain the target feature information of the live video, including: fusing the target feature information under each timestamp in chronological order, and splicing it with the encrypted live video to obtain the target feature information of the live video.

[0065] This embodiment improves data security by performing hash encryption on live video, which is less likely to cause monitoring data leakage, and thus has stronger privacy and confidentiality. The encrypted live video is spliced ​​with the target feature information to retain the information provided by the original data, thereby enhancing the robustness and generalization ability of the model.

[0066] The live broadcast data processing method provided in the embodiment of the present application is described and illustrated below through specific examples. Figure 3 The system architecture diagram of a live data processing method provided by an embodiment of the present application is shown. In the system, video and audio clips of multiple live channels are collected, and the live channel system performs transcoding and slicing, which are transmitted to the live monitoring and recognition system through the FTP file protocol. The live monitoring and recognition system performs slicing synthesis and gene extraction on each video and audio clip, and the video sample is also uploaded to the live monitoring and recognition system for gene extraction. The recognition result is output by the recognition engine and the data processing engine, and the recognition result can be viewed by the result query interface. In addition, each extracted feature is stored in the comparison video gene library, and the video is stored in the live video sample library for subsequent use.

[0067] The live broadcast monitoring and identification system is divided into four layers according to the software level, namely data collection layer, data processing layer, feature comparison layer and result output layer.

[0068] The data collection layer is responsible for live stream capture and segment extraction. Live stream capture refers to obtaining the M3U8 playlist file, which contains the URL address of each segment (such as TS file) in the live stream of each live channel. Segment extraction refers to pulling the video and audio segments in the live stream according to the URL address in the M3U8 file. These segments may be in TS format or other supported formats.

[0069] The data processing layer is responsible for segment merging and feature generation. Segment merging refers to merging multiple audio and video segments to form a longer audio and video stream segment for subsequent processing. Feature generation refers to extracting features from the merged audio and video stream segments to generate unique fingerprint features. These features can be based on keyframe hash values, audio spectrum features, etc. For an introduction to fingerprint features, please refer to the description of target feature information above, which will not be repeated here.

[0070] The feature comparison layer has a built-in feature library to store sample feature information of known audio and video stream segments, which is used to compare with the fingerprint features generated in real time. The feature comparison layer is responsible for feature comparison, that is, comparing the fingerprint features generated in real time with the fingerprints in the feature library to identify copyright-infringing content in the live stream.

[0071] The result output layer is responsible for result analysis, report generation, and alarm notification. Result analysis refers to analyzing whether the content in the live stream meets expectations based on the results of feature comparison, such as whether there is infringement or illegal content. Report generation is used to output the analysis results in the form of a report, including key information such as monitoring time, monitoring content, and comparison results. Alarm notification refers to sending alarm notifications to relevant personnel in a timely manner when abnormal situations are found, so that timely measures can be taken, such as blocking live broadcasts.

[0072] Figure 4 A schematic diagram of a live broadcast data processing method provided by an embodiment of the present application is shown, Figure 4As shown, the target user broadcasts live in the target live broadcast room, the client monitors the live broadcast m3u8 address, dynamically obtains the ts data file list segmented by time axis (equivalent to the video address list above), and further obtains each ts stream data (equivalent to the video clip above) from each ts data address in the ts data file list, and saves each ts stream data as a disk live segment video file (equivalent to the live video above) according to the monitoring pull, frequency and other parameters and merging algorithm. Separate the WAV audio and video frame data of the above video file, and convert the audio into a dense vector. After the video frame image is unified in size, it is converted into feature data through the CNN neural network recognition operator, and the extracted video vector and audio vector are hashed and encrypted, and the time axis is aligned and fused into multi-mode features (equivalent to the target feature information above). Then the multi-mode feature is merged with the ts stream data to obtain the feature, and the CNN neural network, similarity threshold, topK and other parameters and algorithms are used to predict whether the feature exists in the feature library, and the parent identification comparison result is obtained. Finally, the feature comparison result is obtained by grouping, sorting, fault tolerance, compensation and other mechanisms of the comparison result. Based on the feature comparison results, it is determined whether the target live broadcast room is abnormal. If so, an alarm can be issued to the system and corresponding countermeasures can be taken, such as blocking the live broadcast, recording the alarm log, etc.

[0073] In summary, the live data processing method provided by the embodiment of the present application, by extracting features from the m3u8 live stream in slices and warehousing, can achieve the characteristics of one-time warehousing and reuse, not only has the characteristics of high accuracy and fast speed, but also overcomes the data loss and discontinuity in the live broadcast process, improves the rapid positioning and retrieval capabilities of the live content, strengthens copyright protection and anti-piracy measures, and provides system scalability and flexibility, etc., for real-time live broadcast monitoring and management provides an effective solution. By comparing the generated features with the feature library, to detect whether there is a copyright problem or unauthorized live stream, a purpose of copyright monitoring is achieved. In addition, encryption technology is used to protect the security of audio and video clips and features, prevent data leakage, and improve the security of data.

[0074] The embodiments of the present application also provide a live data processing device, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0075] Figure 5 A structural block diagram of a live data processing device provided in an embodiment of the present application is shown. Figure 5 As shown, the device comprises: An acquisition module 51 is used to acquire a video address list of a target live broadcast room, wherein the video address list stores multiple video addresses; A download module 52, used to download a segment video based on each video address, wherein each segment video is a segment of video played in a target live broadcast room; A merging module 53 is used to merge the downloaded video clips into a live video corresponding to the target live broadcast room; The calculation module 54 is used to calculate the similarity between the live video and the pre-stored sample video, wherein the similarity is used to determine whether the target live room corresponding to the live video is abnormal.

[0076] In some embodiments, the device also includes a decision module 55, which is used to determine that the target live broadcast room corresponding to the similarity is an abnormal live broadcast room if the similarity is greater than or equal to the similarity threshold; if the similarity is less than the similarity threshold, then determine that the target live broadcast room corresponding to the similarity is a normal live broadcast room.

[0077] In some of the embodiments, the acquisition module 51 is also used to access the m3u8 file at a regular interval and obtain a video address list of the target live broadcast room by parsing the m3u8 file.

[0078] In some of the embodiments, the merging module 53 is also used to splice the video segments according to the time sequence of each video segment during the playback process in the target live broadcast room to obtain the live video corresponding to the target live broadcast room.

[0079] In some of the embodiments, the calculation module 54 is further used to extract target feature information of the live video and sample feature information of the sample video; and calculate the similarity between the target feature information and the sample feature information.

[0080] In some of the embodiments, the computing module 54 is also used to separate the audio stream from the live video, extract the audio content of the audio stream, and convert the audio content into a dense vector format to obtain audio feature information; obtain image frames from the live video, extract the pixel content of the image frames, and obtain pixel feature information of the image frames; fuse the audio feature information and pixel feature information at the same timestamp to obtain target feature information of the live video; and extract sample feature information of the sample video.

[0081] In some of the embodiments, the computing module 54 is also used to perform hash encryption processing on the audio feature information and the pixel feature information respectively to generate encrypted audio feature information and encrypted pixel feature information; and to splice the encrypted audio feature information and the encrypted pixel feature information at the same timestamp to obtain the target feature information of the live video.

[0082] In some of the embodiments, the calculation module 54 is further used to merge the target feature information under each time stamp in chronological order, and then splice it with the encrypted live video to obtain the target feature information of the live video.

[0083] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0084] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0085] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0086] Optionally, in this embodiment, the processor may be configured to execute the following steps S1 to S4 through a computer program.

[0087] S1. Obtain a video address list of a target live broadcast room, wherein the video address list stores multiple video addresses.

[0088] S2. Downloading a segment video based on each video address, wherein each segment video is a segment of video played in the target live broadcast room.

[0089] S3. Merge the downloaded video clips into a live video corresponding to the target live broadcast room.

[0090] S4. Calculate the similarity between the live video and the pre-stored sample video, wherein the similarity is used to determine whether the target live broadcast room corresponding to the live video is abnormal.

[0091] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0092] In addition, in combination with the method provided in the above embodiment, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the live data processing methods in the above embodiment is implemented.

[0093] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes each function or step executed by the processor in the above method embodiment.

[0094] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0095] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0096] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0097] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A live broadcast data processing method, characterized in that: The method comprises: Obtain a video address list of a target live broadcast room, wherein the video address list stores multiple video addresses; Downloading a video clip based on each of the video addresses, wherein each of the video clips is a video clip played in the target live broadcast room; Merging the downloaded video clips into a live video corresponding to the target live broadcast room; The similarity between the live video and the pre-stored sample video is calculated, wherein the similarity is used to determine whether the target live room corresponding to the live video is abnormal.

2. The live broadcast data processing method according to claim 1, characterized in that: The calculating the similarity between the live video and the pre-stored sample video includes: Extracting target feature information of the live video and sample feature information of the sample video; The similarity between the target feature information and the sample feature information is calculated.

3. The live broadcast data processing method according to claim 2, characterized in that: The extracting target feature information of the live video and sample feature information of the sample video includes: Separating an audio stream from the live video, extracting audio content of the audio stream, and converting the audio content into a dense vector format to obtain audio feature information; Acquire an image frame from the live video, extract pixel content of the image frame, and obtain pixel feature information of the image frame; The audio feature information and the pixel feature information at the same timestamp are fused to obtain target feature information of the live video; Extract sample feature information of the sample video.

4. The live broadcast data processing method according to claim 3, characterized in that: The fusing the audio feature information and the pixel feature information at the same timestamp to obtain the target feature information of the live video includes: The audio feature information and the pixel feature information are respectively subjected to hash encryption processing to generate encrypted audio feature information and encrypted pixel feature information; The encrypted audio feature information and the encrypted pixel feature information at the same timestamp are concatenated to obtain the target feature information of the live video.

5. The live broadcast data processing method according to claim 4, characterized in that: After merging the downloaded video clips into the live video corresponding to the target live room, the method further includes: performing hash encryption processing on the live video to generate an encrypted live video; The step of splicing the encrypted audio feature information and the encrypted pixel feature information at the same timestamp to obtain the target feature information of the live video includes: The target feature information at each time stamp is merged in chronological order and then spliced ​​with the encrypted live video to obtain the target feature information of the live video.

6. The live broadcast data processing method according to any one of claims 1 to 4, characterized in that: The step of merging the downloaded video clips into a live video corresponding to the target live broadcast room includes: According to the chronological order of each of the video clips during the playback process in the target live broadcast room, the downloaded video clips are spliced ​​to obtain the live broadcast video corresponding to the target live broadcast room.

7. The live broadcast data processing method according to any one of claims 1 to 4, characterized in that: The step of obtaining a list of video addresses of a target live broadcast room includes: Access the m3u8 file regularly and obtain the video address list of the target live broadcast room by parsing the m3u8 file.

8. The live broadcast data processing method according to any one of claims 1 to 4, characterized in that: After calculating the similarity between the live video and the pre-stored sample video, the method further includes: If the similarity is greater than or equal to the similarity threshold, determining that the target live broadcast room corresponding to the similarity is an abnormal live broadcast room; If the similarity is less than the similarity threshold, it is determined that the target live broadcast room corresponding to the similarity is a normal live broadcast room.

9. A live broadcast data processing device, characterized in that: include: An acquisition module, used to acquire a video address list of a target live broadcast room, wherein the video address list stores a plurality of video addresses; A download module, used for downloading a video clip based on each of the video addresses, wherein each of the video clips is a video clip played in the target live broadcast room; A merging module, used to merge the downloaded video clips into a live video corresponding to the target live broadcast room; A calculation module is used to calculate the similarity between the live video and a pre-stored sample video, wherein the similarity is used to determine whether the target live room corresponding to the live video is abnormal.

10. A terminal comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the live broadcast data processing method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the live broadcast data processing method according to any one of claims 1 to 8 when running.

Images