Live broadcast data processing method, device, equipment and computer-readable storage medium

Abstract

The application provides a live data processing method and device, equipment and a computer readable storage medium. The method comprises: acquiring a plurality of live data streams uploaded by different terminals, determining at least two target video frames corresponding to each live data stream; merging and processing the at least two target video frames corresponding to each live data stream to obtain a merged image corresponding to each live data stream; determining local feature information of the merged image corresponding to each live data stream, determining the similarity between the live data streams based on the local feature information corresponding to each live data stream; and determining at least two live data streams satisfying a similarity condition based on the similarity between the live data streams. Through the application, the identification efficiency of similar live data streams can be improved while ensuring accuracy.

Description

Technical Field

[0001] This application relates to Internet technology, and more particularly to a live data processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Live streaming is a popular way for many users to share their lives and knowledge. During business operations, it was discovered that multiple accounts were rebroadcasting the same content (referred to as "homogeneous live streams"), which disrupted the WeChat Video Accounts live streaming ecosystem and severely impacted the user experience. To create a healthy live streaming environment for users, it is necessary to detect and filter homogeneous live streams. Existing live stream data processing methods for detecting and filtering homogeneous live streams are either computationally complex and unsuitable for scenarios with time misalignment, or the design and deployment of the neural network models required for live stream data processing are time- and economically costly, failing to meet real-time detection requirements. Summary of the Invention

[0003] This application provides a live streaming data processing method, apparatus, and computer-readable storage medium, which can improve the identification efficiency of similar live streaming data streams while ensuring accuracy.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a live streaming data processing method, including:

[0006] Acquire multiple live data streams uploaded from different terminals and determine at least two target video frames corresponding to each live data stream;

[0007] At least two target video frames corresponding to each of the live data streams are merged to obtain a merged image corresponding to each of the live data streams.

[0008] Determine the local feature information of the merged image corresponding to each of the live data streams, and determine the similarity between the live data streams based on the local feature information corresponding to each of the live data streams;

[0009] Based on the similarity between the various live data streams, at least two live data streams that meet the similarity criteria are identified.

[0010] This application provides a live data processing device, including:

[0011] The first determining module is used to acquire multiple live data streams uploaded by different terminals and determine at least two target video frames corresponding to each live data stream.

[0012] An image merging module is used to merge at least two target video frames corresponding to each live data stream to obtain a merged image corresponding to each live data stream.

[0013] The second determining module is used to determine the local feature information of the merged image corresponding to each of the live data streams, and to determine the similarity between the live data streams based on the local feature information corresponding to each of the live data streams.

[0014] The third determining module is used to determine at least two live streaming data streams that meet the similarity conditions based on the similarity between the various live streaming data streams.

[0015] In some embodiments, the first determining module is further configured to:

[0016] Each live data stream is decoded to obtain multiple live video frames corresponding to each live data stream;

[0017] Frame extraction is performed from multiple live video frames corresponding to each live data stream according to a preset interval, to obtain multiple extracted live video frames corresponding to each live data stream.

[0018] A preset number of live video frames are randomly selected from the multiple extracted live video frames corresponding to each live data stream to obtain at least two target video frames corresponding to each live data stream.

[0019] In some embodiments, the image merging module is further configured to:

[0020] Obtain a preset image target size, and adjust the size of at least two target video frames corresponding to each live data stream based on the image target size to obtain at least two adjusted target video frames corresponding to each live data stream.

[0021] According to the time sequence of at least two adjusted target video frames corresponding to each live data stream, the at least two adjusted target video frames corresponding to each live data stream are merged to obtain the merged image corresponding to each live data stream.

[0022] In some embodiments, the second determining module is further configured to:

[0023] The features of each merged image are extracted according to a preset local feature extraction algorithm to obtain the feature vector of each merged image;

[0024] The feature threshold of each merged image is determined based on the feature vector of each merged image.

[0025] Based on the feature thresholds of each merged image, the corresponding feature vectors are binarized to obtain the local feature information of each merged image.

[0026] In some embodiments, the second determining module is further configured to:

[0027] Based on the local feature information corresponding to each live data stream, determine the number of identical bits in the local feature information of different live data streams.

[0028] The similarity between the different live data streams is determined based on the total number of bits of local feature information and the number of identical bits.

[0029] In some embodiments, the third determining module is further configured to:

[0030] Obtain the preset similarity threshold;

[0031] At least two live streams with a similarity greater than or equal to the similarity threshold are identified as at least two live streams that satisfy the similarity condition;

[0032] The device also includes:

[0033] The fourth determining module is used to determine the target live stream that needs to be deduplicated from the at least two live streams that meet the similarity conditions;

[0034] The deduplication module is used to perform deduplication processing on the target live data stream to obtain the processed live data stream.

[0035] The sending module is used to determine the live data to be sent based on the processed live data stream, and send the live data to be sent to the viewer's terminal.

[0036] In some embodiments, the fourth determining module is further configured to:

[0037] Determine the number of viewers in the live streaming rooms corresponding to at least two live streaming data streams that meet the similarity criteria;

[0038] The live stream with the most viewers among the at least two live streams is identified as the target live stream.

[0039] In some embodiments, the fourth determining module is further configured to:

[0040] Obtain the number of reports for the corresponding live streaming rooms of at least two live streaming data streams that meet the similarity criteria;

[0041] When the difference between the number of reports in the corresponding live rooms of the at least two live data streams is greater than the difference threshold, the other live data stream among the at least two live data streams, excluding the one with the fewest reports, is determined as the target live data stream.

[0042] In some embodiments, the fourth determining module is further configured to:

[0043] When the difference between the number of reports in the live room corresponding to the at least two live data streams is less than or equal to the difference threshold, in response to the live request sent by the viewer terminal, the historical behavior data corresponding to the viewer terminal is obtained.

[0044] Based on the historical behavior data, determine the viewing preference information corresponding to the viewer's terminal;

[0045] Based on the viewing preference information, the other live streaming data stream that has the highest matching degree with the viewing preference information among the at least two live streaming data streams that meet the similarity condition is determined as the target live streaming data stream.

[0046] In some embodiments, the deduplication module is further configured to:

[0047] The target live stream is deleted from the plurality of live streams to obtain a processed live stream; or;

[0048] The target live stream data stream is subjected to traffic weight reduction processing to obtain the processed live stream data stream.

[0049] This application provides a computer device, including:

[0050] Memory, used to store executable instructions;

[0051] The processor, when executing executable instructions stored in the memory, implements the live data processing method provided in the embodiments of this application.

[0052] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the live data processing method provided in this application.

[0053] This application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the live data processing method provided in this application.

[0054] The embodiments of this application have the following beneficial effects:

[0055] After acquiring multiple live streams uploaded from different terminals, the server determines at least two target video frames corresponding to each live stream. Then, it merges these two target video frames to obtain a merged image for each live stream. The server then determines the local feature information of the merged image and, based on this local feature information, determines the similarity between the live streams. Finally, based on the similarity between the live streams, it identifies at least two live streams that meet the similarity criteria. Because the merging process is performed after obtaining at least two target video frames, and the local feature information of the merged image is extracted, the same local feature information can be obtained even if the order of the target video frames in the merged image is different. This ensures accurate similarity calculation for videos with temporal misalignment. Furthermore, the similarity calculation only considers the local feature information of different merged images, without considering semantic feature similarity, thus reducing computational complexity and improving the recognition efficiency of similar live streams. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the network architecture of the live streaming system 100 provided in an embodiment of this application;

[0057] Figure 2 This is a schematic diagram of the structure of the server 400 provided in an embodiment of this application;

[0058] Figure 3 A schematic diagram illustrating an implementation flow of the live data processing method provided in this application embodiment;

[0059] Figure 4 A schematic diagram illustrating the implementation process for determining the similarity of various live streaming data streams, provided in an embodiment of this application;

[0060] Figure 5 A schematic diagram illustrating another implementation flow of the live data processing method provided in this application embodiment;

[0061] Figure 6 This is a schematic diagram of a live streaming interface provided in an embodiment of this application;

[0062] Figure 7A This is the nine-grid image corresponding to live stream room A1;

[0063] Figure 7B This is the nine-grid image corresponding to A2 in the live stream;

[0064] Figure 8A This is the large nine-grid image corresponding to B1 in the live stream room;

[0065] Figure 8BThis is the large nine-grid image corresponding to B2 in the live stream room. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0068] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0070] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0071] 1) Hash calculation is the process of transforming an input of arbitrary length into an output of fixed length using a hash algorithm.

[0072] 2) Homogeneous live streaming refers to live streaming content with the same or similar audio and video.

[0073] 3) Unsupervised learning, learning from unlabeled, unclassified, or unclassified test data.

[0074] 4) Multimodal model: A model is built from multiple modalities, which can process and associate information from multiple modalities.

[0075] 5) Time domain plot: x-axis represents time, y-axis represents amplitude; Frequency domain plot: x-axis represents frequency, y-axis represents amplitude.

[0076] 6) Perceptual Hash (pHash) algorithm: The basic principle is to shrink the image to a computable range, then filter the main features of the image through the Discrete Cosine Transform (DCT) algorithm to obtain data that can reflect the image features to a certain extent, and finally output the image's hash value.

[0077] To better understand the live streaming data processing method for detecting homogeneous live streams provided in the embodiments of this application, the live streaming data processing methods for detecting homogeneous live streams in related technologies and their shortcomings are explained.

[0078] In related technologies, the main methods for processing live stream data to detect homogeneous live streams include the following two:

[0079] Option 1 involves extracting video pairs from a massive amount of video data, sampling multiple short video segments, and performing frame-by-frame similarity analysis. The analysis criteria include image color space, audio spectrum, and amplitude. In live streaming scenarios, this involves storing the video stream as fixed-length videos at fixed intervals and then comparing them frame-by-frame.

[0080] Option 2: Since most homogeneous content undergoes simple processing, such as adding watermarks, intros, and outros, the video content is generally misaligned. To reduce computational costs and minimize the impact of content misalignment on accuracy, another existing approach is to obtain the semantic fingerprint vector of the video segment using a multimodal semantic retrieval model. This fingerprint vector is typically a floating-point vector of a certain length. Then, vector retrieval engines such as Faiss are used to identify homogeneous video groups and determine that the video content within these groups is homogeneous.

[0081] Option 2 can be implemented through the following process:

[0082] Step S001: After the user starts broadcasting, the live data stream is sent to the server.

[0083] Step S002: The server obtains the representation vector of the live content through the multimodal model.

[0084] In step S003, the server uses this representation vector to search and compare with the vectors of all other live streams.

[0085] Step S004: If the live stream vector is too similar to other live stream vectors, then the live stream content is determined to be homogeneous.

[0086] The following explains the shortcomings of Scheme 1 and Scheme 2.

[0087] Option 1, while offering the highest recognition accuracy, is computationally complex and requires perfect alignment of video content with no temporal offset. Otherwise, even a deviation of more than 1 second could cause frames to misalign, resulting in videos that appear identical to the naked eye being judged as different.

[0088] Scheme 2 essentially utilizes a similar video retrieval technique to solve the problem of finding homogeneous content. The main drawbacks of this scheme are: (1) The accuracy is slightly worse, and the semantic retrieval model needs to be updated regularly to adapt to changes in content distribution. The training time for multimodal models is long, and the cost of obtaining the model is too high. The time and economic costs spent from training data preparation, model design to deployment on the server are too high, and this cycle often takes more than half a month. (2) At the same time, the multimodal semantic retrieval model and the Faiss vector retrieval engine consume a lot of resources and have a high latency. The cost of calculating the cosine distance or L1 and L2 distance between semantic fingerprint vectors is higher than the cost of calculating the Hamming distance using binary hash.

[0089] Based on this, this application provides a live streaming data processing method for identifying and detecting homogeneous live streams. In implementation, a sampling method is used instead of a frame-by-frame calculation method to extract local features, obtain binary fingerprint information, and determine the similarity of the live streaming data streams through the binary fingerprint information, thereby realizing the identification of homogeneous live streams.

[0090] The following describes exemplary applications of the computer device provided in the embodiments of this application. The computer device provided in the embodiments of this application can be implemented as a server. Exemplary applications when the device is implemented as a server will be described below.

[0091] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the live streaming system 100 provided in the embodiments of this application, as shown below. Figure 1 As shown, the live streaming system 100 includes a broadcaster terminal 200 (broadcaster terminal 200-1 and broadcaster terminal 200-2 are shown as examples), a network 300, a server 400, and a viewer terminal 500. Broadcaster terminal 200-1, broadcaster terminal 200-2, and viewer terminal 500 are connected to the server 400 through the network 300. The network 300 can be a wide area network, a local area network, or a combination of both.

[0092] The viewer terminal 500 can be equipped with an application (App) capable of watching or listening to live streams. This App can be a dedicated live streaming App or any App with live streaming functionality, such as a short video App. Users can use this App to access the live stream interface. When the viewer terminal 500 receives a touch operation targeting a specific live stream entry point, users can enter that live stream to watch or listen to the live stream content.

[0093] Live streaming apps can also be installed on broadcast terminals 200-1 and 200-2. After starting the live stream, broadcast terminals 201 and 200-2 can send the live streaming data stream to server 400. Server 400 will receive a massive amount of live streaming data stream. In this embodiment, server 400 will perform similarity detection on the live streaming data stream to be pushed to viewer terminal 500, thereby filtering out homogeneous live streaming data streams and ensuring that the live streaming entrance presented by viewer terminal 500 is non-repetitive.

[0094] In this embodiment, server 400 can be a single server or a server cluster or cloud computing center composed of multiple servers. Depending on how the live streaming service is implemented in the viewer terminal 500, server 400 can be deployed in various different ways.

[0095] For example, when the live streaming service is implemented in the viewer terminal 500 as a dedicated live streaming APP, the server 400 can be one or more dedicated servers that provide live streaming video, which communicate directly with the viewer terminal 500 through the network 300 to complete the necessary data and information transmission.

[0096] For example, when the live streaming service is implemented in the viewer terminal 500 as a module or plugin (e.g., a mini-program) coupled to various existing apps (e.g., social apps, shopping apps), the server 400 may include a business server for implementing the basic business functions of these existing apps, and a live streaming server for providing live video. The live streaming server communicates directly with the module or plugin, or indirectly with the module or plugin through the business server. Of course, it is understandable that the main difference between the live streaming server and the business server lies in the business logic it carries. Therefore, the live streaming server and the business server can actually be the same server.

[0097] In the following description, for ease of description, all the servers in the above possible forms will be referred to as servers. Therefore, server 400 should not be simply understood as one or a type of server, but rather as the various possible forms of servers deployed in actual applications to support live streaming services, based on the examples above.

[0098] In some embodiments, server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The broadcast terminal 200 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart in-vehicle device, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0099] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 400 provided in an embodiment of this application. Figure 2 The server 400 shown includes at least one processor 410, at least one network interface 420, a bus system 430, and memory 440. The various components in the server 400 are coupled together via the bus system 430. It is understood that the bus system 430 is used to implement communication between these components. In addition to a data bus, the bus system 430 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 430.

[0100] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0101] The memory 440 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 440 may optionally include one or more storage devices physically located away from the processor 410.

[0102] The memory 440 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 440 described in this application embodiment is intended to include any suitable type of memory.

[0103] In some embodiments, memory 440 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0104] Operating system 441 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0105] The network communication module 442 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0106] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A live data processing device 443 stored in memory 440 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a first determining module 4431, an image merging module 4432, a second determining module 4433, and a third determining module 4434. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0107] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the live data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0108] The live streaming data processing method provided in this application will be described in conjunction with exemplary applications and implementations of the terminal provided in the embodiments of this application. This live streaming data processing method can be implemented by a server. See also Figure 3 , Figure 3This is a schematic diagram illustrating an implementation flow of the live data processing method provided in this application embodiment, which will be combined with... Figure 3 The steps for processing live data provided in the embodiments of this application will be described.

[0109] Step S101: Obtain multiple live data streams uploaded from different terminals and determine at least two target video frames corresponding to each live data stream.

[0110] The different terminals here correspond to the broadcaster terminals in other embodiments. In this step, the server decodes each acquired live data stream to obtain multiple live video frames corresponding to each stream. Then, it performs frame extraction at preset intervals, for example, every 5 seconds. From the multiple live video frames extracted within the preset interval, a preset number of frames are randomly selected as target video frames. For example, 24 live video frames are extracted every two minutes, and then 9 are selected from those 24 to determine the target video frames; alternatively, 12 live video frames are extracted every one minute, and then 4 are selected from those 2 to determine the target video frames.

[0111] Step S102: Merge at least two target video frames corresponding to each live data stream to obtain a merged image corresponding to each live data stream.

[0112] This step can be implemented by merging at least two target video frames in chronological order, or by merging them out of chronological order, to obtain a merged image corresponding to each live stream. In other words, each live stream corresponds to a merged image for its respective similarity recognition duration.

[0113] Step S103: Determine the local feature information of the merged image corresponding to each live data stream, and determine the similarity between the live data streams based on the local feature information corresponding to each live data stream.

[0114] In this embodiment, a preset local feature extraction algorithm can be used to extract features from the merged image corresponding to each live data stream to obtain local feature information. In practical applications, this local feature information is represented in binary form, and in some embodiments, it can also be referred to as binary fingerprint information.

[0115] Since local feature information is represented in binary, determining the similarity between different live streams can be achieved by performing a bitwise AND operation on two local feature information pieces to determine the number of identical bits between them. Then, this identical number of bits is divided by the total number of bits in the local feature information, yielding the similarity between the two pieces of local feature information. This similarity is a real number between 0 and 1. In this embodiment, the similarity is calculated between pairs of live streams.

[0116] Step S104: Based on the similarity between the various live data streams, determine at least two live data streams that meet the similarity condition.

[0117] In implementing this step, a preset similarity threshold can be obtained first, and at least two live data streams with a similarity greater than or equal to the similarity threshold can be identified as at least two live data streams that meet the similarity condition.

[0118] Since step S103 determines the similarity between pairs of live data streams, when this step is implemented, if there are live data streams A, B, and C, assuming that the similarity between live data stream A and live data stream B is greater than or equal to the similarity threshold, and the similarity between live data stream B and live data stream C is greater than or equal to the similarity threshold, it can only be concluded that live data streams A and B are two live data streams that meet the similarity condition, and live data streams B and C are two live data streams that meet the similarity condition. It is not necessarily concluded that live data streams A and C are two live data streams that meet the similarity condition. Only when the similarity between live data streams A and C is also greater than or equal to the similarity threshold can it be concluded that live data streams A, B, and C are three live data streams that meet the similarity condition.

[0119] In the live data processing method provided in this application embodiment, after the server acquires multiple live data streams uploaded by different terminals, it determines at least two target video frames corresponding to each live data stream. Then, it merges the at least two target video frames corresponding to each live data stream to obtain a merged image corresponding to each live data stream, and determines the local feature information of the merged image corresponding to each live data stream. Based on the local feature information, it determines the similarity between the live data streams. Finally, based on the similarity between the live data streams, it determines at least two live data streams that meet the similarity condition. Thus, since the merging process is performed after obtaining at least two target video frames, and the local feature information of the merged image is extracted, the same local feature information can be obtained even if the arrangement order of the target video frames in the merged image is different. This ensures accurate similarity calculation for videos with temporal misalignment. Furthermore, the similarity calculation is performed on the local feature information of different merged images without considering the similarity of semantic features, thereby reducing computational complexity and improving the recognition efficiency of similar live data streams.

[0120] In some embodiments, "determining at least two target video frames corresponding to each live data stream" in step S101 above can be achieved through the following steps:

[0121] Step S1011: Decode each live data stream to obtain multiple live video frames corresponding to each live data stream.

[0122] To improve the transmission efficiency of live data and meet bandwidth requirements, the broadcaster terminals send encoded and compressed live data streams to the server. Therefore, after receiving the live data streams uploaded by each broadcaster terminal, the server needs to decode them to obtain the multiple live video frames corresponding to each live data stream. These multiple live video frames are generally ordered in chronological order.

[0123] Step S1012: Frame extraction is performed from the multiple live video frames corresponding to each live data stream according to a preset interval, to obtain multiple extracted live video frames corresponding to each live data stream.

[0124] In implementation, frame extraction can be performed at regular intervals. Multiple frame extraction operations within a preset duration can yield multiple extracted live video frames corresponding to each live data stream. For example, the interval can be 12 seconds, 6 seconds, or 5 seconds, and the preset duration can be 2 minutes, extracting 10, 20, or 24 live video frames every two minutes.

[0125] Step S1013: Randomly select a preset number of live video frames from the multiple extracted live video frames corresponding to each live data stream to obtain at least two target video frames corresponding to each live data stream.

[0126] In this step, when there are a large number of live video frames extracted in step S1012, a preset number of live video frames can be randomly selected from the multiple extracted live video frames to obtain at least two target video frames corresponding to each live data stream.

[0127] In some embodiments, if the interval used in step S1012 for frame extraction is relatively long, such as 20 seconds, then 6 live video frames are extracted every two minutes, and these 6 live video frames can be identified as target video frames.

[0128] In some embodiments, the above step S102, "merging at least two target video frames corresponding to each live data stream to obtain a merged image corresponding to each live data stream," can be implemented through the following steps:

[0129] Step S1021: Obtain a preset image target size, and adjust the size of at least two target video frames corresponding to each live data stream based on the image target size to obtain at least two adjusted target video frames corresponding to each live data stream.

[0130] The target image size here can be the target size of a single live video frame, such as 360*240. Based on the target image size, the size of at least two target video frames corresponding to each live data stream is adjusted. This can be done by shrinking or enlarging each target video frame to obtain the corresponding adjusted target video frames. The size information of the adjusted target video frames is the target image size.

[0131] Step S1022: According to the time sequence of at least two adjusted target video frames corresponding to each live data stream, merge the at least two adjusted target video frames corresponding to each live data stream to obtain the merged image corresponding to each live data stream.

[0132] In this embodiment of the application, at least two adjusted target video frames corresponding to each live data stream are merged in chronological order. In implementation, at least two adjusted target video frames may be spliced ​​together to obtain a merged image corresponding to each live data stream.

[0133] In the embodiments described in steps S1021 to S1022 above, since the size of each target video frame is adjusted based on the target image size, even if the sizes of the live video frames parsed from the live data streams uploaded by different broadcaster terminals are different, it is still possible to obtain a merged image of the same size.

[0134] In some embodiments, such as Figure 4 As shown, step S103 above, "determining the local feature information of the merged image corresponding to each live data stream, and determining the similarity between the live data streams based on the local feature information corresponding to each live data stream," can be implemented through steps S1031 to S1035. The following is a combination of... Figure 4 Each step is explained.

[0135] Step S1031: Extract features from each merged image according to a preset local feature extraction algorithm to obtain the feature vector of each merged image.

[0136] The preset local feature extraction algorithm can be a perceptual hash algorithm, or it can be a drift (SHIFT) algorithm, a histogram of oriented gradients (HOG) algorithm, etc.

[0137] In this embodiment, the implementation process of this step is described using a perceptual hash algorithm as an example for local feature extraction. First, the merged image is resized, for example, to 8*8 pixels, totaling 64 pixels. This removes image details, retaining only basic information such as structure and brightness, and eliminating differences caused by different sizes and proportions. Then, the resized image is converted to 64 levels of grayscale. That is, all pixels have only 64 colors. The average grayscale value of all 64 pixels is calculated. In this embodiment, the average grayscale value can be determined as a feature value.

[0138] Step S1032: Determine the feature threshold of each merged image based on the feature vector of each merged image.

[0139] In this step, it can be implemented by calculating the arithmetic mean of the feature values ​​in each feature vector to obtain the mean value corresponding to each feature vector, and then determining the mean value as the feature threshold of each merged image; or by sorting the feature values ​​in each feature vector to obtain the sorting result, then determining the feature median value based on the sorting result, and then determining the feature median value as the feature threshold.

[0140] Step S1033: Based on the feature thresholds of each merged image, perform binarization processing on the corresponding feature vectors to obtain the local feature information of each merged image.

[0141] In this step, the feature values ​​in the feature vector that are greater than the feature threshold are set to 1, and the feature values ​​that are less than or equal to the feature threshold are set to 0, thereby achieving binarization of the feature vector and obtaining the local feature information of each merged image.

[0142] Step S1034: Based on the local feature information corresponding to each live data stream, determine the number of identical bits in the local feature information of different live data streams.

[0143] In implementation, this step can be achieved by using a bitwise XOR operation to determine the number of identical bits in the local feature information of different live data streams.

[0144] Step S1035: Based on the total number of bits of local feature information and the number of identical bits, determine the similarity between the different live data streams.

[0145] Here, the similarity between different live stream data streams can be obtained by dividing the number of identical digits by the total number of digits. In other words, the more identical digits there are between different feature vectors, the higher the similarity between the live stream data streams.

[0146] In the embodiments described in steps S1031 to S1035 above, the local feature information of the merged image is extracted. Therefore, even if the order of each live video frame in the merged image is different, it does not affect the final generated local feature information, thereby solving the problem of time alignment. Furthermore, the similarity of different live data streams can be obtained by determining the number of identical bits by bitwise XORing of the local feature information and dividing the number of identical bits by the total number of bits, which can greatly reduce the computational complexity and improve the computational efficiency.

[0147] Based on the foregoing embodiments, this application further provides a live streaming data processing method, applied to... Figure 1 The network architecture shown is Figure 5 This is a schematic diagram illustrating another implementation flow of the live data processing method provided in the embodiments of this application, as follows: Figure 5 As shown, the process includes:

[0148] In step S501, the broadcaster terminal responds to the operation command to start the live streaming App, presents the live streaming window of the live streaming service, and receives the settings of the broadcaster user for the live streaming service that is about to be initialized.

[0149] In this embodiment, the live streaming window before the live streaming service is initialized is used to receive information such as the name and notes of the new live streaming service added by the streamer in the streamer's live streaming room, so that the streamer can find it later.

[0150] In step S502, the broadcaster terminal sends live streaming service initialization data to the server.

[0151] Here, the broadcaster terminal submits the identifiers of the live streaming room to be established and the broadcaster user's identifier to the server to initialize the live streaming service.

[0152] In step S503, the broadcast terminal responds to the start operation for starting live streaming on the broadcast terminal, presents the live streaming playback interface, and obtains the media data to be uploaded.

[0153] Here, when a live video stream is initiated, the media data includes image and audio data. The media data to be uploaded in this step can be captured in real-time by the image acquisition device on the broadcaster's terminal, or it can be transmitted to the broadcaster's terminal from other devices that have established a communication connection with it. Examples include live news broadcasts and television drama rebroadcasts.

[0154] In step S504, the broadcaster terminal encodes the media data to be uploaded to obtain a live data stream, and sends the live data stream to the server.

[0155] Step S505: The server obtains multiple live data streams uploaded by different broadcaster terminals and determines at least two target video frames corresponding to each live data stream.

[0156] In step S506, the server merges at least two target video frames corresponding to each live data stream to obtain a merged image corresponding to each live data stream.

[0157] Step S507: The server determines the local feature information of the merged image corresponding to each live data stream, and determines the similarity between the live data streams based on the local feature information corresponding to each live data stream.

[0158] Step S508: The server determines at least two live streaming data streams that meet the similarity criteria based on the similarity between the various live streaming data streams.

[0159] It should be noted that the implementation process of steps S505 to S508 is similar to that of steps S101 to S104. When implementing these steps, you can refer to the implementation process of steps S101 to S104.

[0160] Step S509: The server determines the target live stream that needs to be deduplicated from the at least two live streams that meet the similarity condition.

[0161] When implementing this step, it can be based on criteria such as the number of viewers, the number of reports, and users' historical behavior data for deduplication filtering. For example, it can identify live streams with fewer viewers, more reports, or that do not match user preference information as target live streams.

[0162] Step S510: The server performs deduplication on the target live data stream to obtain the processed live data stream.

[0163] In implementing this step, the target live stream data stream can be deleted from the plurality of live stream data streams to obtain a processed live stream data stream; or, the target live stream data stream can be subjected to traffic downweighting to obtain a processed live stream data stream. In some embodiments, the target live stream data stream can also be subject to a bottom-feeding penalty.

[0164] In step S511, the viewer terminal initializes the client and the player parameters based on the operation command to start the live streaming client.

[0165] Here, the player program in the viewer's terminal runs as a single instance. This means that as long as the client is running, the player program continues to run and will not be stopped. The player parameters are initialized during client initialization.

[0166] In step S512, the viewer's terminal sends a live data acquisition request to the server.

[0167] Here, the live data retrieval request is at least used to request the interface data of the live room, that is, the interface data of the homepage of the live room.

[0168] Step S513: The server determines the live data to be sent based on the processed live data stream.

[0169] When this step is implemented, after the server receives the data acquisition request, it obtains the user's historical viewing information and followed anchor information based on the user identifier corresponding to the viewer's terminal. Then, it determines the anchors watched by the user and the live streams similar to the user's historical viewing information from the processed live stream data stream, and determines the homepage interface data of the selected live stream data stream as the live stream data to be sent.

[0170] Step S514: The server sends the live data to be sent to the viewer's terminal.

[0171] In step S515, after receiving the live data to be sent, the viewer terminal displays the homepage interface of each live room.

[0172] Since the live streaming data to be sent to the viewer's terminal is determined based on the live streaming data stream obtained after deduplication, the live streaming data to be sent does not include the homepage interface of the live streaming room that plays duplicate live streaming data streams.

[0173] In step S516, the viewer terminal responds to the touch operation on the homepage interface of the target live room by sending a request to the server to obtain the live data stream.

[0174] The live stream data acquisition request carries the identifier of the target live stream room, which is used to request the acquisition of the live stream data of the target live stream room.

[0175] Step S517: The server obtains the live data stream corresponding to the target live room and sends the live data stream to the viewer's terminal.

[0176] In step S518, the viewer's terminal performs live streaming based on the live data stream.

[0177] In the live streaming data processing method provided in this application embodiment, after the broadcaster terminal completes the settings for the live streaming service and starts live streaming, it sends a live streaming data stream to the server. After the server obtains multiple live streaming data streams uploaded by different broadcaster terminals, it determines at least two target video frames corresponding to each live streaming data stream, then merges the at least two target video frames corresponding to each live streaming data stream to obtain a merged image corresponding to each live streaming data stream, and determines the local feature information of the merged image corresponding to each live streaming data stream. Based on the local feature information corresponding to each live streaming data stream, it determines the similarity between the live streaming data streams, and finally, based on the similarity between the live streaming data streams, it determines at least two live streaming data streams that meet the similarity condition. Thus, since it is obtained... At least two target video frames are merged, and the local feature information of the merged image is extracted. This ensures that even if the order of the target video frames in the merged image is different, the same local feature information can be obtained. This guarantees accurate similarity calculation for videos with temporal misalignment. Furthermore, the similarity calculation only considers the local feature information of different merged images, without considering the similarity of semantic features, thereby reducing computational complexity and improving the recognition efficiency of similar live streams. In addition, after the server receives the live data acquisition request sent by the viewer's terminal, it will perform deduplication processing on at least two live streams that meet the similarity conditions. This ensures that the live data sent to the viewer's terminal will not enter the live room playing the same live content, thus improving the viewing experience of the viewer's terminal.

[0178] In some embodiments, the above step S509, "determining the target live stream that needs to be deduplicated from at least two live streams that meet the similarity conditions," can be implemented based on the number of viewers, the number of reports, user historical behavior data, etc., as deduplication filtering criteria. The following are some implementation methods.

[0179] The first method, based on the number of viewers, can be achieved through the following steps:

[0180] Step S5091A: Determine the number of viewers in the live streaming rooms corresponding to at least two live streaming data streams that meet the similarity condition.

[0181] In this embodiment of the application, the number of viewers can be the number of real-time viewers or the cumulative number of viewers.

[0182] Step S5092A: Determine the target live streaming data stream from the at least two live streaming data streams, excluding the one with the most viewers.

[0183] Since the content played in at least two live streams is similar, i.e., homogeneous, only one of the at least two live streams needs to be retained. In implementation, this can be done by retaining only the one with the most viewers, which means that the other live stream among the at least two live streams, excluding the one with the most viewers, is determined as the target live stream.

[0184] The second method, which is based on the number of reports and historical behavior data, can be achieved through the following steps:

[0185] Step S5091B: Obtain the number of reports for the live streaming rooms corresponding to at least two live streaming data streams that meet the similarity condition.

[0186] The number of reports can be the cumulative number of reports from at least two live stream data streams since the start of the broadcast, or the number of reports within a preset historical period starting from the current moment, such as the number of reports within a week, the number of reports within a month, and so on.

[0187] Step S5092B: Determine whether the difference between the number of reports in the corresponding live streaming rooms of the at least two live streaming data streams is greater than the difference threshold.

[0188] When the difference between the number of reports in the live rooms corresponding to the at least two live data streams is greater than the difference threshold, proceed to step S5093B; when the difference between the number of reports in the live rooms corresponding to the at least two live data streams is less than or equal to the difference threshold, proceed to step S5094B.

[0189] Step S5093B: Determine the target live streaming data stream from the at least two live streaming data streams, excluding the one with the fewest reports.

[0190] In this embodiment of the application, when the difference between the number of reports in the live room corresponding to at least two live data streams is greater than the difference threshold, it is determined that the number of reports can be used as the deduplication standard for filtering. At this time, only the live data stream with the fewest reports needs to be retained, that is, the other live data streams except the one with the fewest reports are determined as the target live data streams.

[0191] Step S5094B: In response to the live data acquisition request sent by the viewer terminal, acquire the historical behavior data corresponding to the viewer terminal.

[0192] The live stream data retrieval request carries a user identifier. During this step, the server retrieves the historical behavior data corresponding to the viewer's terminal based on the user identifier. This historical behavior data may include the live stream room identifier, viewing duration, interaction information, and identifiers of followed live stream rooms, etc.

[0193] When the difference between the number of reports in at least two live streaming data streams corresponding to the live streaming rooms is less than or equal to the difference threshold, it means that the number of reports in at least two live streaming rooms is not significantly different. In this case, the number of reports is not used as the standard for deduplication, but is further judged based on the historical behavior data of the audience terminal.

[0194] Step S5095B: Determine the viewing preference information corresponding to the viewer terminal based on the historical behavior data.

[0195] This viewing preference information can be used to characterize the type of live stream preferred by the user corresponding to the viewer's terminal.

[0196] Step S5096B: Based on the viewing preference information, determine the other live streaming data stream that has the highest matching degree with the viewing preference information from the at least two live streaming data streams that meet the similarity condition as the target live streaming data stream.

[0197] When implementing this step, first determine the matching degree between at least two live streaming data streams and the live streaming room corresponding to the viewing preference information, and then retain the live streaming data with the highest matching degree. In other words, other live streaming data streams besides the one with the highest matching degree are determined as the target live streaming data streams.

[0198] It should be noted that in some embodiments, other factors besides the number of viewers, the number of reports, and viewing preference information can be used as the criteria for deduplication. For example, one or more of the following can be used as the criteria for deduplication: the streamer's rating, the popularity of the live stream, and the number of likes.

[0199] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0200] The live streaming data processing method provided in this application can be used in scenarios such as filtering homogeneous live streaming content, reducing traffic ranking, and penalizing content that has been buried in the bottom of the list. Figure 6For example, in the live news category shown, if live rooms 601 and 602 play the same live content within a certain time period, then that live content needs to be filtered out and not displayed. Alternatively, it can be displayed in different user sessions. For instance, user A should only see the live content from live room R1, and user B should only see the live content from live room R2, thus preventing a single user from seeing multiple identical live content streams.

[0201] The following describes the live data processing method provided in the embodiments of this application. In actual implementation, the method can be implemented in two steps: hash calculation of the live stream video and filtering of homogeneous content.

[0202] During the live broadcast, each live broadcast room independently samples and stores its own video frames. The video frames of each live broadcast room do not need to be time aligned. In the embodiment of this application, for all live broadcast rooms, one frame is sampled every approximately 5 seconds, and a 2-minute video segment is accumulated as a prediction interval, with each prediction interval containing 24 frames. Due to the existence of time misalignment, the hash algorithm should meet the following conditions: (1) It can tolerate a certain degree of frame misalignment. For example, the sequence of frame numbers [4, 5, 6, 7, 8, 9] should have a hash similarity to the sequence [5, 6, 7, 8, 9, 10], and there is no problem of excessive distance caused by comparing misaligned frame pairs such as [4, 5] and [5, 6]. (2) The time cost is small. The extraction of hash values ​​should not rely on deep semantic features. Since it is only necessary to discover video content with similar performance rather than to discover semantically similar video pairs, it is not necessary to obtain overly abstract deep semantic information.

[0203] To address these two design goals, the hash calculation of the live video is performed in the embodiments of this application through the following two steps:

[0204] Step S601: First, randomly sample 9 images from the 24 images, and then merge the 9 images into a 3*3 grid image in chronological order.

[0205] in, Figure 7A This is the large nine-grid image corresponding to live stream room A1. Figure 7B This is the large nine-grid image corresponding to A2 in the live stream room. Figure 8A This is the large nine-grid image corresponding to B1 in the live stream room. Figure 8B This is the large nine-grid image corresponding to B2 in the live stream room.

[0206] Step S602: For the obtained nine-square grid image, use a feature extraction algorithm based on local features to extract features and obtain feature vectors.

[0207] The feature extraction algorithm based on local features can be, but is not limited to, the pHash algorithm, the SHIFT algorithm, and the HOG algorithm. The feature values ​​in the resulting feature vector are then represented as floating-point numbers. For example, the first ten feature values ​​of the feature vector corresponding to a large nine-grid image are 0.00776728, 0.0436932, -0.0070386, 0.0783124, -0.0413507, 0.0427038, -0.0814955, 0.130502, -0.0673937, and 0.0512087.

[0208] Step S603: Determine the processing threshold based on each feature vector, and set the feature vectors that are greater than the processing threshold to 1 and those that are less than the processing threshold to 0, to obtain the hash feature vector corresponding to the nine-square grid image.

[0209] Following the example above, assuming the processing threshold is 0, the hash values ​​of the first ten features of this feature vector are 1, 1, 0, 1, 0, 1, 0, 1, 0, 1.

[0210] In this embodiment, after obtaining the hash feature vectors of each 9x9 grid image, the hash feature vectors are stored in the database. Since the image features extracted from the 9x9 grid image are local sensitive features, arbitrarily changing the relative order of the images in the 9x9 grid does not affect the final hash value, thus solving the time alignment problem.

[0211] Based on the hash feature representation obtained in step S603 above, homogeneity filtering can be achieved through the following steps:

[0212] Step S701: Determine the similarity of each live data stream based on the hash feature representation of each live data stream.

[0213] This step is implemented by obtaining the hash feature representations of all live data streams in real time, and then calculating the similarity between the live data streams. For hash values ​​f1 and f2, a bitwise AND operation is used to calculate the number of identical bits, and the number of identical bits is divided by the total number of bits in the hash feature representation to obtain their similarity.

[0214] Step S702: If it is determined that the similarity of at least two live data streams is higher than a preset threshold, it indicates that the at least two live data streams are homogeneous.

[0215] Step S703: Determine one live streaming data stream to be retained from the at least two live streaming data streams and filter out the other live streaming data streams.

[0216] Table 1 is a performance comparison table of Scheme 1 and Scheme 2 in the live data processing method and related technologies provided in the embodiments of this application. As can be seen from Table 1, compared with Scheme 1 and Scheme 2, the live data processing method provided in the embodiments of this application has made significant breakthroughs in resource utilization and latency, while ensuring that the accuracy does not decrease.

[0217]

[0218] In the live streaming data processing method provided in this application embodiment, by implementing a fast audio and video hashing algorithm, homogeneous live streaming content can be filtered in live streaming scenarios, effectively reducing the homogenization of live streaming content and allowing users to watch more diverse live streaming content. It can also serve as a foundational capability to combat illegal accounts such as those that repost content. Due to its lower latency, it can also accurately identify delays caused by rebroadcasting the same content in a live streaming room. Furthermore, this algorithm does not require GPU resources, thus reducing deployment and maintenance costs.

[0219] The following description continues to illustrate the exemplary structure of the live data processing device 443 provided in this application embodiment as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the live data processing device 443 in the memory 440 may include:

[0220] The first determining module is used to acquire multiple live data streams uploaded by different terminals and determine at least two target video frames corresponding to each live data stream.

[0221] An image merging module is used to merge at least two target video frames corresponding to each live data stream to obtain a merged image corresponding to each live data stream.

[0222] The second determining module is used to determine the local feature information of the merged image corresponding to each of the live data streams, and to determine the similarity between the live data streams based on the local feature information corresponding to each of the live data streams.

[0223] The third determining module is used to determine at least two live streaming data streams that meet the similarity conditions based on the similarity between the various live streaming data streams.

[0224] In some embodiments, the first determining module 4431 is further configured to:

[0225] Each live data stream is decoded to obtain multiple live video frames corresponding to each live data stream;

[0226] Frame extraction is performed from multiple live video frames corresponding to each live data stream according to a preset interval, to obtain multiple extracted live video frames corresponding to each live data stream.

[0227] A preset number of live video frames are randomly selected from the multiple extracted live video frames corresponding to each live data stream to obtain at least two target video frames corresponding to each live data stream.

[0228] In some embodiments, the image merging module 4432 is further configured to:

[0229] Obtain a preset image target size, and adjust the size of at least two target video frames corresponding to each live data stream based on the image target size to obtain at least two adjusted target video frames corresponding to each live data stream.

[0230] According to the time sequence of at least two adjusted target video frames corresponding to each live data stream, the at least two adjusted target video frames corresponding to each live data stream are merged to obtain the merged image corresponding to each live data stream.

[0231] In some embodiments, the second determining module 4433 is further configured to:

[0232] The features of each merged image are extracted according to a preset local feature extraction algorithm to obtain the feature vector of each merged image;

[0233] The feature threshold of each merged image is determined based on the feature vector of each merged image.

[0234] Based on the feature thresholds of each merged image, the corresponding feature vectors are binarized to obtain the local feature information of each merged image.

[0235] In some embodiments, the second determining module 4433 is further configured to:

[0236] Based on the local feature information corresponding to each live data stream, determine the number of identical bits in the local feature information of different live data streams.

[0237] The similarity between the different live data streams is determined based on the total number of bits of local feature information and the number of identical bits.

[0238] In some embodiments, the third determining module 4434 is further configured to:

[0239] Obtain the preset similarity threshold;

[0240] At least two live streams with a similarity greater than or equal to the similarity threshold are identified as at least two live streams that satisfy the similarity condition;

[0241] The device also includes:

[0242] The fourth determining module is used to determine the target live stream that needs to be deduplicated from the at least two live streams that meet the similarity conditions;

[0243] The deduplication module is used to perform deduplication processing on the target live data stream to obtain the processed live data stream.

[0244] The sending module is used to determine the live data to be sent based on the processed live data stream, and send the live data to be sent to the viewer's terminal.

[0245] In some embodiments, the fourth determining module is further configured to:

[0246] Determine the number of viewers in the live streaming rooms corresponding to at least two live streaming data streams that meet the similarity criteria;

[0247] The live stream with the most viewers among the at least two live streams is identified as the target live stream.

[0248] In some embodiments, the fourth determining module is further configured to:

[0249] Obtain the number of reports for the corresponding live streaming rooms of at least two live streaming data streams that meet the similarity criteria;

[0250] When the difference between the number of reports in the corresponding live rooms of the at least two live data streams is greater than the difference threshold, the other live data stream among the at least two live data streams, excluding the one with the fewest reports, is determined as the target live data stream.

[0251] In some embodiments, the fourth determining module is further configured to:

[0252] When the difference between the number of reports in the live room corresponding to the at least two live data streams is less than or equal to the difference threshold, in response to the live request sent by the viewer terminal, the historical behavior data corresponding to the viewer terminal is obtained.

[0253] Based on the historical behavior data, determine the viewing preference information corresponding to the viewer's terminal;

[0254] Based on the viewing preference information, the other live streaming data stream that has the highest matching degree with the viewing preference information among the at least two live streaming data streams that meet the similarity condition is determined as the target live streaming data stream.

[0255] In some embodiments, the deduplication module is further configured to:

[0256] The target live stream is deleted from the plurality of live streams to obtain a processed live stream; or;

[0257] The target live stream data stream is subjected to traffic weight reduction processing to obtain the processed live stream data stream.

[0258] It should be noted that the description of the live data processing device in this application is similar to the description of the method embodiments described above, and has similar beneficial effects. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiments of this application for understanding.

[0259] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the live data processing method described above in this application.

[0260] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the live data processing method provided in this application. For example, ... Figure 3 , Figure 4 , Figure 5 The live data processing method is shown.

[0261] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0262] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0263] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0264] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0265] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for processing live streaming data, characterized in that, The method includes: Acquire multiple live data streams uploaded from different terminals and determine at least two target video frames corresponding to each live data stream; At least two target video frames corresponding to each live data stream are merged in a spatial tiling stitching manner to obtain a merged image corresponding to each live data stream. Determine the local feature information of the merged image corresponding to each of the live data streams, and determine the similarity between the live data streams based on the local feature information corresponding to each of the live data streams; Based on the similarity between the various live data streams, at least two live data streams that meet the similarity criteria are identified.

2. The method according to claim 1, characterized in that, Determining at least two target video frames corresponding to each live data stream includes: Each live data stream is decoded to obtain multiple live video frames corresponding to each live data stream; Frame extraction is performed from multiple live video frames corresponding to each live data stream according to a preset interval, to obtain multiple extracted live video frames corresponding to each live data stream. A preset number of live video frames are randomly selected from the multiple extracted live video frames corresponding to each live data stream to obtain at least two target video frames corresponding to each live data stream.

3. The method according to claim 1, characterized in that, The step of merging at least two target video frames corresponding to each of the live data streams in a spatially tiled stitching manner to obtain a merged image corresponding to each of the live data streams includes: Obtain a preset image target size, and adjust the size of at least two target video frames corresponding to each live data stream based on the image target size to obtain at least two adjusted target video frames corresponding to each live data stream. According to the time sequence of at least two adjusted target video frames corresponding to each live data stream, the at least two adjusted target video frames corresponding to each live data stream are merged in a spatial tiling stitching manner to obtain the merged image corresponding to each live data stream.

4. The method according to claim 1, characterized in that, The step of determining the local feature information of the merged image corresponding to each live data stream includes: The features of each merged image are extracted according to a preset local feature extraction algorithm to obtain the feature vector of each merged image; The feature threshold of each merged image is determined based on the feature vector of each merged image. Based on the feature thresholds of each merged image, the corresponding feature vectors are binarized to obtain the local feature information of each merged image.

5. The method according to claim 4, characterized in that, The method determines the similarity between the various live stream data streams based on the local feature information corresponding to each live stream, including: Based on the local feature information corresponding to each live data stream, determine the number of identical bits in the local feature information of different live data streams. The similarity between the different live data streams is determined based on the total number of bits of local feature information and the number of identical bits.

6. The method according to any one of claims 1 to 5, characterized in that, The step of determining at least two live streams that satisfy the similarity condition based on the similarity between the various live streams includes: Obtain the preset similarity threshold; At least two live streams with a similarity greater than or equal to the similarity threshold are identified as at least two live streams that satisfy the similarity condition; The method further includes: From the at least two live stream data streams that meet the similarity criteria, determine the target live stream data stream that needs to be deduplicated; The target live stream data stream is deduplicated to obtain the processed live stream data stream. Based on the processed live data stream, determine the live data to be sent, and send the live data to be sent to the viewer's terminal.

7. The method according to claim 6, characterized in that, The step of determining the target live stream that needs deduplication processing from at least two live streams that meet the similarity criteria includes: Determine the number of viewers in the live streaming rooms corresponding to at least two live streaming data streams that meet the similarity criteria; The live stream with the most viewers among the at least two live streams is identified as the target live stream.

8. The method according to claim 6, characterized in that, The step of determining the target live stream that needs deduplication processing from at least two live streams that meet the similarity criteria includes: Obtain the number of reports for the corresponding live streaming rooms of at least two live streaming data streams that meet the similarity criteria; When the difference between the number of reports in the corresponding live rooms of the at least two live data streams is greater than the difference threshold, the other live data stream among the at least two live data streams, excluding the one with the fewest reports, is determined as the target live data stream.

9. The method according to claim 8, characterized in that, The step of determining the target live stream that needs deduplication processing from at least two live streams that meet the similarity criteria includes: When the difference between the number of reports in the live room corresponding to the at least two live data streams is less than or equal to the difference threshold, in response to the live request sent by the viewer terminal, the historical behavior data corresponding to the viewer terminal is obtained. Based on the historical behavior data, determine the viewing preference information corresponding to the viewer's terminal; Based on the viewing preference information, the other live streaming data stream that has the highest matching degree with the viewing preference information among the at least two live streaming data streams that meet the similarity condition is determined as the target live streaming data stream.

10. The method according to claim 8, characterized in that, The process of deduplicating the target live stream data stream to obtain the processed live stream data stream includes: The target live stream is deleted from the plurality of live streams to obtain a processed live stream; or; The target live stream data stream is subjected to traffic weight reduction processing to obtain the processed live stream data stream.

11. A live streaming data processing device, characterized in that, The device includes: The first determining module is used to acquire multiple live data streams uploaded by different terminals and determine at least two target video frames corresponding to each live data stream. The image merging module is used to merge at least two target video frames corresponding to each live data stream in a spatial tiling splicing manner to obtain a merged image corresponding to each live data stream. The second determining module is used to determine the local feature information of the merged image corresponding to each of the live data streams, and to determine the similarity between the live data streams based on the local feature information corresponding to each of the live data streams. The third determining module is used to determine at least two live streaming data streams that meet the similarity conditions based on the similarity between the various live streaming data streams.

12. The apparatus according to claim 11, characterized in that, The first determining module is further configured to decode each live data stream to obtain multiple live video frames corresponding to each live data stream; perform frame extraction processing from the multiple live video frames corresponding to each live data stream according to a preset interval duration to obtain multiple extracted live video frames corresponding to each live data stream; and randomly select a preset number of live video frames from the multiple extracted live video frames corresponding to each live data stream to obtain at least two target video frames corresponding to each live data stream.

13. The apparatus according to claim 11, characterized in that, The image merging module is further configured to obtain a preset image target size, adjust the size of at least two target video frames corresponding to each live data stream based on the image target size, and obtain at least two adjusted target video frames corresponding to each live data stream; and merge the at least two adjusted target video frames corresponding to each live data stream according to the time sequence of ...

14. A computer device, characterized in that, The computer device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 10.

15. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the method according to any one of claims 1 to 10.

16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method described in any one of claims 1 to 10.

Images