Live streaming information flow delivery method and electronic device

Abstract

Embodiments of the present application disclose a live broadcast information stream putting method and an electronic device. The method comprises: pulling a live broadcast information stream and commodity information in a commodity list associated with the live broadcast information stream; performing real-time analysis on the live broadcast information stream, and performing correlation judgment on a real-time explanation object of the live broadcast information stream and commodities in the commodity list; putting the live broadcast information stream to a traffic distribution system, and providing information of a target commodity corresponding to the real-time explanation object to the traffic distribution system, so that the traffic distribution system pushes the live broadcast information stream to a target user group through a target page according to the information of the target commodity corresponding to the real-time explanation object. Through the embodiments of the present application, more personalized live broadcast room recommendation can be realized, which is beneficial to improving indicators such as commodity click rate and conversion rate.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to methods and electronic devices for delivering live streaming information streams. Background Technology

[0002] With the increasing application of short video and live streaming technologies, live streaming has become an important means for product information service systems to convey product information to consumers due to its ability to provide more detailed shopping decision-making information.

[0003] However, typically, users need to actively enter the live streaming channel page and then select which live stream to watch. To improve metrics such as viewership, some solutions can analyze users' historical viewing history to identify live streams they might be interested in, and then distribute links to currently running live streams to public domain pages, such as "You May Like" sections, allowing users to access specific live streams through these public domain pages. While this method can improve viewership to some extent, it also limits personalized recommendations because the system typically recommends live streams that users already like or that are similar to those they've viewed in terms of main product categories. This also restricts improvements in metrics such as click-through rate and conversion rate. Summary of the Invention

[0004] This application provides a method and electronic device for delivering live streaming information streams, which can achieve more personalized live streaming recommendations and help improve metrics such as product click-through rate and conversion rate.

[0005] This application provides the following solution:

[0006] A method for delivering live streaming information streams, including:

[0007] Retrieve the live stream information stream, as well as the product information from the product list associated with the live stream information stream;

[0008] By performing real-time analysis of the live stream, the relevance between the real-time explanation object of the live stream and the products in the explanation product list is determined;

[0009] The live stream information is delivered to the traffic distribution system, and the traffic distribution system is provided with information about the target product corresponding to the live explanation object, so that the traffic distribution system can push the live stream information to the target user group through the target page based on the information about the target product corresponding to the live explanation object.

[0010] Before the live stream information is distributed to the traffic distribution system, the process also includes:

[0011] By performing model understanding on the multimodal descriptive information associated with the target product corresponding to the real-time explanation object, text content about the current content value point of the live broadcast information stream is generated;

[0012] The step of distributing the live stream information to the traffic distribution system and providing the traffic distribution system with information about the target product corresponding to the live explanation subject includes:

[0013] The live stream information flow is delivered to the traffic distribution system, and the text content of the current content value point is provided to the traffic distribution system so that when the traffic distribution system pushes the live stream information flow to the target user group through the target page, it also provides the text content of the current content value point.

[0014] The relevance determination of the real-time explanation object in the live broadcast information stream and the products in the explanation product list includes:

[0015] Video frames are extracted from the live stream, and the narration intent of the image content in the video frames is located so as to extract the target image content belonging to the current real-time narration object.

[0016] By inputting the target image content into a visual representation model for calculation, a representation vector for representing the semantic information of the target image content is obtained;

[0017] The correlation between the real-time explanation object and the products in the explanation product list is determined by matching the representation vector with the representation vectors corresponding to the product images of multiple products in the pre-obtained explanation product list.

[0018] The step of locating the intended meaning of the image content in the video frame includes:

[0019] Based on the image content in the video frame and the text content converted from the voice information in the live stream, multimodal information is obtained, and the narration intention of the image content in the video frame is located based on the multimodal information.

[0020] The visual representation model structure includes: a visual converter model as the backbone network, with a fully connected layer added on top of the backbone network. The fully connected layer is used to convert the output information of the backbone network into a representation vector of the target dimension. The visual converter model adopts a shifted non-overlapping window design, performs self-attention mechanism calculation at the window level, applies multi-scale pooling in window partitioning, and layers features of different granularities through convolution in the depth direction. The context features of different granularities are aggregated through gating aggregation.

[0021] In the process of training the visual representation model, the objective function used in the training process includes: a multi-similarity loss function;

[0022] Each training round uses a target batch of training samples as input information. The target batch of training samples includes multiple positive samples and multiple negative samples. The positive samples are multiple different images corresponding to the same product, and the negative samples are multiple different images corresponding to other products. The training samples are obtained based on video clips extracted from historical live stream information streams and corresponding product annotation information.

[0023] After the visual representation model outputs representation vectors for each training sample, the training samples are paired up into sample pairs, and the similarity of the representation vectors within multiple sample pairs is compared to optimize the multi-similarity loss function, so as to update the parameter values ​​of the visual representation model.

[0024] This also includes:

[0025] During the optimization of the multi-similarity loss function, hard samples are sampled so that the multi-similarity loss function can be optimized while giving higher weights to the hard sample pairs.

[0026] Specifically, when sampling hard samples, a feature memory queue is created. Historical training samples input into the visual representation model in previous training rounds, as well as new training samples input into the visual representation model in the current training round, are added to the feature memory queue. This allows for the mining of hard samples by calculating the similarity between the representation vectors of sample pairs within the feature memory queue.

[0027] This also includes:

[0028] During the sampling of difficult samples within the feature memory queue, a momentum update model with the same model structure as the visual representation model is constructed. The parameter update direction of the momentum update model is determined by partially approximating the visual representation model.

[0029] When sampling hard samples from the feature memory queue, the momentum update model is used to generate representation vectors for historical training samples.

[0030] The output information of the visual representation model also includes prediction information on the category distribution of goods included in the input image;

[0031] The objective function also includes a product category classification loss function, which is used to assist in training the visual representation model.

[0032] Specifically, when initializing the visual representation model, multiple information pairs consisting of product images and product text descriptions are collected. For each information pair, the product image is input into the visual representation model to extract image features, and the product text descriptions are input into the text feature extraction model to extract text features.

[0033] Based on the similarity between the image features and the text features, cross-modal unsupervised semantic alignment training is performed on the visual representation model and the text feature extraction model. The parameters of the visual representation model after training are used as the initialization parameters of the visual representation model so that the visual representation model can be trained based on the initialization parameters.

[0034] This also includes:

[0035] A dynamic text feature queue is constructed, and the text features from multiple information pairs are added to the dynamic text feature queue. The extended text of the image is obtained by comparing the similarity between the image features and the multiple text features in the dynamic text feature queue, so that the image and the extended text can be combined into an information pair and added to the unsupervised semantic alignment training.

[0036] This also includes:

[0037] The visual representation model is continuously learned using the newly acquired extended training sample set. During the continuous learning process, a distillation learning method is adopted, in which the visual representation model is used as the teacher model, and a student model with the same model structure as the teacher model is constructed. The student model is trained and its parameters are updated using the extended training sample set, with the teacher model as the guide. After training, if the performance of the student model is better than that of the teacher model, the learned model is used as the new visual representation model.

[0038] A product information recommendation method, comprising:

[0039] Receive information about the delivery of live streaming information streams, the delivery information including real-time content value point information of the live streaming information streams, the real-time content value point information being determined based on the information of the target product corresponding to the real-time explanation object of the live streaming information streams;

[0040] During the process of recommending product information to users, the live stream information is recommended to target users based on the real-time content value information of the live stream information stream, so as to provide links to the live stream information stream in the product recommendation information display page shown to the target users.

[0041] The step of recommending the live stream to target users based on the real-time content value information of the live stream includes:

[0042] During the process of recommending product information to users, the degree of interest of the target user in the target product is determined;

[0043] If the level of interest meets the criteria, a link to the live stream is inserted into the set of recommended product information provided to the target user, so that the link to the live stream is provided on the product recommendation information display page shown to the target user.

[0044] A method for providing product search result information includes:

[0045] Receive information about the delivery of live streaming information streams, the delivery information including real-time content value point information of the live streaming information streams, the real-time content value point information being determined based on the information of the target product corresponding to the real-time explanation object of the live streaming information streams;

[0046] During the process of providing product search results based on user-initiated product search requests, links to the live stream are provided on the product search results display page based on the real-time content value information of the live stream.

[0047] The step of providing links to the live stream information on the product search results display page based on the real-time content value information of the live stream includes:

[0048] If the target product corresponding to the real-time explanation object meets the search criteria, then a link to the live stream is inserted into the product search results set so that the link to the live stream is provided on the product search results display page.

[0049] A method for processing difficult samples includes:

[0050] Obtain a training sample set, wherein the training sample set includes multiple images and corresponding annotation information;

[0051] Using a multi-similarity loss function as the objective function, the visual representation model is subjected to comparative learning in multiple training rounds using the training sample set, so as to update the parameter values ​​of the visual representation model.

[0052] In the process of comparative learning, hard sample sampling is also performed so that the multi-similarity loss function can be optimized with the hard samples being given higher weights.

[0053] During the sampling of the hard samples, a feature memory queue is created so that historical training samples input into the visual representation model in previous training rounds, as well as new training samples input into the visual representation model in the current training round, are added to the feature memory queue.

[0054] A momentum update model with the same model structure as the visual representation model is constructed, and the parameter update direction of the momentum update model is determined by partially approximating the visual representation model.

[0055] When sampling hard samples, the momentum update model is used to generate representation vectors for historical training samples in the feature memory queue, and sample pairs are constructed within the feature memory queue. The sampling of hard samples is performed by calculating the similarity between the representation vectors of the sample pairs.

[0056] An initialization method for a visual representation model, comprising:

[0057] During the parameter initialization process of the visual representation model, multiple information pairs consisting of image and text description information are collected.

[0058] For each information pair, the image is input into the visual representation model to extract image features, and the text description information is input into the text feature extraction model to extract text features.

[0059] Based on the similarity between the image features and the text features, cross-modal unsupervised semantic alignment training is performed on the visual representation model and the text feature extraction model. The parameters of the visual representation model after training are used as the initialization parameters of the visual representation model, so that the visual representation model can be trained using the dataset in the downstream application based on the initialization parameters.

[0060] A method for generating product demonstration videos includes:

[0061] Retrieve the live stream information stream, as well as the product information from the product list associated with the live stream information stream;

[0062] By performing real-time analysis of the live stream, the relevance between the real-time explanation object of the live stream and the products in the explanation product list is determined;

[0063] Based on the correlation judgment results, determine the start and end time information of the explanation of the target product in the product list in the live broadcast information stream;

[0064] Based on the start and end time information, corresponding video segments are extracted from the live stream information stream so that the video segments can be used as explanatory videos for the target product and delivered to the target page.

[0065] A live streaming information stream delivery device, comprising:

[0066] An information retrieval unit is used to retrieve the live stream information stream and the product information in the product list associated with the live stream information stream;

[0067] The relevance judgment unit is used to perform real-time analysis of the live broadcast information stream to determine the relevance between the real-time explanation object of the live broadcast information stream and the products in the explanation product list;

[0068] The delivery unit is used to deliver the live stream information flow to the traffic distribution system and provide the traffic distribution system with information on the target product corresponding to the real-time explanation object, so that the traffic distribution system can push the live stream information flow to the target user group through the target page based on the information on the target product corresponding to the real-time explanation object.

[0069] A product information recommendation device, comprising:

[0070] The delivery information receiving unit is used to receive delivery information about the live broadcast information stream. The delivery information includes real-time content value point information of the live broadcast information stream, which is determined based on the information of the target product corresponding to the real-time explanation object of the live broadcast information stream.

[0071] The recommendation unit is used to recommend the live stream to the target user based on the real-time content value information of the live stream during the process of recommending product information to the user, so as to provide links to the live stream in the product recommendation information display page shown to the target user.

[0072] An apparatus for providing product search results information includes:

[0073] The delivery information receiving unit is used to receive delivery information about the live broadcast information stream. The delivery information includes real-time content value point information of the live broadcast information stream, which is determined based on the information of the target product corresponding to the real-time explanation object of the live broadcast information stream.

[0074] The search result providing unit is used to provide links to the live stream in the product search result information display page, based on the real-time content value information of the live stream, during the process of providing product search results according to the product search result request initiated by the user.

[0075] An apparatus for generating product demonstration videos, comprising:

[0076] An information retrieval unit is used to retrieve the live stream information stream and the product information in the product list associated with the live stream information stream;

[0077] The relevance judgment unit performs real-time analysis of the live stream information stream to determine the relevance between the real-time explanation object of the live stream information stream and the products in the explanation product list;

[0078] The time point determination unit is used to determine the start time point information and end time point information of explaining the target product in the product explanation list in the live broadcast information stream based on the correlation judgment result.

[0079] The video clip extraction unit is used to extract corresponding video clips from the live stream information stream based on the start time information and end time information, so as to deliver the video clips as explanatory videos of the target product to the target page.

[0080] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0081] An electronic device, comprising:

[0082] One or more processors; and

[0083] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of any of the preceding methods.

[0084] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0085] This application embodiment allows for the retrieval of product information from a live stream and its associated product listings. Real-time analysis of the live stream enables a correlation assessment between the live stream's content and the products in the product listings. Subsequently, the live stream can be distributed to a traffic distribution system, providing the system with information about the target products corresponding to the live stream's content. This allows the traffic distribution system to push the live stream to target users via target pages based on the target product information. This approach, by reaching users with the live stream content based on the target product information of the live stream's content, enables personalized recommendations at the "highlights" level within the live stream. This improves the efficiency of traffic redirection to the live stream and gives users more opportunities to discover more live streams of interest, enhancing the user experience.

[0086] In the specific process of relevance judgment, a visual representation model can be used to represent video frames in the live stream and product images in the product listing, thereby determining the relevance between the two. When selecting the backbone network structure of the visual representation model, a visual converter model was chosen as the backbone network. Specifically, the visual converter model can employ a shifted non-overlapping window design, perform self-attention mechanism calculations at the window level, apply multi-scale pooling in window partitioning, and layer features of different granularities through depthwise convolution. Gated aggregation is then used to aggregate contextual features of different granularities. This approach ensures that the model's efficiency and performance better meet the requirements of real-time generation of live stream "highlights" in this embodiment of the application.

[0087] Regarding the selection of the objective function for the model, a multi-similarity loss function can be used. In addition, under preferred conditions, a product category classification loss function can also be used as an auxiliary method. This allows different category features to be mapped to different subspaces. With the assistance of classification loss, metric loss can better distinguish between positive and negative samples.

[0088] During model training, a feature memory queue can be created to add historical training samples input into the visual representation model from previous training rounds, as well as new training samples input into the visual representation model from the current training round, to the feature memory queue. This allows for the mining of hard samples by calculating the similarity between the representation vectors of sample pairs within the feature memory queue. This approach enables sampling of hard samples over a wider range, improving the success rate of hard sample sampling. Furthermore, in a preferred embodiment, a momentum update model (a micro-simulation of parameter updates in the visual representation model) with the same model structure as the visual representation model can be constructed. When sampling hard samples from the feature memory queue, this momentum update model can be used to generate representation vectors for historical training samples, thus resolving the feature offset problem between old and new samples in the queue.

[0089] When initializing a visual representation model, a cross-modal unsupervised semantic alignment training method can be adopted. This supervision of different modalities can reduce the cost of sample labeling, enabling a large amount of unlabeled data to be effectively utilized. At the same time, it also brings more discriminative and robust information to image representation training.

[0090] Furthermore, during continuous learning of the visual representation model, a distillation learning approach is employed. The visual representation model is used as the teacher model, and a student model with the same model structure is constructed. The student model is trained and its parameters are updated using an expanded training sample set, guided by the teacher model. This approach reduces knowledge forgetting and performance degradation caused by the model needing to adapt to new data distributions and domain distributions during continuous training.

[0091] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0092] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0093] Figure 1 This is a schematic diagram of the system architecture provided in the embodiments of this application;

[0094] Figure 2 This is a flowchart of the first method provided in the embodiments of this application;

[0095] Figure 3This is a flowchart of the second method provided in the embodiments of this application;

[0096] Figure 4 This is a flowchart of the third method provided in the embodiments of this application;

[0097] Figure 5 This is a flowchart of the fourth method provided in the embodiments of this application;

[0098] Figure 6 This is a flowchart of the fifth method provided in the embodiments of this application;

[0099] Figure 7 This is a flowchart of the sixth method provided in the embodiments of this application;

[0100] Figure 8 This is a schematic diagram of the first device provided in the embodiments of this application;

[0101] Figure 9 This is a schematic diagram of the second device provided in the embodiments of this application;

[0102] Figure 10 This is a schematic diagram of the third device provided in the embodiments of this application;

[0103] Figure 11 This is a schematic diagram of the fourth device provided in the embodiments of this application;

[0104] Figure 12 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0105] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0106] First, it's important to clarify that while existing technologies can push live stream information to users across various traffic scenarios (e.g., recommendation scenarios like "You May Like" or search scenarios), the primary push logic is based on users' historical viewing history of the live stream or the main product categories involved. For example, if a user frequently watches a live stream about children's clothing, after that live stream starts, links to that live stream might be pushed to that user on some pages. Similarly, when other children's clothing-related live streams start, links might also be pushed to that user, allowing them to quickly access the live stream without having to return to the client's homepage to find the live stream channel and then locate the specific live stream within the channel page. While this approach can achieve a certain degree of traffic redirection, the specific recommendation logic is limited to live streams that the user frequently watches or follows, or those similar to those live streams.

[0107] In the process of developing this application, the inventors discovered that when consumers choose whether to enter a live stream, it is often closely related to the specific product being discussed in the live stream. If the product being discussed in a live stream happens to be of interest to the user, there is a higher probability that the user will click to enter the live stream. In this case, the information about the product being discussed in the live stream can become the content value point of the live stream (also known as "highlights," i.e., content or aspects that are worth watching or paying attention to), and can be recommended to the user. Therefore, in the embodiments of this application, live stream recommendations can be made to users based on this logic. That is, based on the product being discussed in the live stream, it is determined whether the user will be interested in the live stream; if so, it can be recommended to that user. For example, suppose a live stream is explaining a women's clothing product. The style of the product might interest a user who happens to be browsing the "You May Also Like" page or searching for women's clothing. A link to the live stream could be inserted (for example, into a specific resource slot on the page) on the "You May Also Like" page or the search results page. The live stream content could then be dynamically played in that resource slot. If the user clicks on that resource slot, they can be redirected to the live stream interface to watch it, and so on.

[0108] Of course, since live stream content is typically generated, streamed, and played in real time, achieving the aforementioned goals requires real-time analysis of the live stream content to determine which product is being discussed. This allows for live stream recommendations based on real-time viewing points. To achieve this, considering that merchants usually add products potentially discussed during a live stream to a list of products associated with the live stream (e.g., a "product bag" on the live stream page), the products being discussed are typically from this list. Therefore, by analyzing the live stream feed in real time and determining the correlation between the live stream's current topic and the products in the associated product list, the current product being discussed can be identified.

[0109] To perform the aforementioned product relevance assessment, various methods can be employed. One approach is to utilize computer vision analysis. Specifically, a visual representation model can be used to generate representation vectors for the live stream images and product images in the product list. Then, the relevance between the two can be determined by calculating the similarity between the representation vectors of the live stream images and the product representation vectors.

[0110] In practical implementation, real-time processing is required during live video streaming, and multiple live streams may be online simultaneously. Therefore, real-time processing of these multiple streams is also necessary to meet real-time requirements. Thus, the visual representation model needs high processing efficiency (specifically related to inference speed and computational complexity) to meet these requirements. Simultaneously, to improve the accuracy of relevance judgment, the performance of the visual representation model is also crucial; that is, the representation vector output by the visual representation model needs to have high accuracy to obtain accurate relevance judgment results. Therefore, to achieve the above objectives, this application also provides corresponding solutions regarding the selection of the backbone network of the visual representation model, the sampling method for hard samples during training, the initialization method of model parameters, and the continuous training method.

[0111] From a system architecture perspective, such as Figure 1As shown, this application embodiment can provide an online service with live stream highlight generation function in a product information service system. After a host starts a live stream through a certain live stream room and lists the products that may be discussed to the "product bag" associated with the live stream room, the above-mentioned online service can retrieve the specific live stream information stream and the product list information in the "product bag". Then, it can generate annotation vectors for the product images of each product in the product list and perform real-time analysis and processing on the video frames in the live stream information stream. This can be done by judging each frame in the live stream information stream separately, or by using a certain frame extraction strategy, such as extracting one frame per second. When analyzing and processing the video frames, since there may be images of multiple products in the live stream, but only one product may be being discussed, the product intent can be located first, selecting the image content that is currently being discussed, and then the representation vector can be extracted and compared with the feature vectors of the products in the "product bag" to determine product relevance. When a video frame's image content is found to be related to a product from a certain moment onwards, this can be used as the starting point for explaining that product. A timestamp can be added to the live stream feed, and the product's information can be used as the current highlight of the live stream. After the highlight is generated, the live stream feed can be distributed to a specific traffic distribution system. The traffic distribution system can review and distribute the content, and during distribution, it can recommend products to users who might be interested in them based on the current highlight information, i.e., the information of the product currently being explained.

[0112] In this application, the generation of representation vectors for product images in the "baby bag" and for image content in video frames can both be achieved using a pre-trained visual representation model. This visual representation model can be pre-trained offline using a model training platform. During offline training, training samples can be collected from media data (e.g., historical live stream data from the product information service system). Offline computing platforms can also perform various processing steps, including low-quality data filtering, manual annotation (correspondence between video clips and products), and training data structuring. After training, the model can be trained on the platform. Once trained, model compression can be used to accelerate the process, and the model can be deployed online for real-time generation of highlights from the live stream. The training process involves difficult sample collection and model initialization, both of which can be optimized in this embodiment. Furthermore, considering that new data will continuously be generated during online operation, which can be used as new training samples, continuous learning can be employed to update the model and improve its performance. During continuous learning, newly generated live stream information can be collected and processed through an offline computing platform, including low-quality data filtering, hard sample labeling, and training data structuring. Furthermore, since newly collected data may include data generated based on the model provided in this application's embodiments, to further improve the learning effect, the quality of specific highlights can be monitored after the live stream information is distributed based on highlight information. For example, the click-through rate after a broadcast based on a particular highlight can be monitored to statistically analyze highlight quality. Therefore, during the collection of new samples, some filtering based on highlight quality can also be performed, and so on. In particular, to prevent the model from forgetting previously learned "knowledge" during continuous learning, distillation learning can be used, and so on.

[0113] The specific implementation schemes provided in the embodiments of this application will be described in detail below.

[0114] Example 1

[0115] First, this first embodiment provides a method for delivering live stream information flow from the perspective of the aforementioned highlight generation service. See [link to previous document]. Figure 2 The method may specifically include:

[0116] S201: Retrieve the live stream information stream and the product information in the product list associated with the live stream information stream.

[0117] The specific live stream information flow can be a real-time live stream generated by the product information service system. Once merchants start their live stream through pre-registered live rooms, a specific live stream information flow is generated, which can include multiple frames of live content and audio information generated by the host explaining the products. The specific highlight generation service can retrieve the live stream information flow from the live channel's server. Since multiple live rooms may be broadcasting simultaneously, multiple live stream information flows can be retrieved, and real-time highlight generation services can be provided for each of them. Alternatively, merchants can select the live stream information flows for which real-time highlight generation services are needed, based on their needs or other strategies.

[0118] In addition, before or during the live stream, merchants or streamers may add links to products that may be involved in the live stream to the list of products to be explained in the live stream. Therefore, it is also possible to retrieve product information from the list of products to be explained in the live stream to make a relevance judgment with the objects being explained in the live stream.

[0119] S202: By performing real-time analysis on the live stream, the relevance between the real-time explanation object of the live stream and the products in the explanation product list is determined.

[0120] After retrieving the live stream feed, it can be analyzed in real time, and the relevance between the live stream's current product being discussed and the products in the product list can be determined. This means identifying which product is currently being discussed in the live stream, and then determining the current highlight of the live stream based on that product's information. In other words, for the same live stream feed, since multiple products are typically discussed in a single live stream, different highlights can exist at different times. Accordingly, recommendations can be made to different user groups based on these different highlights at different times, thus driving traffic to the live stream.

[0121] In this embodiment, when determining the relevance between the explained object in the live stream and the products in the product list, one possible approach is to generate representation vectors for both aspects of information based on visual representation, and then compare these representation vectors to determine the relevance between the explained object and the products in the live stream. In this case, a visual representation model can be used to accomplish the above task. That is, the visual representation model can be used to process the video frames of the live stream to generate representation vectors that can be used to semantically express the product information contained in the video frames. Alternatively, the same visual representation model can be used to process the product images of each product in the "Baby Bag" to generate representation vectors that semantically express each product in the "Baby Bag." Thus, the relevance between the two can be determined by calculating the similarity between the representation vectors corresponding to the video frames in the live stream and the representation vectors corresponding to each product in the "Baby Bag." After determining the correspondence between the video frames and the products, the position of the first video frame containing a certain product can be determined as the starting point for explaining that product in the live stream. For example, suppose a product A appears at frame 50 of a live stream. The location of frame 50 can be designated as the starting point for explaining product A. Accordingly, the live stream will begin showing highlights about product A from that starting point, and so on. Optionally, a timestamp can be added to this starting point to facilitate subsequent extraction of video segments based on the specific product being explained.

[0122] Regarding the video frames in the live stream, representation vectors can be generated for each frame individually, or a frame-by-frame extraction method can be used. For example, one frame can be extracted per second, and only the extracted video frame is used to generate representation vectors and calculate the similarity between the representation vectors and those of the products in the "Baby Bag". Due to the influence of the live stream environment, multiple product images may appear in the same video frame. For instance, suppose the host is explaining a piece of clothing at a certain moment, but other clothes are also hanging in the live stream. In this case, both the clothing being explained and the clothes hanging in the live stream may appear in the live stream frame; that is, the background of a single live stream frame may also include some product-related images. Therefore, to facilitate processing by the visual representation model, before inputting the video frame images into the visual representation model, the explanation intent can be recognized first. This involves identifying the image content that truly matches the current explanation intent from the extracted video frames and extracting this image content for input into the visual representation model for processing. In other words, after extracting video frames from the live stream, the intended meaning of the explanation can be located first in the image content of the video frames. This allows for the extraction of target image content belonging to the current real-time explanation subject. This target image content is then input into a visual representation model for computation, obtaining a representation vector that represents the semantic information of the target image content. Of course, in practical applications, after extracting the target image content, further processing such as cropping and scaling can be applied to ensure that the images input into the visual representation model have the same scale, thus improving the comparability of the output representation vectors.

[0123] In one specific implementation, when locating the explanation intent of image content in video frames, a recognition processing method combining multimodal descriptive information can be used. Specifically, since live streams include not only video frames but also audio information—which explains the specific products—identifying which image content in the video frame belongs to the currently being explained object is very helpful. Therefore, when locating the explanation intent, multimodal information can be obtained based on the image content in the video frame and the text content converted from the audio information in the live stream. This multimodal information is then used to locate the explanation intent of the image content in the video frame. In other words, based on the image frame and the host's narration, it can be analyzed which part of the image content is mainly presented in the current video frame, that is, which part of the image content is being explained in the live stream. This part of the image content is located and then input into a visual representation model to generate a specific representation vector. This vector is then compared with the representation vectors of the products in the "product bag" to determine which product this part of the image content corresponds to.

[0124] Furthermore, the products in the product list are usually pre-configured before the live stream begins and may remain unchanged throughout the entire stream. Therefore, it's advisable to generate representation vectors for each product in the list beforehand and save these vectors for later use in relevance assessments. Of course, if the live stream involves updates to the product list in the "Baby Bag," including adding new products, the representation vectors of the newly added products can be extracted for relevance assessment against the objects being discussed in the live stream. When generating representation vectors for products, a specific visual representation model can be used, which can be the same model used to generate representation vectors for video frames. Specifically, when generating representation vectors for products using this model, the product image can be used as input. If the product image contains complex backgrounds, the multimodal description information in the product description can be used for intent localization to determine the main content presented in the product image, which can then be extracted, cropped, and processed before being input into the visual representation model for further processing, and so on.

[0125] After obtaining the representation vectors of the products in the "Baby Bag," they can be saved. Subsequently, each time a representation vector is generated for a video frame extracted from the live stream, this representation vector can be compared with the representation vectors of each product in the "Baby Bag" to determine which product in the "Baby Bag" the current video frame corresponds to. Since the explanation of the same product in the live stream usually lasts for a period of time, there will be some correlation between different video frames explaining the same product. Therefore, when comparing the representation vector of the video frame with the representation vectors of each product in the "Baby Bag," we can first compare the representation vector of the current video frame with the representation vector of the product in the previous video frame. If the similarity is high, it proves that the same product is still being explained; if the similarity is below a threshold, it proves that the explanation of the next product may have started. In this case, the representation vector of the current video frame is then compared with the representation vectors of other products in the "Baby Bag," and so on.

[0126] In summary, the above methods can identify the corresponding product information for video frames extracted from a live stream. Since this process can be performed in real time, it's possible to determine which product is currently being discussed in the live stream. This product can then be used as the current highlight of the live stream. This highlight information can serve as the basis for specific traffic distribution systems (including product recommendation systems like "You May Also Like" or product search systems) to recommend live stream content. Furthermore, the highlights of the same live stream can differ at different times due to the different products being discussed. Based on this information, the same live stream can be recommended to different audiences, thus more effectively driving traffic to the live stream.

[0127] It's important to note that after identifying the product being discussed in the live stream feed, the product's title and other relevant information can be directly used as the highlight of the live stream. Alternatively, specific recommendation text can be generated based on the product information being discussed. This text can be provided to the traffic distribution system so that it can be displayed to users when recommending specific live stream feeds, thereby improving the success rate of traffic redirection. For example, if a live stream is discussing a particular product, text content can be generated based on the product's detailed images and descriptions. This text content can then serve as a reason for recommending the live stream feed. Furthermore, the generation of this text content can be combined with AI (Artificial Intelligence) models. Specifically, the AI ​​model can transform the multimodal descriptions of the product into its understanding of the product, and then generate the specific text content. This ensures that the generated recommendation text is not merely a simple extraction of product details, but rather text produced after the AI ​​model has fully understood the product's characteristics, guaranteeing vividness and coherence.

[0128] The above provides an overall overview of the process for generating highlights information in real-time from a live stream. This process involves a visual representation model. As mentioned earlier, since this embodiment requires real-time processing of video frames in the live stream and centralized processing of multiple live streams on the server side, the visual representation model faces high demands in terms of efficiency and performance. Next, we will introduce the optimization methods used in this embodiment regarding the selection of the backbone network and training methods of the visual representation model.

[0129] (I) Selection of the backbone network for the visual representation model

[0130] The visual representation model used in the embodiments of this application can be classified as a VIT (Vision Transformer) model. In theory, any general VIT model can be used as the visual representation model in these embodiments. However, the inventors discovered during the implementation of this application that VIT network structures typically lack local prior knowledge, resulting in weaker modeling capabilities for low-level visual features such as edge shapes compared to CNNs (Convolutional Neural Networks), and model training also faces convergence difficulties. Therefore, subsequent academic work has gradually introduced convolutional operations into VITs, or innovatively implemented prior knowledge of hierarchy, locality, and translation invariance into the network. For example, the main idea of ​​the Swin Transformer (SwinT) is to mimic the local convolutional operations of CNNs, employing a shifted, non-overlapping window design, performing self-attention mechanism calculations at the window level, while ensuring information exchange between windows. To address the multi-scale problem, Focal Transformer (FocalT), compared to SwinT, applies multi-scale pooling in window partitioning to capture more visual features of different granularities. Furthermore, due to the low computational efficiency of secondary self-attention, which employs a process of first interacting with contextual information and then aggregating it, the implementation of focal attention in FocalT is relatively complex. Therefore, FocalNET introduces Focal Modulation, specifically, using depthwise convolutions to stratify features of different granularities, and then aggregating the contextual features of different granularities through gating aggregation.

[0131] After determining the structure of the actual representation model, the next step is to train the model. Specifically, this can be done by selecting manually selected viewpoint data from historical live streaming data (for example, the host adds markers during the live stream, generating video clips after the stream ends, each associated with product information). This data is then filtered out using methods such as identifying "spotted images," watermarked images, and detailed images. Furthermore, after balancing the sample distribution (e.g., achieving a certain balance in the data distribution across multiple product categories), manual annotation can be performed to further identify the correspondence between specific video clips and products. Since each video clip can include multiple video frames, multiple image-product correspondences can be obtained. This process constructs an offline training and evaluation training set, iterates the algorithm model, and after model compression and acceleration, it can be deployed online.

[0132] (II) Objective Function in the Training Process of Visual Representation Models

[0133] In the specific training process of the aforementioned visual annotation model, the objective function used may include a multi-similarity loss function. For example, multi-similarity loss provides a general sample weighting framework in which the weight of positive sample pairs decreases as similarity increases, while the weight of negative sample pairs increases as similarity increases (positive and negative sample pairs will be discussed later). Additionally, in a preferred implementation, a classification loss function can be added to assist model training. Specifically, the classification loss optimizes the model's learning of a hyperplane in the feature space, mapping different class features to different subspaces. With the assistance of the classification loss, the metric loss can better distinguish between positive and negative samples.

[0134] Specifically, during the training process, each round of training can use a target batch (e.g., 128) of training samples as input information. This target batch of training samples can include multiple positive samples and multiple negative samples. Specifically, positive samples are multiple different images corresponding to the same product, and negative samples are multiple different images corresponding to other products. That is, the same batch of input samples can guarantee that it includes multiple different images corresponding to the same product (which can be any product) to construct positive samples. Additionally, this batch of samples can include images corresponding to other products, or even images corresponding to products from other categories, etc. In this way, different images of the same product can constitute positive samples, and images of different products become negative samples. During training, after the visual representation model outputs representation vectors for each training sample, the training samples can be paired into sample pairs. By comparing the similarity of the representation vectors within multiple sample pairs, the multi-similarity loss function can be optimized to update the parameter values ​​of the visual representation model. For example, during the optimization process, the goal can be to reduce the distance between positive sample pairs and increase the distance between negative sample pairs. This can be achieved by optimizing the multi-similarity loss function. Optimizing the multi-similarity loss function involves adjusting the model parameters, and the adjusted model parameters can be used for the next round of training, and so on.

[0135] In addition to using a classification loss function, the aforementioned visual annotation model can predict the category distribution of a specific image and compare it with the actual category distribution of the image. The goal is to optimize the model parameters by making the two distributions as close as possible. When predicting the category distribution, categories at the same level can be used. For example, the prediction can focus on the first-level categories within a category system. Assuming there are 10 first-level categories, and an image actually belongs to the first category, the actual category distribution for that image would be (1, 0, 0, 0, 0, 0, 0, 0, 0), meaning the first category is 0 and all other categories are 0, indicating that the image belongs to the first category. If, after inputting this image into the visual representation model, the predicted category distribution is (0.6, 0.3, 0.18, 0.02, 0.01…), the performance of the current model can be evaluated by comparing the similarity between the predicted and actual category distributions. The model parameters can then be optimized to make the predicted distributions as close as possible to the actual distributions.

[0136] (III) On the sampling of hard samples in the training process of visual representation models

[0137] In the aforementioned process of optimizing the multi-similarity loss function, deep metric learning is involved. An important issue in deep metric learning is the mining of hard samples so that the multi-similarity loss function can be optimized while assigning higher weights to these hard samples.

[0138] To facilitate understanding, a brief introduction to the concept of "hard samples" is provided below. Generally speaking, training sets can be divided into hard samples and easy samples based on the ease of learning. As the names suggest, hard samples are those that are difficult to learn (high loss), while easy samples are those that are easy to learn (low loss). In product recognition scenarios, hard samples are usually located in the ambiguous regions where different products are distinguished, reflecting discriminability. This difference and informationality of hard samples can improve the model's ability to distinguish between different products.

[0139] In the scenario corresponding to the embodiments of this application, since the same batch of samples needs to include positive samples (that is, multiple different images corresponding to the same product) and negative samples (that is, images corresponding to different products or different categories of products), the specific difficult samples can be divided into: difficult samples in positive samples and difficult samples in negative samples.

[0140] Specifically, "difficult samples" among positive samples refer to samples that correspond to the same product but have very different features. For example, suppose a batch of input images contains 80 positive samples, all of which correspond to the same product (product A). However, two of these samples have image features that are very different from the other samples. These two samples can be called difficult samples among positive samples.

[0141] The difficult samples in the negative samples can specifically refer to samples that correspond to different products or different categories of products, but whose image features have very small differences. For example, suppose there are 40 negative samples in the same batch of input images, that is, the products corresponding to these samples are not the aforementioned product A. However, two of the samples have very small differences in image features from product A. In other words, although they correspond to different products, the images are very similar. These two samples can be called the difficult samples in the negative samples.

[0142] During contrastive learning, harder samples can be assigned higher weights. This leads to better optimization results when using contrastive learning to refine the multi-similarity loss function, resulting in a more robust model that performs better in complex scenarios. Therefore, identifying harder samples is crucial for improving model performance.

[0143] In general, hard sample sampling is performed within the input samples of the current training epoch. For example, after generating representation vectors for each input sample in the current epoch for a specific model, these 128 samples are paired, and the similarity between the representation vectors within each pair is calculated. Then, knowing whether the products corresponding to each sample are the same, it can be determined whether a specific sample is a hard sample. In other words, assuming 128 samples are input into the model in each training epoch, the hard samples can only be determined from these 128 samples, which may result in a relatively small number of hard samples being identified. For example, a sample may actually be a hard sample, but due to the limited number of samples input in the current training epoch, it may be difficult to discover this hard sample among these samples. Of course, the success rate of sampling hard samples can be increased by increasing the number of input samples in each training round (usually known as mini-batch). However, the number of input samples in each round is related to the size and quantity of GPU (Graphics Processing Unit) memory. In other words, the number of input samples in each round cannot be expanded indefinitely, but is limited by the size and quantity of GPU memory.

[0144] To overcome the limitation on the number of input samples in each training round and sample over a wider range to obtain more valuable hard samples, a preferred embodiment of this application employs a feature memory queue method. That is, after each training round, the feature vectors of the input samples (which can be called new samples) from that round are added to the queue. This ensures that the queue includes not only the input samples from the current round but also those from previous rounds. The queue length is finite (but greater than the number of samples input into the model in each round). Once the queue is full, samples that entered the queue earlier can be removed first, and so on. In this case, within the queue range, hard sample pairs are mined by calculating the similarity between the representation vectors of sample pairs. In other words, assuming that the number of samples in each mini-batch is 128 and the length of the queue is 1000, the above scheme can sample difficult samples within the range of 1000 samples instead of within the range of 128 samples. Therefore, it can overcome the limitation of mini-batch with low memory consumption and sample more difficult samples with higher information value.

[0145] However, in practical implementation, the above method of sampling hard samples through a feature memory queue has a drawback: the feature shift within the queue due to model optimization updates affects the representation vectors of new and old samples. In other words, assuming a queue length of 1000 and a mini-batch length of 128, the earliest samples entering the queue might still be samples input from the first 9 rounds. Correspondingly, the representation vectors of these samples are still generated using the "old" model from 9 rounds ago. Meanwhile, the representation vectors of the latest samples entering the queue are generated by the "new" model after 9 rounds of model updates. The parameter values ​​of the updated model after 9 rounds may have changed significantly compared to 9 rounds ago, which reduces the comparability between the representation vectors of new and old samples.

[0146] To address the aforementioned issues, in a preferred embodiment of this application, momentum updates can be used to control the feature shift problem. Specifically, during the sampling of hard samples within the feature memory queue, a momentum update model (temporarily referred to as model B) with the same model structure as the visual representation model (for ease of description, let's call it model A) can be constructed. When the parameter values ​​of the visual representation model (i.e., model A) are updated, the parameter update direction of the momentum update model (model B) depends not only on the update direction of model B itself but also on the parameter update direction of model A. For example, assuming the parameter α of model A is α = a, and the parameter α of model B is α = b, then the new value of parameter α after momentum update in model B can be: α = γ*b + (1-γ)a, where γ can be a decimal between 0 and 1, for example, 0.9, etc. That is, model A is a normally trained model, while model B can have the same model structure as model A, but its parameter update method is different from that of model A. Model A updates its parameters normally based on the results of comparative learning, while Model B approximates the normally trained Model A to a certain extent after each parameter value update, but only by a small amount. Therefore, when sampling hard samples from the queue, the momentum update model, i.e., Model B, can be used to generate representation vectors for historical training samples in the queue.

[0147] In other words, given the existence of both Model A and Model B (Model A is a normally trained model, and Model B is a momentum update model that partially approximates Model A), when sampling hard samples from the feature memory queue, the representation vector of the newly entered sample can be generated by Model A. However, for historical samples input into the model in previous rounds, Model B can be used to generate the representation vector. This makes the representation vectors of each sample in the queue more comparable. Then, by pairing the samples in the queue, multiple sample pairs are obtained. By comparing the similarity between the representation vectors of the two samples within the same pair, it is possible to determine which samples are hard samples, given that the products corresponding to each sample are the same. This allows for improved sampling efficiency for these hard samples when optimizing the multi-similarity loss function.

[0148] (iv) Initialization before training the visual representation model

[0149] The above has introduced the objective function determination and hard sample sampling in the training process of visual representation models. In practical implementation, before training a specific model, model initialization is usually involved. Model initialization, after determining the model's structure, involves initializing the parameter values; that is, obtaining a set of initial parameter values ​​for the model, and then performing the training process based on these initialized parameter values. Model initialization is a crucial issue affecting the convergence speed and performance of deep learning model training.

[0150] However, the inventors of this application discovered in the process of realizing this application that in the field of vision, when performing downstream tasks such as image classification, retrieval, detection and segmentation, the previous model initialization paradigm is generally to pre-train a set of model parameters in a supervised manner on the basis of a public image dataset, load them into the model for initialization, and then continue to train the model according to the specific downstream task in order to fine-tune the model parameters.

[0151] However, the aforementioned model initialization scheme involves image annotation (specifically classifying images into a limited number of label clusters). Large-scale image annotation is costly and renders a large amount of unlabeled image data unusable. Therefore, in this embodiment, during model initialization, self-supervised contrastive learning can be performed with a larger dataset, allowing visual representations to be mapped to a larger feature space and achieving more generalized representation capabilities.

[0152] For example, a massive number (e.g., up to 20 million or more) of live-stream product images can be extracted and detected from historical live-stream data to build a dataset. Then, self-supervised comparative learning can be performed, and the trained model can be used as an initial model for offline fine-tuning training of fine-grained visual representations of live-stream products.

[0153] Furthermore, cross-modal pre-training methods have received considerable attention in recent years. By encoding the input image and text separately using image encoders and text encoders, and then performing comparative learning training, superior learning performance can be achieved. Therefore, in a preferred embodiment of this application, the limitations of a single modality can be further overcome. A massive amount of image-text pairs collected through sampling (i.e., image-text pairs consisting of product images and text information such as product titles) can be used to obtain a pre-trained image representative with stronger semantic representation through cross-modal comparative learning methods. Here, the matching of text modality and image modality can be seen as data augmentation for the image modality, and supervision from different modalities can bring more discriminative and robust information to the image representation training.

[0154] In order to improve the sample utilization rate of initial training, the specific initial training implementation schemes can be divided into three types of tasks: single-model self-supervised contrastive learning, cross-modal contrastive learning, and nearest-neighbor modal contrastive learning.

[0155] Regarding single-model self-supervised contrastive learning, this means that self-supervised learning can be performed separately on the image side and the text side. On the image side, methods such as random cropping and deformation can be used to produce multiple enhanced versions of the same original image. Self-supervised learning is achieved by comparing the feature vectors corresponding to the original image and the enhanced version image.

[0156] On the text side, models such as BERT (Bidirectional Encoder Representation from Transformers, a pre-trained language representation model) can be used as the backbone network, and self-supervised learning can be performed based on the MLM (Masked Image Modeling) paradigm. For example, enhanced versions of text data can be generated through text rewriting, and self-supervised learning can be achieved by comparing the feature vectors corresponding to the original text and the enhanced text. For example, an original text of "Hong Kong style white shirt men's short-sleeved summer solid color Korean style slim fit business shirt trendy shirt youth shirt cotton" can be rewritten as "Hong Kong style trendy white shirt men's short-sleeved summer solid color Korean style tight fit business shirt trendy shirt youth shirt", and so on.

[0157] Furthermore, cross-modal contrastive learning can be performed. On the image side, a set of positive image sample pairs is generated through data augmentation; on the text side, a set of positive text sample pairs is generated through text rewriting. These can then be combined pairwise to form four cross-modal sample pairs. For example, an image-text pair may include the original image and original text about a product. After data augmentation, corresponding augmented images and augmented text can be obtained. These can then be combined pairwise to obtain four cross-modal sample pairs: (original image, original text), (original image, augmented text), (augmented image, original text), and (augmented image, augmented text). Contrastive learning can then be performed based on these cross-modal sample pairs. That is, the representation vector generated by the image model for the original image can be compared with the representation vector generated by the text model for the original text. The similarity between the two can be used to optimize specific model parameter values, and so on.

[0158] In other words, when initializing the visual representation model, multiple information pairs consisting of product images and product text descriptions can be collected. For each information pair, the product image is input into the visual representation model to extract image features, and the product text description is input into the text feature extraction model to extract text features. Then, based on the similarity between the image features and the text features, unsupervised semantic alignment training is performed on the visual representation model and the text feature extraction model. The parameters of the trained visual representation model are used as the initialization parameters of the visual representation model so that the visual representation model can be trained based on the initialization parameters.

[0159] Regarding nearest-neighbor modality contrastive learning, it's considered that identical images may have similar linguistic descriptions. Therefore, during training, a dynamic text feature queue of a certain length (e.g., 64K) can be constructed. That is, multiple image pairs and their corresponding texts can be added to a single text feature queue. Then, the similarity between the image features and the texts in the queue is calculated, and the texts with higher similarity are selected as extended image-text pairs for contrastive learning training. This allows for sampling of semantically similar samples across a wider range, not only comparing within the current image-text pair, thereby further improving the initial training effect.

[0160] (V) Continuous Learning of Visual Representations

[0161] Typically, a model needs to possess automated self-updating capabilities and good scalability. In the live streaming content scenario of this application embodiment, the live streaming format and live streaming product trends evolve with consumer preferences. For example, the rapid listing of new seasonal products and related products for trending events leads to significant changes in the relevant products within the live streaming ecosystem. Therefore, the need for a live streaming product content representation algorithm with continuous learning capabilities has become a pressing issue. Considering the unique attributes of live streaming highlights, daily live streaming product data monitored by the streamer provides a natural source of noisy, labeled training samples for continuous learning solutions.

[0162] However, during continuous learning, training the model with new extended data may cause the model to "forget" the knowledge learned from the original data. To address this, in this embodiment, to reduce knowledge forgetting and performance degradation caused by adapting to the domain distribution of new data during training, a distillation learning approach can be used. The online model currently in operation is used as the teacher model, and a copy is used as the student model. The constructed extended dataset can be input into the teacher and student models respectively, yielding feature representation vectors for each model. If a classification loss function is also present, category distribution vectors obtained from category prediction by both models can also be obtained. Then, the consistency of the distribution can be calculated using the representation vectors output by the teacher and student models. When calculating the loss, the parameters of the teacher model are not updated; only the parameters of the student model are updated. The representation vectors of the teacher model and the predicted category distribution vectors are used to supervise the learning of the student model. After training is complete, the performance of the teacher model and the student model can be compared using the evaluation dataset. If the student model is found to be better optimized, meaning it outperforms the teacher model on the evaluation set, then the student model can be replaced online as the teacher model for the next round. Of course, if the student model does not show any performance gain compared to the teacher model, then the original teacher model will continue to be used online.

[0163] Specifically, in the distillation learning process described above, two distillation loss functions can be introduced: the KL divergence (Kullback-Leibler divergence, a method for measuring the degree of matching between two distributions) loss of the category prediction distribution and the distance loss of the mini-batch feature similarity matrix.

[0164] By cleaning and reusing online data and continuously training and updating the model, the performance degradation caused by changes in product content trends can be effectively alleviated, the problem of low-cost or cost-free annotation can be solved, and low-cost knowledge learning can be achieved.

[0165] S203: The live stream information flow is delivered to the traffic distribution system, and the traffic distribution system is provided with information on the target product corresponding to the real-time explanation object, so that the traffic distribution system can push the live stream information flow to the target user group through the target page based on the information on the target product corresponding to the real-time explanation object.

[0166] The above provides a detailed introduction to the process of generating highlights in live stream information feeds, as well as the visual representation models involved and their optimization methods during training. After analyzing a specific live stream information feed in real time using the aforementioned visual representation model and determining the correspondence between the real-time explanation object of the live stream information feed and the products in the explained product list, the target product corresponding to the current real-time explanation object of the live stream information feed can be determined. Then, the information of the target product can be used as the current highlight of the live stream information feed and delivered to the traffic distribution system. It should be noted that when the specific highlight generation system discovers that a live stream information feed has started explaining a product, it can deliver the live stream information feed to the traffic distribution system and provide information about the highlights of the live stream information feed (e.g., associated product information). Since the specific object delivered to the traffic distribution system is the live stream information feed, that is, the link address of the live stream information feed can be directly provided to the traffic distribution system. Correspondingly, the traffic distribution system can also directly recommend this link address of the live stream information feed to consumer users. In other words, what is actually recommended to consumers is a link to this live stream information stream (rather than product demonstrations or clips), allowing users to directly enter the corresponding live stream room by clicking the link.

[0167] The traffic distribution system can specifically include product recommendation systems such as "You May Like," or product search systems, etc. In this way, the specific traffic distribution system can determine users who might be interested in a particular live stream based on its current viewing status, and recommend this live stream through the corresponding front-end page. That is, when a user visits this "You May Like" page, assuming a live stream is explaining a product, the "You May Like" server (i.e., the traffic distribution end) determines that the user might be interested in that product, and can then insert a link to that live stream into a resource slot (such as a display area, typically used to show recommended product information cards) on the "You May Like" page, so that the user can see the live stream in one of these resource slots when browsing "You May Like" products. The live stream feed can play automatically within the resource slots on pages such as "You May Also Like," allowing users to view real-time explanations within the feed (there may be a short delay between this feed and the content played on the actual live stream page). If a user clicks on the resource slot, they can be redirected to the corresponding live stream page to watch, and so on.

[0168] Furthermore, as mentioned earlier, if text content about the current content value points (highlights) is generated based on the specific target product information being explained in the live stream, this text content can be provided to the traffic distribution system when the live stream is delivered. Correspondingly, when the traffic distribution system recommends the live stream to users, it can also provide this text content to users. This helps to further increase users' interest in the live stream and thus improve the success rate of traffic redirection to the live stream.

[0169] In summary, through the embodiments of this application, information about products in a live stream and its associated product list can be retrieved. By performing real-time analysis of the live stream, the relevance between the live stream's content and the products in the product list can be determined. Subsequently, the live stream can be distributed to a traffic distribution system, providing the system with information about the target products corresponding to the live stream's content. This allows the traffic distribution system to push the live stream to target users through target pages based on the target product information. In this way, because the live stream can reach users based on the target product information of the live stream's content, personalized recommendations can be achieved at the "highlights" level within the live stream, improving the efficiency of traffic redirection to the live stream and giving users more opportunities to discover more live streams of interest, thus enhancing the user experience.

[0170] In the specific process of relevance judgment, a visual representation model can be used to represent video frames in the live stream and product images in the product listing, thereby determining the relevance between the two. When selecting the backbone network structure of the visual representation model, a visual converter model was chosen as the backbone network. Specifically, the visual converter model can employ a shifted non-overlapping window design, perform self-attention mechanism calculations at the window level, apply multi-scale pooling in window partitioning, and layer features of different granularities through depthwise convolution. Gated aggregation is then used to aggregate contextual features of different granularities. This approach ensures that the model's efficiency and performance better meet the requirements of real-time generation of live stream "highlights" in this embodiment of the application.

[0171] Regarding the selection of the objective function for the model, a multi-similarity loss function can be used. In addition, under preferred conditions, a product category classification loss function can also be used as an auxiliary method. This allows different category features to be mapped to different subspaces. With the assistance of classification loss, metric loss can better distinguish between positive and negative samples.

[0172] During model training, a feature memory queue can be created to add historical training samples input into the visual representation model from previous training rounds, as well as new training samples input into the visual representation model from the current training round, to the feature memory queue. This allows for the mining of hard samples by calculating the similarity between the representation vectors of sample pairs within the feature memory queue. This approach enables sampling of hard samples over a wider range, improving the success rate of hard sample sampling. Furthermore, in a preferred embodiment, a momentum update model (a micro-simulation of parameter updates in the visual representation model) with the same model structure as the visual representation model can be constructed. When sampling hard samples from the feature memory queue, this momentum update model can be used to generate representation vectors for historical training samples, thus resolving the feature offset problem between old and new samples in the queue.

[0173] When initializing a visual representation model, a cross-modal unsupervised semantic alignment training method can be adopted. This supervision of different modalities can reduce the cost of sample labeling, enabling a large amount of unlabeled data to be effectively utilized. At the same time, it also brings more discriminative and robust information to image representation training.

[0174] Furthermore, during continuous learning of the visual representation model, a distillation learning approach is employed. The visual representation model is used as the teacher model, and a student model with the same model structure is constructed. The student model is trained and its parameters are updated using an expanded training sample set, with the teacher model serving as guidance. This method can reduce the problems of knowledge forgetting and performance degradation caused by the domain distribution of new data during continuous training.

[0175] Example 2

[0176] This second embodiment primarily introduces a live stream recommendation method based on real-time viewing points, specifically when the product recommendation system is used as a traffic distribution system. Specifically, this second embodiment provides a product information recommendation method. (See [link to relevant documentation]). Figure 3 The method may include:

[0177] S301: Receive information about the delivery of live streaming information stream, the delivery information including real-time content value point information of the live streaming information stream, the real-time content value point information being determined based on the information of the target product corresponding to the real-time explanation object of the live streaming information stream;

[0178] The specific delivery information can be submitted by the aforementioned real-time highlights generation service, and the correspondence between the current subject of the live broadcast information stream and the target product can be determined by real-time online analysis using the aforementioned visual representation model.

[0179] S302: During the process of recommending product information to users, the live stream information stream is recommended to the target user based on the real-time content value information of the live stream information stream, so as to provide a link to the live stream information stream in the product recommendation information display page shown to the target user.

[0180] In practice, when recommending products to a target user, the user's level of interest in the target product can be determined first. If the level of interest meets the criteria, a link to the live stream can be inserted into the set of recommended product information provided to the target user, so that the link to the live stream can be displayed on the product recommendation information display page.

[0181] Example 3

[0182] This third embodiment primarily introduces a method for reaching users with live streaming information based on real-time viewpoints when the product search system is used as a traffic distribution system. Specifically, this third embodiment provides a method for providing product search result information. See [link to documentation]. Figure 4 The method may include:

[0183] S401: Receive information about the delivery of live streaming information stream, the delivery information including real-time content value point information of the live streaming information stream, the real-time content value point information being determined based on the information of the target product corresponding to the real-time explanation object of the live streaming information stream;

[0184] S402: During the process of providing product search results based on the product search request initiated by the user, the link information about the live broadcast information stream is provided on the product search results information display page based on the real-time content value point information of the live broadcast information stream.

[0185] Specifically, after a target user initiates a search, it can be determined that the target product corresponding to the current real-time explanation object of the specific live broadcast information stream meets the search criteria. If so, a link to the live broadcast information stream can be inserted into the product search results set so that the link to the live broadcast information stream can be provided on the product search results information display page.

[0186] Example 4

[0187] This fourth embodiment mainly addresses the difficult sample acquisition scheme involved in the first embodiment, providing a method for training a visual representation model. For details, see [link to example]. Figure 5 The method may include:

[0188] S501: Obtain a training sample set, wherein the training sample set includes multiple images and corresponding annotation information;

[0189] S502: Using the multi-similarity loss function as the objective function, the visual representation model is subjected to comparative learning in multiple training rounds using the training sample set, so as to update the parameter values ​​of the visual representation model.

[0190] In the process of comparative learning, hard sample sampling is also performed so that the multi-similarity loss function can be optimized with the hard samples being given higher weights.

[0191] During the sampling of difficult samples, a feature memory queue is created so that historical training samples input into the visual representation model in previous training rounds, as well as new training samples input into the visual representation model in the current training round, are added to the feature memory queue.

[0192] A momentum update model with the same model structure as the visual representation model is constructed, and the parameter update direction of the momentum update model is determined by partially approximating the visual representation model.

[0193] When sampling hard samples, the momentum update model is used to generate representation vectors for historical training samples in the feature memory queue, and sample pairs are constructed within the feature memory queue. The sampling of hard samples is performed by calculating the similarity between the representation vectors of the sample pairs.

[0194] Example 5

[0195] This fifth embodiment addresses the implementation scheme mentioned in Embodiment 1, which initializes the visual annotation model in a cross-modal, unsupervised manner. It provides a method for initializing the visual representation model. See [link to previous embodiment]. Figure 6 The method may include:

[0196] S601: During the parameter initialization process of the visual representation model, multiple information pairs consisting of image and text description information are collected.

[0197] S602: For each information pair, the image is input into the visual representation model to extract image features, and the text description information is input into the text feature extraction model to extract text features.

[0198] S603: Based on the similarity between the image features and the text features, perform cross-modal unsupervised semantic alignment training on the visual representation model and the text feature extraction model, and use the parameters of the visual representation model after training as the initialization parameters of the visual representation model, so that the visual representation model can be trained using the dataset in the downstream application based on the initialization parameters.

[0199] Example 6

[0200] In the aforementioned Embodiment 1, after generating real-time highlights of the live stream, the live stream is delivered based on these highlights, allowing it to reach users through a specific traffic distribution system. In practical applications, because the correspondence between specific real-time explained objects and products in the live stream can be established, the automatic generation of product explanation video clips can also be achieved, eliminating reliance on manual adjustments by the broadcaster or other users. Therefore, Embodiment 6 of this application also provides a method for generating product explanation videos, see [link to embodiment]. Figure 7 The method may specifically include:

[0201] S701: Retrieve the live stream information stream and the product information in the product list associated with the live stream information stream;

[0202] S702: By performing real-time analysis on the live stream, the relevance between the real-time explanation object of the live stream and the products in the explanation product list is determined;

[0203] S703: Based on the correlation judgment result, determine the start time and end time information of the explanation of the target product in the product list in the live broadcast information stream.

[0204] Since the correlation between the real-time narration object in the live stream and the products in the narrated product list can be determined, the start and end time information of the narration of the target product in the narrated product list in the live stream can be obtained accordingly. For example, assuming that video frame 1 is extracted from the live stream at time t1, and video frame 2 is extracted from the live stream at time t2 (where t1 and t2 are two adjacent frame extraction times), and it is determined that the main narrated product corresponding to video frame 1 is product 1, and the main narrated product corresponding to video frame 2 is product 2, then the start time of the narration of product 2 can be roughly determined to be time t2. That is, time t2 is the start point of the narration of product 2, and time t1 is the end point of the narration of product 1, and so on. Of course, in the above-mentioned embodiment one solution, at time t2, the live stream can be delivered to the traffic distribution system, providing information about product 2 being narrated at that time. At the same time, a timestamp can be added to time t2 to record that time as the start time of the narration of product 2. When the end time of product 2 is determined later, a corresponding timestamp can also be added.

[0205] S704: Based on the start time information and end time information, extract the corresponding video segment from the live stream information stream so that the video segment can be used as an explanation video for the target product and delivered to the target page.

[0206] Because the start and end times of the product demonstration in the live stream can be determined in real time, the video explanatory segments can be automatically extracted based on this time information, eliminating the need for manual adjustments by the host or other users. After generating the video explanatory segments, they can be provided to merchants and other users, allowing them to distribute them across target pages. For example, they can be placed on product detail pages, or used as the main product image in product recommendations or search systems. This way, users can view the video explanatory segments in specific locations on product recommendation pages or search results pages, instead of just static product images.

[0207] For the parts of Embodiments 2 to 6 that are not described in detail above, please refer to Embodiment 1 and other parts of this specification. They will not be repeated here.

[0208] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0209] Corresponding to Embodiment 1, this application also provides a live streaming information stream delivery device, see [link to embodiment]. Figure 8 The device may include:

[0210] Information retrieval unit 801 is used to retrieve the live broadcast information stream and the product information in the product list associated with the live broadcast information stream;

[0211] The relevance judgment unit 802 is used to perform real-time analysis of the live broadcast information stream to judge the relevance between the real-time explanation object of the live broadcast information stream and the products in the explanation product list;

[0212] The delivery unit 803 is used to deliver the live stream information flow to the traffic distribution system and provide the traffic distribution system with information on the target product corresponding to the real-time explanation object, so that the traffic distribution system can push the live stream information flow to the target user group through the target page according to the information on the target product corresponding to the real-time explanation object.

[0213] In a specific implementation, the device may further include:

[0214] The text content generation unit is used to generate text content about the current content value points of the live information stream by performing model understanding on the multimodal descriptive information associated with the target product corresponding to the real-time explanation object before the live information stream is delivered to the traffic distribution system.

[0215] The delivery unit can specifically be used for:

[0216] The live stream information flow is delivered to the traffic distribution system, and the text content of the current content value point is provided to the traffic distribution system so that when the traffic distribution system pushes the live stream information flow to the target user group through the target page, it also provides the text content of the current content value point.

[0217] Specifically, the correlation determination unit may include:

[0218] The explanation intent localization subunit is used to extract video frames from the live information stream and locate the explanation intent of the image content in the video frames so as to extract the target image content belonging to the current real-time explanation object.

[0219] The representation vector generation subunit is used to obtain a representation vector that represents the semantic information of the target image content by inputting the target image content into the visual representation model for calculation.

[0220] The matching calculation subunit is used to determine the correlation between the real-time explanation object and the products in the explanation product list by matching the representation vector with the representation vectors corresponding to the product images of multiple products in the pre-obtained explanation product list.

[0221] Specifically, the explained intention positioning subunit can be used for:

[0222] Based on the image content in the video frame and the text content converted from the voice information in the live stream, multimodal information is obtained, and the narration intention of the image content in the video frame is located based on the multimodal information.

[0223] The visual representation model structure includes: a visual converter model as the backbone network, with a fully connected layer added on top of the backbone network. The fully connected layer is used to convert the output information of the backbone network into a representation vector of the target dimension. The visual converter model adopts a shifted non-overlapping window design, performs self-attention mechanism calculation at the window level, applies multi-scale pooling in window partitioning, and layers features of different granularities through convolution in the depth direction. The context features of different granularities are aggregated through gating aggregation.

[0224] During the training of the visual representation model, the objective function used in the training process includes: a multi-similarity loss function;

[0225] Each training round uses a target batch of training samples as input information. The target batch of training samples includes multiple positive samples and multiple negative samples. The positive samples are multiple different images corresponding to the same product, and the negative samples are multiple different images corresponding to other products. The training samples are obtained based on video clips extracted from historical live stream information streams and corresponding product annotation information.

[0226] After the visual representation model outputs representation vectors for each training sample, the training samples are paired up into sample pairs, and the similarity of the representation vectors within multiple sample pairs is compared to optimize the multi-similarity loss function, so as to update the parameter values ​​of the visual representation model.

[0227] Furthermore, during the optimization of the multi-similarity loss function, hard samples can be sampled to optimize the multi-similarity loss function while assigning higher weights to the hard sample pairs. Specifically, when sampling hard samples, a feature memory queue can be created. Historical training samples input into the visual representation model from previous training rounds, as well as new training samples input into the visual representation model from the current training round, are added to the feature memory queue. Within the feature memory queue, hard samples are mined by calculating the similarity between the representation vectors of sample pairs.

[0228] Specifically, during the sampling of difficult samples within the feature memory queue, a momentum update model with the same model structure as the visual representation model is constructed, and the parameter update direction of the momentum update model is determined by partially approximating the visual representation model.

[0229] When sampling hard samples from the feature memory queue, the momentum update model is used to generate representation vectors for historical training samples.

[0230] In addition, the output information of the visual representation model may also include prediction information on the category distribution of goods included in the input image; in this case, the objective function also includes a product category classification loss function to assist in training the visual representation model.

[0231] Furthermore, when initializing the visual representation model, multiple information pairs consisting of product images and product text descriptions can be collected. For each information pair, the product image is input into the visual representation model to extract image features, and the product text descriptions are input into the text feature extraction model to extract text features. Then, based on the similarity between the image features and the text features, cross-modal unsupervised semantic alignment training is performed on the visual representation model and the text feature extraction model. The parameters of the trained visual representation model are used as the initialization parameters of the visual representation model so that the visual representation model can be trained based on the initialization parameters.

[0232] In addition, a dynamic text feature queue can be constructed, and the text features from multiple information pairs can be added to the dynamic text feature queue. The extended text of the image can be obtained by comparing the similarity between the image features and multiple text features in the dynamic text feature queue, so that the image and the extended text can be combined into an information pair and added to the unsupervised semantic alignment training.

[0233] Furthermore, the visual representation model can be continuously learned using the newly acquired extended training sample set. During the continuous learning process, a distillation learning method is adopted, using the visual representation model as the teacher model and constructing a student model with the same model structure as the teacher model. The student model is trained and its parameters are updated using the extended training sample set, with the teacher model serving as guidance. After training, if the performance of the student model is better than that of the teacher model, the learned model is used as the new visual representation model.

[0234] Corresponding to Embodiment 2, this application also provides a product information recommendation device, see [link to embodiment]. Figure 9 The device may include:

[0235] The delivery information receiving unit 901 is used to receive delivery information about the live broadcast information stream. The delivery information includes real-time content value point information of the live broadcast information stream. The real-time content value point information is determined based on the information of the target product corresponding to the real-time explanation object of the live broadcast information stream.

[0236] The recommendation unit 902 is used to recommend the live stream to the target user based on the real-time content value information of the live stream during the process of recommending product information to the user, so as to provide link information about the live stream in the product recommendation information display page shown to the target user.

[0237] Specifically, the recommendation unit can be used for:

[0238] During the process of recommending product information to users, the degree of interest of the target user in the target product is determined;

[0239] If the level of interest meets the criteria, a link to the live stream is inserted into the set of recommended product information provided to the target user, so that the link to the live stream is provided on the product recommendation information display page shown to the target user.

[0240] Corresponding to Embodiment 3, this application also provides a device for providing product search result information, see [link to embodiment]. Figure 10 The device may include:

[0241] The delivery information receiving unit 1001 is used to receive delivery information about the live broadcast information stream. The delivery information includes real-time content value point information of the live broadcast information stream. The real-time content value point information is determined based on the information of the target product corresponding to the real-time explanation object of the live broadcast information stream.

[0242] The search result providing unit 1002 is used to provide links to the live stream in the product search result information display page based on the real-time content value point information of the live stream, during the process of providing product search results according to the product search result request initiated by the user.

[0243] Specifically, the search result providing unit can be used to: if the target product corresponding to the real-time explanation object meets the search conditions, insert link information about the live broadcast information stream into the product search result set so as to provide link information about the live broadcast information stream in the product search result information display page.

[0244] Corresponding to Embodiment Six, this application also provides an apparatus for generating product demonstration videos, see [link to embodiment six]. Figure 11 The device may include:

[0245] Information retrieval unit 1101 is used to retrieve the live broadcast information stream and the product information in the product list associated with the live broadcast information stream;

[0246] The correlation judgment unit 1102 performs real-time analysis of the live broadcast information stream to determine the correlation between the real-time explanation object of the live broadcast information stream and the products in the explanation product list.

[0247] The time point determination unit 1103 is used to determine the start time point information and end time point information of explaining the target product in the product explanation list in the live broadcast information stream based on the correlation judgment result.

[0248] The video segment extraction unit 1104 is used to extract corresponding video segments from the live broadcast information stream according to the start time information and end time information, so as to deliver the video segments as explanatory videos of the target product to the target page.

[0249] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0250] And an electronic device, comprising:

[0251] One or more processors; and

[0252] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0253] in, Figure 12 An exemplary architecture of an electronic device is shown, which may include a processor 1210, a video display adapter 1211, a disk drive 1212, an input / output interface 1213, a network interface 1214, and a memory 1220. The processor 1210, video display adapter 1211, disk drive 1212, input / output interface 1213, network interface 1214, and memory 1220 can communicate with each other via a communication bus 1230.

[0254] The processor 1210 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solution provided in this application.

[0255] The memory 1220 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1220 can store the operating system 1221 for controlling the operation of the electronic device 1200, and the basic input / output system (BIOS) for controlling the low-level operations of the electronic device 1200. Additionally, it can store a web browser 1223, a data storage management system 1224, and a live streaming information stream delivery processing system 1225, etc. The aforementioned live streaming information stream delivery processing system 1225 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 1220 and is called and executed by the processor 1210.

[0256] Input / output interface 1213 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0257] Network interface 1214 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0258] Bus 1230 includes a pathway for transmitting information between various components of the device, such as processor 1210, video display adapter 1211, disk drive 1212, input / output interface 1213, network interface 1214, and memory 1220.

[0259] It should be noted that although the above-described device only shows the processor 1210, video display adapter 1211, disk drive 1212, input / output interface 1213, network interface 1214, memory 1220, bus 1230, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0260] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0261] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0262] The live streaming information stream delivery method and electronic device provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for delivering live streaming information streams, characterized in that, include: Retrieve the live stream information stream, as well as the product information from the product list associated with the live stream information stream; By performing real-time analysis on the live stream to extract video frames, and obtaining multimodal information based on the image content in the video frames and the text content converted from the audio information in the live stream, the explanation intent in the image content of the video frames is located based on the multimodal information, and the target image content belonging to the current real-time explanation object is extracted. The target image content is input into a visual representation model for calculation to obtain a representation vector for semantic expression of the target image content. The representation vector corresponding to the target image is matched with the representation vector corresponding to the product image in the pre-generated explanation product list to determine the relevance between the real-time explanation object in the live stream and the products in the explanation product list, and the target product corresponding to the real-time explanation object is determined based on the judgment result. The live stream information is delivered to the traffic distribution system, and the traffic distribution system is provided with information about the target product corresponding to the live explanation object, so that the traffic distribution system can push the live stream information to the target user group through the target page based on the information about the target product corresponding to the live explanation object.

2. The method according to claim 1, characterized in that, Before the live stream information is distributed to the traffic distribution system, the following steps are also included: By performing model understanding on the multimodal descriptive information associated with the target product corresponding to the real-time explanation object, text content about the current content value point of the live broadcast information stream is generated; The step of distributing the live stream information to the traffic distribution system and providing the traffic distribution system with information about the target product corresponding to the live explanation subject includes: The live stream information flow is delivered to the traffic distribution system, and the text content of the current content value point is provided to the traffic distribution system so that when the traffic distribution system pushes the live stream information flow to the target user group through the target page, it also provides the text content of the current content value point.

3. The method according to claim 1, characterized in that, The model structure of the visual representation model includes: using a visual converter model as the backbone network, and adding a fully connected layer on the backbone network. The fully connected layer is used to transform the output information of the backbone network into a representation vector of the target dimension. The visual converter model adopts a shifted non-overlapping window design, performs self-attention mechanism calculation at the window level, applies multi-scale pooling in window partitioning, and layers features of different granularities through convolution in the depth direction. The context features of different granularities are aggregated through gating aggregation.

4. The method according to claim 1, characterized in that, During the training of the visual representation model, the objective function used in the training process includes: a multi-similarity loss function; Each training round uses a target batch of training samples as input information. The target batch of training samples includes multiple positive samples and multiple negative samples. The positive samples are multiple different images corresponding to the same product, and the negative samples are multiple different images corresponding to other products. The training samples are obtained based on video clips extracted from historical live stream information streams and corresponding product annotation information. After the visual representation model outputs representation vectors for each training sample, the training samples are paired up into sample pairs, and the similarity of the representation vectors within multiple sample pairs is compared to optimize the multi-similarity loss function, so as to update the parameter values ​​of the visual representation model.

5. The method according to claim 4, characterized in that, Also includes: During the optimization of the multi-similarity loss function, hard samples are sampled so that the multi-similarity loss function is optimized when the weights assigned to the hard sample pairs are higher than those assigned to the simple samples. Specifically, when sampling hard samples, a feature memory queue is created. Historical training samples input into the visual representation model in previous training rounds, as well as new training samples input into the visual representation model in the current training round, are added to the feature memory queue. This allows for the mining of hard samples by calculating the similarity between the representation vectors of sample pairs within the feature memory queue.

6. The method according to claim 5, characterized in that, Also includes: During the sampling of difficult samples within the feature memory queue, a momentum update model with the same model structure as the visual representation model is constructed. The parameter update direction of the momentum update model is determined by partially approximating the visual representation model. When sampling hard samples from the feature memory queue, the momentum update model is used to generate representation vectors for historical training samples.

7. The method according to claim 1, characterized in that, When initializing the visual representation model, multiple information pairs consisting of product images and product text description information are collected. For each information pair, the product image is input into the visual representation model to extract image features, and the product text description information is input into the text feature extraction model to extract text features. Based on the similarity between the image features and the text features, cross-modal unsupervised semantic alignment training is performed on the visual representation model and the text feature extraction model. The parameters of the visual representation model after training are used as the initialization parameters of the visual representation model so that the visual representation model can be trained based on the initialization parameters.

8. The method according to claim 7, characterized in that, Also includes: A dynamic text feature queue is constructed, and the text features from multiple information pairs are added to the dynamic text feature queue. The extended text of the image is obtained by comparing the similarity between the image features and the multiple text features in the dynamic text feature queue, so that the image and the extended text can be combined into an information pair and added to the unsupervised semantic alignment training.

9. The method according to claim 1, characterized in that, Also includes: The visual representation model is continuously learned using the newly acquired extended training sample set; wherein, in the continuous learning process, a distillation learning method is adopted, the visual representation model is used as the teacher model, and a student model with the same model structure as the teacher model is constructed. The student model is trained and its parameters are updated using the extended training sample set, and the teacher model is used as guidance. After training, if the student model outperforms the teacher model, the student model is used as the new visual representation model.

10. A method for recommending product information, characterized in that, include: The system receives information about the delivery of live stream information, including real-time content value point information of the live stream information. This real-time content value point information is determined based on the target product information corresponding to the real-time explanation object in the live stream information. The target product is determined by extracting video frames from the live stream information in real-time, converting the image content in the video frames and the text content obtained from the audio information in the live stream information to obtain multimodal information, locating the explanation intent of the image content in the video frames based on the multimodal information, and extracting the target image content belonging to the current real-time explanation object. This target image content is then input into a visual representation model for calculation to obtain a representation vector for semantically expressing the target image content. The representation vector corresponding to the target image is matched with the representation vector corresponding to the product image in a pre-generated explanation product list to determine the relevance between the real-time explanation object in the live stream information and the products in the explanation product list, and the determination is made based on the judgment result. During the process of recommending product information to users, the live stream information is recommended to target users based on the real-time content value information of the live stream information stream, so as to provide links to the live stream information stream in the product recommendation information display page shown to the target users.

11. A method for providing product search result information, characterized in that, include: The system receives information about the delivery of live stream information, including real-time content value point information of the live stream information. This real-time content value point information is determined based on the target product information corresponding to the real-time explanation object in the live stream information. The target product is determined by extracting video frames from the live stream information in real-time, converting the image content in the video frames and the text content obtained from the audio information in the live stream information to obtain multimodal information, locating the explanation intent of the image content in the video frames based on the multimodal information, and extracting the target image content belonging to the current real-time explanation object. This target image content is then input into a visual representation model for calculation to obtain a representation vector for semantically expressing the target image content. The representation vector corresponding to the target image is matched with the representation vector corresponding to the product image in a pre-generated explanation product list to determine the relevance between the real-time explanation object in the live stream information and the products in the explanation product list, and the determination is made based on the judgment result. During the process of providing product search results based on user-initiated product search requests, links to the live stream are provided on the product search results display page based on the real-time content value information of the live stream.

12. A method for processing difficult samples, characterized in that, include: Obtain a training sample set, wherein the training sample set includes multiple images and corresponding annotation information; Using the multi-similarity loss function as the objective function, the visual representation model is subjected to comparative learning through multiple training rounds using the training sample set in order to update the parameter values ​​of the visual representation model; wherein, during the comparative learning process, hard samples are also sampled so that the multi-similarity loss function is optimized when the weights assigned to the hard samples are higher than those assigned to the simple samples. During the sampling of the hard samples, a feature memory queue is created so that historical training samples input into the visual representation model in previous training rounds, as well as new training samples input into the visual representation model in the current training round, are added to the feature memory queue. A momentum update model with the same model structure as the visual representation model is constructed, and the parameter update direction of the momentum update model is determined by partially approximating the visual representation model. When sampling hard samples, for historical training samples in the feature memory queue, the momentum update model is used to generate representation vectors, and sample pairs are constructed within the feature memory queue. By calculating the similarity between the representation vectors of the sample pairs, hard samples are sampled. The visual representation model is used to calculate the input target image content to obtain a representation vector for semantically expressing the target image content. The representation vector corresponding to the target image is matched with the representation vector corresponding to the product image in a pre-generated product list to determine the relevance between the real-time explanation object in the live stream and the products in the product list. Based on the determination result, the target product corresponding to the real-time explanation object is identified. The target image content includes video frames extracted through real-time analysis of the live stream. Multimodal information is obtained based on the image content in the video frames and the text content converted from the audio information in the live stream. The explanation intent is located based on the multimodal information, and the image content belonging to the current real-time explanation object is extracted.

13. An initialization processing method for a visual representation model, characterized in that, include: During the parameter initialization process of the visual representation model, multiple information pairs consisting of image and text description information are collected. For each information pair, the image is input into the visual representation model to extract image features, and the text description information is input into the text feature extraction model to extract text features. Based on the similarity between the image features and the text features, cross-modal unsupervised semantic alignment training is performed on the visual representation model and the text feature extraction model. The parameters of the visual representation model after training are used as the initialization parameters of the visual representation model, so that the visual representation model can be trained using the dataset in the downstream application based on the initialization parameters. The visual representation model is used to calculate the input target image content to obtain a representation vector for semantically expressing the target image content. The representation vector corresponding to the target image is matched with the representation vector corresponding to the product image in a pre-generated product list to determine the relevance between the real-time explanation object in the live stream and the products in the product list. Based on the determination result, the target product corresponding to the real-time explanation object is identified. The target image content includes video frames extracted through real-time analysis of the live stream. Multimodal information is obtained based on the image content in the video frames and the text content converted from the audio information in the live stream. The explanation intent is located based on the multimodal information, and the image content belonging to the current real-time explanation object is extracted.

14. A method for generating product demonstration videos, characterized in that, include: Retrieve the live stream information stream, as well as the product information from the product list associated with the live stream information stream; By performing real-time analysis on the live stream to extract video frames, and obtaining multimodal information based on the image content in the video frames and the text content converted from the audio information in the live stream, the explanation intent in the image content of the video frames is located based on the multimodal information, and the target image content belonging to the current real-time explanation object is extracted. The target image content is input into a visual representation model for calculation to obtain a representation vector for semantic expression of the target image content. The representation vector corresponding to the target image is matched with the representation vector corresponding to the product image in the pre-generated explanation product list to determine the relevance between the real-time explanation object in the live stream and the products in the explanation product list. Based on the correlation judgment results, determine the start and end time information of the explanation of the target product in the product list in the live broadcast information stream; Based on the start and end time information, corresponding video segments are extracted from the live stream information stream so that the video segments can be used as explanatory videos for the target product and delivered to the target page.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program performs the steps of the method described in any one of claims 1 to 14.

16. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 14.

Images