Live scene commodity detection method and device, medium and computer equipment

Abstract

Embodiments of the present disclosure provide a live scene commodity detection method and device, a medium and a computer device, the method comprising: acquiring a live video frame collected by a host client; detecting a commodity in the live video frame through a pre-trained detection model to obtain category information of the commodity; wherein the detection model comprises a feature extraction sub-model and a detection sub-model, the feature extraction sub-model comprises a convolution module and a transformer module, the convolution module is configured to extract features of the live video frame to obtain first output features, the transformer module is configured to extract features of the live video frame to obtain second output features, and the detection sub-model is configured to acquire the category information based on the first output features and the second output features.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to methods, devices, media and computer equipment for detecting goods in live streaming scenarios. Background Technology

[0002] In live video streaming platforms, it is necessary to monitor the products sold during the live stream to achieve product supervision. However, due to the complex environment of the live streaming scene, and the fact that live streaming products are usually displayed from multiple angles by the host, the products are in a state of real-time change, and multiple products can easily be obscured when displayed, resulting in relatively low accuracy in detecting live streaming products. Summary of the Invention

[0003] In a first aspect, embodiments of this disclosure provide a product detection method for a live streaming scenario. The method includes: acquiring live video frames collected by a broadcaster's client; detecting products in the live video frames using a pre-trained detection model to obtain product category information; wherein the detection model includes a feature extraction sub-model and a detection sub-model, the feature extraction sub-model includes a convolution module and a transformer module, the convolution module is used to extract features from the live video frames to obtain a first output feature, the transformer module is used to extract features from the live video frames to obtain a second output feature, and the detection sub-model is used to obtain the category information based on the first output feature and the second output feature.

[0004] In some embodiments, the number of convolutional modules and the number of transformer modules are both greater than 1; each transformer module corresponds to one convolutional module; wherein each transformer module shares input features with the convolutional module corresponding to it, and the output features of each transformer module and the convolutional module corresponding to it are fused and used as the input features of the next transformer module and the convolutional module corresponding to the next transformer module, respectively.

[0005] In some embodiments, the i-th transformer module corresponds to the (i+1)-th convolutional module, and the output features of the i-th transformer module and the (i+1)-th convolutional module are fused and then input into the (i+1)-th transformer module and the (i+2)-th convolutional module, respectively. The output features of the first convolutional module are used as the input features of the first transformer module and the second convolutional module, respectively, where i is a positive integer.

[0006] In some embodiments, the output features of each convolutional module are used as the input features of each subsequent convolutional module.

[0007] In some embodiments, the convolution module includes: a convolutional layer for performing convolution processing on the input features of the convolution module; a regularization layer for performing normalization processing on the output features of the convolutional layer; and an activation layer for performing delinearization processing on the output features of the regularization layer to obtain the first output feature.

[0008] In some embodiments, the transformer module includes: a first feature extraction submodule, configured to perform self-attention calculation on the input features of the transformer module using a query matrix, a key matrix, and a feature matrix to obtain a first intermediate feature; a second feature extraction submodule, configured to perform weighted processing on the input features of the transformer module using self-attention weights to obtain a second intermediate feature; and a merging submodule, configured to fuse the first intermediate feature and the second intermediate feature to obtain a second output feature.

[0009] In some embodiments, the first feature extraction submodule obtains the first intermediate feature by: dividing the input features of the transformer module into multiple feature blocks; converting each feature block into a feature vector; and performing self-attention calculation on each feature vector through a query matrix, a key matrix, and a feature value matrix to obtain the first intermediate feature.

[0010] In some embodiments, the second feature extraction submodule obtains the second intermediate feature by: downsampling the input feature of the transformer module; multiplying the downsampled input feature with the transpose of the downsampled input feature to obtain a third intermediate feature; weighting the third intermediate feature with the self-attention weight to obtain a fourth intermediate feature; and restoring the fourth intermediate feature to the size corresponding to the input feature of the transformer module to obtain the second intermediate feature.

[0011] In some embodiments, the method further includes: obtaining relevant information of the live video frame, the relevant information including audio information and / or subtitle information extracted from the live video frame; matching the detected category information with the relevant information to determine a matching result; and reviewing the product based on the matching result to obtain a review result.

[0012] In some embodiments, the method further includes: generating current evaluation information corresponding to the broadcaster account of the broadcaster client based on the review result, wherein the current evaluation information is used to characterize the live streaming quality of the current live stream of the broadcaster account.

[0013] In some embodiments, the method further includes: updating the cumulative evaluation information of the streamer account based on the current evaluation information, wherein the cumulative evaluation information is used to characterize the average live streaming quality of the streamer account in previous live streams.

[0014] In some embodiments, the method further includes: sending the cumulative evaluation information to the viewer client upon receiving a query instruction sent by the viewer client.

[0015] In some embodiments, the live video frame includes multiple video frames in a live video stream; the method further includes: obtaining product category information in each video frame stream of the multiple video frames; and determining the proportion of live streaming time for each type of product in the live video stream based on the product category information in each video frame.

[0016] Secondly, embodiments of this disclosure provide a product detection device for a live streaming scenario. The device includes: an acquisition module for acquiring live video frames collected by a broadcaster's client; and a detection module for detecting products in the live video frames using a pre-trained detection model to obtain product category information. The detection model includes a feature extraction sub-model and a detection sub-model. The feature extraction sub-model includes a convolution module and a transformer module. The convolution module extracts features from the live video frames to obtain a first output feature, and the transformer module extracts features from the live video frames to obtain a second output feature. The detection sub-model obtains the category information based on the first output feature and the second output feature.

[0017] Thirdly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the methods described in any of the embodiments.

[0018] Fourthly, embodiments of this disclosure provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any embodiment.

[0019] This embodiment of the disclosure integrates the convolutional module and the Transformer module. By merging the features between the convolutional module and the Transformer module, the overall feature richness of the network is improved, thereby effectively improving the detection efficiency of product information appearing in live video frames and achieving high detection accuracy.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0022] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure.

[0023] Figure 2 This is a schematic diagram of a product detection method in a live streaming scenario according to an embodiment of this disclosure.

[0024] Figure 3 This is a schematic diagram of the feature extraction sub-model of an embodiment of this disclosure.

[0025] Figure 4 This is a schematic diagram of a feature extraction sub-model according to another embodiment of this disclosure.

[0026] Figure 5 This is a schematic diagram of a feature extraction sub-model according to another embodiment of the present disclosure.

[0027] Figure 6 This is a schematic diagram of a convolution module according to an embodiment of the present disclosure.

[0028] Figure 7 This is a schematic diagram of the transformer module according to an embodiment of this disclosure.

[0029] Figure 8 This is a schematic diagram of a product detection device in a live streaming scenario according to an embodiment of this disclosure.

[0030] Figure 9 This is a schematic diagram of the structure of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0032] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0033] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0034] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0035] like Figure 1 The diagram shown is an illustration of an application scenario of this disclosure. It should be noted that this application scenario is merely an illustrative example and is not intended to limit the scope of this disclosure. Figure 1In the illustrated application scenario, during a live broadcast, the broadcaster's client 101 can capture video frames F and send them to the video server 102. The video server 102 can then send video frames F to the viewer's client 103 for viewing. The video server 102 can also send video frames F to the monitoring server 104. The monitoring server 104 can deploy a detection model to detect the goods in video frames F and obtain the category information of the goods. The monitoring server 104 can also monitor the live broadcast process based on the detected category information, obtaining a monitoring result R. For example, it can determine whether the goods sold during the live broadcast are "not as described" or "falsely advertised," etc. The monitoring result R can be stored in the database 105 for regulatory personnel to query and take certain regulatory measures based on the query results, such as banning the broadcaster's account for serious violations.

[0036] However, on the one hand, live streaming scenarios are quite complex. For example, the rich colors of the background can easily interfere with the detection results. On the other hand, live streaming products are usually displayed from multiple angles by the host, making the products constantly changing, and multiple products displayed can easily obscure each other. Therefore, the accuracy of detecting live streaming products is often relatively low.

[0037] Based on this, this disclosure provides a product detection method, apparatus, medium, and computer equipment for live streaming scenarios. The following is in conjunction with... Figure 1 The application scenarios will be described to illustrate the solutions of the embodiments of this disclosure.

[0038] See Figure 2 The product detection method in a live streaming scenario according to this embodiment includes:

[0039] Step 201: Obtain the live video frame F captured by the broadcaster client 101;

[0040] Step 202: Detect the products in the live video frame F using the pre-trained detection model 201 to obtain the product category information C;

[0041] The detection model 201 includes a feature extraction sub-model 2011 and a detection sub-model 2012. The feature extraction sub-model 2011 includes a convolution module 2011a and a transformer module 2011b. The convolution module 2011a is used to extract features from the live video frame F to obtain a first output feature f. out1 The transformer module 2011b is used to extract features from the live video frame F to obtain the second output feature f. out2The detection sub-model 2012 is used to detect based on the first output feature f out1 and the second output feature f out2 Obtain the category information C.

[0042] In step 201, the broadcaster client 101 refers to the broadcaster user's client, which can be an application installed on a smart device such as a smartphone or tablet. During the broadcast, the broadcaster client 101 can access the camera on the smart device to capture the live video stream. In this embodiment, the live video frame F may include some or all of the video frames in the live video stream. A single live video frame F may not contain any products, or it may include one or more products.

[0043] In step 202, the detection model 201 can detect each live video frame F to obtain the category information C of the goods in each live video frame F. If a live video frame F includes multiple goods, the detection model 201 can detect the category information C of each goods in the live video frame F.

[0044] To improve detection performance, this embodiment employs a convolutional module 2011a and a transformer module 2011b to jointly construct the feature extraction sub-model 2011 in the detection model 201. Specifically, the convolutional module 2011a is primarily responsible for extracting local features (i.e., the first output feature f). out1 The transformer module 2011b is mainly responsible for constructing long-distance feature relationships, that is, extracting global features (i.e., the second output feature f). out2 Therefore, by fusing the convolution module 2011a and the transformer module 2011b, the feature extraction sub-model 2011 extracts a richer quantity and variety of features. The two modules complement each other in feature extraction, and the amount of feature information extracted is richer than that obtained by using a single feature extraction module. This is beneficial for the detection sub-model 2012 to perform detection and analysis, thereby effectively improving detection accuracy.

[0045] After extracting the first output feature f out1 Second output feature f out2 Then, the first output feature f can be... out1 Second output feature f out2 The features are then fused to obtain the fused features. The fusion method can be based on the first output feature f. out1 Second output feature f out2 Perform concatenation, or process the first output feature f out1 Second output feature f out2Perform an arithmetic average or weighted average, or use other fusion methods. Because by applying the first output feature f... out1 Second output feature f out2 In the method of feature fusion by concatenation, the first output feature f out1 Second output feature f out2 The dimension selection can be more flexible. Therefore, the embodiments of this disclosure can generate fused features by splicing.

[0046] After obtaining the fused features, the detection sub-model 2012 can obtain the category information C of the goods in the live video frame F based on the fused features. Optionally, the detection sub-model 2012 can use a YOLO detector, which uses linear regression fitting to simultaneously predict the target location and classify the category, and has the advantages of fast detection speed and low complexity. In addition to the YOLO detector, other prediction and localization decoupled detectors can also be used as the detection sub-model 2012 in this embodiment of the disclosure, which will not be listed here.

[0047] In addition to outputting product category information C, the detection model 201 can also output the quantity information of each product category in the live video frame F, and / or the location information of the products. The location information of a product can be represented by its bounding box. Location information can be output for each category of products, or category information can be output only for specific categories. For example, category information can be output for pre-defined categories of products, for the product with the highest quantity in live video frame F, or for products detected from a specified location in live video frame F.

[0048] In some embodiments, the number of convolutional modules 2011a and the number of transformer modules 2011b are both greater than 1. Each transformer module 2011b may correspond to one convolutional module 2011a. Each transformer module 2011b shares input features with its corresponding convolutional module 2011a, and the output features of each transformer module 2011b and its corresponding convolutional module 2011a are fused and used as input features for the next transformer module 2011b and its corresponding convolutional module 2011a, respectively.

[0049] See Figure 3Assume that the number of convolutional modules 2011a and transformer modules 2011b is 3, and each convolutional module 2011a corresponds to one transformer module 2011b. Those skilled in the art will understand that the above numbers are merely illustrative, and the number of convolutional modules 2011a and transformer modules 2011b can be other values ​​besides 3. The number of convolutional modules 2011a and transformer modules 2011b can be determined based on both detection accuracy and detection time requirements. A higher number of convolutional modules 2011a and transformer modules 2011b results in longer detection time but higher detection accuracy; conversely, a lower number of convolutional modules 2011a and transformer modules 2011b results in shorter detection time but lower detection accuracy.

[0050] Besides the one-to-one correspondence between convolutional modules 2011a and transformer modules 2011b, as described above, the number of convolutional modules 2011a and the number of transformer modules 2011b may not be equal. For example, the number of convolutional modules 2011a can be greater than the number of transformer modules 2011b. See also... Figure 4 The number of convolutional modules 2011a can be one more than the number of transformer modules 2011b. Based on this, the i-th transformer module 2011b corresponds to the (i+1)-th convolutional module 2011a, and the output features of the i-th transformer module 2011b and the (i+1)-th convolutional module 2011a are fused and then input into the (i+1)-th transformer module 2011b and the (i+2)-th convolutional module 2011a, respectively. The output features of the 1st convolutional module 2011a are used as the input features of the 1st transformer module 2011b and the 2nd convolutional module 2011a, respectively, where i is a positive integer. Figure 4 The illustration shows a case where the number of convolutional modules 2011a is equal to 6 and the number of transformer modules 2011b is equal to 5. Those skilled in the art will understand that the above values ​​are merely illustrative and the solutions of the embodiments disclosed herein are not limited thereto.

[0051] In some embodiments, the output features of each convolutional module 2011a can be used as the input features of each subsequent convolutional module 2011a. See also Figure 5The output features of the first convolutional module 2011a are used as the input features of the second, third, fourth, and fifth convolutional modules 2011a, respectively; the output features of the second convolutional module 2011a are used as the input features of the third, fourth, and fifth convolutional modules 2011a, respectively; and so on. The output of each convolutional module 2011a is passed to each subsequent convolutional module 2011a in a skip-pass manner. The skip-pass of multiple convolutional modules 2011a constitutes dense connections between them. By adopting dense connections, the feature reuse rate can be improved, allowing the feature information extracted by the preceding convolutional module 2011a to be further processed and extracted by the subsequent convolutional modules 2011a. This allows for the acquisition of more feature information, achieving the effect of increasing the number of network layers to acquire more features. Consequently, the number of layers in the feature extraction sub-model 2011 can be reduced, making the structure of the feature extraction sub-model 2011 more lightweight. The specific structures of the convolution module 2011a and the transformer module 2011b are illustrated below.

[0052] In some embodiments, the convolution module 2011a includes: a convolutional layer Conv for performing convolution processing on the input features of the convolution module; a regularization layer BN for normalizing the output features of the convolutional layer; and an activation layer ELU for delinearizing the output features of the regularization layer to obtain the first output feature f. out1 . Figure 6 The specific structure of convolution module 2011a is shown. For simplicity, only the structure of the first convolution module 2011a is shown in the figure. The structures of other convolution modules 2011a can be found in the first convolution module 2011a.

[0053] Optionally, the convolutional layer Conv includes a 3x3 convolutional layer. The regularization layer BN can accelerate model training and convergence, control gradient explosion to prevent gradient vanishing and overfitting, and perform batch regularization. The activation layer ELU can use the ELU activation function to delinearize the features output by the BN layer. The calculation formula for the ELU activation function is as follows:

[0054]

[0055] Where α is the learnable weight parameter in the convolution module 2011a, x represents the input feature of the activation layer ELU, and f(x) represents the output feature of the activation layer ELU.

[0056] In some embodiments, transformer module 2011b includes: a first feature extraction submodule, configured to extract the input features f of transformer module 2011b using a query matrix (Q), a key matrix (K), and a feature matrix (V). in Perform self-attention calculation to obtain the first intermediate feature f. med1 The second feature extraction submodule is used to apply self-attention weights to the input features f of the transformer module 2011b. in After weighting, the second intermediate feature f is obtained. med2 The merging submodule is used to process the first intermediate feature f. med1 and the second intermediate feature f med2 The fusion is performed to obtain the second output feature f. out2 .

[0057] In this embodiment, see Figure 7 The input feature f of the transformer module 2011b in The features are divided into two sub-features (denoted as f) by grouping. in1 and f in2 Specifically, the input features f of the transformer module 2011b can be... in The channels are split in a 1:1 ratio to obtain the two sets of sub-features mentioned above. Of course, channels can be split in other ratios besides 1:1. However, after multiple experiments, it has been confirmed that proportional splitting is more adaptable to various detection scenarios.

[0058] One set of sub-features f in1 After conversion to tokens, they are fed into the first feature extraction submodule for global spatial attention calculation of low-level features. Sub-feature f in1 First, divide it into feature blocks. For example, the sub-features f can be divided into equal parts. in1 Divide into multiple feature blocks of a preset size, or divide the sub-features f in1 Divide into a predetermined number of feature blocks. Use sub-features f in1 Taking a size of 16x16x256 as an example, the first feature extraction submodule can be divided into 16 feature blocks of size 4x4, each of which is 4x4x256. Then, each feature block can be converted into a one-dimensional vector. Multi-head self-attention computation refers to multiple self-attention calculations performed simultaneously, with the results merged. The self-attention computation of a single one-dimensional vector uses a query matrix Q, a key-value matrix K, and an eigenvalue matrix V to achieve dynamic aggregation of global features. The calculation process can be represented as follows:

[0059]

[0060] Where, d k Sub-feature f in1 The number of one-dimensional vectors obtained from the segmented feature blocks. Attention(Q,K,V) represents the self-attention weights calculated based on the query matrix Q, key matrix K, and feature matrix V. The self-attention calculation is performed on all one-dimensional vectors using the above formula, and the results are combined to obtain the multi-head self-attention calculation result. The multi-head self-attention calculation formula can be expressed as:

[0061] MultiHead(Q,K,V)=Concat(head1,...,head n W 0

[0062] Where MultiHead(Q,K,V) represents the output features of the first feature extraction submodule, and head i (1≤i≤n) represents the output feature of the i-th self-attention head, which is obtained by weighting the input feature of the i-th self-attention head with its self-attention weight. The weight of each self-attention head can be calculated using the formula Attention(Q,K,V), where W... 0 Here are the learnable weight parameters, and n is the value of the sub-features f. in1 The number of feature blocks that are divided, i.e., the number of tokens, is related to the above d. k They are equal. After multi-head self-attention computation, all one-dimensional vectors are re-encoded and concatenated to the size of the feature map before input, thus completing the global fine-grained self-attention computation of the upper-layer structure.

[0063] Another set of sub-features f in2 The data is then fed into the second feature extraction submodule for spatial attention calculation of high-level features. In the second feature extraction submodule, the sub-feature f is processed... in2 Perform downsampling to obtain the downsampled sub-features f in21 Two 1x1 convolutional layers are used to process the downsampled sub-features f. in21 By performing dimensional transformation and transpose calculation, we obtain a sub-feature f of size (HxW)xC. in21_1 With sub-feature f of size Cx(HxW) in21_2 Where H is the sub-feature f in21 The height, W is the sub-feature f in21 The width of f, C is the sub-feature f in21 The number of channels. Sub-feature f in21_1 and sub-feature f in21_2 Multiplying them together yields a feature f of size (HxW)x(HxW). in22And use the Softmax activation function to calculate feature f in22 A heatmap of spatial attention weights was generated. Then, the spatial attention weight heatmap was compared with the sub-feature f. in21 Multiply the features and then pad them with zeros to restore the resulting features to their original size (i.e., sub-features f). in2 (size).

[0064] Sub-feature f in2 Taking a size of 16x16x256 as an example, firstly, max pooling with double downsampling is used to obtain a sub-feature f of size 8x8x256. in21 Then, two parallel 1x1 convolutional layers are used to perform dimensionality transformation and transpose calculation on the reduced feature map, resulting in a sub-feature f of size (8x8)x256. in21_1 With a sub-feature f of size 256x(8x8) in21_2 Multiplying the two together yields the characteristic f of the size of (8x8)x(8x8). in22 A spatial attention weight heatmap of size 8x8x1 was calculated using the Softmax activation function. Finally, the spatial attention weight heatmap was compared with the 8x8x256 sub-features f. in21 Multiply the features to obtain features containing spatial weight information, and restore the features containing spatial weight information to a size of 16x16x256 by zero-padding upsampling.

[0065] Then, the output features of the first and second feature extraction submodules are calculated separately and then merged to restore the number of channels to the input feature f. in The number of channels is used to obtain the output of a single transformer module 2011b. Tokens are 1x1xC feature matrices, which are transformed through matrix calculations, where C represents the number of channels.

[0066] In the above embodiments, the input feature f in After grouping the channels, the number of token parameters for multi-head self-attention computation in the first feature extraction submodule is reduced, thus reducing the number of computational parameters. Furthermore, the second feature extraction submodule employs a convolutional computation mechanism, which has a much smaller number of computational parameters than multi-head self-attention computation. Therefore, the overall number of parameters is reduced, thereby improving feature extraction speed and, consequently, detection speed.

[0067] This disclosure integrates convolutional and Transformer modules, enhancing the overall feature richness of the network by merging features from both modules. Furthermore, it incorporates a lightweight design philosophy, employing dense connections in the convolutional modules to improve feature reuse and reduce the number of network layers, and using a layered Transformer module to reduce computational parameters. This disclosure effectively improves the detection efficiency of product information appearing in live video frames while achieving high detection accuracy.

[0068] It should be noted that the feature extraction sub-model 2011 in the detection model 201 used in the product detection method for live streaming scenarios in this disclosure may only include the transformer module 2011b and not the convolution module 2011a. The transformer module 2011b includes a first feature extraction sub-module, used to perform self-attention calculation on the input features of the transformer module 2011b through a query matrix, a key-value matrix, and a feature-value matrix to obtain a first intermediate feature; a second feature extraction sub-module, used to weight the input features of the transformer module 2011b through self-attention weights to obtain a second intermediate feature; and a merging sub-module, used to fuse the first intermediate feature and the second intermediate feature to obtain the second output feature. The transformer module 2011b, by fusing spatial attention and self-attention algorithms, can extract richer features, which is beneficial for the detection sub-model 2012 to perform analysis and detection, thereby effectively improving detection accuracy. Specific details of the transformer module 2011b are detailed in the foregoing embodiments and will not be repeated here.

[0069] In some embodiments, the sample live video stream can be preprocessed, including frame extraction and data cleaning, to obtain multiple sample live video frames. These preprocessed sample live video frames are then labeled to construct a live sample dataset. Frame extraction involves extracting video frames from the sample live video stream at set intervals to filter out video frames with excessively high similarity. Data cleaning filters out low-quality video frames. Data labeling identifies the category and location of goods in each preprocessed sample live video frame. The sample dataset can include sample live video frames corresponding to goods in each of multiple categories. A detection model 201 can then be trained based on the sample dataset.

[0070] Furthermore, for training the detection model 201, this disclosure has set different hyperparameters through multiple experimental tests. One possible hyperparameter setting is shown in Table 1:

[0071] Table 1 Hyperparameter Settings

[0072] Hyperparameters Parameter value Epoch 800 Batch size 16 Learning rate (starting) 0.001 Learning rate (end) 0.00025 Momentum 0.857 Weight decay 0.0005

[0073] This disclosed method for product detection in live streaming scenarios combines product detection tasks within live video frames with deep learning, providing an effective method for detecting and recording products appearing in live video frames. This method exhibits better detection performance compared to other deep learning networks; specific test comparison results are shown in Table 2.

[0074] Table 2 Comparison of F1-Score and mAP experimental results between the disclosed method and four existing methods

[0075]

[0076]

[0077] Based on the data in the table above, it is demonstrated that the detection model 201 designed in this disclosure is superior to other models in terms of overall performance and average detection accuracy, and also has higher detection accuracy for goods of different sizes.

[0078] In some embodiments, the category information of the goods in each live video frame of the video stream can be obtained through the above method. Then, based on the category information of the goods in each video frame, the proportion of live streaming time for each type of goods in the live video stream is determined. Specifically, for each type of goods, the number of live video frames in which that type of goods is included can be calculated, and the proportion of live streaming time for that type of goods is determined based on the ratio between the number of live video frames including that type of goods and the total number of live video frames in the video stream.

[0079] Furthermore, the live stream time percentage of products in a preset category can be obtained, and the current live stream content can be reviewed based on this percentage to obtain a review result. This review result indicates whether any violations have occurred in the current live stream. The preset category can be a category pre-reported by the streamer's account corresponding to the streamer's client. Based on this, if the live stream time percentage of products in the preset category is less than the preset percentage, it can be determined that the product category in the current live stream does not match the pre-reported category, thus determining that a violation has occurred in the current live stream.

[0080] In some embodiments, relevant information of the live video frame may also be obtained, including audio information and / or subtitle information extracted from the live video frame; the detected category information is matched with the relevant information to determine the matching result; the product is reviewed based on the matching result to obtain the review result.

[0081] For example, reference category information can be extracted from the relevant information in the live video frames, and it can be determined whether the detected category information matches the reference category information. If they do not match, it can be determined that the product category currently recommended in the live stream is inconsistent with the product category detected from the live video frames, thus determining that the current live stream has violated regulations. In some application scenarios, in order to circumvent review, some streamers may display products of one category in their live stream, but actually recommend products of another category during the live stream. In this case, the detection results of the live video frames and relevant information can be combined to review the live stream content, thereby obtaining a more accurate review result.

[0082] For example, target-related information including preset semantics can be extracted from the relevant information of the live video frame. For example, preset semantics may include, but are not limited to, "counterfeit goods" or "goods not as described". When the number of target-related information with preset semantics extracted is greater than a preset threshold, it can be determined that the current live broadcast has violated regulations.

[0083] In some embodiments, current evaluation information corresponding to the streamer account on the streamer client can also be generated based on the review results. This current evaluation information is used to characterize the quality of the streamer account's current live stream. The current evaluation information can be a rating, for example, using a 5-point or 100-point scale to rate the quality of the current live stream. A higher rating indicates higher quality, and vice versa. Alternatively, the current evaluation information can also be evaluation keywords, such as positive keywords like "genuine product," "good quality," and "high cost-performance ratio," and / or negative keywords like "product not as advertised," "poor quality," and "low cost-performance ratio."

[0084] Furthermore, the cumulative evaluation information of the streamer's account can be updated based on the current evaluation information. This cumulative evaluation information represents the average quality of the streamer's account's various live streams. For example, when the current evaluation information is a rating, the average of the streamer's current rating and historical ratings can be calculated, and this average can be used as the streamer's account's cumulative rating. When the current evaluation information is an evaluation keyword, the radar chart corresponding to the streamer's account's cumulative evaluation keyword can be updated based on the current evaluation keyword.

[0085] Upon receiving a query command from a viewer's client, the cumulative rating information can be sent to that viewer's client. Furthermore, the average cumulative rating information for each streamer's account on the live streaming platform can also be sent to the viewer's client, allowing users of the viewer's client (i.e., viewers) to make a horizontal comparison of the rating information for multiple streamer accounts on the live streaming platform.

[0086] In some embodiments, the cumulative rating information can also be sent to the broadcaster's client so that broadcasters can be aware of their own live stream ratings, thereby incentivizing them to improve subsequent live stream content. Furthermore, the average cumulative rating information of all broadcaster accounts on the live streaming platform can also be sent to the broadcaster's client so that broadcasters can make horizontal comparisons of the rating information of multiple broadcaster accounts on the live streaming platform, thereby incentivizing them to improve subsequent live stream content.

[0087] See Figure 8 This disclosure also provides a product detection device for live streaming scenarios, the device comprising:

[0088] The acquisition module 801 is used to acquire live video frames captured by the broadcaster's client;

[0089] The detection module 802 is used to detect products in the live video frames using a pre-trained detection model to obtain the category information of the products;

[0090] The detection model includes a feature extraction sub-model and a detection sub-model. The feature extraction sub-model includes a convolution module and a transformer module. The convolution module is used to extract features from the live video frames to obtain a first output feature. The transformer module is used to extract features from the live video frames to obtain a second output feature. The detection sub-model is used to obtain the category information based on the first output feature and the second output feature.

[0091] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0092] This specification also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the foregoing embodiments.

[0093] Figure 9 This diagram illustrates a more specific hardware structure of a computing device provided in an embodiment of this specification. The device may include a processor 901, a memory 902, an input / output interface 903, a communication interface 904, and a bus 905. The processor 901, memory 902, input / output interface 903, and communication interface 904 are interconnected internally via the bus 905.

[0094] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor 901 may also include a graphics card, such as an Nvidia Titan X graphics card or a 1080Ti graphics card.

[0095] The memory 902 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901.

[0096] The input / output interface 903 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0097] The communication interface 904 is used to connect the communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as sports networks, WIFI, Bluetooth, etc.).

[0098] Bus 905 includes a pathway for transmitting information between various components of the device, such as processor 901, memory 902, input / output interface 903, and communication interface 904.

[0099] It should be noted that although the above-described device only shows the processor 901, memory 902, input / output interface 903, communication interface 904, and bus 905, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0100] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0101] Computer-readable media include both permanent and non-permanent, movable and non-movable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0102] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this specification can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this specification, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this specification.

[0103] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0104] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0105] The above description is merely a specific implementation of the embodiments of this specification. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this specification, and these improvements and modifications should also be considered within the protection scope of the embodiments of this specification.

Claims

1. A product detection method for a live streaming scenario, characterized in that, The method includes: Obtain live video frames captured by the broadcaster's client; The product category information is obtained by detecting the products in the live video frames using a pre-trained detection model. The detection model includes a feature extraction sub-model and a detection sub-model. The feature extraction sub-model includes a convolutional module and a transformer module. The convolutional module is used to extract features from the live video frames to obtain a first output feature. The transformer module is used to extract features from the live video frames to obtain a second output feature. The detection sub-model is used to obtain the category information based on the first output feature and the second output feature. The first output feature is a local feature, and the second output feature is a global feature. The number of convolutional modules and the number of transformer modules are both greater than 1; each transformer module corresponds to one convolutional module; wherein, each transformer module shares input features with the convolutional module corresponding to it, and the output features of each transformer module and the convolutional module corresponding to it are fused and used as the input features of the next transformer module and the convolutional module corresponding to the next transformer module, respectively.

2. The method according to claim 1, characterized in that, The i-th transformer module corresponds to the (i+1)-th convolutional module, and the output features of the i-th transformer module and the (i+1)-th convolutional module are fused and then input into the (i+1)-th transformer module and the (i+2)-th convolutional module, respectively. The output features of the first convolutional module are used as the input features of the first transformer module and the second convolutional module, respectively, where i is a positive integer.

3. The method according to claim 1 or 2, characterized in that, The output features of each convolutional module serve as the input features for each subsequent convolutional module.

4. The method according to claim 1, characterized in that, The convolutional module includes: Convolutional layers are used to perform convolution processing on the input features of the convolutional module; The regularization layer is used to normalize the output features of the convolutional layer; An activation layer is used to delinearize the output features of the regularization layer to obtain the first output feature.

5. The method according to claim 1, characterized in that, The transformer module includes: The first feature extraction submodule is used to perform self-attention calculation on the input features of the transformer module through a query matrix, a key matrix, and a feature matrix to obtain the first intermediate feature; The second feature extraction submodule is used to weight the input features of the transformer module using self-attention weights to obtain the second intermediate features; The merging submodule is used to fuse the first intermediate feature and the second intermediate feature to obtain the second output feature.

6. The method according to claim 5, characterized in that, The first feature extraction submodule obtains the first intermediate feature based on the following method: The input features of the transformer module are divided into multiple feature blocks; Each feature block is converted into a feature vector; The first intermediate feature is obtained by performing self-attention calculation on each feature vector using the query matrix, key matrix, and eigenvalue matrix.

7. The method according to claim 5, characterized in that, The second feature extraction submodule obtains the second intermediate feature based on the following method: The input features of the transformer module are downsampled; The third intermediate feature is obtained by multiplying the downsampled input feature with the transpose of the downsampled input feature. The third intermediate feature is weighted using the self-attention weights to obtain the fourth intermediate feature; The fourth intermediate feature is restored to the size corresponding to the input feature of the transformer module to obtain the second intermediate feature.

8. The method according to claim 1, characterized in that, The method further includes: Obtain relevant information of the live video frame, including audio information and / or subtitle information extracted from the live video frame; The detected category information is matched with the relevant information to determine the matching result; The product is reviewed based on the matching results to obtain the review results.

9. The method according to claim 8, characterized in that, The method further includes: Based on the review results, the current evaluation information corresponding to the streamer account on the streamer client is generated. The current evaluation information is used to characterize the live streaming quality of the streamer account's current live stream.

10. The method according to claim 9, characterized in that, The method further includes: The cumulative evaluation information of the streamer's account is updated based on the current evaluation information. The cumulative evaluation information is used to characterize the average live streaming quality of the streamer's account throughout its live streams.

11. The method according to claim 10, characterized in that, The method further includes: Upon receiving a query command from a viewer's client, the cumulative evaluation information is sent to the viewer's client.

12. The method according to claim 1, characterized in that, The live video frame includes multiple video frames in the live video stream; the method further includes: Obtain the category information of the goods in each of the multiple video frames; The percentage of live streaming time for each type of product in the live video stream is determined based on the product category information in each video frame.

13. A product detection device for live streaming scenarios, characterized in that, The device includes: The acquisition module is used to acquire live video frames captured by the broadcaster's client. The detection module is used to detect products in the live video frames using a pre-trained detection model to obtain the category information of the products; The detection model includes a feature extraction sub-model and a detection sub-model. The feature extraction sub-model includes a convolutional module and a transformer module. The convolutional module is used to extract features from the live video frames to obtain a first output feature. The transformer module is used to extract features from the live video frames to obtain a second output feature. The detection sub-model is used to obtain the category information based on the first output feature and the second output feature. The first output feature is a local feature, and the second output feature is a global feature. The number of convolutional modules and the number of transformer modules are both greater than 1; each transformer module corresponds to one convolutional module; wherein, each transformer module shares input features with the convolutional module corresponding to it, and the output features of each transformer module and the convolutional module corresponding to it are fused and used as the input features of the next transformer module and the convolutional module corresponding to the next transformer module, respectively.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1 to 12.

15. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 12.

Images