Video information processing method, device and electronic equipment

Abstract

The embodiment of the present application discloses a video information processing method, device and system, wherein the method includes the following steps: determining a target commodity object set, and images and associated text information of multiple commodity objects in the target commodity object set; obtaining multiple image frames and their associated audio information from the video to be processed, and performing voice recognition on the audio information to obtain text information associated with the image frame; obtaining feature information of the commodity object based on the image of the commodity object and its associated text information; obtaining feature information of the image frame based on the image information of the image frame and its associated text information; determining the matching relationship between the image frame and the commodity object by calculating the similarity between the feature information of the image frame and the feature information of the commodity object. Through this method, the success rate and accuracy of matching can be improved when matching image frames with commodity objects.

Description

Technical Field

[0001] The present application relates to the technical field of video information detection, and in particular to a video information processing method, device and electronic equipment. Background Art

[0002] In the commodity object information service system, merchants or sellers can introduce the information of specific commodity objects to buyers through live broadcast. Compared with traditional information such as pictures and videos, live video can provide users with more intuitive information.

[0003] For a live video that lasts for several hours, it usually involves a serial introduction of multiple product objects in the time dimension. If the live video can be analyzed through an algorithm, combined with the information of the product objects, and marked on the timeline of the live video, or a short video of "product highlights" is generated for each product object, it can bring a richer experience to users. For example, when watching the live broadcast replay, users can quickly locate the specific product explanation; or, after the user enters the product details page in some way, a "product highlight explanation" button can be provided to allow users to better perceive the selling points of this product; or, in a variety of search and recommendation scenarios, these "product highlights" short videos can be mixed with other regular images to improve conversion and purchase (compared to static images, users often prefer to watch dynamic videos), etc.

[0004] In the process of implementing the above processing, how to mark the starting point of a certain commodity object on the long live video timeline is the key to the problem. In fact, this is a commodity retrieval problem: there are frames of live video images and some commodity object information. An algorithm can be used to extract the features of the commodity object and the video frame, and then use these features to calculate the similarity between the live video image frame and the commodity object. If the similarity is greater than a certain threshold, then it is determined that this frame of video image corresponds to a specific commodity object, so it can be marked on the live timeline.

[0005] In the prior art, features are extracted from the images of the video frame and the main image of the product object, and then the extracted features are compared and calculated for similarity. However, due to various problems such as low resolution, large angle, noisy background, color distortion, etc. of the live broadcast screen or the images taken by the user themselves, it is often difficult to obtain a good matching relationship.

[0006] Therefore, how to more effectively detect specific commodity objects from videos has become a technical problem that needs to be solved by those skilled in the art. Summary of the invention

[0007] The present application provides a video information processing method, device and electronic device, which can more effectively detect specific commodity objects from a video.

[0008] This application provides the following solutions:

[0009] A video information processing method, comprising:

[0010] Determine a target commodity object set, and images of multiple commodity objects in the target commodity object set and associated text information;

[0011] Acquire multiple image frames and their associated audio information from the video to be processed, and perform speech recognition on the audio information to obtain text information associated with the image frames;

[0012] Acquiring feature information of the commodity object based on the image of the commodity object and associated text information;

[0013] Acquire feature information of the image frame based on image information of the image frame and associated text information thereof;

[0014] By determining the similarity between the feature information of the image frame and the feature information of the commodity object, a matching relationship between the image frame and the commodity object is determined.

[0015] A live video stream processing method, comprising:

[0016] Acquire commodity object index data, wherein the index data includes feature information of a plurality of commodity objects, wherein the feature information of the commodity objects is obtained based on image features and associated text features of the commodity objects;

[0017] Acquire a live video stream, acquire multiple image frames and associated audio information from the live video stream, and perform voice recognition on the audio information to obtain text information associated with the image frames;

[0018] Acquire feature information of the image frame based on image information of the image frame and associated text information thereof;

[0019] Determining a matching relationship between the image frame and the commodity object by calculating a similarity between the feature information of the image frame and the feature information of the commodity object in the index data;

[0020] According to the matching relationship, information of the corresponding commodity object is provided in the playback interface of the live video stream.

[0021] A video information processing device, comprising:

[0022] An object information determination unit, used to determine a target commodity object set, and images of multiple commodity objects in the target commodity object set and associated text information;

[0023] An associated text recognition unit, used to obtain multiple image frames and their associated audio information from the video to be processed, and perform speech recognition on the audio information to obtain text information associated with the image frames;

[0024] A commodity object feature information extraction unit, configured to obtain feature information of the commodity object based on the image of the commodity object and associated text information;

[0025] An image frame feature extraction unit, configured to obtain feature information of the image frame based on image information of the image frame and associated text information thereof;

[0026] The matching relationship determining unit is used to determine the matching relationship between the image frame and the commodity object by calculating the similarity between the feature information of the image frame and the feature information of the commodity object.

[0027] A live video stream processing device, comprising:

[0028] An index data acquisition unit, used to acquire commodity object index data, wherein the index data includes feature information of a plurality of commodity objects, wherein the feature information of the commodity objects is obtained based on image features and associated text features of the commodity objects;

[0029] A live stream acquisition unit, used to acquire a live video stream, acquire multiple image frames and their associated audio information from the live video stream, and perform voice recognition on the audio information to obtain text information associated with the image frames;

[0030] An image frame feature acquisition unit, configured to acquire feature information of the image frame based on image information of the image frame and associated text information thereof;

[0031] a matching relationship determination unit, configured to determine a matching relationship between an image frame and a commodity object by calculating a similarity between feature information of the image frame and feature information of the commodity object in the index data;

[0032] A commodity object information providing unit is used to provide corresponding commodity object information in the playback interface of the live video stream according to the matching relationship.

[0033] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0034] Through the embodiments of the present application, when detecting commodity objects from a video, the image and text information of the commodity objects can be fused to extract the features of the commodity objects, and the image information of the image frames in the video and the related audio information can be fused to extract the features of the image frames. Since multimodal information such as images, texts, and audio can be fused, the accuracy of feature extraction can be improved, and more comprehensive and accurate information can be used to represent commodity objects and video image frames, so that when matching image frames with commodity objects by comparing feature similarity, the success rate and accuracy of matching can be improved.

[0035] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0038] Figure 2 is a flow chart of the first method provided in an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of a first feature extraction network provided in an embodiment of the present application;

[0040] Figure 4 is a schematic diagram of a second feature extraction network provided in an embodiment of the present application;

[0041] Figure 5 is a schematic diagram of a third feature extraction network provided in an embodiment of the present application;

[0042] Figure 6 is a flow chart of the second method provided in an embodiment of the present application;

[0043] Figure 7 is a schematic diagram of a first device provided in an embodiment of the present application;

[0044] Figure 8 is a schematic diagram of a second device provided in an embodiment of the present application;

[0045] Fig. 9 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0047] In an embodiment of the present application, in order to more effectively detect specific commodity objects from a video, feature extraction can be performed in a manner that integrates "multimodal" information. The so-called "multimodal" information means that in addition to image information, relevant features can also be mined from other types of information such as associated text information and audio information, thereby improving the success rate and accuracy of commodity object detection from the video.

[0048] Specifically, the information of commodity objects stored in the commodity object information database of the system can usually include multiple types of information, for example, it can include image information such as the main image of the commodity object, and can also include text information such as titles and attributes to describe the characteristics of the commodity object. Moreover, not only can the image information of the main image reflect the characteristics of the commodity object, but this text information can also reflect the characteristics of the commodity object to a certain extent, or it can be enhanced on the basis of the image information or the feature extraction results of the image information can be verified, etc. Therefore, by combining with the text information, the characteristic information of the commodity object can be extracted more accurately.

[0049] On the other hand, in addition to the images frame by frame, specific videos (e.g., live videos, etc.) also include audio data, and the audio information usually corresponds to the image content. For example, in a live video broadcast of a specific category of merchandise such as clothing, the "anchor" user usually "try on" a specific piece of clothing while giving a voice explanation of the texture, details, etc. of the clothing. When trying on one piece of clothing and then another, the corresponding voice explanation content can usually reflect this change, and so on. Therefore, the voice information in the video can also reflect the characteristics of the associated image frame.

[0050] In other words, the live broadcast scene contains voice information, and the product object itself also carries text information such as title, category, and attributes. Therefore, these multimodal (image, voice, text) information can be used to extract better features about the image frames and product objects in the video, and then perform similarity comparison to improve the success rate and accuracy of product object detection from the video.

[0051] Among them, in order to be able to use these multimodal information to extract better features about the image frames and commodity objects in the video, the embodiment of the present application also provides a more specific implementation scheme. In this scheme, the algorithm model can be trained in advance, and then the algorithm model can be used to extract features of the commodity objects or image frames in the video. Among them, specifically when extracting features, whether it is a commodity object or an image frame in a video, the algorithm model can extract features from both image and text information separately, and then fuse the two features to obtain the feature information of the specific commodity object or image frame in the video. That is, for the commodity object to be predicted or the image frame in the video, both types of information, image and text, can be obtained first, and then, when extracting features of the specific commodity object or the image frame in the video, the image and text information associated with the commodity object, or the image and text information associated with the image frame in the video, can be used as input information of the algorithm model.

[0052] Among them, the image information of the commodity object can be obtained from the specific commodity main picture and other pictures, and the text information can be obtained from the title, category, attributes and other information of the commodity object. Regarding the image frame in the video, its image information can be directly obtained from the image frame, and the text information can be obtained by performing voice recognition on the audio information associated with the image frame. For example, regarding a certain image frame in a certain video, the audio content of a period of time before and after the time axis of the image frame (for example, 20S) can be used as the audio information associated with the image frame, and then the voice recognition algorithm is used to recognize the voice contained in this audio information and convert it into text information. The text information can then be used as the text information associated with the image frame, and so on.

[0053] It can be seen that both the product object and the image frame in the video can obtain two types of information, image and text. Therefore, the same algorithm model can be used to extract image features and text features of the product object or the image frame in the video, and then the feature information of the specific product object or the image frame in the video can be determined by fusing the image features with the text features. After that, the features of multiple image frames in the specific video are compared with multiple product objects respectively, and whether a specific image frame corresponds to a product object is determined based on the comparison results.

[0054] From the perspective of system architecture, see Figure 1, the embodiment of the present application can provide relevant video processing functions in the commodity object information service system, and the system can be divided into a client and a server. Among them, the client can be divided into a merchant client and a buyer client. After the merchant client initiates the live broadcast, the live video stream can be pushed to the address according to the push stream address applied by the server in the cloud service system. Correspondingly, the buyer client can "pull the stream" from the address, obtain the live video stream and play it. In specific implementation, the commodity object detection of the video can be performed on the buyer client, or it can be completed on the server, or the two can be coordinated to complete the training of the algorithm model on the server. For example, in a specific implementation method, the algorithm model provided by the server may include a "lightweight" version for sending to the client. In the process of introducing the commodity object information by a specific merchant user through a live video, the live video stream can be detected in real time on the buyer client. The real-time detection results of the client can be directly displayed in the client. For example, when a commodity object is detected, a link to the corresponding commodity object can be provided in the live interface, and so on. In addition, the real-time detection results of the client can also be submitted to the server, which is marked or split into multiple short videos, and so on. In addition, the "enhanced" algorithm model can be saved on the server side, and specific videos can be processed in batches after the live broadcast is completed. Because it is an enhanced algorithm, more and more accurate product object detection can be achieved.

[0055] The specific implementation scheme provided in the embodiments of the present application is described in detail below.

[0056] Embodiment 1

[0057] First, this embodiment 1 provides a video information processing method, which can be applied to a server or a client. Figure 2 , the method may include:

[0058] S210: Determine a target commodity object set, and images of multiple commodity objects in the target commodity object set and associated text information.

[0059] The target product object set includes multiple product objects. The purpose of the first embodiment of the present application is to determine, for the video to be processed, which product object in the set the multiple image frames are respectively associated with. Then, the position of the image frame where the product object has changed is the starting point corresponding to the new product object. Based on this, the video can be marked, or truncated into multiple short videos, and so on.

[0060] Among them, which commodity objects are included in the specific target commodity object set can be determined according to the actual scenario. For example, if the video to be processed has prior information-explanation, the set can be determined within a relatively small range. Specifically, assuming that it is known that a certain video to be processed is a live video related to a certain store, the target commodity object set can be determined based on the commodity objects associated with the store. If there is not enough prior information about the video to be processed, all commodity objects that have been published in the system can be combined into the target commodity object set, and so on.

[0061] Since specific commodity objects are usually published by specific merchant users in the current system, the system stores a commodity object information library, which stores commodity object image (picture, video, etc., where the video is usually a pre-recorded short video, etc.) information, title information, category information, attribute information, etc. In the embodiment of the present application, after determining a specific set, the image information and text information respectively associated with multiple commodity objects in the set can also be determined at the same time, wherein the text information can be determined based on the title, category, attribute and other information.

[0062] S220: Acquire multiple image frames and their associated audio information from the video to be processed, and perform speech recognition on the audio information to obtain text information associated with the image frames;

[0063] There may be multiple videos to be processed. For example, in a specific scenario, it may specifically refer to live videos related to commodity objects, and it may be videos collected in real time during the live broadcast, or videos recorded after the live broadcast, etc. For the former, real-time commodity object detection can be performed on the client side, and for the latter, batch detection can be performed on the server side.

[0064] The video to be processed is composed of a number of video image frames and audio. There are many ways to obtain image frames from a specific video. For example, if the processing capacity is sufficient, each image frame in the video can be processed separately. Alternatively, frame extraction processing can be performed in a certain way, for example, image frames are extracted from the video at a certain time interval, and only the extracted image frames are processed, such as extracting one frame every 1 second, etc.

[0065] For the audio information in the video, a short audio segment can be intercepted from the audio for each image frame to be processed as the audio content related to the corresponding image frame content. For example, for each image frame, the audio content of several seconds (such as 20 seconds) before and after the image frame position can be intercepted as the audio content associated with the image frame. In addition, the voice information contained in the audio content can be recognized by a speech recognition algorithm, and the corresponding text information can be output to determine the text information corresponding to the image frame. Of course, in specific implementation, in order to improve the accuracy of speech recognition, speech recognition can also be performed on the audio information in the video first, and after converting it into text, the speech text of several seconds before and after the specific image frame position can be intercepted as the text information of the current image frame, and so on. Among them, when processing the video in the live state, the specific speech recognition processing can be implemented through the streaming speech recognition model.

[0066] S230: Acquire feature information of the commodity object based on the image of the commodity object and associated text information.

[0067] S240: Acquire feature information of the image frame based on the image information of the image frame and its associated text information.

[0068] Since multiple product objects in the product object set are respectively associated with image information and text information, and multiple image frames extracted from the video to be processed are also respectively associated with image information and text information, the same method can be used to extract features from the product objects and image frames.

[0069] In specific implementation, a pre-trained algorithm model can be used to extract features from the multiple commodity objects and multiple image frames. There can be multiple specific algorithm models, one of which can be a convolutional neural network. Overall, the specific processing process can be as follows: Figure 3 As shown, after obtaining the image frame and audio information in the video, the image information of the commodity object, and the text information such as the title, category, and attributes, in an optional manner, a preprocessing module can be used for preprocessing first, and then the preprocessed image information and text information are input into the algorithm model. For each specific image frame or commodity object, the information input into the algorithm model can include two parts: image information and text information. The algorithm model can be used to extract features from the input image and input text respectively, and then fuse them, and then output the feature information of the commodity object or image frame.

[0070] Among them, Figure 4As shown in FIG. 1 , for the image information of the commodity object or the image information of the image frame, the specific preprocessing operation may be to detect and intercept the image of the commodity object itself and scale it to the target size to obtain the input image of the algorithm model. In specific implementation, the pre-established commodity detection model may be used to predict the area where the commodity is located in the image. For example, Figure 4 The image is then cut out as shown in the "commodity frame" in the image. Through this step, the content in the original image that is irrelevant to the specific commodity object, such as the head of the model, can be deleted to reduce the interference to the feature extraction process. For ease of description, the cut-out image is recorded as I.

[0071] For the text information associated with the commodity object or the text information associated with the image frame (for the convenience of description, the unprocessed text information of the commodity object or the text information of the image frame in the video is denoted as R), the specific preprocessing operation may include: performing word segmentation on the text information, and extracting a target number of words from the multiple words obtained by the word segmentation to obtain the input text of the algorithm model. In other words, for the text information associated with a commodity object or the text information associated with the image frame in the video, the number of words obtained by the specific word segmentation is not fixed. Therefore, in order to facilitate calculation in the algorithm model, the text information can be standardized into the same number of words. In this way, the input text of the algorithm model is unified into a text composed of several words (marked as T).

[0072] Of course, in the specific implementation, since both the voice text and the product text may contain some pure numbers, pure English, modal particles (especially common in voice recognition), or other words that are irrelevant to the product object description (such as "free shipping", etc.), these words may cause interference in the subsequent feature extraction process. Therefore, in an optional implementation method, it is also possible to first filter out words that are irrelevant to the product object description information from the word segmentation results according to certain rules. Then extract the target number of words.

[0073] Specifically, when performing vocabulary extraction, if the number of words in the word segmentation results of the same text information is greater than the target number, the target number of words can be extracted from the word segmentation results, and the extraction can be performed randomly to obtain the input text of the algorithm model. In actual applications, there are also cases where the number of words in the word segmentation results of the same text information is less than the target number, or even empty (especially the image frame of the video, the associated audio information may not have voice or all irrelevant words). At this time, the concept of a DUMMY (placeholder) word can be introduced, and by inserting this placeholder word, an input text T containing the target number of words can be obtained. For example, in specific implementation, a maximum length K (for example, K=5) can be set first. If the number of words contained in the word segmentation result of a certain text information (which can be filtered) is greater than K, K words can be randomly selected as the input text T of the algorithm model. In the case where the number of words after word segmentation and filtering is less than K, several DUMMY words can be added to the input text T of the algorithm model so that the input text T of the algorithm model contains just K words.

[0074] After completing the preprocessing of the image and text information, they can be input into the algorithm model as input images and input texts respectively. That is to say, for the same product object or the same image frame, the information input into the algorithm model always includes two parts: image information and text information. The algorithm model extracts image features and text features, and then fuses the two parts of the features to obtain the feature information of the specific product object or image frame.

[0075] In a specific implementation, such as Figure 5As shown, the specific algorithm model may include two convolutional neural networks, which are used to extract image features and text features respectively. After that, the image features and text features are fused through a feature fusion module, and the output features can be used as feature information of a specific commodity object or image frame for further comparison. For example, for the preprocessed image I, it can be scaled to a size of 224*224 (or other values), and then input into network A. A can be a conventional convolutional neural network (CNN), such as ResNet (residual network), ResNetFPN (image pyramid network), VGG (Visual Geometry Group, a convolutional neural network), AlexNet (Alex network, another convolutional neural network), etc. Network A can extract an n-dimensional feature feature vector x from the input image. In specific implementation, since image information can usually contain richer information than text information, the final fusion result is often mainly based on image features, supplemented by other features. Therefore, the network can be called a "trunk network", and the network that extracts features from other information can be called a "branch network", and so on.

[0076] In the text branch, the text information T consisting of K words can first be input into an embedding (using a low-dimensional vector to represent an object) layer to obtain the initial word vector, and then the word vector is input into the text feature extraction network C to obtain a global text feature vector t. Here, the network C can be a convolutional neural network, or an RNN (Recurrent Neural Network) network with LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Neural Network). The DUMMY word D mentioned above can also be trained with the network to obtain a specific expression.

[0077] In addition, during the specific implementation, although the area where the main image of the commodity object is located is cut out from the image information that has been preprocessed, the area may also include some background images in addition to the main image of the commodity object. Especially in live broadcast scenes, such background images may be very messy, which seriously affects the accuracy of feature extraction. Therefore, in an optional implementation, another branch network (for example, it can be called a Mask branch) can be provided in the specific algorithm model. The branch network can perform feature extraction based on the predicted results of the image contour of the commodity object in the input image. At this time, the specific algorithm model can fuse the feature extraction results of the backbone network with the feature extraction results of the sub-network, and then fuse them with the feature extraction results of the input text to output the feature information of the commodity object or image frame.

[0078] For example, in the specific implementation, you can take one or more layers of the backbone network A output F, and then use F to predict the mask M (the outline of the product) of the product in the image I. Each point in M ​​is a number between 0 and 1, where 0 represents the non-product image area and 1 represents the product image area. Then you can multiply M and F and input them into a sub-network B to extract a feature y. The significance of the Mask branch is that it can better allow the network to extract global features. The sub-network B can be a convolutional neural network composed of several convolutions.

[0079] After the main network and the branch network obtain their own extracted features, feature fusion processing can be performed. In a specific implementation, feature fusion can be implemented by a feature fusion module based on an attention mechanism. For example, Figure 5 In the model shown, two feature fusion modules based on the attention mechanism can be used to fuse feature x and feature y into feature z, and then fuse feature z and feature t into feature vector o.

[0080] The feature vector output by the specific calculation can be used as the feature of a specific commodity object or the feature of an image frame in a specific video. In this way, feature information of multiple commodity objects and multiple image frames can be obtained respectively.

[0081] In the specific implementation, the feature information of the commodity object can be extracted and fused in advance, and then the feature information corresponding to multiple commodity objects is constructed into an index library. After determining the specific video to be processed, the image frame is extracted, and the specific image frame features are extracted and fused to obtain the feature information of the image frame.

[0082] It should be noted that, when the algorithm model is trained, the structure of the algorithm model can be determined in advance, for example, a convolutional neural network structure can be used, and a specific feature fusion module can be determined, etc. In addition, multiple known "<image frame, commodity object> pairs" can be obtained in advance as training samples. For example, if it is known that a certain image frame is introducing a certain commodity object, the two can be combined into an <image frame, commodity object> pair, and multiple such <image frame, commodity object> pairs can be used as training samples for the algorithm model. When training is carried out specifically, information of the image frame and the commodity object in both image and text types can also be obtained as input information of the algorithm model. In each round of iterative operation, the algorithm model can extract image features and text features of the image frame respectively, and fuse them into feature information of the image frame. In addition, the image features and text features of the commodity object can be extracted, and then fused into feature information of the commodity object. Then, the distance between the feature information of the image frame and the feature information of the commodity object in the same <image frame, commodity object> pair is calculated, and the parameters in the algorithm model (for example, the weight matrix in the neural network structure, etc.) are adjusted with the goal of gradually reducing the distance between the two. Once the algorithm converges and the final parameters in the algorithm model are determined, the training of the algorithm model can be completed.

[0083] Among them, in a specific implementation, the expression of the feature fusion module E based on the attention mechanism can be:

[0084] z=x+(α+b)*s*y

[0085]

[0086] Where α is the cosine of the angle θ between the two vectors x and y. b is the bias term, s is the scaling term, and both b and s are parameters that the network needs to learn.

[0087] The loss function (Loss) during training can be:

[0088] L total =L mask +L triplet

[0089] Where L mask It is the loss of calculating the predicted mask M, and the cross entropy loss function of binary classification can be used. triplet The triplet loss function can be used, and so on.

[0090] S250: Determine a matching relationship between the image frame and the commodity object by calculating a similarity between the feature information of the image frame and the feature information of the commodity object.

[0091] After determining the feature information of multiple image frames and the feature information of multiple commodity objects, the multiple image frames can be respectively calculated for similarity with the features of multiple commodity objects, thereby determining the matching relationship between the image frames and the commodity objects. For example, if the feature similarity between an image frame and a commodity object is higher than the target threshold, it proves that the image frame may be used to introduce the commodity object, thereby completing the matching between the two. After determining the matching relationship between multiple image frames and commodity objects, corresponding marks can be added to the corresponding positions in the playback progress bar of the video according to the positions (time points) where the commodity objects in the video change. Alternatively, it is also possible to truncate at the corresponding positions to generate a "highlight" short video about the specific corresponding commodity object, and so on.

[0092] In the case of generating the above-mentioned short video, there can be a variety of specific applications. For example, when providing search results according to a commodity object search request, if the commodity object in the search result has a corresponding short video, information about the short video can be provided in the search result page. In addition to the ordinary search initiated by inputting keywords, the search request can also include: a request to search for the same commodity object as the specified commodity object, or a request to search for commodity objects that have a matching relationship with the specified commodity object, etc.

[0093] In addition, when providing the user with recommendation information of a target commodity object that has a matching relationship with the commodity object currently being displayed, if the target commodity object has a corresponding short video, information about the short video may also be provided in the recommendation information.

[0094] Specifically, during the process of searching or recommending, in addition to interacting through traditional screen input methods, you can also interact through voice, etc.

[0095] In summary, through the embodiments of the present application, when detecting commodity objects from a video, the image and text information of the commodity object can be fused to extract the features of the commodity object, and the image information of the image frame in the video and the related audio information can be fused to extract the features of the image frame. Since multimodal information such as images, texts, and audio can be fused, the accuracy of feature extraction can be improved, so that when matching image frames with commodity objects by comparing feature similarities, the success rate and accuracy of matching can be improved.

[0096] Embodiment 2

[0097] This embodiment 2 provides a live video stream processing method from the perspective of the client. Figure 6 , the method may include:

[0098] S610: Acquire commodity object index data, wherein the index data includes feature information of a plurality of commodity objects, wherein the feature information of the commodity objects is obtained based on image features and associated text features of the commodity objects;

[0099] Specifically, when obtaining the commodity object index data, it can be obtained from the server. The index data provided by the server can also have multiple copies, corresponding to different sets of commodity objects. For example, separate indexes can be established for different store objects, and the index corresponding to each store object can only include the characteristic information of multiple commodity objects associated with the store object. In this way, when the client obtains the commodity object index data, it can also first determine the target store object information associated with the currently playing live video stream, and then obtain the commodity object index data associated with the target store object, so that in the process of playing the live video stream, the index data associated with the target store object can be used to determine the commodity objects corresponding to the multiple image frames in the live video stream, thereby reducing the difficulty of retrieval.

[0100] In the specific index data, the feature information of the commodity object may be obtained by the method in the aforementioned first embodiment, that is, by fusing the image features and text features of the commodity object, feature information that can more accurately describe the commodity object is obtained.

[0101] S620: Acquire a live video stream, acquire multiple image frames and associated audio information from the live video stream, and perform voice recognition on the audio information to obtain text information associated with the image frames;

[0102] S630: Acquire feature information of the image frame based on image information of the image frame and associated text information;

[0103] S640: Determine a matching relationship between the image frame and the commodity object by calculating a similarity between the feature information of the image frame and the feature information of the commodity object in the index data;

[0104] The details of steps S620 to S640 may first refer to the description in the first embodiment.

[0105] S650: According to the matching relationship, provide information of the corresponding commodity object in the playback interface of the live video stream.

[0106] During the live video broadcast, after the client obtains the real-time matching relationship between the image frame and the commodity object, the client can promptly provide the user with corresponding information. For example, the corresponding commodity object information can be provided in the playback interface of the live video stream, for example, a link to the commodity object can be included. In this way, it is convenient for users to obtain information about the commodity object introduced in the video while watching the live video.

[0107] In addition, during the specific implementation, the client can use the first algorithm model to perform image feature extraction and text feature extraction on the image information of the multiple image frames and the associated text information, and merge the image features with the text features to obtain the feature information of the multiple image frames. Among them, the first algorithm model used by the client may be relatively lightweight, so the ability to detect commodity objects in the live video stream may be relatively limited. In addition, the specific detection results can also provide users with richer information in a variety of other application scenarios. Therefore, in this case, the client can also submit the matching relationship to the server. After the server subsequently completes the commodity object detection of the recording results of the video stream through the enhanced second algorithm model, it can also be combined with the results submitted by the client. The two parts of the detection results can complement or verify each other, further improving the success rate and accuracy of the detection.

[0108] In addition, in a specific implementation, environmental category information (for example, indoor or outdoor, etc.) in the live video stream can also be identified. In this way, when the live video stream is recorded as a video file, a label can be added to the video file according to the environmental category information, so as to push the video file to the user according to the label, or provide video file search results.

[0109] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0110] Corresponding to the first embodiment, the present application embodiment also provides a video information processing device, see Figure 7 , the device may include:

[0111] The object information determining unit 710 is used to determine a target commodity object set, and images of multiple commodity objects in the target commodity object set and associated text information;

[0112] The associated text recognition unit 720 is used to obtain multiple image frames and their associated audio information from the video to be processed, and perform speech recognition on the audio information to obtain text information associated with the image frames;

[0113] A commodity object feature information extraction unit 730, configured to obtain feature information of the commodity object based on the image of the commodity object and associated text information;

[0114] An image frame feature extraction unit 740, configured to obtain feature information of the image frame based on image information of the image frame and associated text information thereof;

[0115] The matching relationship determining unit 750 is configured to determine the matching relationship between the image frame and the commodity object by calculating the similarity between the feature information of the image frame and the feature information of the commodity object.

[0116] When implementing it specifically, it may also include:

[0117] The feature extraction unit is used to extract features from the multiple commodity objects and multiple image frames using a pre-trained algorithm model; the algorithm model is used to extract features from the input image and the input text respectively and after fusing them, output feature information of the commodity objects or image frames.

[0118] Additionally, it may include:

[0119] The preprocessing unit is used to preprocess the image of the commodity object and the associated text information, the image information of the image frame and the associated text information before using the algorithm model to extract features, so as to obtain the input image and input text of the algorithm model.

[0120] Specifically, the preprocessing unit is specifically used for:

[0121] For the image information of the commodity object or the image information of the image frame, the commodity object body image is detected and captured, and scaled to a target size to obtain an input image for the algorithm model.

[0122] Alternatively, the preprocessing unit is specifically used for:

[0123] The text information associated with the commodity object or the text information associated with the image frame is segmented, and a target number of words are extracted from the multiple words obtained by the segmentation to obtain the input text of the algorithm model.

[0124] In addition, the device may also include:

[0125] The vocabulary extraction unit is used to extract the target number of words from the word segmentation results if the number of words in the word segmentation results of the same text information is greater than the target number, so as to obtain the input text of the algorithm model.

[0126] The placeholder word insertion unit is used to insert placeholder words if the number of words in the word segmentation result of the same text information is less than the target number, so as to obtain an input text containing the target number of words.

[0127] The vocabulary filtering unit is used to filter out words that are irrelevant to the description information of the product object from the word segmentation results.

[0128] The algorithm model includes a backbone network, a first sub-network, a second sub-network, a first feature fusion module, and a second feature fusion module;

[0129] The backbone network is used to extract features from the input image, the first sub-network is used to extract features from the input text, and the second sub-network is used to extract features based on the prediction result of the image contour of the commodity object in the input image;

[0130] The first feature fusion module is used to fuse the feature extraction result of the backbone network with the feature extraction result of the second sub-network;

[0131] The second feature fusion module is used to fuse the output result of the first feature fusion module with the feature extraction result of the input text, and output feature information of the commodity object or image frame.

[0132] Specifically, the algorithm model is specifically used to perform feature fusion processing through a feature fusion module based on an attention mechanism.

[0133] The associated text recognition unit can be specifically used to extract image frames from the video to be processed at target time intervals, and determine audio information of a target time length before and / or after the extracted image frames as audio information associated with the image frames.

[0134] In addition, the device may also include:

[0135] A change position determination unit is used to determine the position where the commodity object changes in the video based on the matching relationship between the image frame and the commodity object, so as to add a corresponding mark in the playback progress bar of the video according to the position, or split the video into multiple short videos corresponding to different commodity objects.

[0136] In one scenario, the device may further include:

[0137] The search result processing unit is used to provide search results according to a product object search request, and if there is a corresponding short video for the product object in the search result, provide information about the short video in the search result page.

[0138] The search request includes: a request to search for a commodity object of the same type as the specified commodity object, or a request to search for a commodity object that has a matching relationship with the specified commodity object.

[0139] It may also include:

[0140] The recommendation result processing unit is used to provide the user with recommendation information of a target commodity object that has a matching relationship with the commodity object currently being displayed, and if the target commodity object has a corresponding short video, provide information about the short video in the recommendation information.

[0141] Corresponding to the second embodiment, the present application embodiment also provides a live video stream processing device, see Figure 8 , the device may include:

[0142] An index data acquisition unit 810 is used to acquire commodity object index data, wherein the index data includes feature information of a plurality of commodity objects, wherein the feature information of the commodity objects is obtained based on image features and associated text features of the commodity objects;

[0143] A live stream acquisition unit 820 is used to acquire a live video stream, obtain multiple image frames and their associated audio information from the live video stream, and perform speech recognition on the audio information to obtain text information associated with the image frames;

[0144] An image frame feature acquisition unit 830, configured to acquire feature information of the image frame based on image information of the image frame and associated text information thereof;

[0145] A matching relationship determination unit 840, configured to determine a matching relationship between an image frame and a commodity object by calculating a similarity between feature information of the image frame and feature information of the commodity object in the index data;

[0146] The commodity object information providing unit 850 is used to provide the corresponding commodity object information in the playback interface of the live video stream according to the matching relationship.

[0147] The index data acquisition unit may be specifically used for:

[0148] Determine target store object information associated with the live video stream;

[0149] Acquire the commodity object index data associated with the target store object, so as to determine the commodity objects corresponding to the multiple image frames in the live video stream by using the index data associated with the target store object during the playing of the live video stream;

[0150] The commodity object index data associated with the target store object includes feature information of multiple commodity objects associated with the target store object.

[0151] The image frame feature acquisition unit can be specifically used for:

[0152] The first algorithm model is used to perform image feature extraction and text feature extraction on the image information of the multiple image frames and the associated text information, respectively, and the image features are merged with the text features to obtain the feature information of the multiple image frames.

[0153] In addition, the device may also include:

[0154] A matching relationship submission unit is used to submit the matching relationship to a server. After the live broadcast ends, the server is used to use a second algorithm model to match multiple image frames in the recording results of the live video stream with the commodity objects, and obtain a final matching relationship by combining with the matching relationship corresponding to the first algorithm model.

[0155] Furthermore, the device may further include:

[0156] An environment category identification unit, used to identify environment category information in the live video stream;

[0157] A tag adding unit is used to add tags to the video file according to the environment category information when recording the live video stream as a video file, so as to push the video file to the user according to the tag, or provide video file search results.

[0158] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0159] And an electronic device, comprising:

[0160] one or more processors; and

[0161] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0162] in, Fig. 9The architecture of the electronic device is shown as an example, which may include a processor 910, a video display adapter 911, a disk drive 912, an input / output interface 913, a network interface 914, and a memory 920. The processor 910, the video display adapter 911, the disk drive 912, the input / output interface 913, the network interface 914, and the memory 920 may be communicatively connected via a communication bus 930.

[0163] Among them, the processor 910 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in this application.

[0164] The memory 920 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 920 can store an operating system 921 for controlling the operation of the electronic device 900, and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device 900. In addition, a web browser 923, a data storage management system 924, and a video matching processing system 925, etc. can also be stored. The above-mentioned video matching processing system 925 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 920 and is called and executed by the processor 910.

[0165] The input / output interface 913 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0166] The network interface 914 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0167] The bus 930 comprises a pathway for transmitting information between the various components of the device (eg, the processor 910, the video display adapter 911, the disk drive 912, the input / output interface 913, the network interface 914, and the memory 920).

[0168] It should be noted that, although the above device only shows a processor 910, a video display adapter 911, a disk drive 912, an input / output interface 913, a network interface 914, a memory 920, a bus 930, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0169] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0170] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0171] The video information processing method, device and electronic device provided by the present application are introduced in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A video information processing method, characterized in that: include: Determine a target commodity object set, and images of multiple commodity objects in the target commodity object set and associated text information; Acquire multiple image frames and their associated audio information from the video to be processed, and perform speech recognition on the audio information to obtain text information associated with the image frames; Acquiring feature information of the commodity object based on the image of the commodity object and associated text information; Acquire feature information of the image frame based on image information of the image frame and associated text information thereof; By determining the similarity between the feature information of the image frame and the feature information of the commodity object, a matching relationship between the image frame and the commodity object is determined.

2. The method according to claim 1, characterized in that Also includes: Using a pre-trained algorithm model, extract features from the commodity object and the image frame; The algorithm model is used to extract features from an input image and an input text respectively, and after fusing the features, output feature information of a commodity object or an image frame.

3. The method according to claim 2, characterized in that Before using the algorithm model to extract features, the method also includes: The image of the commodity object and the associated text information, the image information of the image frame and the associated text information are preprocessed respectively to obtain the input image and input text of the algorithm model.

4. The method according to claim 3, characterized in that The pre-processing comprises: For the image information of the commodity object or the image information of the image frame, the commodity body image is detected and captured, and scaled to a target size to obtain an input image for the algorithm model.

5. The method according to claim 3, characterized in that: The pre-processing comprises: The text information associated with the commodity object or the text information associated with the image frame is segmented, and a target number of words are extracted from the multiple words obtained by the segmentation to obtain the input text of the algorithm model.

6. The method according to claim 5, characterized in that Also includes: If the number of words in the word segmentation results of the same text information is greater than the target number, the target number of words are extracted from the word segmentation results to obtain the input text of the algorithm model.

7. The method according to claim 5, characterized in that Also includes: If the number of words in the word segmentation results of the same text information is less than the target number, placeholder words are inserted to obtain an input text containing the target number of words.

8. The method according to claim 5, characterized in that Also includes: Filter out words that are irrelevant to the product object description information from the word segmentation results.

9. The method according to any one of claims 2 to 8, characterized in that: The algorithm model includes a backbone network, a first sub-network, a second sub-network, a first feature fusion module, and a second feature fusion module; The backbone network is used to extract features from the input image, the first sub-network is used to extract features from the input text, and the second sub-network is used to extract features based on the prediction result of the contour of the commodity body image in the input image; The first feature fusion module is used to fuse the feature extraction result of the backbone network with the feature extraction result of the second sub-network; The second feature fusion module is used to fuse the output result of the first feature fusion module with the feature extraction result of the input text, and output feature information of the commodity object or image frame.

10. The method according to any one of claims 2 to 8, characterized in that: The algorithm model is specifically used to perform feature fusion processing through a feature fusion module based on an attention mechanism.

11. The method according to any one of claims 1 to 8, characterized in that: The step of obtaining a plurality of image frames and associated audio information from the video to be processed includes: Image frames are extracted from the video to be processed at target time intervals, and audio information of a target time length before and / or after the extracted image frames is determined as audio information associated with the image frames.

12. The method according to any one of claims 1 to 8, characterized in that: Also includes: According to the matching relationship between the image frame and the commodity object, the position where the commodity object changes in the video is determined, so as to add a corresponding mark in the playback progress bar of the video according to the position, or split the video into multiple short videos corresponding to different commodity objects.

13. The method according to claim 12, characterized in that Also includes: When providing search results according to a commodity object search request, if a corresponding short video exists for a commodity object in the search results, information about the short video is provided in the search result page.

14. The method according to claim 13, characterized in that The search request includes: a request to search for a commodity object of the same type as the designated commodity object, or a request to search for a commodity object that has a matching relationship with the designated commodity object.

15. The method according to claim 12, characterized in that Also includes: When providing the user with recommendation information of a target commodity object that has a matching relationship with a commodity object currently being displayed, if the target commodity object has a corresponding short video, information about the short video is provided in the recommendation information.

16. A method for processing a live video stream, characterized in that: include: Acquire commodity object index data, wherein the index data includes feature information of a plurality of commodity objects, wherein the feature information of the commodity objects is obtained based on image features and associated text features of the commodity objects; Acquire a live video stream, acquire multiple image frames and associated audio information from the live video stream, and perform voice recognition on the audio information to obtain text information associated with the image frames; Acquire feature information of the image frame based on image information of the image frame and associated text information thereof; Determining a matching relationship between the image frame and the commodity object by calculating a similarity between the feature information of the image frame and the feature information of the commodity object in the index data; According to the matching relationship, information of the corresponding commodity object is provided in the playback interface of the live video stream.

17. The method according to claim 16, characterized in that The obtaining of commodity object index data includes: Determine target store object information associated with the live video stream; Acquire the commodity object index data associated with the target store object, so as to determine the commodity objects corresponding to the multiple image frames in the live video stream by using the index data associated with the target store object during the playing of the live video stream; The commodity object index data associated with the target store object includes feature information of multiple commodity objects associated with the target store object.

18. The method according to claim 16, characterized in that The acquiring feature information of the image frame based on the image information of the image frame and the associated text information thereof includes: The first algorithm model is used to perform image feature extraction and text feature extraction on the image information of the image frame and the associated text information, respectively, and the image features are merged with the text features to obtain the feature information of the image frame.

19. The method according to claim 18, characterized in that The method further comprises: The matching relationship is submitted to the server, which is used to match multiple image frames in the recording results of the live video stream with the product objects using the second algorithm model after the live broadcast ends, and obtain the final matching relationship by combining with the matching relationship corresponding to the first algorithm model.

20. The method according to claim 16, characterized in that Also includes: Identifying environment category information in the live video stream; When the live video stream is recorded as a video file, a tag is added to the video file according to the environment category information, so as to push the video file to the user according to the tag, or provide a video file search result.

21. A video information processing device, characterized in that: include: An object information determination unit, used to determine a target commodity object set, and images of multiple commodity objects in the target commodity object set and associated text information; An associated text recognition unit, used to obtain multiple image frames and their associated audio information from the video to be processed, and perform speech recognition on the audio information to obtain text information associated with the image frames; A commodity object feature information extraction unit, configured to obtain feature information of the commodity object based on the image of the commodity object and associated text information; An image frame feature extraction unit, configured to obtain feature information of the image frame based on image information of the image frame and associated text information thereof; The matching relationship determining unit is used to determine the matching relationship between the image frame and the commodity object by calculating the similarity between the feature information of the image frame and the feature information of the commodity object.

22. A live video stream processing device, characterized in that: include: An index data acquisition unit, used to acquire commodity object index data, wherein the index data includes feature information of a plurality of commodity objects, wherein the feature information of the commodity objects is obtained based on image features and associated text features of the commodity objects; A live stream acquisition unit, used to acquire a live video stream, acquire multiple image frames and their associated audio information from the live video stream, and perform voice recognition on the audio information to obtain text information associated with the image frames; An image frame feature acquisition unit, configured to acquire feature information of the image frame based on image information of the image frame and associated text information thereof; a matching relationship determination unit, configured to determine a matching relationship between an image frame and a commodity object by calculating a similarity between feature information of the image frame and feature information of the commodity object in the index data; A commodity object information providing unit is used to provide corresponding commodity object information in the playback interface of the live video stream according to the matching relationship.

23. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 20 are implemented.

24. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 20.

Images