Method, device, electronic device and storage medium for determining target recommendation object

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for determining a target recommendation object. The method includes performing target detection on a live frame image of a target live room during the live streaming playback of a target live room, obtaining an object image of the target detection object in the live frame image and first category information corresponding to the target detection object; performing live object category recognition based on live content information corresponding to the live frame image, obtaining second category information of the live object in the target live room; determining a live intent object from the target detection object based on the second category information and the first category information; and determining a target recommendation object from objects to be recommended in the target live room based on the object image of the live intent object. The disclosed embodiments can be used to quickly and accurately automatically match the target recommendation object corresponding to the explanation intent from the objects to be recommended, thereby improving the convenience of the target recommendation object delivery operation.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a method, device, electronic device and storage medium for determining a target recommendation object. Background Art

[0002] With the development of the Internet and live broadcast technology, information promotion based on live broadcast has become a new form of information dissemination. In the promotion process of commodities, novels, applications and other objects, it is often necessary to put the corresponding object recommendation information into the live broadcast room.

[0003] In the related art, when the host starts explaining a certain object to be recommended, he can manually set the explanation status of a certain object to the explanation status, and then display the object recommendation information of the corresponding object to be recommended in the live broadcast page in the form of a card or the like. However, this manual method has problems such as low object coverage and inconvenient operation. Therefore, how to automatically and accurately determine the object to be explained in the live broadcast stream from a large number of objects to be recommended has become an urgent problem to be solved. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device and storage medium for determining a target recommendation object, which can quickly and accurately automatically match the recommended object corresponding to the explanation intention from the recommended objects, thereby improving the convenience of the recommended object delivery operation. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for determining a target recommendation object is provided, comprising:

[0006] During the live streaming playback of the target live room, target detection is performed on the live frame image of the target live room to obtain an object image of the target detection object in the live frame image and first category information corresponding to the target detection object;

[0007] Performing live object category recognition based on the live content information corresponding to the live frame image to obtain second category information of the live object in the target live room;

[0008] Based on the second category information and the first category information, determining a live broadcast intention object from the target detection objects;

[0009] Based on the object image of the live broadcast intention object, a target recommended object is determined from the objects to be recommended in the target live broadcast room.

[0010] Optionally, the determining the target recommended object from the objects to be recommended in the target live broadcast room based on the object image of the live broadcast intention object comprises:

[0011] Acquire an object image of the object to be recommended;

[0012] Input the object image of the object to be recommended and the object image of the live broadcast intention object into the fine-grained representation network for fine-grained representation, respectively, to obtain first fine-grained feature information corresponding to the live broadcast intention object and second fine-grained feature information corresponding to the object to be recommended;

[0013] Based on the first fine-grained feature information and the second fine-grained feature information, matching an initial recommended object from the objects to be recommended;

[0014] Inputting the object image of the live broadcast intention object and the speech recognition text information corresponding to the live broadcast intention object into a multimodal representation network for multimodal representation to obtain first multimodal feature information corresponding to the live broadcast intention object;

[0015] Inputting the object image of the initial recommended object and the associated text information corresponding to the initial recommended object into a multimodal representation network for multimodal representation to obtain second multimodal feature information corresponding to the initial recommended object;

[0016] Based on the first multimodal feature information and the second multimodal feature information, the target recommended object is matched from the initial recommended objects.

[0017] Optionally, matching an initial recommended object from the objects to be recommended based on the first fine-grained feature information and the second fine-grained feature information includes:

[0018] Determining first matching information between the live broadcast intention object and the object to be recommended according to the first fine-grained feature information and the second fine-grained feature information;

[0019] Based on the first matching information, matching the initial recommended object from the objects to be recommended;

[0020] The matching the target recommended object from the initial recommended objects based on the first multimodal feature information and the second multimodal feature information includes:

[0021] Determining second matching information between the live broadcast intended object and the initial recommended object according to the first multimodal feature information and the second multimodal feature information;

[0022] Based on the second matching information, the target recommended object is matched from the initial recommended objects.

[0023] Optionally, the method further includes:

[0024] Determining target matching information according to the first matching information and the second matching information;

[0025] The matching the target recommended object from the initial recommended objects based on the second matching information includes:

[0026] Based on the target matching information, the target recommended object is matched from the initial recommended objects.

[0027] Optionally, the method further includes:

[0028] Acquire a sample object image of a first sample object and a first target category identifier of the first sample object;

[0029] Inputting the sample object image into a first deep learning network for fine-grained characterization to obtain sample fine-grained feature information corresponding to the first sample object;

[0030] Inputting the sample fine-grained feature information into a first preset classification network for classification processing to obtain first predicted classification information;

[0031] Determining first target loss information based on the first predicted classification information and the first target category identifier;

[0032] Based on the first target loss information, training the first preset classification network and the first deep learning network;

[0033] The trained first deep learning network is used as the fine-grained representation network.

[0034] Optionally, the first predicted classification information includes a first predicted category identifier of the first sample object; the first target loss information includes a category identifier loss;

[0035] The determining of first target loss information based on the first prediction classification information and the first target category identifier includes:

[0036] The category identification loss is determined according to the first predicted category identification and the first target category identification.

[0037] Optionally, the first target loss information further includes category loss and / or category metric loss;

[0038] In a case where the first target loss includes the category loss, the first predicted classification information further includes predicted category information of the first sample object;

[0039] In the case where the first target loss includes the category metric loss, the first sample object includes a positive sample object pair and a negative sample object pair, the positive sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to the same sample object, and the negative sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to different sample objects; the first predicted classification information also includes first classification feature information of objects in the positive sample object pair and second classification feature information of objects in the negative sample object pair;

[0040] The determining the first target loss information based on the first prediction classification information and the first target category identifier further includes:

[0041] Acquire target category information of the first sample object;

[0042] determining the category loss according to the target category information and the predicted category information;

[0043] and / or,

[0044] Determining a first category similarity between objects in the positive sample object pair according to the first classification feature information;

[0045] Determining a second category similarity between objects in the negative sample object pair according to the second classification feature information;

[0046] The category metric loss is determined according to the first category similarity and the second category similarity.

[0047] Optionally, the method further includes:

[0048] Acquire a sample object image of a second sample object, object text information of the second sample object, and a second target category identifier of the second sample object;

[0049] Inputting the sample object image and the object text information into a second deep learning network for multimodal representation to obtain sample multimodal feature information corresponding to the second sample object;

[0050] Inputting the sample multimodal feature information into a second preset classification network for classification processing to obtain second predicted classification information;

[0051] Determining second target loss information based on the second predicted classification information and the second target category identifier;

[0052] Based on the second target loss information, training the second preset classification network and the second deep learning network;

[0053] The trained second deep learning network is used as the multimodal representation network.

[0054] Optionally, the determining the target recommended object from the objects to be recommended in the target live broadcast room based on the object image of the live broadcast intention object comprises:

[0055] Acquire an object image of the object to be recommended;

[0056] Inputting the object image of the object to be recommended and the object image of the live broadcast intention object into a feature characterization network respectively for object feature characterization, and obtaining first object feature information of the object to be recommended and second object feature information of the live broadcast intention object respectively;

[0057] Based on the first object feature information and the second object feature information, the target recommended object is matched from the objects to be recommended.

[0058] Optionally, the performing target detection on the live frame image of the target live room to obtain the object image of the target detection object in the live frame image and the first category information corresponding to the target detection object includes:

[0059] The live frame image is input into a target detection network for target detection to obtain an object image of the target detection object and first category information corresponding to the target detection object.

[0060] Optionally, the method further includes:

[0061] Acquire a labeled sample image and an unlabeled sample image, wherein the labeled sample image is a sample object image labeled with object detection information;

[0062] Based on the labeled sample images, performing target detection training on the detection network to be trained to obtain an initial detection network;

[0063] Inputting the unlabeled sample image into the initial detection network to perform target detection, and obtaining sample category information and category confidence data of a third sample object in the unlabeled sample image;

[0064] Determining a target sample image from the non-annotated sample images according to the category confidence data;

[0065] Annotate the target sample image based on the sample category information corresponding to the target sample image to obtain an incrementally annotated sample image;

[0066] Based on the labeled sample images and the incremental labeled sample images, target detection training is performed on an initial detection network to obtain a target detection network.

[0067] Optionally, the performing of the live object category identification based on the live content information corresponding to the live frame image to obtain the second category information of the live object in the target live room includes:

[0068] The live broadcast content information is input into a category recognition network for category recognition to obtain the second category information.

[0069] Optionally, the method further includes:

[0070] The object recommendation information of the target recommended object is displayed on the live broadcast page corresponding to the target live broadcast room.

[0071] According to a second aspect of an embodiment of the present disclosure, there is provided a device for determining a target recommendation object, comprising:

[0072] The first target detection module is configured to perform target detection on the live frame image of the target live room during the live stream playback of the target live room, and obtain an object image of the target detection object in the live frame image and first category information corresponding to the target detection object;

[0073] A category identification module is configured to perform category identification of a live object based on the live content information corresponding to the live frame image, and obtain second category information of the live object in the target live room;

[0074] A live broadcast intention object determination module is configured to determine a live broadcast intention object from the target detection object based on the second category information and the first category information;

[0075] The target recommendation object determination module is configured to execute the object image based on the live broadcast intention object, and determine the target recommendation object from the objects to be recommended in the target live broadcast room.

[0076] Optionally, the target recommendation object determination module includes:

[0077] A first object image acquisition unit is configured to acquire an object image of the object to be recommended;

[0078] A fine-grained characterization unit is configured to input the object image of the object to be recommended and the object image of the live broadcast intention object into a fine-grained characterization network for fine-grained characterization, respectively, to obtain first fine-grained feature information corresponding to the live broadcast intention object and second fine-grained feature information corresponding to the object to be recommended;

[0079] an initial recommended object matching unit, configured to match an initial recommended object from the objects to be recommended based on the first fine-grained feature information and the second fine-grained feature information;

[0080] The first multimodal representation unit is configured to input the object image of the live broadcast intention object and the speech recognition text information corresponding to the live broadcast intention object into the multimodal representation network for multimodal representation, so as to obtain the first multimodal feature information corresponding to the live broadcast intention object;

[0081] A second multimodal representation unit is configured to input the object image of the initial recommended object and the associated text information corresponding to the initial recommended object into a multimodal representation network for multimodal representation, so as to obtain second multimodal feature information corresponding to the initial recommended object;

[0082] The first target recommendation object matching unit is configured to match the target recommendation object from the initial recommendation objects based on the first multimodal feature information and the second multimodal feature information.

[0083] Optionally, the initial recommendation object matching unit includes:

[0084] A first matching information determining unit is configured to determine first matching information between the live broadcast intention object and the to-be-recommended object according to the first fine-grained feature information and the second fine-grained feature information;

[0085] an initial recommended object determining unit, configured to match the initial recommended object from the objects to be recommended based on the first matching information;

[0086] The target recommendation object matching unit comprises:

[0087] A second matching information determining unit is configured to determine second matching information between the live broadcast intended object and the initial recommended object according to the first multimodal feature information and the second multimodal feature information;

[0088] The target recommended object determination unit is configured to match the target recommended object from the initial recommended objects based on the second matching information.

[0089] Optionally, the device further comprises:

[0090] a target matching information determining unit, configured to determine target matching information according to the first matching information and the second matching information;

[0091] The target recommendation object determination unit is further configured to perform matching of the target recommendation object from the initial recommendation objects based on the target matching information.

[0092] Optionally, the device further comprises:

[0093] A first sample data acquisition module is configured to acquire a sample object image of a first sample object and a first target category identifier of the first sample object;

[0094] A fine-grained characterization module is configured to input the sample object image into a first deep learning network for fine-grained characterization to obtain sample fine-grained feature information corresponding to the first sample object;

[0095] A first classification processing module is configured to input the sample fine-grained feature information into a first preset classification network for classification processing to obtain first predicted classification information;

[0096] A first target loss information determination module is configured to determine first target loss information based on the first predicted classification information and the first target category identifier;

[0097] A first network training module is configured to train the first preset classification network and the first deep learning network based on the first target loss information;

[0098] The fine-grained representation network determination module is configured to execute the trained first deep learning network as the fine-grained representation network.

[0099] Optionally, the first predicted classification information includes a first predicted category identifier of the first sample object; the first target loss information includes a category identifier loss;

[0100] The first target loss information determination module includes:

[0101] The category identification loss determining unit is configured to determine the category identification loss according to the first predicted category identification and the first target category identification.

[0102] Optionally, the first target loss information further includes category loss and / or category metric loss;

[0103] In a case where the first target loss includes the category loss, the first predicted classification information further includes predicted category information of the first sample object;

[0104] In the case where the first target loss includes the category metric loss, the first sample object includes a positive sample object pair and a negative sample object pair, the positive sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to the same sample object, and the negative sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to different sample objects; the first predicted classification information also includes first classification feature information of objects in the positive sample object pair and second classification feature information of objects in the negative sample object pair;

[0105] The first target loss information determination module also includes:

[0106] a target category information acquiring unit, configured to acquire target category information of the first sample object;

[0107] a category loss determining unit, configured to determine the category loss according to the target category information and the predicted category information;

[0108] and / or,

[0109] A first category similarity determination unit is configured to determine a first category similarity between objects in the positive sample object pair according to the first classification feature information;

[0110] A second category similarity determination unit is configured to determine a second category similarity between objects in the negative sample object pair according to the second classification feature information;

[0111] The category metric loss determining unit is configured to determine the category metric loss according to the first category similarity and the second category similarity.

[0112] Optionally, the device further comprises:

[0113] A second sample data acquisition module is configured to acquire a sample object image of a second sample object, object text information of the second sample object, and a second target category identifier of the second sample object;

[0114] a multimodal characterization module configured to input the sample object image and the object text information into a second deep learning network for multimodal characterization to obtain sample multimodal feature information corresponding to the second sample object;

[0115] A second classification processing module is configured to input the sample multimodal feature information into a second preset classification network for classification processing to obtain second predicted classification information;

[0116] A second target loss information determination module is configured to determine second target loss information based on the second predicted classification information and the second target category identifier;

[0117] A second network training module is configured to train the second preset classification network and the second deep learning network based on the second target loss information;

[0118] The multimodal representation network determination module is configured to execute the trained second deep learning network as the multimodal representation network.

[0119] Optionally, the target recommendation object determination module includes:

[0120] A second object image acquisition unit is configured to acquire an object image of the object to be recommended;

[0121] The object feature characterization unit is configured to input the object image of the object to be recommended and the object image of the live broadcast intention object into a feature characterization network for object feature characterization, respectively, to obtain first object feature information of the object to be recommended and second object feature information of the live broadcast intention object;

[0122] The second target recommendation object matching unit is configured to match the target recommendation object from the objects to be recommended based on the first object feature information and the second object feature information.

[0123] Optionally, the first target detection module is further configured to input the live frame image into a target detection network for target detection, and obtain an object image of the target detection object and first category information corresponding to the target detection object.

[0124] Optionally, the device further comprises:

[0125] A sample image acquisition module is configured to acquire an annotated sample image and an unannotated sample image, wherein the annotated sample image is a sample object image annotated with object detection information;

[0126] A first detection network training module is configured to perform target detection training on the detection network to be trained based on the labeled sample image to obtain an initial detection network;

[0127] A second target detection module is configured to input the unlabeled sample image into the initial detection network to perform target detection, and obtain sample category information and category confidence data of a third sample object in the unlabeled sample image;

[0128] a target sample image determination module, configured to determine a target sample image from the non-annotated sample images according to the category confidence data;

[0129] A category annotation module is configured to perform category annotation on the target sample image based on sample category information corresponding to the target sample image to obtain an incrementally annotated sample image;

[0130] The second detection network training module is configured to perform target detection training on the initial detection network based on the labeled sample images and the incremental labeled sample images to obtain a target detection network.

[0131] Optionally, the category identification module is further configured to input the live content information into a category identification network for category identification to obtain the second category information.

[0132] Optionally, the device further comprises:

[0133] The object recommendation information display module is configured to display the object recommendation information of the target recommended object on the live broadcast page corresponding to the target live broadcast room.

[0134] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method as described in any one of the above-mentioned first aspects.

[0135] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any method described in the first aspect of the embodiment of the present disclosure.

[0136] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any one of the methods described in the first aspect of the embodiment of the present disclosure.

[0137] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0138] During the live streaming process, after performing target detection on the live frame image and extracting the object image of the display object (target detection object) in the live screen and the first category information corresponding to the display object, the live object category recognition is performed in combination with the live content information corresponding to the live frame image, and the second category information of the object being explained in the live room can be identified; and based on the first category information and the second category information, the detected target detection objects are screened, and the target recommendation objects of the current live screen explanation intention can be accurately identified, effectively avoiding the problem of low accuracy of target recommendation objects identified only from the visual dimension, greatly improving the accuracy and coverage of live intent object recognition, and thereby improving the convenience and efficiency of subsequent target recommendation object delivery operations.

[0139] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0140] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0141] Figure 1 is a flow chart of a method for determining a target recommendation object according to an exemplary embodiment;

[0142] Figure 2 is a flowchart showing a pre-trained target detection network according to an exemplary embodiment;

[0143] Figure 3 It is a flowchart showing a method of determining a target recommended object from among the objects to be recommended in a target live broadcast room based on an object image of a live broadcast intention object according to an exemplary embodiment;

[0144] Figure 4 is a flowchart of pre-training a fine-grained representation network according to an exemplary embodiment;

[0145] Figure 5 is a flowchart of pre-training a multimodal representation network according to an exemplary embodiment;

[0146] Figure 6 is a flowchart of another method for determining a target recommendation object according to an exemplary embodiment;

[0147] Figure 7 is a block diagram of a device for determining a target recommendation object according to an exemplary embodiment;

[0148] Figure 8 It is a block diagram of an electronic device for determining a target recommendation object according to an exemplary embodiment. DETAILED DESCRIPTION

[0149] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0150] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0151] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0152] Figure 1 is a flowchart of a method for determining a target recommendation object according to an exemplary embodiment. Figure 1 As shown, the target recommendation object determination method is used in a terminal electronic device and includes the following steps.

[0153] In step S101, during the live streaming playback of the target live room, target detection is performed on the live frame image of the target live room to obtain an object image of the target detection object in the live frame image and first category information corresponding to the target detection object.

[0154] In a specific embodiment, the live streaming playback process of the target live streaming room can be during the live streaming process of the target live streaming room, or can be during the live streaming playback process of the target live streaming room. Specifically, the target live streaming room can be any live streaming room that can perform object recommendation.

[0155] In a specific embodiment, target detection can be performed on the live broadcast frame image frame by frame, and accordingly, a live broadcast image frame of the target live broadcast room can be extracted according to a preset frequency, for example, a live broadcast image frame (live broadcast frame image) is extracted every 10 seconds. Specifically, the preset frequency can be pre-set in combination with actual applications. Accordingly, the live broadcast frame image of the target live broadcast room can be a live broadcast image frame of the target live broadcast room extracted according to the preset frequency.

[0156] In another specific embodiment, target detection can also be performed by segment, and accordingly, the live frame image of the target live room can be each live frame image in the live segment of the preset detection period. Specifically, the preset detection period can be pre-set in combination with actual applications.

[0157] In the embodiments of this specification, target detection is performed by frame or by segment, and can be set according to the performance requirements in the actual application scenario.

[0158] In an optional embodiment, target detection may be performed in combination with a target detection network. Accordingly, the target detection is performed on the live frame image of the target live room, and the object image of the target detection object in the live frame image and the first category information corresponding to the target detection object may include:

[0159] The live frame image is input into the target detection network for target detection, and an object image of the target detection object and first category information corresponding to the target detection object are obtained.

[0160] In a specific embodiment, the target detection network can be obtained by pre-training the detection network to be trained for target detection based on labeled sample images. Specifically, the labeled sample images can be sample object images labeled with object detection information. Specifically, the object detection information can include the location information of the area where the object is located in the sample object image and the object category information. Optionally, the target detection network can output the location information of the target detection object in the live frame image and the category information (first category information) corresponding to the target detection object. Accordingly, the object image of the target detection object can be determined based on the location information of the target detection object in the live frame image.

[0161] In a specific embodiment, the network structure of the detection network to be trained (deep learning network) can be set in combination with actual applications.

[0162] In addition, it should be noted that in practical applications, target detection is not limited to the combination of the above-mentioned target detection network based on deep learning, but can also be combined with traditional target detection algorithms, that is, usually using a sliding window method, sliding the sliding window on the detection image, selecting the region of interest in turn, and performing feature extraction on each sliding window, such as feature extraction based on HOG (Histogram of Oriented Gradient) and other feature extraction algorithms, and then classifying the extracted features using machine learning algorithms, such as support vector machines, to finally obtain detection information on whether the window contains a certain type of object.

[0163] In the above embodiment, the live broadcast frame image is input into the target detection network for target detection, so that the object image and the corresponding category information in the live broadcast picture can be extracted quickly and accurately.

[0164] In an optional embodiment, the above method further includes: a step of pre-training a target detection network, specifically, Figure 2 As shown, pre-training the target detection network can include the following steps:

[0165] In step S201, a labeled sample image and an unlabeled sample image are obtained;

[0166] In step S203, target detection training is performed on the detection network to be trained based on the labeled sample image to obtain an initial detection network;

[0167] In step S205, the unlabeled sample image is input into the initial detection network for target detection to obtain sample category information and category confidence data of the third sample object in the unlabeled sample image;

[0168] In step S207, a target sample image is determined from the non-labeled sample images according to the category confidence data;

[0169] In step S209, the target sample image is annotated according to the category information corresponding to the target sample image to obtain an incrementally annotated sample image;

[0170] In step S211, target detection training is performed on the initial detection network based on the labeled sample images and the incremental labeled sample images to obtain a target detection network.

[0171] In a specific embodiment, the annotated sample image may be a sample object image annotated with object detection information, and correspondingly, the unannotated sample image may be a sample object image not annotated with object detection information.

[0172] In a specific embodiment, based on the labeled sample images, the target detection training of the detection network to be trained is performed to obtain the initial detection network, which may include inputting the labeled sample images into the detection network to be trained for target detection, obtaining the predicted detection information in the labeled sample images (the location information and object category information of the area where the object is located in the labeled sample images predicted by the detection network to be trained); and determining the detection loss information based on the predicted detection information and the labeled object detection information; updating the network parameters in the detection network to be trained based on the detection loss information; then, based on the updated detection network to be trained, repeating the above training iteration operation of inputting the labeled sample images into the detection network to be trained for target detection, obtaining the predicted detection information in the labeled sample images, and updating the network parameters in the detection network to be trained based on the detection loss information, until the detection convergence condition is reached. When the detection convergence condition is reached, the current detection network to be trained is used as the above initial detection network.

[0173] In an optional embodiment, the above-mentioned detection convergence condition can be that the number of training iteration operations reaches a first preset training number. Optionally, the detection convergence condition can also be that the detection loss information is less than a specified threshold. In the embodiments of this specification, the first preset training number and the specified threshold can be pre-set in combination with the training speed and accuracy of the network in actual applications.

[0174] In a specific embodiment, after obtaining the initial detection network, the non-labeled sample image can be input into the initial detection network for target detection to obtain the sample category information and category confidence data of the third sample object in the non-labeled sample image. Specifically, the category confidence data can characterize the accurate probability of the sample category information of the third sample object predicted by the initial detection network. Optionally, the non-labeled sample image with higher category confidence data can be used as the target sample image. For example, the non-labeled sample image with category confidence data greater than the first preset confidence can be used as the target sample image, or the first preset number of non-labeled sample images with larger category confidence data can be used as the target sample image. Optionally, the sample object image with medium confidence (the sample object image with category confidence data less than or equal to the first preset confidence and greater than the second preset confidence) can also be added to the sample object image after manual correction. Further, the target sample image is annotated based on the sample category information corresponding to the target sample image predicted by the initial detection network, and an incrementally labeled sample image can be obtained.

[0175] In a specific embodiment, the first preset number, the first preset reliability, and the second preset reliability can be preset in combination with actual applications, and the first preset reliability is greater than the second preset reliability.

[0176] In a specific embodiment, based on the labeled sample images and the incremental labeled sample images, the initial detection network is trained for target detection to obtain the specific refinement steps of the target detection network. Please refer to the above-mentioned steps of performing target detection training on the detection network to be trained based on the labeled sample images to obtain the specific refinement of the initial detection network, which will not be repeated here.

[0177] In the above embodiment, after the to-be-trained detection network is trained based on the labeled sample images, incremental labeled sample images are extracted from the unlabeled sample images in combination with the initial detection network, so that a large amount of training data can be quickly obtained, the labeling cost can be effectively reduced, and the network training speed and efficiency can be improved.

[0178] In step S103, the live broadcast object category is identified based on the live broadcast content information corresponding to the live broadcast frame image to obtain the second category information of the live broadcast object in the target live broadcast room.

[0179] In a specific embodiment, a collection period of live content information (hereinafter referred to as preset collection period) can be preset. Accordingly, during the live streaming playback of the target live broadcast room, live content information can be collected according to the preset collection period, and the live object category can be identified in combination with the collected live content information. Optionally, the live content information can be information that can reflect the object explained during the live broadcast of the target live broadcast room. Optionally, the live content information can include live frame images and live voice information.

[0180] In a specific embodiment, when target detection is performed on live frame images frame by frame, the end time of the preset acquisition period may be the acquisition time corresponding to the live frame images. When target detection is performed on segment by segment, the preset acquisition period may be the same as the preset detection period.

[0181] In a specific embodiment, a second preset number of live images (live frame images) and live voice information corresponding to the second preset number of live images can be equally frequency extracted in each preset acquisition cycle; and the second preset number of live images and the live voice information corresponding to the second preset number of live images are used as the live content information in the corresponding preset acquisition cycle. Specifically, the frame extraction frequency can be preset in combination with actual applications, for example, extracting a live image frame every 10 seconds.

[0182] In another specific embodiment, each live image frame (live frame image) in each preset acquisition period and the live voice information corresponding to each live image frame can be obtained; and each live image frame and the live voice information corresponding to each live image frame can be used as the live content information in the corresponding preset acquisition period.

[0183] In an optional embodiment, a pre-trained category recognition network can be used to perform live object category recognition. Accordingly, the live object category recognition based on the live content information corresponding to the live frame image can obtain the second category information of the live object in the target live room, which may include:

[0184] The live broadcast content information is input into the category recognition network for category recognition to obtain second category information.

[0185] In a specific embodiment, the objects may include but are not limited to objects that need to be promoted during the live broadcast, such as commodities, applications, etc. The categories of the objects may be set in combination with actual applications. For example, the categories corresponding to commodities may include electronic products, cosmetics, clothing, etc.

[0186] In a specific embodiment, live object category identification is performed based on live content information, and the second category information of the live object in the identified target live room may be the category information of the object being explained during the live stream playback process determined by semantic understanding of the live content information.

[0187] In a specific embodiment, the category recognition network can be obtained by pre-training the third deep learning network for category recognition based on the live content information corresponding to the sample object (for example, including the image of the object and the voice information used to explain the object) and the category information corresponding to the sample object. In a specific embodiment, the network structure of the category recognition network (the third deep learning network) can be pre-set in combination with actual application requirements. Accordingly, the trained category recognition network can be used for category recognition.

[0188] In addition, it should be noted that, when the live content information includes live voice information, the live voice information can be converted into corresponding text information and then input into the above-mentioned category recognition network. And the live object category recognition is not limited to the above-mentioned combined category recognition network. For example, based on the live content information, HOG features and color histograms are extracted, and then, the live object category recognition is performed in combination with SVM (support vector machine).

[0189] In the above embodiment, the live broadcast content information is input into the category recognition network for category recognition, which can be combined with the semantic understanding of the live broadcast content information to quickly and accurately identify the category information of the object being explained in the live broadcast room.

[0190] In step S105, based on the second category information and the first category information, the live broadcast intention object is determined from the target detection objects.

[0191] In a specific embodiment, the detection object whose category information (first category information) in the target detection object is the same as the second category information can be used as the live broadcast intention object.

[0192] In the embodiments of the present specification, the detected target detection objects are screened by combining the second category information of the object being explained in the live broadcast room identified by the semantic understanding of the live broadcast content information, so that the recommended objects based on the explanation intention can be identified, thereby avoiding the problems of low accuracy of recommended objects identified only from the visual dimension and inability to cope with object recognition in complex live broadcast scenes (scenes with more objects in the live broadcast screen), thereby greatly improving the accuracy of live broadcast intention object recognition.

[0193] In step S107, based on the object image of the live broadcast intention object, a target recommended object is determined from the objects to be recommended in the target live broadcast room.

[0194] In a specific embodiment, the object to be recommended in the target live broadcast room may be an object that can be recommended in the target live broadcast room. Specifically, the object to be recommended may include multiple objects.

[0195] In an optional embodiment, if Figure 3As shown, based on the above-mentioned object image of the live broadcast intention object, determining the target recommended object from the objects to be recommended in the target live broadcast room may include:

[0196] In step S301, an object image of an object to be recommended is obtained.

[0197] In step S303, the object image of the object to be recommended and the object image of the live broadcast intention object are respectively input into the fine-grained representation network for fine-grained representation, and the first fine-grained feature information corresponding to the live broadcast intention object and the second fine-grained feature information corresponding to the object to be recommended are respectively obtained;

[0198] In step S305, based on the first fine-grained feature information and the second fine-grained feature information, an initial recommended object is matched from the objects to be recommended;

[0199] In step S307, the object image of the live broadcast intention object and the speech recognition text information corresponding to the live broadcast intention object are input into the multimodal representation network for multimodal representation to obtain the first multimodal feature information corresponding to the live broadcast intention object;

[0200] In step S309, the object image of the initial recommended object and the associated text information corresponding to the initial recommended object are input into the multimodal representation network for multimodal representation to obtain second multimodal feature information corresponding to the initial recommended object;

[0201] In step S311, based on the first multimodal feature information and the second multimodal feature information, a target recommended object is matched from the initial recommended objects.

[0202] In practical applications, a recommended image of the object to be recommended is obtained in advance. Generally, the recommended image may include the object to be recommended. Optionally, target detection may be performed on the recommended image of the object to be recommended in advance to obtain the object image of the object to be recommended. Optionally, target detection may also be performed on the recommended image of the object to be recommended during the live stream playback of the target live room to obtain the object image of the object to be recommended. In a specific embodiment, the specific refinement of target detection on the recommended image of the object to be recommended can refer to the specific refinement of target detection on the live frame image of the target live room mentioned above, which will not be repeated here.

[0203] In a specific embodiment, target detection can be performed on the recommended image of the object to be recommended in combination with the target detection network, and accordingly, the object image of the object to be recommended and the corresponding object confidence data can be obtained. Specifically, the object confidence data can represent the accurate probability of the position information of the object to be recommended in the recommended image predicted by the target detection network; optionally, the object image whose object confidence data is greater than the preset object confidence can be used as the object image of the object to be recommended. Specifically, the preset object confidence can be pre-set in combination with the actual application.

[0204] In a specific embodiment, the fine-grained representation network may be obtained by performing fine-grained representation training on a first deep learning network based on sample object images annotated with category identifiers in advance.

[0205] In a specific embodiment, the multimodal representation network can be obtained by performing multimodal representation training on a second deep learning network based on sample object images annotated with category identifiers and object text information corresponding to the sample object images.

[0206] In a specific embodiment, the voice recognition text information corresponding to the live broadcast intention object can be the text information of the live broadcast voice corresponding to the live broadcast intention object; the associated text information corresponding to the initial recommended object can be the text information used to describe the initial recommended object, such as the object recommendation title, object recommendation introduction information and text information recognized from the object recommendation image, etc.

[0207] In a specific embodiment, the first fine-grained feature information may be information that can reflect the fine-grained features of the live broadcast intention object; the second fine-grained feature information corresponding to the object to be recommended may be information that can reflect the fine-grained features of the object to be recommended.

[0208] In a specific embodiment, the first multimodal feature information may be information that can reflect the multimodal features of the live broadcast intention object; the second multimodal feature information corresponding to the object to be recommended may be information that can reflect the multimodal features of the initial recommended object.

[0209] In an optional embodiment, matching the initial recommended object from the objects to be recommended based on the first fine-grained feature information and the second fine-grained feature information may include:

[0210] Determine first matching information between the live broadcast intention object and the object to be recommended according to the first fine-grained feature information and the second fine-grained feature information;

[0211] Based on the first matching information, matching an initial recommended object from the objects to be recommended;

[0212] In a specific embodiment, the above-mentioned objects to be recommended may include multiple objects. Accordingly, the first matching information between the live broadcast intention object and the objects to be recommended can characterize the degree of matching between the live broadcast intention object and each object in the objects to be recommended from the fine-grained feature dimension corresponding to the object. Optionally, the distance between the first fine-grained feature information and the second fine-grained feature information of each object can be determined and used as the first matching information between the live broadcast intention object and each object in the objects to be recommended.

[0213] In a specific embodiment, the distance between the first fine-grained feature information and the second fine-grained feature information may include, but is not limited to, cosine distance, Euclidean distance, Manhattan distance, and the like.

[0214] In an optional embodiment, the object whose corresponding value of the first matching information between the live broadcast intention object is greater than or equal to the first preset association threshold can be used as the above-mentioned initial recommendation object, or the first third preset number of objects whose corresponding values ​​of the first matching information between the live broadcast intention object are relatively large can be used as the above-mentioned initial recommendation objects.

[0215] In a specific embodiment, the first preset association threshold and the third preset number can be preset in combination with actual applications.

[0216] In a specific embodiment, matching the target recommended object from the initial recommended objects based on the first multimodal feature information and the second multimodal feature information may include:

[0217] Determining second matching information between the live broadcast intended object and the initial recommended object according to the first multimodal feature information and the second multimodal feature information;

[0218] Based on the second matching information, a target recommended object is matched from the initial recommended objects.

[0219] In a specific embodiment, the initial recommended object may include multiple objects. Accordingly, the second matching information between the live broadcast intention object and the initial recommended object can characterize the degree of matching between the live broadcast intention object and each object in the initial recommended object from the multimodal feature dimension corresponding to the object. Optionally, the distance between the first multimodal feature information and the second multimodal feature information of each object can be determined and used as the second matching information between the live broadcast intention object and each object in the initial live broadcast pair.

[0220] In a specific embodiment, the distance between the first multimodal feature information and the second multimodal feature information may include, but is not limited to, cosine distance, Euclidean distance, Manhattan distance, and the like.

[0221] In an optional embodiment, the live broadcast intention object can be an object, and optionally, the target recommendation object can also be an object. Accordingly, the object with the largest value corresponding to the second matching information between the live broadcast intention object can be used as the above-mentioned target recommendation object.

[0222] In an optional embodiment, the target recommendation object may also be multiple objects. Optionally, the object whose second matching information corresponding value between the live broadcast intention object is the largest and is greater than or equal to the second preset association threshold can be used as the above-mentioned target recommendation object, or the first fourth preset number of objects with larger values ​​corresponding to the second matching information between the live broadcast intention object can be used as the above-mentioned target recommendation objects.

[0223] In a specific embodiment, the second preset association threshold and the fourth preset number can be preset in combination with actual applications.

[0224] In the above embodiment, two-stage object matching is performed by combining the fine-grained features and multimodal features corresponding to the object; in the first stage, the objects to be screened are matched and screened by combining the fine-grained features of the object image, and then, in the second stage, the multimodal features corresponding to the object are extracted using the object image and related text information, and matched again by combining the multimodal features, which can greatly improve the accuracy of the matched target recommended objects. In addition, in the process of combining fine-grained features and multimodal features for object matching, the fine-grained features and multimodal features are combined successively to characterize the degree of matching between objects, which can effectively improve the accuracy of the measurement of the degree of matching between objects.

[0225] In an optional embodiment, the above method may further include:

[0226] Determining target matching information according to the first matching information and the second matching information;

[0227] Based on the second matching information, matching the target recommended object from the initial recommended objects includes:

[0228] Based on the target matching information, the target recommended object is matched from the initial recommended objects.

[0229] In a specific embodiment, the first matching information and the second matching information between the live broadcast intention object and the initial recommended object can be weighted to obtain the above-mentioned target matching information. Specifically, the weights corresponding to the first matching information and the second matching information can be set in combination with the proportion of fine-grained features and multimodal features in the object characterization features in the actual application requirements. Accordingly, the target matching information can characterize the degree of matching between the live broadcast intention object and each object in the initial recommended object from the dimensions of the fine-grained features and multimodal features corresponding to the object.

[0230] In an optional embodiment, the target recommendation object may be an object, and the object having the largest corresponding value of the target matching information with the live broadcast intention object may be used as the above-mentioned target recommendation object.

[0231] In an optional embodiment, the target recommendation object may also be multiple objects. Optionally, the object whose target matching information corresponding value with the live broadcast intention object is the largest and is greater than or equal to a third preset association threshold value can be used as the above-mentioned target recommendation object, or the first five preset number of objects with larger target matching information corresponding values ​​with the live broadcast intention object can be used as the above-mentioned target recommendation objects.

[0232] In a specific embodiment, the third preset association threshold and the fifth preset number can be preset in combination with actual applications.

[0233] In the above embodiment, a two-stage object matching process is performed by combining the fine-grained features and multimodal features corresponding to the object. After the object matching is performed in the first stage by combining the matching information between objects represented by the fine-grained features, the object matching is performed in the second stage by integrating the matching information between objects represented by the fine-grained features and the multimodal features. This can better improve the accuracy of the measurement of the degree of matching between objects, and further better match the target recommended objects corresponding to the explanation intention from the objects to be recommended, thereby improving the stability of the subsequent object automatic delivery service.

[0234] In an optional embodiment, the above method may further include: a step of pre-training a fine-grained representation network, specifically, Figure 4 As shown, pre-training a fine-grained representation network may include the following steps:

[0235] In step S401, a sample object image of a first sample object and a first target category identifier of the first sample object are obtained;

[0236] In step S403, the sample object image is input into a first deep learning network for fine-grained characterization to obtain sample fine-grained feature information corresponding to the first sample object;

[0237] In step S405, the sample fine-grained feature information is input into a first preset classification network for classification processing to obtain first predicted classification information;

[0238] In step S407, first target loss information is determined based on the first predicted classification information and the first target category identifier;

[0239] In step S409, based on the first target loss information, a first preset classification network and a first deep learning network are trained;

[0240] In step S411, the trained first deep learning network is used as a fine-grained representation network.

[0241] In a specific embodiment, the sample object image of the first sample object may be an object image used to train a fine-grained representation network. Optionally, the sample object image of the first sample object may be extracted from a large number of live images including objects and / or object recommendation images including objects. In a specific embodiment, the sample object image of the first sample object may include object images of multiple objects. Accordingly, each object corresponds to a category identifier. For example, the category identifier corresponding to the object "dress" may be 0, and the category identifier corresponding to the object "sneakers" may be 1. Specifically, the category identifier may be identification information that can distinguish different objects and is set in combination with actual applications.

[0242] In a specific embodiment, in the process of training the fine-grained representation network, the processing of the classification task can be combined, and accordingly, a first preset classification network can be connected after the first deep learning network (the fine-grained representation network to be trained).

[0243] In an optional embodiment, the first predicted classification information may include a first predicted category identifier of the first sample object; correspondingly, the first target loss information may include a category identifier loss;

[0244] Accordingly, the determining of the first target loss information based on the first prediction classification information and the first target category identifier may include:

[0245] A category identification loss is determined based on the first predicted category identification and the first target category identification.

[0246] In a specific embodiment, determining the category identification loss according to the first predicted category identification and the first target category identification may include determining the category identification loss between the first predicted category identification and the first target category identification based on a preset loss function.

[0247] In a specific embodiment, the category identification loss can characterize the difference between the first predicted category identification and the first target category identification. In the fine-grained representation network training process, the category identification loss can be combined to train the fine-grained representation network to represent the same object in different object images (such as live broadcast images and object recommendation images) consistently, thereby improving the object representation accuracy of the fine-grained representation network while ensuring that when the fine-grained features are subsequently combined for object matching, the accuracy of object matching in different object images can be better.

[0248] In a specific embodiment, the preset loss function may include but is not limited to a cross entropy loss function, a logical loss function, an exponential loss function, etc.

[0249] In an optional embodiment, based on the first target loss information, training the first preset classification network and the first deep learning network may include: based on the first target loss information, updating the network parameters of the first deep learning network and the first preset classification network; based on the updated first deep learning network and the first preset classification network, repeating step S403 to the classification training iterative operation of updating the network parameters of the first deep learning network based on the first target loss information, to the classification iterative operation of updating the network parameters of the first deep learning network and the first preset classification network based on the first target loss information, until the classification convergence condition is reached; the first deep learning network obtained when the classification convergence condition is reached is used as a fine-grained characterization network.

[0250] In a specific embodiment, the specific details of achieving the classification convergence condition can refer to the specific details of achieving the detection convergence condition mentioned above, which will not be repeated here.

[0251] In the above embodiment, during the training process of the fine-grained representation network, the classification network is combined to process the classification task, which can better improve the representation accuracy of the fine-grained representation network for different object images.

[0252] In an optional embodiment, the first target loss information may also include category loss and / or category metric loss;

[0253] In the case where the first target loss includes category loss, the first predicted classification information may further include predicted category information of the first sample object;

[0254] In the case where the first target loss includes a category metric loss, the first sample object includes a positive sample object pair and a negative sample object pair, the positive sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to the same sample object, and the negative sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to different sample objects; the first predicted classification information also includes first classification feature information of the object in the positive sample object pair and second classification feature information of the object in the negative sample object pair;

[0255] Accordingly, the determining of the first target loss information based on the first prediction classification information and the first target category identifier may also include:

[0256] Obtain target category information of the first sample object;

[0257] Determine the category loss based on the target category information and the predicted category information;

[0258] and / or,

[0259] Determining a first category similarity between objects in a positive sample object pair based on the first classification feature information;

[0260] Determining a second category similarity between objects in the negative sample object pair based on the second classification feature information;

[0261] A class metric loss is determined based on the first class similarity and the second class similarity.

[0262] In a specific embodiment, the target category information may be the category information of the first sample object that is pre-labeled. The specific details of determining the category loss based on the target category information and the predicted category information may refer to the specific details of determining the category identification loss based on the first predicted category identification and the first target category identification, which will not be repeated here.

[0263] In a specific embodiment, the category loss can characterize the difference between the target category information and the predicted category information. During the network training process, the category loss can be combined to train the fine-grained representation network to distinguish between different objects in the same category, thereby improving the object representation accuracy of the fine-grained representation network, while ensuring that when the subsequent combination of fine-grained features cannot match the same object, similar objects in the same category can be found, thereby better improving the accuracy of object matching.

[0264] In a specific embodiment, the category metric loss can characterize the accuracy of the fine-grained representation network for the same object. In the network training process, combining the category metric loss can further improve the fine-grained representation network's ability to distinguish between the same object and similar objects, thereby better improving the object representation accuracy of the fine-grained representation network.

[0265] In a specific embodiment, the positive sample object pairs may include multiple pairs of objects; determining the first category similarity between objects in the positive sample object pairs based on the first classification feature information may include determining the distance between the first classification feature information corresponding to each pair of objects, and using the distance as the first category similarity between the pair of objects.

[0266] In a specific embodiment, the negative sample object pairs may include multiple pairs of objects; determining the second category similarity between objects in the negative sample object pairs based on the second classification feature information may include determining the distance between the second classification feature information corresponding to each pair of objects, and using the distance as the second category similarity between the pair of objects.

[0267] In an optional embodiment, determining the category metric loss according to the first category similarity and the second category similarity may include subtracting the first category similarity from the second category similarity to obtain the category metric loss.

[0268] In addition, it should be noted that, when the first target loss information includes at least two losses, the at least two losses can be weighted and summed, and based on the loss information after the weighted summation, the network parameters of the first deep learning network and the first preset classification network are updated. Specifically, the weights of the at least two losses can be preset in combination with the actual application. Generally, the weight of the category identification loss is larger than that of other losses.

[0269] In the above embodiment, during the training process of the fine-grained representation network, combined with category loss and / or category metric loss, the fine-grained representation network can be better trained to better distinguish the representation of different objects under the same category, as well as the representation distinction of the same object and similar objects, thereby better improving the object representation accuracy of the fine-grained representation network, ensuring that when the subsequent combination of fine-grained features cannot match the same object, similar objects under the same category can be found, thereby better improving the accuracy of subsequent object matching combined with fine-grained features.

[0270] In an optional embodiment, the above method may further include: a step of pre-training a multimodal representation network, specifically, Figure 5 As shown, pre-training a multimodal representation network may include the following steps:

[0271] In step S501, a sample object image of a second sample object, object text information of the second sample object, and a second object category identifier of the second sample object are obtained;

[0272] In step S503, the sample object image and object text information are input into a second deep learning network for multimodal representation to obtain sample multimodal feature information corresponding to the second sample object;

[0273] In step S505, the sample multimodal feature information is input into a second preset classification network for classification processing to obtain second predicted classification information;

[0274] In step S507, second target loss information is determined based on the second predicted classification information and the second target category identifier;

[0275] In step S509, based on the second target loss information, a second preset classification network and a second deep learning network are trained;

[0276] In step S511, the trained second deep learning network is used as a multimodal representation network.

[0277] In a specific embodiment, the sample object image of the second sample object may be an object image for training a multimodal representation network. Optionally, the sample object image of the second sample object may be extracted from a large number of live images including objects and / or object recommendation images including objects. In a specific embodiment, the sample object image of the second sample object may include object images of multiple objects, and accordingly, each object corresponds to this one category identifier, for example, the category identifier corresponding to the object "dress" may be 0, and the category identifier corresponding to the object "sneakers" may be 1. Specifically, the category identifier may be identification information that can distinguish different objects in combination with actual applications. The object text information of the second sample object may be text information that can describe the second sample object. For example, when the second sample object is an object in a live image, the object text information may be the corresponding live speech recognition text information. When the second sample object is an object in an object recommendation image, the object text information may be an object recommendation title, object recommendation introduction information, and text information recognized from the object recommendation image, etc.

[0278] In a specific embodiment, the specific details of the above steps S507 to S509 can refer to the specific details of the above steps S407 and S409, which will not be repeated here.

[0279] In the above embodiment, during the training process of the multimodal representation network, the classification network is combined to process the classification task, which can better improve the accuracy of the multimodal representation of different objects by the multimodal representation network.

[0280] In an optional embodiment, in the process of determining the target recommended object, a single-stage matching may also be performed. Accordingly, based on the above-mentioned object image of the live broadcast intention object, determining the target recommended object from the objects to be recommended in the target live broadcast room may include:

[0281] Obtain an object image of an object to be recommended;

[0282] Inputting the object image of the object to be recommended and the object image of the live broadcast intention object into the feature characterization network respectively to perform object feature characterization, and obtaining first object feature information of the object to be recommended and second object feature information of the live broadcast intention object respectively;

[0283] Based on the first object feature information and the second object feature information, a target recommended object is matched from the objects to be recommended.

[0284] In a specific embodiment, the feature characterization network can be obtained by jointly training the fourth deep learning network and the third preset classification network for feature characterization and classification for sample object images (extracted from a large number of live images including objects and / or object recommendation images including objects) and category identifiers corresponding to the sample object images. In a specific embodiment, the network structure of the feature characterization network (fourth deep learning network) can be preset in combination with actual application requirements. Accordingly, the trained feature characterization network can be used for object feature characterization.

[0285] In a specific embodiment, the above-mentioned matching of the target live broadcast pair from the objects to be recommended based on the first object feature information and the second object feature information may include: determining the third matching information between the live broadcast intention object and the object to be recommended based on the first object feature information and the second object feature information; and determining the target recommendation object from the objects to be recommended based on the third matching information.

[0286] In the above embodiment, by combining the object feature information corresponding to the object, the target recommended object corresponding to the explanation intention can be matched quickly and accurately from the objects to be recommended.

[0287] In an optional embodiment, if Figure 6 As shown, the above method may further include the following steps:

[0288] In step S109, the object recommendation information of the target recommended object is displayed on the live broadcast page corresponding to the target live broadcast room.

[0289] In a specific embodiment, the object recommendation information may be information used to describe the target recommended object, and may be used to trigger an object conversion operation corresponding to the target recommended object.

[0290] In the above embodiment, the object recommendation information of the target recommended object determined based on the explanation intention is displayed on the live broadcast page, which can realize the automatic delivery of the recommended object, greatly improving the convenience and efficiency of the recommended object delivery operation.

[0291] According to the technical solution provided by the above embodiments of this specification, during the live streaming playback process, after target detection is performed on the live frame image and the object image of the display object (target detection object) in the live screen and the first category information corresponding to the display object are extracted, the live object category recognition is performed in combination with the live content information corresponding to the live frame image, so that the second category information of the object being explained in the live room can be identified; and based on the first category information and the second category information, the detected target detection objects are screened, so that the target recommendation objects in the current live screen explanation intention can be accurately identified, effectively avoiding the problem of low accuracy of the target recommendation objects identified only from the visual dimension, greatly improving the accuracy and coverage of live intent object recognition, and thereby improving the convenience and efficiency of subsequent target recommendation object delivery operations.

[0292] Figure 7 is a block diagram of a device for determining a target recommendation object according to an exemplary embodiment. Figure 7 , the device comprises:

[0293] The first target detection module 710 is configured to perform target detection on the live frame image of the target live room during the live stream playback of the target live room, and obtain an object image of the target detection object in the live frame image and first category information corresponding to the target detection object;

[0294] The category identification module 720 is configured to perform category identification of the live broadcast object based on the live broadcast content information corresponding to the live broadcast frame image, and obtain second category information of the live broadcast object in the target live broadcast room;

[0295] The live broadcast intention object determination module 730 is configured to determine the live broadcast intention object from the target detection object based on the second category information and the first category information;

[0296] The target recommendation object determination module 740 is configured to execute an object image based on the live broadcast intention object and determine the target recommendation object from the objects to be recommended in the target live broadcast room.

[0297] Optionally, the target recommendation object determination module 740 includes:

[0298] A first object image acquisition unit is configured to acquire an object image of an object to be recommended;

[0299] The fine-grained representation unit is configured to input the object image of the object to be recommended and the object image of the live broadcast intention object into the fine-grained representation network for fine-grained representation, respectively, to obtain first fine-grained feature information corresponding to the live broadcast intention object and second fine-grained feature information corresponding to the object to be recommended;

[0300] An initial recommended object matching unit is configured to match an initial recommended object from the objects to be recommended based on the first fine-grained feature information and the second fine-grained feature information;

[0301] The first multimodal representation unit is configured to input the object image of the live broadcast intention object and the speech recognition text information corresponding to the live broadcast intention object into the multimodal representation network for multimodal representation, and obtain the first multimodal feature information corresponding to the live broadcast intention object;

[0302] The second multimodal representation unit is configured to input the object image of the initial recommended object and the associated text information corresponding to the initial recommended object into the multimodal representation network for multimodal representation, so as to obtain second multimodal feature information corresponding to the initial recommended object;

[0303] The first target recommendation object matching unit is configured to match the target recommendation object from the initial recommendation objects based on the first multimodal feature information and the second multimodal feature information.

[0304] Optionally, the initial recommendation object matching unit includes:

[0305] A first matching information determining unit is configured to determine first matching information between a live broadcast intention object and an object to be recommended according to the first fine-grained feature information and the second fine-grained feature information;

[0306] an initial recommended object determining unit, configured to perform matching of an initial recommended object from the objects to be recommended based on the first matching information;

[0307] The target recommendation object matching unit includes:

[0308] A second matching information determining unit is configured to determine second matching information between the live broadcast intended object and the initial recommended object according to the first multimodal feature information and the second multimodal feature information;

[0309] The target recommendation object determination unit is configured to perform matching of the target recommendation object from the initial recommendation objects based on the second matching information.

[0310] Optionally, the above device further includes:

[0311] a target matching information determining unit, configured to determine the target matching information according to the first matching information and the second matching information;

[0312] The target recommendation object determination unit is further configured to perform matching of the target recommendation object from the initial recommendation objects based on the target matching information.

[0313] Optionally, the above device further includes:

[0314] A first sample data acquisition module is configured to acquire a sample object image of a first sample object and a first target category identifier of the first sample object;

[0315] The fine-grained characterization module is configured to input the sample object image into the first deep learning network for fine-grained characterization to obtain sample fine-grained feature information corresponding to the first sample object;

[0316] A first classification processing module is configured to input sample fine-grained feature information into a first preset classification network for classification processing to obtain first predicted classification information;

[0317] A first target loss information determination module is configured to determine first target loss information based on the first predicted classification information and the first target category identifier;

[0318] A first network training module is configured to train a first preset classification network and a first deep learning network based on the first target loss information;

[0319] The fine-grained representation network determination module is configured to execute the trained first deep learning network as the fine-grained representation network.

[0320] Optionally, the first predicted classification information includes a first predicted category identifier of the first sample object; the first target loss information includes a category identifier loss;

[0321] The first target loss information determination module includes:

[0322] The category identification loss determination unit is configured to determine the category identification loss according to the first predicted category identification and the first target category identification.

[0323] Optionally, the first target loss information further includes category loss and / or category metric loss;

[0324] In the case where the first target loss includes category loss, the first predicted classification information also includes predicted category information of the first sample object;

[0325] In the case where the first target loss includes a category metric loss, the first sample object includes a positive sample object pair and a negative sample object pair, the positive sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to the same sample object, and the negative sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to different sample objects; the first predicted classification information also includes first classification feature information of the object in the positive sample object pair and second classification feature information of the object in the negative sample object pair;

[0326] The first target loss information determination module also includes:

[0327] A target category information acquisition unit, configured to acquire target category information of a first sample object;

[0328] A category loss determination unit is configured to determine the category loss according to the target category information and the predicted category information;

[0329] and / or,

[0330] A first category similarity determination unit is configured to determine a first category similarity between objects in a positive sample object pair according to the first classification feature information;

[0331] A second category similarity determination unit is configured to determine a second category similarity between objects in a negative sample object pair according to the second classification feature information;

[0332] The category metric loss determination unit is configured to determine the category metric loss according to the first category similarity and the second category similarity.

[0333] Optionally, the above device further includes:

[0334] A second sample data acquisition module is configured to execute acquisition of a sample object image of a second sample object, object text information of the second sample object, and a second target category identifier of the second sample object;

[0335] The multimodal representation module is configured to input the sample object image and the object text information into the second deep learning network for multimodal representation to obtain sample multimodal feature information corresponding to the second sample object;

[0336] A second classification processing module is configured to input the sample multimodal feature information into a second preset classification network for classification processing to obtain second predicted classification information;

[0337] A second target loss information determination module is configured to determine the second target loss information based on the second predicted classification information and the second target category identifier;

[0338] A second network training module is configured to train a second preset classification network and a second deep learning network based on the second target loss information;

[0339] The multimodal representation network determination module is configured to execute the trained second deep learning network as the multimodal representation network.

[0340] Optionally, the target recommendation object determination module 740 includes:

[0341] A second object image acquisition unit is configured to acquire an object image of an object to be recommended;

[0342] The object feature characterization unit is configured to input the object image of the object to be recommended and the object image of the live broadcast intention object into the feature characterization network for object feature characterization, and obtain first object feature information of the object to be recommended and second object feature information of the live broadcast intention object respectively;

[0343] The second target recommendation object matching unit is configured to match a target recommendation object from the objects to be recommended based on the first object feature information and the second object feature information.

[0344] Optionally, the first target detection module 710 is further configured to input the live frame image into the target detection network for target detection, and obtain an object image of the target detection object and first category information corresponding to the target detection object.

[0345] Optionally, the above device further includes:

[0346] A sample image acquisition module is configured to acquire annotated sample images and unannotated sample images, wherein the annotated sample images are sample object images annotated with object detection information;

[0347] The first detection network training module is configured to perform target detection training on the detection network to be trained based on the labeled sample images to obtain an initial detection network;

[0348] A second target detection module is configured to input the unlabeled sample image into the initial detection network to perform target detection, and obtain sample category information and category confidence data of a third sample object in the unlabeled sample image;

[0349] The target sample image determination module is configured to determine the target sample image from the non-labeled sample images according to the category confidence data;

[0350] A category annotation module is configured to perform category annotation on the target sample image based on the sample category information corresponding to the target sample image to obtain an incrementally annotated sample image;

[0351] The second detection network training module is configured to perform target detection training on the initial detection network based on the labeled sample images and the incremental labeled sample images to obtain the target detection network.

[0352] Optionally, the category identification module 720 is further configured to input the live content information into a category identification network for category identification to obtain second category information.

[0353] Optionally, the above device further includes:

[0354] The object recommendation information display module is configured to display the object recommendation information of the target recommended object on the live broadcast page corresponding to the target live broadcast room.

[0355] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0356] Figure 8 is a block diagram of an electronic device for determining a target recommendation object according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for determining a target recommendation object is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0357] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0358] In an exemplary embodiment, an electronic device is also provided, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the target recommendation object determination method as in the embodiment of the present disclosure.

[0359] In an exemplary embodiment, a computer-readable storage medium is also provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the target recommendation object determination method in the embodiment of the present disclosure.

[0360] In an exemplary embodiment, a computer program product including instructions is also provided. When the computer program product is executed on a computer, the computer is enabled to execute the target recommendation object determination method in the embodiment of the present disclosure.

[0361] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0362] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0363] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining a target recommendation object, characterized in that: include: During the live streaming playback of the target live room, target detection is performed on the live frame image of the target live room to obtain an object image of the target detection object in the live frame image and first category information corresponding to the target detection object; Performing live object category recognition based on the live content semantic information corresponding to the live frame image to obtain second category information of the live object in the target live room; Based on the second category information and the first category information, determining a live broadcast intention object from the target detection objects; Acquire an object image of an object to be recommended in the target live broadcast room; Inputting the object image of the object to be recommended and the object image of the live broadcast intention object into a feature characterization network for object feature characterization, respectively, to obtain first object feature information of the live broadcast intention object and second object feature information of the object to be recommended; Based on the first object feature information and the second object feature information, a target recommended object is matched from the objects to be recommended.

2. The method for determining a target recommendation object according to claim 1, characterized in that: The feature characterization network includes: a fine-grained characterization network, the first object feature information includes: first fine-grained feature information, the second object feature information includes: second fine-grained feature information, the object image of the object to be recommended and the object image of the live broadcast intention object are respectively input into the feature characterization network for object feature characterization, and the first object feature information of the live broadcast intention object and the second object feature information of the object to be recommended are respectively obtained, including: Inputting the object image of the object to be recommended and the object image of the live broadcast intention object into the fine-grained representation network for fine-grained representation, respectively, to obtain first fine-grained feature information corresponding to the live broadcast intention object and second fine-grained feature information corresponding to the object to be recommended; The matching a target recommended object from the objects to be recommended based on the first object feature information and the second object feature information includes: Based on the first fine-grained feature information and the second fine-grained feature information, matching an initial recommended object from the objects to be recommended; Inputting the object image of the live broadcast intention object and the speech recognition text information corresponding to the live broadcast intention object into a multimodal representation network for multimodal representation to obtain first multimodal feature information corresponding to the live broadcast intention object; Inputting the object image of the initial recommended object and the associated text information corresponding to the initial recommended object into a multimodal representation network for multimodal representation to obtain second multimodal feature information corresponding to the initial recommended object; Based on the first multimodal feature information and the second multimodal feature information, the target recommended object is matched from the initial recommended objects.

3. The method for determining a target recommendation object according to claim 2, characterized in that: The matching of the initial recommended object from the objects to be recommended based on the first fine-grained feature information and the second fine-grained feature information includes: Determining first matching information between the live broadcast intention object and the object to be recommended according to the first fine-grained feature information and the second fine-grained feature information; Based on the first matching information, matching the initial recommended object from the objects to be recommended; The matching the target recommended object from the initial recommended objects based on the first multimodal feature information and the second multimodal feature information includes: Determining second matching information between the live broadcast intended object and the initial recommended object according to the first multimodal feature information and the second multimodal feature information; Based on the second matching information, the target recommended object is matched from the initial recommended objects.

4. The method for determining a target recommendation object according to claim 3, characterized in that: The method further comprises: Determining target matching information according to the first matching information and the second matching information; The matching the target recommended object from the initial recommended objects based on the second matching information includes: Based on the target matching information, the target recommended object is matched from the initial recommended objects.

5. The method for determining a target recommendation object according to claim 2, characterized in that: The method further comprises: Acquire a sample object image of a first sample object and a first target category identifier of the first sample object; Inputting the sample object image into a first deep learning network for fine-grained characterization to obtain sample fine-grained feature information corresponding to the first sample object; Inputting the sample fine-grained feature information into a first preset classification network for classification processing to obtain first predicted classification information; Determining first target loss information based on the first predicted classification information and the first target category identifier; Based on the first target loss information, training the first preset classification network and the first deep learning network; The trained first deep learning network is used as the fine-grained representation network.

6. The method for determining a target recommendation object according to claim 5, characterized in that: The first predicted classification information includes a first predicted category identifier of the first sample object; the first target loss information includes a category identifier loss; The determining of first target loss information based on the first prediction classification information and the first target category identifier includes: The category identification loss is determined according to the first predicted category identification and the first target category identification.

7. The method for determining a target recommendation object according to claim 6, characterized in that: The first target loss information also includes category loss and / or category metric loss; In a case where the first target loss includes the category loss, the first predicted classification information further includes predicted category information of the first sample object; In the case where the first target loss includes the category metric loss, the first sample object includes a positive sample object pair and a negative sample object pair, the positive sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to the same sample object, and the negative sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to different sample objects; the first predicted classification information also includes first classification feature information of objects in the positive sample object pair and second classification feature information of objects in the negative sample object pair; The determining the first target loss information based on the first prediction classification information and the first target category identifier further includes: Acquire target category information of the first sample object; determining the category loss according to the target category information and the predicted category information; and / or, Determining a first category similarity between objects in the positive sample object pair according to the first classification feature information; Determining a second category similarity between objects in the negative sample object pair according to the second classification feature information; The category metric loss is determined according to the first category similarity and the second category similarity.

8. The method for determining a target recommendation object according to claim 2, characterized in that: The method further comprises: Acquire a sample object image of a second sample object, object text information of the second sample object, and a second target category identifier of the second sample object; Inputting the sample object image and the object text information into a second deep learning network for multimodal representation to obtain sample multimodal feature information corresponding to the second sample object; Inputting the sample multimodal feature information into a second preset classification network for classification processing to obtain second predicted classification information; Determining second target loss information based on the second predicted classification information and the second target category identifier; Based on the second target loss information, training the second preset classification network and the second deep learning network; The trained second deep learning network is used as the multimodal representation network.

9. The method for determining a target recommendation object according to claim 1, characterized in that: The performing target detection on the live frame image of the target live room to obtain the object image of the target detection object in the live frame image and the first category information corresponding to the target detection object includes: The live frame image is input into a target detection network for target detection to obtain an object image of the target detection object and first category information corresponding to the target detection object.

10. The method for determining a target recommendation object according to claim 9, characterized in that: The method further comprises: Acquire a labeled sample image and an unlabeled sample image, wherein the labeled sample image is a sample object image labeled with object detection information; Based on the labeled sample images, performing target detection training on the detection network to be trained to obtain an initial detection network; Inputting the unlabeled sample image into the initial detection network to perform target detection, and obtaining sample category information and category confidence data of a third sample object in the unlabeled sample image; Determining a target sample image from the non-annotated sample images according to the category confidence data; Annotate the target sample image based on the sample category information corresponding to the target sample image to obtain an incrementally annotated sample image; Based on the labeled sample images and the incremental labeled sample images, target detection training is performed on an initial detection network to obtain a target detection network.

11. The method for determining a target recommendation object according to claim 1, characterized in that: The performing of the live object category identification based on the live content semantic information corresponding to the live frame image to obtain the second category information of the live object in the target live room includes: The live content semantic information is input into a category recognition network for category recognition to obtain the second category information.

12. The method for determining a target recommendation object according to any one of claims 1 to 11, characterized in that: The method further comprises: The object recommendation information of the target recommended object is displayed on the live broadcast page corresponding to the target live broadcast room.

13. A device for determining a target recommendation object, characterized in that: include: The first target detection module is configured to perform target detection on the live frame image of the target live room during the live stream playback of the target live room, and obtain an object image of the target detection object in the live frame image and first category information corresponding to the target detection object; A category recognition module is configured to perform category recognition of a live object based on the semantic information of the live content corresponding to the live frame image, and obtain second category information of the live object in the target live room; A live broadcast intention object determination module is configured to determine a live broadcast intention object from the target detection object based on the second category information and the first category information; An object image acquisition module is configured to acquire an object image of an object to be recommended in the target live broadcast room; The object feature characterization module is configured to input the object image of the object to be recommended and the object image of the live broadcast intention object into a feature characterization network for object feature characterization, respectively, to obtain first object feature information of the live broadcast intention object and second object feature information of the object to be recommended; The target recommendation object matching module is configured to match a target recommendation object from the objects to be recommended based on the first object feature information and the second object feature information.

14. The target recommendation object determination device according to claim 13, characterized in that: The feature representation network includes: a fine-grained representation network, the first object feature information includes: first fine-grained feature information, the second object feature information includes: second fine-grained feature information, and the object feature representation module includes: A fine-grained characterization unit is configured to input the object image of the object to be recommended and the object image of the live broadcast intention object into the fine-grained characterization network for fine-grained characterization, respectively, to obtain first fine-grained feature information corresponding to the live broadcast intention object and second fine-grained feature information corresponding to the object to be recommended; The target recommendation object matching module includes: an initial recommended object matching unit, configured to match an initial recommended object from the objects to be recommended based on the first fine-grained feature information and the second fine-grained feature information; The first multimodal representation unit is configured to input the object image of the live broadcast intention object and the speech recognition text information corresponding to the live broadcast intention object into the multimodal representation network for multimodal representation, so as to obtain the first multimodal feature information corresponding to the live broadcast intention object; A second multimodal representation unit is configured to input the object image of the initial recommended object and the associated text information corresponding to the initial recommended object into a multimodal representation network for multimodal representation, so as to obtain second multimodal feature information corresponding to the initial recommended object; The first target recommendation object matching unit is configured to match the target recommendation object from the initial recommendation objects based on the first multimodal feature information and the second multimodal feature information.

15. The target recommendation object determination device according to claim 14, characterized in that: The initial recommendation object matching unit comprises: A first matching information determining unit is configured to determine first matching information between the live broadcast intention object and the to-be-recommended object according to the first fine-grained feature information and the second fine-grained feature information; an initial recommended object determining unit, configured to match the initial recommended object from the objects to be recommended based on the first matching information; The first target recommendation object matching unit includes: A second matching information determining unit is configured to determine second matching information between the live broadcast intended object and the initial recommended object according to the first multimodal feature information and the second multimodal feature information; The target recommended object determination unit is configured to match the target recommended object from the initial recommended objects based on the second matching information.

16. The target recommendation object determination device according to claim 15, characterized in that: The device also includes: a target matching information determining unit, configured to determine target matching information according to the first matching information and the second matching information; The target recommendation object determination unit is further configured to perform matching of the target recommendation object from the initial recommendation objects based on the target matching information.

17. The target recommendation object determination device according to claim 14, characterized in that: The device also includes: A first sample data acquisition module is configured to acquire a sample object image of a first sample object and a first target category identifier of the first sample object; A fine-grained characterization module is configured to input the sample object image into a first deep learning network for fine-grained characterization to obtain sample fine-grained feature information corresponding to the first sample object; A first classification processing module is configured to input the sample fine-grained feature information into a first preset classification network for classification processing to obtain first predicted classification information; A first target loss information determination module is configured to determine first target loss information based on the first predicted classification information and the first target category identifier; A first network training module is configured to train the first preset classification network and the first deep learning network based on the first target loss information; The fine-grained representation network determination module is configured to execute the trained first deep learning network as the fine-grained representation network.

18. The target recommendation object determination device according to claim 17, characterized in that: The first predicted classification information includes a first predicted category identifier of the first sample object; the first target loss information includes a category identifier loss; The first target loss information determination module includes: The category identification loss determining unit is configured to determine the category identification loss according to the first predicted category identification and the first target category identification.

19. The target recommendation object determination device according to claim 18, characterized in that: The first target loss information also includes category loss and / or category metric loss; In a case where the first target loss includes the category loss, the first predicted classification information further includes predicted category information of the first sample object; In the case where the first target loss includes the category metric loss, the first sample object includes a positive sample object pair and a negative sample object pair, the positive sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to the same sample object, and the negative sample object pair is a sample live broadcast intention object and a sample recommended object corresponding to different sample objects; the first predicted classification information also includes first classification feature information of the object in the positive sample object pair and second classification feature information of the object in the negative sample object pair; The first target loss information determination module also includes: a target category information acquiring unit, configured to acquire target category information of the first sample object; a category loss determining unit, configured to determine the category loss according to the target category information and the predicted category information; and / or, A first category similarity determination unit is configured to determine a first category similarity between objects in the positive sample object pair according to the first classification feature information; A second category similarity determination unit is configured to determine a second category similarity between objects in the negative sample object pair according to the second classification feature information; The category metric loss determining unit is configured to determine the category metric loss according to the first category similarity and the second category similarity.

20. The target recommendation object determination device according to claim 14, characterized in that: The device also includes: A second sample data acquisition module is configured to acquire a sample object image of a second sample object, object text information of the second sample object, and a second target category identifier of the second sample object; a multimodal characterization module configured to input the sample object image and the object text information into a second deep learning network for multimodal characterization to obtain sample multimodal feature information corresponding to the second sample object; A second classification processing module is configured to input the sample multimodal feature information into a second preset classification network for classification processing to obtain second predicted classification information; A second target loss information determination module is configured to determine second target loss information based on the second predicted classification information and the second target category identifier; A second network training module is configured to train the second preset classification network and the second deep learning network based on the second target loss information; The multimodal representation network determination module is configured to execute the trained second deep learning network as the multimodal representation network.

21. The target recommendation object determination device according to claim 13, characterized in that: The first target detection module is also configured to input the live frame image into a target detection network for target detection, and obtain an object image of the target detection object and first category information corresponding to the target detection object.

22. The target recommendation object determination device according to claim 21, characterized in that: The device also includes: A sample image acquisition module is configured to acquire an annotated sample image and an unannotated sample image, wherein the annotated sample image is a sample object image annotated with object detection information; A first detection network training module is configured to perform target detection training on the detection network to be trained based on the labeled sample image to obtain an initial detection network; A second target detection module is configured to input the unlabeled sample image into the initial detection network to perform target detection, and obtain sample category information and category confidence data of a third sample object in the unlabeled sample image; a target sample image determination module, configured to determine a target sample image from the non-annotated sample images according to the category confidence data; A category annotation module is configured to perform category annotation on the target sample image based on sample category information corresponding to the target sample image to obtain an incrementally annotated sample image; The second detection network training module is configured to perform target detection training on the initial detection network based on the labeled sample images and the incremental labeled sample images to obtain a target detection network.

23. The target recommendation object determination device according to claim 13, characterized in that: The category identification module is further configured to input the live content semantic information into a category identification network for category identification to obtain the second category information.

24. The target recommendation object determination device according to any one of claims 13 to 23, characterized in that: The device also includes: The object recommendation information display module is configured to display the object recommendation information of the target recommended object on the live broadcast page corresponding to the target live broadcast room.

25. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the target recommendation object determination method as described in any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the target recommendation object determination method as described in any one of claims 1 to 12.

27. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the target recommendation object determination method according to any one of claims 1 to 12 is implemented.

Images