Information recommendation method and device based on artificial intelligence, electronic equipment and storage medium

Abstract

The application provides an information recommendation method and device based on artificial intelligence, an electronic device and a computer readable storage medium. The method comprises: performing feature extraction on target recommendation information to obtain features of multiple modes of the target recommendation information; fusing the features of the multiple modes of the target recommendation information to obtain multi-modal fusion features of the target recommendation information; performing feature extraction on historical interaction behavior data of multiple candidate users to obtain interest features of each candidate user; performing similarity processing on the multi-modal fusion features of the target recommendation information and the interest features of the candidate users to obtain similarities between the target recommendation information and the candidate users; determining a target user matched with the target recommendation information from the multiple candidate users based on the similarities between the target recommendation information and each candidate user; and performing a recommendation operation on the corresponding target user based on the target recommendation information. Through the application, the user can be recommended with recommendation information meeting the user's interest.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an information recommendation method and device based on artificial intelligence, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] Artificial intelligence (AI) is a comprehensive technology of computer science, which enables machines to have the functions of perception, reasoning and decision-making by studying the design principles and implementation methods of various intelligent machines. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] A recommendation system is one of the important applications in the field of artificial intelligence, which can help users discover information that may interest them in an information overload environment and push information to users who are interested in them.

[0004] Although the recommendation system in the related art can determine information that may interest a user from a large amount of information to be recommended and recommend the information that may interest the user to the user. However, the accuracy of the recommendation system in the related art to recommend to the user needs to be improved. SUMMARY

[0005] The embodiments of the present application provide an information recommendation method and device based on artificial intelligence, an electronic device, and a computer readable storage medium, which can recommend recommendation information meeting the interests of a user to the user and improve the accuracy of recommendation.

[0006] The technical solutions of the embodiments of the present application are implemented as follows:

[0007] The embodiments of the present application provide an information recommendation method based on artificial intelligence, comprising:

[0008] performing feature extraction processing on target recommendation information to obtain features of multiple modalities of the target recommendation information;

[0009] performing fusion processing on the features of multiple modalities of the target recommendation information to obtain multi-modality fusion features of the target recommendation information;

[0010] performing feature extraction processing on historical interaction behavior data of multiple candidate users to obtain interest features of each candidate user;

[0011] performing similarity processing on the multi-modality fusion features of the target recommendation information and the interest features of each candidate user to obtain similarities between the target recommendation information and each candidate user.

[0012] determine a target user matching the target recommendation information from the plurality of candidate users based on the similarity between the target recommendation information and each of the candidate users;

[0013] perform a recommendation operation corresponding to the target user based on the target recommendation information.

[0014] Embodiments of the present application provide an information recommendation device, comprising:

[0015] a first extraction module configured to perform feature extraction processing on target recommendation information to obtain features of multiple modalities of the target recommendation information;

[0016] a fusion module configured to perform fusion processing on the features of multiple modalities of the target recommendation information to obtain multi-modal fusion features of the target recommendation information;

[0017] a second extraction module configured to perform feature extraction processing on historical interaction behavior data of a plurality of candidate users to obtain interest features of each of the candidate users;

[0018] a processing module configured to perform similarity processing on the multi-modal fusion features of the target recommendation information and the interest features of each of the candidate users to obtain a similarity between the target recommendation information and each of the candidate users;

[0019] a screening module configured to determine a target user matching the target recommendation information from the plurality of candidate users based on the similarity between the target recommendation information and each of the candidate users;

[0020] a recommendation module configured to perform a recommendation operation corresponding to the target user based on the target recommendation information.

[0021] In the above technical solution, the device further comprises:

[0022] a determination module configured to obtain to-be-recommended information from a recommendation information library;

[0023] when the behavior data of the to-be-recommended information is greater than a behavior data threshold, performing information recommendation processing based on the behavior data of the to-be-recommended information;

[0024] when the behavior data of the to-be-recommended information is less than or equal to the behavior data threshold, taking the to-be-recommended information as the target recommendation information.

[0025] In the technical solution, when the target recommendation information is a video, the first extraction module is further configured to perform image-form coding processing on each image frame in the target recommendation information to obtain a vector representation of each image frame, and perform fusion processing on the vector representation of each image frame to obtain an image representation feature of the target recommendation information.

[0026] perform audio-form coding processing on each audio frame in the target recommendation information to obtain a vector representation of each audio frame, and perform fusion processing on the vector representation of each audio frame to obtain an audio representation feature of the target recommendation information.

[0027] perform text-form coding processing on each text in the target recommendation information to obtain a vector representation of each text, and perform fusion processing on the vector representation of each text to obtain a text representation feature of the target recommendation information.

[0028] use the image representation feature, the audio representation feature, and the text representation feature of the target recommendation information as features of multiple modalities of the target recommendation information.

[0029] In the technical solution, the first extraction module is further configured to perform the following processing on each audio frame in the target recommendation information:

[0030] perform windowing processing on the audio frame, and perform Fourier transform on the audio feature after the windowing processing to obtain a frequency domain feature of the audio frame.

[0031] perform logarithmic processing on a mel frequency band of the frequency domain feature to obtain the vector representation of the audio frame.

[0032] In the technical solution, the image-form coding processing is implemented by an image encoder, and the image encoder includes a plurality of cascaded coding layers. The first extraction module is further configured to perform the following processing on each image frame in the target recommendation information:

[0033] perform image coding processing on the image frame by a first coding layer in the plurality of cascaded coding layers.

[0034] output a coding result of the first coding layer to a subsequent cascaded coding layer, so as to continue image coding processing and coding result output in the subsequent cascaded coding layer until output to a last coding layer, and

[0035] use a coding result output by the last coding layer as the vector representation of the image frame.

[0036] In the technical solution, the first extraction module is further configured to perform the following processing through the ith encoding layer: performing dimension increasing processing on the encoding result output by the (i-1)th encoding layer to obtain a dimension-increased image feature;

[0037] performing attention processing on the dimension-increased image feature to obtain an attention feature;

[0038] performing dimension decreasing processing on the dimension-increased image feature to obtain a dimension-decreased image feature;

[0039] performing product processing on the attention feature and the dimension-decreased image feature, performing sum processing on the result of the product processing and the encoding result output by the (i-1)th encoding layer, and taking the result of the sum processing as the encoding result of the ith encoding layer;

[0040] outputting the encoding result of the ith encoding layer to the (i+1)th encoding layer;

[0041] wherein i is a positive integer greater than 1 and less than N, and N is the number of the plurality of cascaded encoding layers.

[0042] In the technical solution, when the target recommendation information is a video, the features of the plurality of modalities include: a text representation feature, an audio representation feature, and an image representation feature; and the fusion module is further configured to perform sum processing on the text representation feature, the audio representation feature, and the image representation feature, and take the result of the sum processing as the multi-modal fusion feature of the target recommendation information; or,

[0043] performing splicing processing on the text representation feature, the audio representation feature, and the image representation feature, and taking the result of the splicing processing as the multi-modal fusion feature of the target recommendation information.

[0044] In the technical solution, when the target recommendation information is a video, the features of the plurality of modalities include: a text representation feature, an audio representation feature, and an image representation feature; and the fusion module is further configured to perform mapping processing on the text representation feature, the audio representation feature, and the image representation feature to obtain an intermediate feature vector of the target recommendation information.

[0045] performing affine transformation on the intermediate feature vector of the target recommendation information to obtain the multi-modal fusion feature of the target recommendation information.

[0046] In the technical solution, the second extraction module is further configured to perform the following processing for any candidate user in the plurality of candidate users:

[0047] performing statistical processing on the historical interaction behavior data of the candidate user to obtain a plurality of interest labels of the candidate user;

[0048] performing encoding processing on the plurality of interest labels of the candidate user in a text form to obtain a vector representation of each of the interest labels;

[0049] performing weighted sum processing on the vector representations of the plurality of interest labels based on weights of the plurality of interest labels, and taking a result of the weighted sum processing as an interest feature of the candidate user.

[0050] In the technical solution, the feature extraction processing is implemented by a text encoder, and the text encoder is obtained by interest label prediction training; the apparatus further comprises:

[0051] a training module configured to perform the following processing by using the text encoder:

[0052] performing mask processing on interest label samples for masking in a plurality of interest label samples of a user sample to obtain masked interest label samples;

[0053] performing feature extraction processing on the masked interest label samples and interest label samples that are not masked in the plurality of interest label samples to obtain an interest feature of the user sample;

[0054] performing interest prediction processing based on the interest feature of the user sample to obtain predicted interest labels for masking;

[0055] training the text encoder based on the interest label samples for masking and the predicted interest labels for masking to obtain a trained text encoder.

[0056] In the technical solution, the processing module is further configured to perform similarity processing on the text representation feature of the target recommendation information and the interest feature of each of the candidate users to obtain a text similarity between the target recommendation information and each of the candidate users;

[0057] performing similarity processing on the multi-modal fusion feature of the target recommendation information and the interest feature of each of the candidate users to obtain a multi-modal similarity between the target recommendation information and each of the candidate users;

[0058] performing weighted sum processing on the text similarity and the multi-modal similarity, and taking a result of the weighted sum processing as a similarity between the target recommendation information and each of the candidate users.

[0059] In the technical solution, the screening module takes the candidate user as a target user matched with the target recommendation information when the similarity between the target recommendation information and the candidate user is greater than a similarity threshold; or,

[0060] rank the plurality of candidate users in a descending order based on the similarity between the target recommendation information and the candidate users, and take the candidate user in the front of the descending order as a target user matched with the target recommendation information.

[0061] An embodiment of the present application provides an electronic device for information recommendation, the electronic device comprising:

[0062] a memory configured to store executable instructions;

[0063] a processor configured to execute the executable instructions stored in the memory, so as to implement the information recommendation method based on artificial intelligence provided by the embodiments of the present application.

[0064] An embodiment of the present application provides a computer readable storage medium storing executable instructions, so as to cause the processor to execute the information recommendation method based on artificial intelligence provided by the embodiments of the present application.

[0065] The embodiments of the present application have the following beneficial effects:

[0066] By comprehensively integrating the features of multiple modalities of the target recommendation information, the target recommendation information is fully understood, and the target user matched with the target recommendation information is determined from the plurality of candidate users in combination with the interests of the candidate users, and the recommendation operation on the corresponding target user is performed based on the target recommendation information, so that the user is recommended with the recommendation information meeting the interests of the user, and the accuracy of the recommendation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 is an application scenario diagram of a recommendation system provided by the embodiments of the present application;

[0068] Figure 2 is a structural diagram of an electronic device for information recommendation provided by the embodiments of the present application;

[0069] Figures 3-5 is a flow diagram of the information recommendation method based on artificial intelligence provided by the embodiments of the present application;

[0070] Figure 6 is a structural diagram of an image encoder provided by the embodiments of the present application;

[0071] Figure 7 is a structural diagram of an encoding layer provided by the embodiments of the present application;

[0072] Figure 8 is a flow diagram of user interest and multi-dimensional understanding of video cold start provided by the embodiments of the present application;

[0073] Figure 9 is a user interest representation model diagram provided by the embodiments of the present application;

[0074] Figure 10 is a video multi-dimension representation model diagram provided by an embodiment of the present application;

[0075] Figure 11 is a user interest-video multi-dimension interest correlation interest relevance model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0077] In the following description, the terms "first\second" are merely to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first\second" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0079] The related data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carry out subsequent data use and processing.

[0080] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0081] 1) Target user: the user currently using the recommendation system, i.e. the current user, for example, user A is using the text recommendation system to watch news, and user A is the target user.

[0082] 2) User interest: the behavior tendency shown by the user when using the recommendation system. The recommendation system will determine the user's interest according to a series of behavior performances of the user.

[0083] 3) Modality: the source or form of each kind of information, for example, information in the form of voice, video, text, etc. Each form of information can be referred to as a modality. Modalities can be classified into single modalities and multi-modalities. Single modalities are information represented as numerical vectors that can be processed by computers or further abstracted as higher-level feature vectors. Multi-modalities are better feature representations learned by eliminating the redundancy between modalities through the complementarity between multi-modalities.

[0084] For example, the recommended information can include information in the following modalities: image information (i.e., information in the video modality), voice information (i.e., information in the audio modality), and text information (i.e., information in the text modality).

[0085] 4) Video cold start: When a video is just published on a video site, the video has not been exposed enough and has no historical user behavior data for the recommendation system. Such a video is defined as a cold video, and cold start needs to be performed for the cold video to expose the video to appropriate users to obtain user behavior and assist in accurate recommendation later.

[0086] 5) User portrait: also known as user role, as an effective tool for sketching target users, contacting user demands, and designing directions. User portraits have been widely used in various fields. In the actual operation process, the attributes, behaviors, and expectations of users are often connected as virtual representatives of actual users in the most simple and life-like language.

[0087] The embodiments of the present application provide an information recommendation method and device based on artificial intelligence, electronic equipment and computer readable storage medium, which can recommend recommended information meeting user interests to the user and improve the accuracy of recommendation.

[0088] The information recommendation method based on artificial intelligence provided by the embodiments of the present application can be implemented by a terminal or a server alone; or can be implemented by a terminal and a server in cooperation, for example, the terminal alone bears the information recommendation method based on artificial intelligence described below, or the terminal sends an information recommendation request for a candidate user to the server, the server executes the information recommendation method based on artificial intelligence according to the received information recommendation request for the candidate user, determines a target user matching the target recommended information from multiple candidate users, and performs a recommendation operation on the corresponding target user based on the target recommended information.

[0089] The electronic device for information recommendation provided by the embodiments of the present application can be various types of terminals or servers, wherein the server can be a stand-alone physical server, a server cluster composed of multiple physical servers or a distributed system, or a cloud server providing cloud computing services; the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart television, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0090] Taking the server as an example, for example, it can be a server cluster deployed in the cloud, and an artificial intelligence cloud service (AI as a Service, AIaaS) is opened to users. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service mode is similar to an AI theme mall, and all users can access and use one or more artificial intelligence services provided by the AIaaS platform through the application programming interface.

[0091] For example, one of the artificial intelligence cloud services can be an information recommendation service, that is, the server in the cloud encapsulates the information recommendation program provided by the embodiments of the present application. The user calls the information recommendation service in the cloud service through the terminal (running the client, such as a music client, a video client, etc.) to make the server deployed in the cloud call the encapsulated information recommendation program, determine the target user matched with the target recommendation information from multiple candidate users based on the multi-modal fusion features of the target recommendation information and the interest features of each candidate user, and perform the recommendation operation on the corresponding target user based on the target recommendation information, for example, for the video application, since the target video is a cold video (without enough behavior data), it cannot perform relevant recommendation processing based on the behavior data of the target video, so the interest features of the candidate user are determined based on the historical interaction behavior of the candidate user, the target user matched with the target video is determined from multiple candidate users based on the multi-modal fusion features of the target video and the interest features of each candidate user, and the target video is recommended to the target user, so that the target video is distributed to the user meeting the interest demand, the user behavior data is quickly obtained, the video cold start efficiency is accelerated, and the effect of video recommendation based on user behavior in the later period is improved.

[0092] Referring to Figure 1 , Figure 1 is a schematic diagram of an application scenario of the recommendation system 10 provided by the embodiments of the present application. The terminal (exemplarily shown as the terminal 200-1, the terminal 200-2 and the terminal 200-3) is connected to the server 100 through the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two.

[0093] The terminal (running a client, such as a music client, a video playing client, a short video client, an instant messaging client, a learning client, etc.) can be used to obtain an information recommendation request for a candidate user, for example, when the candidate user opens a video client running on the terminal, the terminal automatically obtains a news recommendation request for the candidate user.

[0094] In some embodiments, after the terminal obtains the information recommendation request for the candidate user, the information recommendation interface of the server 100 (which can be provided in the form of a cloud service, i.e., an information recommendation service) is called, the server 100 obtains the historical interaction behavior data of the candidate user based on the information recommendation request for the candidate user, and performs feature extraction processing to obtain the interest features of the candidate user, determines the target user matching the target recommendation information from multiple candidate users based on the multi-modal fusion features of the target recommendation information and the interest features of each candidate user, and performs a recommendation operation on the corresponding target user based on the target recommendation information to respond to the information recommendation request for the candidate user.

[0095] As an application example, for a video application, since the target video is a cold video (without sufficient behavior data), the related recommendation processing cannot be performed based on the behavior data of the target video, the terminal calls the information recommendation interface of the server based on the video recommendation request for the candidate user, the server obtains the historical interaction behavior data of the candidate user based on the video recommendation request for the candidate user, and performs feature extraction processing to obtain the interest features of the candidate user, determines the target user matching the target video from multiple candidate users based on the multi-modal fusion features of the target video and the interest features of each candidate user, and performs a recommendation operation on the corresponding target user based on the target video to recommend the target video to the target user, thereby distributing the target video to the user meeting the interest demand, quickly obtaining user behavior data, accelerating the video cold start efficiency, and improving the effect of video recommendation based on user behavior in the later period.

[0096] As another application example, for a music application, since the target music is cold music (not enough exposure and historical user behavior data for recommendation), the related recommendation process cannot be performed based on the behavior data of the target music, the terminal calls the information recommendation interface of the server based on the music recommendation request for the candidate user, the server obtains the historical interaction behavior data of the candidate user based on the music recommendation request for the candidate user, and performs feature extraction processing to obtain the interest feature of the candidate user, determines the target user matched with the target music from the multiple candidate users based on the multi-modal fusion feature of the target music and the interest feature of each candidate user, and performs a recommendation operation on the corresponding target user based on the target music to recommend the target music to the target user, thereby distributing the target music to the user meeting the interest demand, quickly obtaining the user behavior data, accelerating the music cold start efficiency, and improving the effect of music recommendation based on the user behavior in the later period.

[0097] The structure of the electronic device for information recommendation provided by the embodiment of the present application is described below. Referring to Figure 2 , Figure 2 is a structural schematic diagram of an electronic device 500 for information recommendation provided by the embodiment of the present application, which is taken as an example of a server, Figure 2 The electronic device 500 for information recommendation shown in the figure comprises at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. Each component in the electronic device 500 is coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection communication between the components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 540 in Figure 2 .

[0098] The processor 510 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0099] The memory 550 includes volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read only memory (ROM), and the volatile memory can be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550 can optionally include one or more storage devices that are physically separate from the processor 510.

[0100] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily described below.

[0101] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks.

[0102] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), etc.

[0103] In some embodiments, the artificial intelligence-based information recommendation apparatus provided by the embodiments of the present application can be implemented in a software manner, for example, it can be implemented in the form of an information recommendation service in the server described above. Of course, it is not limited thereto, and the artificial intelligence-based information recommendation apparatus provided by the embodiments of the present application can be provided in various software embodiments, including various forms such as application programs, software, software modules, scripts, or codes.

[0104] Figure 2 An artificial intelligence-based information recommendation apparatus 555 stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, such as an information recommendation plug-in, and includes a series of modules, including a first extraction module 5551, a fusion module 5552, a second extraction module 5553, a processing module 5554, a screening module 5555, a recommendation module 5556, a determination module 5557, and a training module 5558; among them, the first extraction module 5551, the fusion module 5552, the second extraction module 5553, the processing module 5554, the screening module 5555, the recommendation module 5556, and the determination module 5557 are used to implement the information recommendation function provided by the embodiments of the present application, and the training module 5558 is used to train the text encoder.

[0105] As mentioned above, the information recommendation method based on artificial intelligence provided by the embodiments of the present application can be implemented by various types of electronic devices. Referring to Figure 3 , Figure 3 is a flowchart of the information recommendation method based on artificial intelligence provided by the embodiments of the present application, which is described in combination with the steps shown in Figure 3

[0106] In the following steps, the target recommendation information and the to-be-recommended information can be audio, video and the like. For example, for a music application, the target recommendation information can be audio data, and the features of multiple modalities include text representation features and audio representation features. For a video application, the target recommendation information can be video data, and the features of multiple modalities include text representation features, audio representation features and image representation features.

[0107] Before performing the following steps, it is necessary to determine whether the recommendation information is cold start information (the recommendation information does not have enough exposure and behavior data for recommendation). The to-be-recommended information is obtained from the recommendation information library. When the behavior data of the to-be-recommended information is greater than the behavior data threshold (for example, the number of plays within the valid period is greater than X times, the number of likes is greater than Y times, etc., wherein X and Y are positive integers greater than 0), it means that the to-be-recommended information is hot start information (there is enough exposure and behavior data for recommendation), so that the subsequent information recommendation processing can be performed based on the behavior data of the to-be-recommended information. When the behavior data of the to-be-recommended information is less than or equal to the behavior data threshold, it means that the to-be-recommended information is cold start information, and the to-be-recommended information is taken as the target recommendation information to perform the following steps.

[0108] In step 101, feature extraction processing is performed on the target recommendation information to obtain the features of multiple modalities of the target recommendation information.

[0109] The target recommendation information can be video, audio or other media. For example, when the target recommendation information is video, the type of the video can be TV series, movie, variety show, music, game, animation and the like.

[0110] ​For example, when the target recommendation information is a video, image-form encoding processing is performed on each image frame in the target recommendation information to obtain vector representation of each image frame, and fusion processing is performed on the vector representation of each image frame to obtain image representation features of the target recommendation information; audio-form encoding processing is performed on each audio frame in the target recommendation information to obtain vector representation of each audio frame, and fusion processing is performed on the vector representation of each audio frame to obtain audio representation features of the target recommendation information; text-form encoding processing is performed on each text in the target recommendation information to obtain vector representation of each text, and fusion processing is performed on the vector representation of each text to obtain text representation features of the target recommendation information; and the image representation features, the audio representation features, and the text representation features of the target recommendation information are taken as features of multiple modalities of the target recommendation information.

[0111] Referring to Figure 4 , Figure 4 is an optional flowchart of an information recommendation method based on artificial intelligence provided by the embodiments of the present application, Figure 4 illustrates Figure 3 Step 101 in the method can be implemented through steps 1011-1014: in step 1011, when the target recommendation information is a video, image-form encoding processing is performed on each image frame in the target recommendation information to obtain vector representation of each image frame, and fusion processing is performed on the vector representation of each image frame to obtain image representation features of the target recommendation information; in step 1012, audio-form encoding processing is performed on each audio frame in the target recommendation information to obtain vector representation of each audio frame, and fusion processing is performed on the vector representation of each audio frame to obtain audio representation features of the target recommendation information; in step 1013, text-form encoding processing is performed on each text in the target recommendation information to obtain vector representation of each text, and fusion processing is performed on the vector representation of each text to obtain text representation features of the target recommendation information; and in step 1014, the image representation features, the audio representation features, and the text representation features of the target recommendation information are taken as features of multiple modalities of the target recommendation information.

[0112] For example, when the target recommendation information is a video, the target recommendation information is subjected to feature extraction processing by a classification model (user interest-video multi-dimensional associated interest relevance model), to obtain image representation features, audio representation features, and text representation features of the target recommendation information. The image frames of the target recommendation information include various images such as image frames in the video and a cover, and the text of the target recommendation information includes various texts such as a title, dialogue, and subtitles of the video. The user interest-video multi-dimensional associated interest relevance model includes a classification model (video multi-dimensional representation model), and the video multi-dimensional representation model can be trained by a text encoder (ALBERT model), an audio encoder (for example, VGGish NeXtVlad model for audio feature encoding), and an image encoder (EfficientNet NeXtVlad model for image feature encoding).

[0113] The video multi-dimensional representation model is pre-trained by a video classification task, that is, the video multi-dimensional representation model is subjected to feature extraction processing on a video sample to obtain image representation features, audio representation features, and text representation features of the video sample, the video sample is subjected to classification processing based on the image representation features, the audio representation features, and the text representation features of the video sample to obtain a predicted video type of the video sample, and the value of the loss function of the video multi-dimensional representation model is determined based on the predicted video type of the video sample and a video type label. When the value of the loss function of the video multi-dimensional representation model exceeds a preset threshold, an error signal of the video multi-dimensional representation model is determined based on the loss function of the video multi-dimensional representation model, the error information is back-propagated in the video multi-dimensional representation model, and the model parameters of each layer are updated during the propagation.

[0114] Here, the back-propagation is described. The training sample data is input to the input layer of the neural network model, passes through the hidden layer, finally reaches the output layer and outputs the result, which is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated, and the error is back-propagated from the output layer to the hidden layer until it propagates to the input layer. In the process of back-propagation, the value of the model parameters is adjusted according to the error; the above process is iterated continuously until convergence. The video multi-dimensional representation model belongs to the neural network model.

[0115] In some embodiments, each audio frame in the target recommendation information is encoded in audio form to obtain a vector representation of each audio frame, including: performing the following processing on each audio frame in the target recommendation information: windowing the audio frame and performing Fourier transform on the windowed audio features to obtain the frequency domain features of the audio frame; performing logarithmic processing on the Mel frequency band of the frequency domain features to obtain a vector representation of the audio frame.

[0116] For example, the specific encoding process using the VGGish NeXtVlad model is as follows: 1) Resample the audio frame into mono audio; 2) Window the audio using a Hamming window and perform a short-time Fourier transform to obtain a spectrogram (frequency domain features); 3) Calculate the Mel spectrum (Mel band) by mapping the spectrogram to the filter bank; 4) Perform logarithmic processing on the Mel band to obtain a vector representation of the audio frame.

[0117] In some embodiments, image-based encoding processing is implemented by an image encoder, which includes multiple cascaded encoding layers. Image-based encoding processing is performed on each image frame in the target recommendation information to obtain a vector representation of each image frame. This includes performing the following processing on each image frame in the target recommendation information: performing image encoding processing on the image frame through the first encoding layer in the multiple cascaded encoding layers; outputting the encoding result of the first encoding layer to subsequent cascaded encoding layers to continue image encoding processing and encoding result output in subsequent cascaded encoding layers until the last encoding layer is output, and using the encoding result output by the last encoding layer as the vector representation of the image frame.

[0118] like Figure 6 As shown, an image frame is input to the first coding layer of the image encoder. The image frame is encoded by the first coding layer to obtain the encoding result of the first coding layer. The encoding result of the first coding layer is then input to the second coding layer. The encoding result of the first coding layer is encoded by the second coding layer to obtain the encoding result of the second coding layer. This encoding process is repeated sequentially, with the Nth coding layer encoding the encoding results of the (N-1)th coding layer to obtain the vector representation of the image frame. Here, N is the number of cascaded coding layers. Through cascaded coding, the features of the image frame can be progressively learned, allowing for accurate representation of image features using vector representation.

[0119] In some embodiments, the image encoding processing and the encoding result output continue in the subsequent cascaded encoding layers, including: performing the following processing by the i-th encoding layer: performing dimension increasing processing on the encoding result output by the (i-1)-th encoding layer to obtain dimension-increased image features; performing attention processing on the dimension-increased image features to obtain attention features; performing dimension decreasing processing on the dimension-increased image features to obtain dimension-decreased image features; performing product processing on the attention features and the dimension-decreased image features, and performing sum processing on the product processing result and the encoding result output by the (i-1)-th encoding layer, and taking the sum processing result as the encoding result of the i-th encoding layer; outputting the encoding result of the i-th encoding layer to the (i+1)-th encoding layer; wherein i is a positive integer greater than 1 and less than N.

[0120] As shown in Figure 7 , the encoding layer is implemented by a Mobile inverte dBottleneck Convolution (MBConv) module, which introduces the attention idea of Squeeze-and-Excitation Network (SENet) and can better learn image features. The MBConv module includes dimension increasing, dimension decreasing and attention functions. The dimension increasing processing is implemented by convolution, BatchNormlation, Swish function, Depthwise Convolution, BatchNormlation and Swish function as shown in Figure 7 , the attention processing is implemented by global pooling, convolution, Swish function, convolution and Sigmoid function, and the dimension decreasing processing is implemented by convolution, BatchNormlation and Dropout function.

[0121] In step 102, the features of multiple modalities of the target recommendation information are fused to obtain multi-modal fusion features of the target recommendation information.

[0122] When the target recommendation information is audio, the text representation feature and the audio representation feature of the audio are fused to obtain the multi-modal feature of the audio; when the target recommendation information is video, the text representation feature, the audio representation feature and the image representation feature of the audio are fused to obtain the multi-modal feature of the video. By fusing the features of multiple modalities, multi-modal fusion features containing multiple modalities are obtained, so that features of multiple dimensions are fused to better represent the target recommendation information through features of multiple dimensions for subsequent accurate information recommendation.

[0123] For example, when the target recommendation information is a video, the features of multiple modalities include: text representation features, audio representation features, and image representation features, the text representation features, the audio representation features, and the image representation features are added, and the result of the addition is taken as the multi-modal fusion features of the target recommendation information, so that the fusion processing is realized through a simple addition operation, and the calculation resources are saved.

[0124] For example, when the target recommendation information is a video, the features of multiple modalities include: text representation features, audio representation features, and image representation features, the text representation features, the audio representation features, and the image representation features are added, and the result of the addition is taken as the multi-modal fusion features of the target recommendation information, so that the fusion processing is realized through a simple addition operation, and the calculation resources are saved.

[0125] For example, when the target recommendation information is a video, the features of multiple modalities include: text representation features, audio representation features, and image representation features; the text representation features, the audio representation features, and the image representation features are mapped to obtain an intermediate feature vector of the target recommendation information, and the intermediate feature vector of the target recommendation information is subjected to affine transformation to obtain multi-modal fusion features of the target recommendation information, so that accurate multi-modal fusion features are obtained through layer-by-layer decreasing mapping processing.

[0126] The intermediate feature vector is represented as The multi-modal fusion features are represented as A, The learning parameters are represented as

[0127] In step 103, the historical interaction behavior data of the multiple candidate users is subjected to feature extraction processing to obtain the interest features of each candidate user.

[0128] Since the target recommendation video does not have sufficient behavior data, the features of the candidate users are obtained based on the historical interaction behavior data of the candidate users to obtain the interest features (interest representation) of the candidate users, so that the interest bias of the candidate users is represented through the interest features, and the subsequent user screening processing is performed based on the interest features of the candidate users and the multi-modal features of the target video to determine the appropriate target user for information recommendation.

[0129] Referring to Figure 5 , Figure 5 is an optional flowchart of the information recommendation method based on artificial intelligence provided by the embodiments of the present application, Figure 5 shows Figure 3The step 103 in the method can be implemented through steps 1031-1033. In step 1031, the following processing is performed for any candidate user in the plurality of candidate users: statistical processing is performed on the historical interaction behavior data of the candidate user to obtain a plurality of interest labels of the candidate user; in step 1032, text-form encoding processing is performed on the plurality of interest labels of the candidate user to obtain vector representation of each interest label; in step 1033, weighted summation processing is performed on the vector representation of the plurality of interest labels based on the weights of the plurality of interest labels, and the result of the weighted summation processing is taken as the interest feature of the candidate user.

[0130] For example, the interest weight of the user is introduced to distinguish the interest degree, and in this way, the user interest can be more accurately modeled and represented to represent the preference of the user. The historical interaction behavior data includes interaction data such as playing, clicking, and liking, and by statistically processing the historical interaction behavior data of the candidate user, a plurality of interest labels of the candidate user can be obtained. For example, when the number of times that the candidate user plays variety videos within the valid period exceeds 5 times, it is determined that the candidate user includes the interest label of variety.

[0131] In some embodiments, the feature extraction processing is implemented through a text encoder, and the text encoder is obtained through interest label prediction training; the method further includes: performing the following processing through the text encoder: performing mask processing on the interest label samples for masking in the plurality of interest label samples of the user sample to obtain masked interest label samples; performing feature extraction processing on the masked interest label samples and the interest label samples that are not masked in the plurality of interest label samples to obtain the interest feature of the user sample; performing interest prediction processing based on the interest feature of the user sample to obtain predicted interest labels for masking; and training the text encoder based on the interest label samples for masking and the predicted interest labels for masking to obtain the trained text encoder.

[0132] As Figure 9As shown, the text encoder is implemented by the following steps: first, the interest label sample for masking is masked by the input layer (for example, interest label 2 is masked), to obtain the masked interest label sample, and the masked interest label sample and the interest label sample not masked in the plurality of interest label samples are subjected to feature extraction processing to obtain the interest feature of the user sample, the interest prediction processing is performed based on the interest feature of the user sample, to obtain the predicted interest label for masking (for example, the predicted interest label 2 is obtained), and based on the interest label sample for masking and the predicted interest label for masking, the value of the loss function of the text encoder is determined. When the value of the loss function of the text encoder exceeds the preset threshold, the error signal of the text encoder is determined based on the loss function of the text encoder, the error information is back propagated in the text encoder, and the model parameters of each layer are updated in the propagation process.

[0133] In step 104, the multi-modal fusion feature of the target recommendation information is subjected to similarity processing with the interest feature of each candidate user to obtain the similarity between the target recommendation information and each candidate user.

[0134] For example, after obtaining the multi-modal fusion feature of the target recommendation information and the interest feature of each candidate user, the similarity between the multi-modal fusion feature of the target recommendation information and the interest feature of each candidate user is determined, and the similarity between the multi-modal fusion feature of the target recommendation information and the interest feature of each candidate user is taken as the similarity between the target recommendation information and each candidate user, so as to determine the interest or preference of each candidate user for the target recommendation information, so as to subsequently determine the target user matching the target recommendation information from the plurality of candidate users.

[0135] In some embodiments, before determining the target user matching the target recommendation information from the plurality of candidate users, the text representation feature of the target recommendation information is subjected to similarity processing with the interest feature of each candidate user to obtain the text similarity between the target recommendation information and each candidate user; the multi-modal fusion feature of the target recommendation information is subjected to similarity processing with the interest feature of each candidate user to obtain the similarity between the target recommendation information and each candidate user, including: the multi-modal fusion feature of the target recommendation information is subjected to similarity processing with the interest feature of each candidate user to obtain the multi-modal similarity between the target recommendation information and each candidate user; the text similarity and the multi-modal similarity are subjected to weighted sum processing, and the result of the weighted sum processing is taken as the similarity between the target recommendation information and each candidate user.

[0136] For example, since the interest features of the users are mainly represented in the form of text, the text representation features of the target recommendation information can also be processed for similarity with the interest features of each candidate user to obtain the text similarity between the target recommendation information and each candidate user, the multi-modal fusion features of the target recommendation information are processed for similarity with the interest features of each candidate user to obtain the multi-modal similarity between the target recommendation information and each candidate user, the text similarity and the multi-modal similarity are processed for weighted summation based on the weight of the text similarity and the weight of the multi-modal similarity, and the result of the weighted summation processing is taken as the similarity between the target recommendation information and each candidate user, so that the similarity between the target recommendation information and each candidate user is represented from multiple aspects to obtain an accurate similarity between the target recommendation information and each candidate user, so as to accurately recommend information subsequently.

[0137] In step 105, based on the similarity between the target recommendation information and each candidate user, a target user matching the target recommendation information is determined from the plurality of candidate users.

[0138] For example, when the similarity between the target recommendation information and the candidate user is greater than a similarity threshold, the candidate user is taken as the target user matching the target recommendation information; the plurality of candidate users can also be sorted in descending order based on the similarity between the target recommendation information and the candidate user, and the candidate user sorted in the front is taken as the target user matching the target recommendation information, for example, 100 candidate users are sorted in descending order according to the similarity between the target recommendation information and the candidate user from large to small, and the first 40 candidate users sorted in the front are taken as the target user matching the target recommendation information.

[0139] In step 106, a recommendation operation on the corresponding target user is performed based on the target recommendation information.

[0140] For example, after determining the target user matching the target recommendation information, the target recommendation information is distributed to the target user, so that the target recommendation information is exposed to the user meeting the interest demand, the user behavior is quickly constructed for the target recommendation information to quickly obtain user behavior data, and the effect of information recommendation based on user behavior in the later stage is improved, so that the information recommendation in the later stage is more accurate, while the invalid information exposure is reduced and the recommendation interference on the user is reduced.

[0141] In the following, an exemplary application of the embodiment of the present application in an actual application scenario will be described.

[0142] In the related art, the scheme of video cold start is to calculate the similarity between the cold start video and the user historical watching video, distribute the cold start video to the user who has watched similar videos, so as to help the user discover the video that may interest them in the information overload environment, and push the video to the user who is interested in them.

[0143] The applicant finds in the process of implementing the embodiments of the application that the video cold start scheme in the related art does not fully understand the multi-dimensional content of the video, and does not fully combine the user interest and the multi-dimensional (multi-modal) content of the video, so that the user interest and the interest degree of the video content cannot be well captured, resulting in low cold start efficiency.

[0144] To solve the above problems, the embodiments of the application provide a user interest and multi-dimensional understanding video cold start method, which jointly models the user interest and the multi-dimensional content of the video, fully understands the video that needs to be cold started based on the multi-dimensional content of the video, exposes the cold start video to the user who meets the interest demand, and thus quickly constructs the user behavior for the new cold start video through the interaction behavior of the appropriate user, so as to quickly obtain the user behavior data, speed up the video cold start efficiency, improve the effect of video recommendation based on the user behavior in the later stage, play a positive role in the video recommendation effect of the platform, make the later video recommendation more accurate, and reduce the exposure of invalid new cold video and the recommendation interference to the user.

[0145] As shown in Figure 8 The user interest and multi-dimensional understanding video cold start method provided by the embodiments of the application includes the steps as follows:

[0146] Step 1) Constructing a user interest representation model (text encoder)

[0147] Through deep learning of the user interest portrait, the user portrait is modeled by a Lite Bidirectional Encoder Representations from Transformers (ALBERT) model. As shown in Figure 9 The input layer of the ALBERT model masks part of the user's interest, and the output layer of the ALBERT model predicts the interest label that is masked, while introducing a user interest weight to distinguish the interest degree. In this way, the user interest can be more accurately modeled and represented to represent the user. The user interest label and the corresponding weight are iteratively counted on the user's historical playing behavior.

[0148] The ALBERT model is pre-trained on user interest label data, so that the ALBERT model has the function of inputting user interest and returning user interest representation (interest feature). The ALBERT model is further trained as a sub-module of the user interest-video multidimensional representation associated interest representation model, so that the user interest representation model can learn more fully, and the joint learning with the video multidimensional representation model can converge more quickly.

[0149] Step 2) Building a video multidimensional representation model (classification model)

[0150] The video multidimensional representation model is pre-trained on the video text content (title, automatic speech recognition (ASR, Automatic Speech Recognition) dialogue text, optical character recognition (OCR, Optical Character Recognition) subtitle text), audio content, cover picture and video image frame image content, so that the video multidimensional representation model has the function of inputting video text, audio and image features and returning video multidimensional representation. The video multidimensional representation model is further trained as a sub-module of the user interest-video multidimensional representation associated interest representation model to accelerate convergence. Figure 10 The video multidimensional representation model is pre-trained on the video text content (title, automatic speech recognition (ASR, Automatic Speech Recognition) dialogue text, optical character recognition (OCR, Optical Character Recognition) subtitle text), audio content, cover picture and video image frame image content, so that the video multidimensional representation model has the function of inputting video text, audio and image features and returning video multidimensional representation. The video multidimensional representation model is further trained as a sub-module of the user interest-video multidimensional representation associated interest representation model to accelerate convergence.

[0151] The video multidimensional representation model is pre-trained on the video text content (title, automatic speech recognition (ASR, Automatic Speech Recognition) dialogue text, optical character recognition (OCR, Optical Character Recognition) subtitle text), audio content, cover picture and video image frame image content, so that the video multidimensional representation model has the function of inputting video text, audio and image features and returning video multidimensional representation. The video multidimensional representation model is further trained as a sub-module of the user interest-video multidimensional representation associated interest representation model to accelerate convergence.

[0152] Step 3) Building a user interest-video multidimensional associated interest relevance model

[0153] As shown in Figure 11 The user interest-video multidimensional associated interest relevance model is built by jointly learning the user interest representation model in step 1) and the video multidimensional representation model in step 2).

[0154] The video multidimensional vector representation and the user interest representation are interacted, and the model is trained on the video data set of whether the user is interested. The similarity between the multidimensional representation of the video and the interest representation of the user is constructed, that is, the similarity between the interest representation vector of the user and the multidimensional representation vector of the video represents the interest degree of the user to the video. The video data set of whether the user is interested is automatically constructed based on the exposure or playback record of a large number of users on the platform, and the video is recommended to the user. If the user's playback completion degree is higher than a certain threshold or a certain time length, it is considered that the user is interested in the video; if the user does not play a certain video, or the playback time length or completion degree is lower than a certain threshold, it is considered that the user is not interested in the video.

[0155] As Figure 11 shown in the

[0156] Through the above Figure 11 model, the user interest and the video multi-dimensional content are jointly associated with interest learning, so that the user interest representation sub-model and the video multi-dimensional representation sub-model have interest relevance representation ability, which provides a basis for constructing a cold start strategy based on the user interest representation vector and the video multi-dimensional representation vector.

[0157] Step 4) Constructing vector retrieval library for platform users and cold start video library

[0158] To obtain the interest labels of the users of the platform, the user interest model shown in Figure 11 is used to construct the user interest representation, and a vector retrieval library is constructed for the user interest representation of the platform. In addition, for the new cold start video library of the platform, the text, audio, and image modal features are obtained, and the multi-dimensional representation and the text representation vector are constructed through the video multi-dimensional representation model in the above Figure 11 , and then the video cold start text representation vector retrieval index and the multi-dimensional representation vector retrieval index are constructed, which are used for later video cold start distribution implementation.

[0159] Step 5) Realizing video cold start distribution based on user interest representation and video multi-dimensional representation relevance

[0160] For each video in the cold start video library, search for appropriate users for exposure, i.e. determine the users whose interest degree meets a certain interest degree threshold, and improve the exposure efficiency of the cold start video. The cold start interest degree of the user to the video = x1*user interest representation and video multi-dimensional representation vector similarity (UVM) + x2*user interest representation and video text representation vector similarity (UVT), wherein x1 represents the weight of UVM, and x2 represents the weight of UVT, wherein x1+x2=1.0. The above UVM interest degree is obtained by using the user interest vector retrieval library and the video multi-dimensional representation vector retrieval index, and the above UVT interest degree is obtained by using the video text representation vector retrieval index. When the user behavior data obtained by the cold start video meets the threshold, for example, the exposure amount meets X times, and the play times meets Y times, the cold start video is removed from the cold start video library.

[0161] To sum up, the embodiment of the present application proposes a user interest and multi-dimensional understanding video cold start method, which fully models the user interest and multi-dimensional content of the video, more accurately distributes the cold start video to the appropriate user, makes the cold start video obtain the user behavior data more quickly, improves the cold start efficiency, reduces the interference of invalid cold start distribution on the user, provides the behavior data support for the later video recommendation, and improves the video recommendation effect.

[0162] By now, the exemplary application and implementation of the server provided by the embodiment of the present application have been combined to describe the information recommendation method based on artificial intelligence provided by the embodiment of the present application. The embodiment of the present application also provides an information recommendation device. In actual application, each function module in the information recommendation device can be cooperatively implemented by hardware resources of an electronic device (such as a terminal device, a server or a server cluster), such as a processor and other computing resources, communication resources (such as those used to support various communication modes such as optical cable and cellular network), and a memory. Figure 2 An information recommendation device 555 stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, for example, software modules designed using software C / C++, Java and other programming languages, special software modules in application software or large software systems designed using C / C++, Java and other programming languages, application program interfaces, plug-ins, cloud services and other implementation manners. Different implementation manners are exemplified below.

[0163] Example 1: The information recommendation device is a mobile terminal application program and module

[0164] The information recommendation device 555 in the embodiment of the present application can be provided as a software module designed using software C / C++, Java and other programming languages, embedded in various mobile terminal applications based on Android or iOS systems (stored in the storage medium of the mobile terminal in the form of executable instructions and executed by the processor of the mobile terminal), so as to directly use the computing resources of the mobile terminal itself to complete the related information recommendation task, and periodically or irregularly transmit the processing result to the remote server through various network communication modes, or save it locally on the mobile terminal.

[0165] Example 2: The information recommendation device is a server application program and platform

[0166] The information recommendation device 555 in the embodiment of the present application can be provided as a special software module in an application software or a large software system designed using C / C++, Java and other programming languages, running on a server side (stored in the storage medium of the server side in the form of executable instructions and executed by the processor of the server side), and the server uses its own computing resources to complete the related information recommendation task.

[0167] The embodiment of the present application can also be provided as a customized, easy-to-interact web interface or other user interface (UI) on a distributed, parallel computing platform composed of multiple servers, forming an information recommendation platform (for recommendation list) for individuals, groups or units.

[0168] Example Three, the information recommendation device is a server-side application program interface (API) and plug-in

[0169] The information recommendation device 555 in the embodiment of the present application can be provided as a server-side API or plug-in for users to call to execute the artificial intelligence-based information recommendation method of the embodiment of the present application and embedded into various application programs.

[0170] Example Four, the information recommendation device is a mobile device client-side API and plug-in

[0171] The information recommendation device 555 in the embodiment of the present application can be provided as a mobile device-side API or plug-in for users to call to execute the artificial intelligence-based information recommendation method of the embodiment of the present application.

[0172] Example Five, the information recommendation device is a cloud-side open service

[0173] The information recommendation device 555 in the embodiment of the present application can be provided as an information recommendation cloud service developed for users for individuals, groups or units to make information recommendations.

[0174] The information recommendation device 555 includes a series of modules, including a first extraction module 5551, a fusion module 5552, a second extraction module 5553, a processing module 5554, a screening module 5555, a recommendation module 5556, a determination module 5557 and a training module 5558. The following continues to describe the scheme for the modules in the information recommendation device 555 provided by the embodiment of the present application to cooperate to implement information recommendation.

[0175] The first extraction module 5551 is configured to perform feature extraction processing on the target recommendation information to obtain features of multiple modalities of the target recommendation information; the fusion module 5552 is configured to perform fusion processing on the features of multiple modalities of the target recommendation information to obtain multi-modal fusion features of the target recommendation information; the second extraction module 5553 is configured to perform feature extraction processing on historical interaction behavior data of multiple candidate users to obtain interest features of each candidate user; the processing module 5554 is configured to perform similarity processing on the multi-modal fusion features of the target recommendation information and the interest features of each candidate user to obtain similarities between the target recommendation information and each candidate user; the screening module 5555 is configured to determine a target user that matches the target recommendation information from the multiple candidate users based on the similarities between the target recommendation information and each candidate user; and the recommendation module 5556 is configured to perform a recommendation operation corresponding to the target user based on the target recommendation information.

[0176] In some embodiments, the information recommendation apparatus 555 further includes a determination module 5557 configured to obtain to-be-recommended information from a recommendation information library; when behavior data of the to-be-recommended information is greater than a behavior data threshold, performing information recommendation processing based on the behavior data of the to-be-recommended information; and when the behavior data of the to-be-recommended information is less than or equal to the behavior data threshold, taking the to-be-recommended information as the target recommendation information.

[0177] In some embodiments, when the target recommendation information is a video, the first extraction module 5551 is further configured to perform image-form encoding processing on each image frame in the target recommendation information to obtain vector representation of each image frame, and perform fusion processing on the vector representation of each image frame to obtain image representation features of the target recommendation information; perform audio-form encoding processing on each audio frame in the target recommendation information to obtain vector representation of each audio frame, and perform fusion processing on the vector representation of each audio frame to obtain audio representation features of the target recommendation information; perform text-form encoding processing on each text in the target recommendation information to obtain vector representation of each text, and perform fusion processing on the vector representation of each text to obtain text representation features of the target recommendation information; and take the image representation features, the audio representation features, and the text representation features of the target recommendation information as the features of multiple modalities of the target recommendation information.

[0178] In some embodiments, the first extraction module 5551 is further configured to, for each audio frame in the target recommendation information, perform the following processing: windowing the audio frame, and performing Fourier transform on the windowed audio feature to obtain a frequency domain feature of the audio frame; performing logarithmic processing on a mel frequency band of the frequency domain feature to obtain a vector representation of the audio frame.

[0179] In some embodiments, the encoding processing of the image form is implemented by an image encoder, the image encoder comprising a plurality of cascaded encoding layers; the first extraction module 5551 is further configured to, for each image frame in the target recommendation information, perform the following processing: performing image encoding processing on the image frame by a first encoding layer in the plurality of cascaded encoding layers; outputting the encoding result of the first encoding layer to a subsequent cascaded encoding layer, so as to continue the image encoding processing and the encoding result output in the subsequent cascaded encoding layer until the last encoding layer, and outputting the encoding result output by the last encoding layer as the vector representation of the image frame.

[0180] In some embodiments, the first extraction module 5551 is further configured to, by the i-th encoding layer, perform the following processing: performing dimension increasing processing on the encoding result output by the i-1-th encoding layer to obtain a dimension-increased image feature; performing attention processing on the dimension-increased image feature to obtain an attention feature; performing dimension decreasing processing on the dimension-increased image feature to obtain a dimension-decreased image feature; performing product processing on the attention feature and the dimension-decreased image feature, and performing sum processing on the result of the product processing and the encoding result output by the i-1-th encoding layer, and outputting the result of the sum processing as the encoding result of the i-th encoding layer; outputting the encoding result of the i-th encoding layer to the i+1-th encoding layer; wherein i is a positive integer greater than 1 and less than N, and N is the number of the plurality of cascaded encoding layers.

[0181] In some embodiments, when the target recommendation information is a video, the features of the plurality of modalities comprise: text representation features, audio representation features, and image representation features; the fusion module 5552 is further configured to perform sum processing on the text representation features, the audio representation features, and the image representation features, and output the result of the sum processing as the multi-modal fusion feature of the target recommendation information; or, perform concatenation processing on the text representation features, the audio representation features, and the image representation features, and output the result of the concatenation processing as the multi-modal fusion feature of the target recommendation information.

[0182] In some embodiments, when the target recommendation information is a video, the features of the plurality of modalities include: a text representation feature, an audio representation feature, and an image representation feature; the fusion module 5552 is further configured to perform mapping processing on the text representation feature, the audio representation feature, and the image representation feature to obtain an intermediate feature vector of the target recommendation information; and perform affine transformation on the intermediate feature vector of the target recommendation information to obtain a multi-modal fusion feature of the target recommendation information.

[0183] In some embodiments, the second extraction module 5553 is further configured to perform the following processing for any candidate user in the plurality of candidate users: perform statistical processing on historical interaction behavior data of the candidate user to obtain a plurality of interest labels of the candidate user; perform text-form encoding processing on the plurality of interest labels of the candidate user to obtain vector representations of each of the interest labels; perform weighted summation processing on the vector representations of the plurality of interest labels based on weights of the plurality of interest labels, and take a result of the weighted summation processing as an interest feature of the candidate user.

[0184] In some embodiments, the feature extraction processing is implemented by a text encoder, and the text encoder is obtained by interest label prediction training; the information recommendation apparatus 555 further includes a training module 5558 configured to perform the following processing by using the text encoder: perform mask processing on interest label samples for masking in a plurality of interest label samples of a user sample to obtain masked interest label samples; perform feature extraction processing on the masked interest label samples and interest label samples that are not masked in the plurality of interest label samples to obtain an interest feature of the user sample; perform interest prediction processing based on the interest feature of the user sample to obtain predicted interest labels for masking; and train the text encoder based on the interest label samples for masking and the predicted interest labels for masking to obtain a trained text encoder.

[0185] In some embodiments, the processing module 5554 is further configured to perform similarity processing on the text representation feature of the target recommendation information and the interest feature of each of the candidate users to obtain a text similarity between the target recommendation information and each of the candidate users; perform similarity processing on the multi-modal fusion feature of the target recommendation information and the interest feature of each of the candidate users to obtain a multi-modal similarity between the target recommendation information and each of the candidate users; and perform weighted summation processing on the text similarity and the multi-modal similarity to take a result of the weighted summation processing as a similarity between the target recommendation information and each of the candidate users.

[0186] In some embodiments, the screening module 5555 screens the candidate users as target users matched with the target recommendation information when a similarity between the target recommendation information and the candidate user is greater than a similarity threshold; or, based on the similarity between the target recommendation information and the candidate user, the plurality of candidate users are sorted in descending order, and the candidate user in front of the descending order is screened as a target user matched with the target recommendation information.

[0187] The embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the information recommendation method based on artificial intelligence provided by the embodiment of the present application.

[0188] The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the executable instructions are stored in the computer readable storage medium. When the executable instructions are executed by a processor, the processor will execute the information recommendation method based on artificial intelligence provided by the embodiment of the present application, for example, as shown in the information recommendation method based on artificial intelligence. Figures 3-5 The information recommendation method based on artificial intelligence.

[0189] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices including one or any combination of the above memories.

[0190] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0191] As an example, the executable instructions can but not necessarily correspond to files in a file system, can be stored in part of a file storing other programs or data, for example, stored in one or more scripts in a hyper text markup language (HTML, Hyper Text Markup Language) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code portions).

[0192] As an example, the executable instructions can be deployed to execute on one computing device, or on multiple computing devices located at one site, or distributed across multiple sites and interconnected by a communication network.

[0193] The above merely provides illustration of the embodiment of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and range of the present application shall fall within the protection scope of the present application.

Claims

1. An information recommendation method based on artificial intelligence, characterized in that, The method comprises: obtaining to-be-recommended information from a recommendation information library; when behavior data of the to-be-recommended information is greater than a behavior data threshold, performing information recommendation processing based on the behavior data of the to-be-recommended information; when the behavior data of the to-be-recommended information is less than or equal to the behavior data threshold, taking the to-be-recommended information as target recommended information; performing feature extraction processing on the target recommended information to obtain features of multiple modalities of the target recommended information; performing fusion processing on the features of multiple modalities of the target recommended information to obtain multi-modality fusion features of the target recommended information; performing feature extraction processing on historical interaction behavior data of multiple candidate users to obtain interest features of each candidate user; performing similarity processing on the multi-modality fusion features of the target recommended information and the interest features of each candidate user to obtain similarity between the target recommended information and each candidate user; determining a target user matched with the target recommended information from the multiple candidate users based on the similarity between the target recommended information and each candidate user; performing a recommendation operation corresponding to the target user based on the target recommended information to construct behavior data of the target recommended information.

2. The method of claim 1, wherein, When the target recommended information is a video, the feature extraction processing on the target recommended information to obtain features of multiple modalities of the target recommended information comprises: performing image-form coding processing on each image frame in the target recommended information to obtain vector representation of each image frame, and performing fusion processing on the vector representation of each image frame to obtain image representation features of the target recommended information; performing audio-form coding processing on each audio frame in the target recommended information to obtain vector representation of each audio frame, and performing fusion processing on the vector representation of each audio frame to obtain audio representation features of the target recommended information; performing text-form coding processing on each text in the target recommended information to obtain vector representation of each text, and performing fusion processing on the vector representation of each text to obtain text representation features of the target recommended information; taking the image representation features, the audio representation features, and the text representation features of the target recommended information as the features of multiple modalities of the target recommended information.

3. The method of claim 2, wherein, The audio-form coding processing on each audio frame in the target recommended information to obtain vector representation of each audio frame comprises: performing the following processing on each audio frame in the target recommended information: performing windowing processing on the audio frame, and performing Fourier transform on the audio features after the windowing processing to obtain frequency domain features of the audio frame; performing logarithmic processing on a mel frequency band of the frequency domain features to obtain the vector representation of the audio frame.

4. The method of claim 2, wherein the image-form coding processing is implemented by an image encoder, and the image encoder comprises a plurality of cascaded coding layers. The image form encoding processing is performed on each image frame in the target recommendation information to obtain a vector representation of each image frame, including: The following processing is performed on each image frame in the target recommendation information: image encoding processing is performed on the image frame through a first encoding layer in the plurality of cascaded encoding layers; the encoding result of the first encoding layer is output to a subsequent cascaded encoding layer, so that image encoding processing and encoding result output are continued in the subsequent cascaded encoding layer until the last encoding layer is output, and the encoding result output by the last encoding layer is taken as the vector representation of the image frame.

5. The method of claim 4, wherein, The image encoding processing and the encoding result output are continued in the subsequent cascaded encoding layer, including: The following processing is performed through the i-th encoding layer: dimensionality increasing processing is performed on the encoding result output by the i-1-th encoding layer to obtain dimensionality increased image features; attention processing is performed on the dimensionality increased image features to obtain attention features; dimensionality reducing processing is performed on the dimensionality increased image features to obtain dimensionality reduced image features; the attention features are multiplied with the dimensionality reduced image features, and the result of the multiplication processing is added with the encoding result output by the i-1-th encoding layer, and the result of the addition processing is taken as the encoding result of the i-th encoding layer; the encoding result of the i-th encoding layer is output to the i+1-th encoding layer; wherein i is a positive integer greater than 1 and less than N, and N is the number of the plurality of cascaded encoding layers.

6. The method of claim 1, wherein when the target recommendation information is a video, the plurality of modal features include: text representation features, audio representation features, and image representation features; the fusion processing of the plurality of modal features of the target recommendation information includes: the text representation features, the audio representation features, and the image representation features are added, and the result of the addition is taken as the multi-modal fusion feature of the target recommendation information; or the text representation features, the audio representation features, and the image representation features are spliced, and the result of the splicing is taken as the multi-modal fusion feature of the target recommendation information.

7. The method of claim 1, wherein when the target recommendation information is a video, the plurality of modal features include: text representation features, audio representation features, and image representation features; the fusion processing of the plurality of modal features of the target recommendation information includes: the text representation features, the audio representation features, and the image representation features are mapped to obtain an intermediate feature vector of the target recommendation information; the intermediate feature vector of the target recommendation information is subjected to affine transformation to obtain the multi-modal fusion feature of the target recommendation information.

8. The method of claim 1, wherein, The feature extraction processing of the historical interaction behavior data of the plurality of candidate users includes: The following processing is performed for any candidate user in the plurality of candidate users: statistical processing is performed on the historical interaction behavior data of the candidate user to obtain a plurality of interest labels of the candidate user; textual form encoding processing is performed on the plurality of interest labels of the candidate user to obtain vector representations of each of the interest labels; weighting sum processing is performed on the vector representations of the plurality of interest labels based on the weights of the plurality of interest labels, and the result of the weighting sum processing is taken as the interest feature of the candidate user.

9. The method of claim 8, wherein the feature extraction processing is implemented by a text encoder, and the text encoder is obtained by interest label prediction training; the method further comprises: performing the following processing by the text encoder: masking processing is performed on interest label samples for masking in a plurality of interest label samples of a user sample to obtain masked interest label samples; feature extraction processing is performed on the masked interest label samples and interest label samples that are not masked in the plurality of interest label samples to obtain an interest feature of the user sample; interest prediction processing is performed based on the interest feature of the user sample to obtain predicted interest labels for masking; the text encoder is trained based on the interest label samples for masking and the predicted interest labels for masking to obtain a trained text encoder.

10. The method of claim 1, wherein before the determining of the target user matching the target recommendation information from the plurality of candidate users, the method further comprises: similarity processing is performed on a text representation feature of the target recommendation information and an interest feature of each of the candidate users to obtain a text similarity between the target recommendation information and each of the candidate users; the similarity processing comprises: similarity processing is performed on a multi-modal fusion feature of the target recommendation information and an interest feature of each of the candidate users to obtain a multi-modal similarity between the target recommendation information and each of the candidate users; weighting sum processing is performed on the text similarity and the multi-modal similarity, and the result of the weighting sum processing is taken as the similarity between the target recommendation information and each of the candidate users.

11. The method of claim 1, wherein, the determining of the target user matching the target recommendation information from the plurality of candidate users based on the similarity between the target recommendation information and each of the candidate users comprises: when the similarity between the target recommendation information and the candidate user is greater than a similarity threshold, the candidate user is taken as the target user matching the target recommendation information; or the plurality of candidate users are sorted in descending order based on the similarity between the target recommendation information and each of the candidate users, and a candidate user in front of the descending order is taken as the target user matching the target recommendation information.

12. An information recommendation apparatus based on artificial intelligence, characterized by comprising: the apparatus comprises: The first extraction module is configured to obtain to-be-recommended information from a recommendation information library; when behavior data of the to-be-recommended information is greater than a behavior data threshold, performing information recommendation processing based on the behavior data of the to-be-recommended information; when the behavior data of the to-be-recommended information is less than or equal to the behavior data threshold, taking the to-be-recommended information as target recommended information; and performing feature extraction processing on the target recommended information to obtain features of multiple modalities of the target recommended information; The fusion module is configured to perform fusion processing on the features of multiple modalities of the target recommended information to obtain multi-modality fusion features of the target recommended information; The second extraction module is configured to perform feature extraction processing on historical interaction behavior data of multiple candidate users to obtain interest features of each candidate user; The processing module is configured to perform similarity processing on the multi-modality fusion features of the target recommended information and the interest features of each candidate user to obtain a similarity between the target recommended information and each candidate user; The screening module is configured to determine a target user matched with the target recommended information from the multiple candidate users based on the similarity between the target recommended information and each candidate user. The recommendation module is configured to perform a recommendation operation corresponding to the target user based on the target recommended information to construct behavior data of the target recommended information.

13. The apparatus of claim 12, wherein, When the target recommended information is a video, the apparatus further comprises: The first extraction module is further configured to perform image-form coding processing on each image frame in the target recommended information to obtain vector representation of each image frame, perform fusion processing on the vector representation of each image frame to obtain image representation features of the target recommended information, perform audio-form coding processing on each audio frame in the target recommended information to obtain vector representation of each audio frame, perform fusion processing on the vector representation of each audio frame to obtain audio representation features of the target recommended information, perform text-form coding processing on each text in the target recommended information to obtain vector representation of each text, and perform fusion processing on the vector representation of each text to obtain text representation features of the target recommended information; and take the image representation features, the audio representation features, and the text representation features of the target recommended information as the features of multiple modalities of the target recommended information.

14. The apparatus of claim 12, wherein, The apparatus further comprises: The first extraction module is further configured to perform the following processing on each audio frame in the target recommended information: performing windowing processing on the audio frame, and performing Fourier transform on the audio features after the windowing processing to obtain frequency domain features of the audio frame; and performing logarithmic processing on a mel frequency band of the frequency domain features to obtain vector representation of the audio frame.

15. An electronic device, comprising: The electronic device comprises: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the artificial intelligence-based information recommendation method in any one of claims 1 to 11.

16. A computer-readable storage medium, characterized in that, The computer instructions are executed by the processor to implement the artificial intelligence-based information recommendation method according to any one of claims 1-11.

17. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the artificial intelligence-based information recommendation method according to any one of claims 1-11.

Images