Information recommendation method and device, model training method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an information recommendation method and device, a model training method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: based on received first voice data, determining first text data of the first voice data, performing object recognition on the first text data to obtain a to-be-recommended first object, and performing feature extraction on multi-modal data of the first object to obtain a first multi-modal feature; and performing feature fusion on the first text feature and the first multi-modal feature to obtain a first fusion feature, and determining recommendation information of the first voice data based on the first fusion feature. Through the method, the accuracy of the recommended information can be improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to an information recommendation method, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of information technology, personalized recommendations have become a crucial component of various fields, including e-commerce, online video, and social media. Providing users with accurate recommendations of products, works, and other information can enhance user experience and improve the platform's commercial benefits. However, due to the diverse ways in which product information is carried, key product information may be distributed across different information types. Related technologies that rely solely on textual product information to generate recommendations can easily result in inaccurate product information, leading to lower accuracy in the resulting recommendations. Summary of the Invention

[0003] Embodiments of the present application provide an information recommendation method, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of recommended information.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides an information recommendation method, the method comprising:

[0006] Determining first text data of the first voice data based on the received first voice data;

[0007] Performing text feature extraction on the first text data to obtain a first text feature;

[0008] Performing object recognition on the first text data to obtain a first object to be recommended, and performing feature extraction on multimodal data of the first object to obtain a first multimodal feature;

[0009] Performing feature fusion on the first text feature and the first multimodal feature to obtain a first fused feature;

[0010] Based on the first fusion feature, recommendation information for the first speech data is determined.

[0011] The present invention provides an information recommendation device, including:

[0012] a data determining module, configured to determine first text data of the first voice data based on the received first voice data;

[0013] A feature extraction module, configured to extract text features from the first text data to obtain first text features;

[0014] The feature extraction module is further configured to perform object recognition on the first text data to obtain a first object to be recommended, and perform feature extraction on the multimodal data of the first object to obtain a first multimodal feature;

[0015] a feature fusion module, configured to fuse the first text feature and the first multimodal feature to obtain a first fused feature;

[0016] An information determination module is used to determine recommendation information of the first speech data based on the first fusion feature.

[0017] The present invention provides a model training method, which includes:

[0018] Determining third text data of the voice sample based on the received voice sample;

[0019] Performing text feature extraction on the third text data to obtain third text features;

[0020] performing object recognition on the third text data to obtain a third object to be recommended, and performing feature extraction on the multimodal data of the third object to obtain a second multimodal feature;

[0021] Performing feature fusion on the third text feature and the second multimodal feature to obtain a third fused feature;

[0022] Predicted recommendation information of the voice sample is determined based on the third fusion feature, and the first model is trained based on a difference between the predicted recommendation information and actual recommendation information of the voice sample.

[0023] The present invention provides an information recommendation device, including:

[0024] a data determination module, configured to determine third text data of the voice sample based on the received voice sample;

[0025] A feature extraction module, configured to extract text features from the third text data to obtain third text features;

[0026] The feature extraction module is further configured to perform object recognition on the third text data to obtain a third object to be recommended, and perform feature extraction on the multimodal data of the third object to obtain a second multimodal feature;

[0027] a feature fusion module, configured to fuse the third text feature and the second multimodal feature to obtain a third fused feature;

[0028] A model training module is used to determine the predicted recommendation information of the voice sample based on the third fusion feature, and train the first model based on the difference between the predicted recommendation information and the actual recommendation information of the voice sample.

[0029] An embodiment of the present application provides an electronic device, comprising:

[0030] a memory for storing computer-executable instructions or computer programs;

[0031] The processor is used to implement the information recommendation method provided in the embodiment of the present application or the model training method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0032] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the information recommendation method or the model training method provided in the embodiment of the present application.

[0033] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the information recommendation method provided by the embodiment of the present application or the model training method provided by the embodiment of the present application is implemented.

[0034] The embodiments of the present application have the following beneficial effects:

[0035] Through the above embodiment, when recommending information to a user, the first text data of the first voice data can be determined based on the received first voice data, and the recommended objects that match the voice data can be first determined through the received voice data. The recommended objects are first screened to avoid subsequent data processing of all recommended objects, thereby improving the efficiency of data processing. Afterwards, text feature extraction can be performed on the first text data to obtain the first text feature, object recognition can be performed on the first text data to obtain the first object to be recommended, and feature extraction can be performed on the multimodal data of the first object to obtain the first multimodal feature. By performing feature fusion on the first text feature and the first multimodal feature, the first fusion feature is obtained, and based on the first fusion feature, the recommended information for the first voice data is determined. In this way, when processing data of different modalities of the recommended object, the data of multiple modalities can be accurately combined, and the recommended information of the user can be generated by combining the data of multiple modalities, so as to obtain recommended information that better meets the user's needs and improve the accuracy of the recommended information pushed to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 1 is a schematic diagram of the architecture of the information recommendation system 100 provided in an embodiment of the present application;

[0037] Figure 2A is a structural diagram of an electronic device 500-1 provided in an embodiment of the present application;

[0038] Figure 2B is a schematic structural diagram of an electronic device 500-2 provided in an embodiment of the present application;

[0039] Figure 3A This is a schematic diagram of the first flow chart of the information recommendation method provided in an embodiment of the present application;

[0040] Figure 3B This is a second flow chart of the information recommendation method provided in an embodiment of the present application;

[0041] Figure 4A This is a first flow chart of the model training method provided in an embodiment of the present application;

[0042] Figure 4B This is a second flow chart of the model training method provided in an embodiment of the present application;

[0043] Figure 5 This is a structural diagram of the first model provided in the embodiment of the present application;

[0044] Figure 6 This is a flowchart of the implementation of the embodiment of the present application in a practical application scenario.

[0045] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0047] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0048] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0049] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0050] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0051] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0052] Personalized recommendations have become a crucial component of various sectors, including e-commerce, online video, and social media. Providing users with precise recommendations on products, works, and other information improves user experience and the platform's commercial success. For example, in e-commerce platforms, users can enter questions, which then provide product recommendations based on the user's input. Another example is in intelligent telemarketing systems, where users can ask intelligent voice assistants questions to request recommendations. The intelligent voice assistants then respond with recommendations based on the user's voice input.

[0053] However, with the development of globalization and the diversification of user needs, systems or models used for personalized recommendations face many challenges in multilingual processing, multimodal data integration, etc., especially the information of the recommended objects that is difficult to understand in combination with multimodal data. This can easily lead to insufficient understanding of the recommended objects, resulting in low accuracy of the generated recommendation information.

[0054] The present application provides an information recommendation method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy of recommended information. The following describes exemplary applications of the information recommendation device provided by the present application. The device provided by the present application can be implemented as various types of terminals, such as laptops, tablet computers, desktop computers, set-top boxes, smartphones, smart watches, smart TVs, and in-vehicle terminals, and can also be implemented as a server. The following describes exemplary applications of the device implemented as a server.

[0055] See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the information recommendation system 100 provided in an embodiment of the present application. To support an information recommendation application, the terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0056] Terminal 400 is configured with an application, mini-program or web page for recommending information to the user, and obtains the first voice data input by the user in the application, mini-program or web page through a data acquisition device deployed on terminal 400, and transmits the obtained first voice data to server 200 through network 300.

[0057] The server 200 is used to receive the first voice data and determine the first text data of the first voice data; perform text feature extraction on the first text data to obtain a first text feature; perform object recognition on the first text data to obtain a first object to be recommended, and perform feature extraction on the multimodal data of the first object to obtain a first multimodal feature; perform feature fusion on the first text feature and the first multimodal feature to obtain a first fusion feature; based on the first fusion feature, determine recommendation information for the first voice data, and then the server 200 can return the recommendation information to the terminal 400 through the network 300, and the terminal 400 will display the recommendation information to the user.

[0058] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0059] See also Figures 2A-2B , Figures 2A-2Bis a structural diagram of an electronic device 400 provided in an embodiment of the present application, Figures 2A-2B The electronic device 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figures 2A-2B Various buses are labeled as bus system 440 .

[0060] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0061] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0062] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0063] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0064] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0065] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0066] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0067] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0068] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0069] In some embodiments, the information recommendation device provided in the embodiments of the present application can be implemented in software. Figure 2A An information recommendation device 455A stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a data determination module 4551A, a feature extraction module 4552A, a feature fusion module 4553A, and an information determination module 4554A. These modules are logical and can be arbitrarily combined or further separated based on the functions they implement. The functions of each module will be described below.

[0070] In some embodiments, the model training device provided in the embodiments of the present application can be implemented in software. Figure 2B The model training device 455B stored in the memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a data determination module 4551B, a feature extraction module 4552B, a feature fusion module 4553B, and a model training module 4554B. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0071] In other embodiments, the information recommendation device or model training device provided in the embodiments of the present application can be implemented in hardware. As an example, the information recommendation device or model training device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the information recommendation method or model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0072] In some embodiments, the terminal or server can implement the information recommendation method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as an information recommendation APP or an instant messaging APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0073] The following describes the information recommendation method provided by the embodiment of the present application. As mentioned above, the electronic device that implements the information recommendation method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0074] See also Figure 3A , Figure 3A This is a first flow chart of the information recommendation method provided by the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0075] In step 101 , first text data of the first voice data is determined based on received first voice data.

[0076] In actual implementation, the terminal can obtain the first voice data input by the user through the audio acquisition device. The first voice data input by the user can be voice data expressing the user's needs. For example, the first voice data can be "Recommend me a product with a price less than 100", and the corresponding first object is the product with a price less than 100. For example, the first voice data can also be "Recommend me a shower gel with a high rating", and the corresponding first object is a shower gel with a high rating. The number of first objects can be multiple.

[0077] In some embodiments, determining the first text data of the first voice data based on the first voice data input by the user in step 101 can be achieved by the following technical solution: performing scene mapping on the first voice data to obtain second voice data that conforms to the first scene; performing text conversion on the second voice data to obtain first text data of the first voice data in the first scene.

[0078] In actual applications, if the voice data input by the user is not clearly expressed, or the application scenario cannot be identified from a single sentence of voice data, the voice data input by the user can be scene mapped to determine the application scenario of the voice data. For example, if the voice data input by the user is "Please recommend me a product priced below 100 yuan", it is impossible to determine what type of "product" the user wants to recommend based on this sentence alone. However, it can be combined with other voice data input by the user to perform scene mapping or inference, and the category of the "product" the user wants to recommend can be determined based on the mapped scenario. For example, if the mapped scenario is a pen purchase scenario, it can be determined that the category of the "product" the user wants to recommend is a pen, and the first object is a pen priced below 100 yuan.

[0079] If the voice data input by the user often contains interference such as noise and useless information, the voice of the user expressing their needs may not be recognized, and the voice data input by the user may not be suitable for the current scene, resulting in inaccurate object recognition based on the voice data. For example, in a telemarketing scenario, the first voice data input by the user is "This toothpaste is really not easy to use today, I will buy a new toothpaste tomorrow." Since this scenario is a daily conversation scenario of the user, if object recognition is performed based on the first voice data, the first object recognized may be all toothpastes, while the actual user wants to obtain a toothpaste different from the one currently used. Therefore, after obtaining the voice input by the user, the first voice can be mapped to the scene to obtain the second voice data that is convenient for data processing.

[0080] As an example, the first voice data is "I found today that the toothpaste I bought before is not easy to use. Can you recommend me a toothpaste that is different from the one I bought last time?" Then, by performing scene mapping on the first voice data, the second voice data can be obtained as "Recommend a toothpaste that is different from the toothpaste I bought before."

[0081] As an example, the method of converting the second voice data into text can be to use automatic speech recognition (ASR) to realize speech-to-text conversion. Specifically, the second voice data is processed using a deep learning algorithm, such as a convolutional neural network (CNN), to extract the voice features of the second voice data, which may include Mel Frequency Cepstral Coefficients (MFCCs), filter group features, etc.; the extracted voice features are mapped to the phoneme or word level, and commonly used models include recurrent neural network (RNN), etc.; a language model is used to understand and generate sentences to ensure the grammatical and semantic rationality of the output text; the decoder combines the results of the acoustic model and the language model to output the first text data.

[0082] It should be noted that the first text data may also include relevant data of the user. The terminal extracts relevant data of the user, such as age, occupation, location, historical purchase records, etc., based on the first voice data input by the user.

[0083] In step 102, text feature extraction is performed on the first text data to obtain a first text feature.

[0084] In actual implementation, the method of extracting text features from the first text data can use text extraction models such as Bag of Words (BoW), word vector generation model, and Bidirectional Encoder Representations from Transformers (BERT) to perform feature extraction.

[0085] In some embodiments, the text feature extraction of the first text data in step 102 to obtain the first text feature can be achieved by the following technical solution: dividing the first text data to obtain multiple sub-texts; for each sub-text, extracting the text features of the sub-text to obtain the text features of the sub-text; determining the distance between the text features of the sub-text and the text features of the adjacent sub-texts, and adjusting the weight of the text features of the sub-text according to the distance; based on the adjusted weight of the text features of the sub-texts, performing feature fusion on the text features of each sub-text to obtain the first text feature.

[0086] In some embodiments, the above-mentioned adjustment of the weight of the text feature of the sub-text according to the distance can be achieved through the following technical solution: if the distance is lower than the first distance threshold, the weight of the text feature of the sub-text is reduced; if the distance is higher than the second distance threshold, the weight of the text feature of the sub-text is increased.

[0087] In actual implementation, the first text data may include multiple texts with the same meaning. If too much attention is paid to texts with the same meaning during the feature extraction process, the accuracy and comprehensiveness of the text features obtained by feature extraction of the first text data may be reduced. Therefore, in the process of feature extraction of the first text data, if the distance between the text features of two sub-texts is lower than the first distance threshold (which can be set according to actual needs), the weight of the sub-text is reduced so that when performing feature extraction, the text features with the same meaning are not repeatedly paid attention to, wherein the distance between text features is the spatial distance between text features, and the distance between text features can be specifically determined by Euclidean distance and other methods; accordingly, if it is determined that the distance between the text features of two sub-texts is higher than the second distance threshold (which can be set according to actual needs), it indicates that the two sub-texts are different and are sub-texts including different information. Therefore, the weight of the sub-text can be increased so that when performing feature extraction, all the information included in the first text data can be extracted.

[0088] In the above manner, by adjusting the weights of the sub-texts, all text information in the first text data can be extracted during the feature extraction process, thereby improving the accuracy and comprehensiveness of the first text features.

[0089] In step 103 , object recognition is performed on the first text data to obtain a first object to be recommended, and feature extraction is performed on the multimodal data of the first object to obtain a first multimodal feature.

[0090] In actual implementation, object recognition can be performed on the first text data to preliminarily screen out the first object. For example, if the first text data is "Recommend a highly rated coat," object recognition can be performed on the first text to determine that the recommended object is a coat. The restriction condition for the recommended object is a high rating, so all highly rated coats in the database can be used as the first object. It should be noted that non-specific descriptions such as high ratings and good sales can be used to set corresponding thresholds based on practical experience. For example, objects with a score above 8 points are identified as highly rated objects, and objects ranked in the top 10 in sales are identified as good sales objects.

[0091] In actual implementation, the first object can be the object of the user's consultation. For example, if the user's consultation information is "I want a toothpaste", then all toothpastes here can be used as the first object. In order to describe the first object as accurately as possible, the description information of the first object often includes data from multiple modalities. For example, if the first object is toothpaste, the description information of the recommended object can include the text "Toothpaste made of XXX material, with a score of 9 points, and sales of 1,000 pieces this month", and can also include the appearance of the toothpaste, a photo of the test certificate obtained through product testing, etc.

[0092] In some embodiments, the feature extraction of the multimodal data of the first object in step 103 to obtain the first multimodal feature can be achieved by the following technical solution: extracting the second text data and the first visual data from the multimodal data of the first object; performing text feature extraction on the second text data to obtain the second text feature; performing visual feature extraction on the first visual data to obtain the first visual feature; and combining the second text feature and the first visual feature to obtain the first multimodal feature.

[0093] In actual implementation, multimodal data can be composed of second text data and first visual data. For example, the first object is toothpaste, and the second text data of the first object can be the introduction text of the toothpaste: "The main ingredients of this toothpaste are calcium carbonate, glycerin, water, etc.", and the first visual data of the first object can be a photo of the toothpaste.

[0094] In actual implementation, the method of extracting text features from the second text data can be to extract features using text extraction models such as the BoW model, the word vector generation model (Word2Vec), and the BERT model. It should be noted that the method of extracting text features from the second text data can be the same as the method of extracting text features from the first text data mentioned above. For example, the second text data is divided into segments to obtain multiple sub-texts; for each sub-text, text features are extracted from the sub-text to obtain text features of the sub-text; the distance between the text features of the sub-text and the text features of the adjacent sub-text is determined, and when the distance is lower than a first distance threshold, the weight of the text features of the sub-text is reduced, and when the distance is higher than a second distance threshold, the weight of the text features of the sub-text is increased; based on the adjusted weights, the text features of each sub-text are feature fused to obtain second text features.

[0095] In actual implementation, the second text data may include multiple texts with the same meaning. If too much attention is paid to texts with the same meaning during the feature extraction process, the accuracy and comprehensiveness of the text features obtained by feature extraction of the second text data may be reduced. Therefore, in the process of feature extraction of the second text data, if the distance between the text features of two sub-texts is lower than the third distance threshold (which can be set according to actual needs), the weight of the sub-text is reduced so that when feature extraction is performed, the text features with the same meaning are not repeatedly paid attention to; correspondingly, if it is determined that the distance between the text features of the two sub-texts is higher than the fourth distance threshold (which can be set according to actual needs), it is characterized that the two sub-texts are different and are sub-texts including different information. Therefore, by increasing the weight of the sub-text, all the information included in the second text data can be extracted when feature extraction is performed.

[0096] As an example, the first visual information may be image information such as an image or video generated by light stimulation received by the eyes.

[0097] In actual implementation, the second text feature and the first visual feature can be combined by feature splicing, that is, the second text feature and the first visual feature are directly spliced ​​together, that is, the two features are merged into a longer feature, and the spliced ​​feature is the first multimodal feature.

[0098] In some embodiments, the first visual feature can be obtained by extracting the visual feature of the first visual data in step 103. Figure 3B Steps 1031 to 1034 are implemented as shown.

[0099] In step 1031 , visual features are extracted from the first visual data to obtain second visual features.

[0100] As an example, the method of extracting visual features from the first visual data can be to extract the color histogram, color moment, etc. of the image, extract the texture features of the image, such as local binary patterns (LBP), etc., extract the shape description of the object in the image, such as edges, corners, contours, etc., and automatically learn the feature representation of the image by training a neural network model.

[0101] In step 1032 , when the first visual data is an image, the image is partitioned to obtain a plurality of visual areas, and visual features are extracted from each visual area to obtain a third visual feature of each visual area.

[0102] In actual implementation, the process of partitioning the visual data can be performed using a preset partitioning algorithm. Taking the first visual data including an image as an example, when partitioning the first visual data, a pixel value range can be set, and adjacent pixels in the first visual data that are in the same pixel value range can be regarded as the same visual area, based on which multiple visual areas are obtained; it is also possible to first identify the area of ​​rapid change in pixel intensity in the image, and use the area as the boundary area of ​​each visual area, and divide the first visual data into multiple visual areas based on the boundary area); it is also possible to first arbitrarily select multiple target pixel points in the image, and by determining the similarity between each target pixel point and the adjacent pixel point, use the pixel point with a similarity higher than a similarity threshold as the pixel point belonging to the same visual area as the target pixel point, based on which the first visual data is divided into multiple visual partitions; it is also possible to determine multiple local minima of the pixels of the visual data, connect the local minima, obtain the boundary line of the visual area, and divide the first visual data into multiple visual areas according to the obtained boundary line.

[0103] In actual implementation, feature extraction may be performed on each region included in the partitioned visual data to obtain a third visual feature.

[0104] In step 1033, based on the second visual feature, weight estimation is performed on each visual area to obtain the weight of each visual area.

[0105] In some embodiments, step 1033 performs weight estimation on each visual area based on the second visual feature, and obtaining the weight of each visual area can be achieved by the following technical solution: using the second visual feature as the query feature in the attention mechanism and using the third visual feature of each visual area as the key feature in the attention mechanism; for each visual area, performing a dot product operation on the query feature and the key feature to obtain the attention score of each visual area; normalizing the attention score of each visual area to obtain the weight of each visual area.

[0106] In actual implementation, since the importance of each visual area in the visual data is different, more precise feature extraction is required for important visual areas to obtain more detailed features of the visual area. For unimportant visual areas, only the basic features of the visual area can be extracted without wasting computing resources.

[0107] In actual implementation, in order to determine the importance of each visual area, the attention mechanism can be used to determine the weight of each area. Specifically, the weight of each area can be determined using the following formula:

[0108] A=V*T (1)

[0109] In formula (1), A is the attention score of the visual area, V is the query feature (the second visual feature), and T is the key feature (the third visual feature of the visual area). Through the above method, the attention score of each visual area can be calculated separately, and then the attention score of each visual area can be normalized by the normalized exponential function to obtain the weight of each visual area.

[0110] Through the above method, computing resources can be allocated according to the importance of each visual area, thereby improving the utilization efficiency of computing resources.

[0111] In step 1034 , based on the weights of the visual areas, feature fusion is performed on the third visual features and the second visual features to obtain the first visual feature.

[0112] As an example, the third visual feature and the second visual feature may be fused by feature splicing.

[0113] In step 104 , feature fusion is performed on the first text feature and the first multimodal feature to obtain a first fused feature.

[0114] In some embodiments, the feature fusion of the first text feature and the first multimodal feature in step 104 to obtain the first fused feature can be achieved by the following technical solution: feature fusion of the first text feature and the second text feature in the first multimodal feature to obtain the second fused feature; determining the weight of the first visual feature in the first multimodal feature based on the second fused feature, and determining the weight of the second fused feature based on the first visual feature; based on the weight of the first visual feature and the weight of the second fused feature, feature fusion of the second fused feature and the first visual feature to obtain the first fused feature.

[0115] As an example, a method of fusing the first text feature and the second text feature may be to perform feature splicing on the first text feature and the second text feature to obtain a second fused feature.

[0116] In actual implementation, the attention mechanism can be used to determine the weight of the first visual feature and the weight of the second fused feature. Specifically, the weight of the first visual feature can be determined by using the second fused feature as the query feature of the attention mechanism, using the second fused feature as the key feature of the attention mechanism, performing a dot product on the first visual feature and the second fused feature, and obtaining the attention score of the first visual feature. Similarly, the weight of the second fused feature can be determined by using the second fused feature as the query feature of the attention mechanism, using the first visual feature as the key feature of the attention mechanism, performing a dot product on the second fused feature and the first visual feature, and obtaining the attention score of the second fused feature. The attention score of the first visual feature and the attention score of the second fused feature are normalized by a normalized exponential function to obtain the weights of the first visual feature and the second fused feature.

[0117] As an example, the first visual feature and the second fused feature are weightedly fused using the weight of the first visual feature and the weight of the second fused feature to obtain a first fused feature.

[0118] In step 105 , recommendation information of the first speech data is determined based on the first fusion feature.

[0119] In some embodiments, determining the recommendation information of the first voice data based on the first fusion feature in step 105 can be achieved through the following technical solution: based on the first fusion feature, determining the second object that meets the first voice data from the first object; and outputting the recommendation information for the second object.

[0120] In actual implementation, the first fusion feature can be input into the first model, and the first model determines the second object from the first object, and then outputs recommendation information of the target game object.

[0121] Below, the model training method provided by the embodiment of the present application is described. As mentioned above, the electronic device that implements the model training method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeated below.

[0122] See also Figure 4A , Figure 4A This is a first flow chart of the model training method provided in the embodiment of the present application, which will be combined with Figure 4A The steps shown are explained.

[0123] In step 201 , third text data of a received voice sample is determined based on the received voice sample.

[0124] In some embodiments, the first model includes a first network, the first network includes a mapping layer and a conversion layer, and the third text data of the voice sample is determined based on the received voice sample in step 201 by Figure 4B Steps 2011 to 2014 are implemented as shown.

[0125] In step 2011, scene mapping is performed on the voice sample through the mapping layer to obtain third voice data that conforms to the second scene.

[0126] In practical applications, the first model can be used as follows Figure 5 The structure shown, Figure 5 This is a structural diagram of the first model provided in the embodiment of the present application. Figure 5 In the example, the first model includes a mapping layer, which can perform scene mapping on the voice data input into the mapping layer and output third voice data that conforms to the second scene.

[0127] In step 2012, the third voice data is converted into text by a conversion layer to obtain third text data of the voice sample.

[0128] In practical applications, the first model may also include Figure 5 The conversion layer shown can convert the input third voice data into text and output third text data.

[0129] In step 2013, a first loss of a mapping layer is constructed based on the third speech data and the first label of the speech sample, and a second loss of a conversion layer is constructed based on the third text data and the second label of the speech sample.

[0130] The first label is the text content of the speech sample corresponding to the speech sample in different scenarios. For example, if the speech sample is "I am going out this afternoon" spoken by the speaker in dialect A, the corresponding first label can be the text content of the Mandarin version of "I am going out this afternoon". For another example, if the speech sample is "I am not eating breakfast today" output by the speaker in dialect B, the corresponding first label can be the text content of the Cantonese version of "I am not eating breakfast today". The second label can be the standard text content of the speech sample. For example, if the speech sample is "I am going to work today" spoken by the speaker, the corresponding second label can be the text content of "I am going to work today".

[0131] In some embodiments, the speech samples include a first speech sample and a second speech sample that are annotated with the same first label and belong to different scenes. Step 2211 of performing scene mapping on the speech samples through a mapping layer to obtain third speech data that conforms to the second scene can be achieved by the following technical solution: mapping the first speech sample and the second speech sample to a space of the second scene through the mapping layer to obtain fourth speech data corresponding to the first speech sample in the second scene and fifth speech data corresponding to the second speech sample in the second scene;

[0132] In actual implementation, in order to train the scene mapping capability of the mapping layer, it is first necessary to determine two different training samples, where the first labels corresponding to the first voice sample and the second voice sample are the same, that is, the texts obtained after the scene conversion of the first voice sample and the second voice sample are the same.

[0133] As an example, the first voice sample is the Mandarin version of "Recommend a mobile phone", and the second voice sample is the Cantonese version of "Recommend a mobile phone". The first voice sample and the second voice sample are both "Recommend a mobile phone", but the speaking voices are different.

[0134] Constructing the first loss of the mapping layer based on the third voice data and the first label of the voice sample in step 2013 can be achieved by the following technical solution: determining the first similarity between the fourth voice data and the fifth voice data, and determining the second similarity between the fourth voice data and the first label; constructing the first loss of the mapping layer based on the first similarity and the second similarity.

[0135] As an example, the method of determining the first loss can refer to the following formula:

[0136]

[0137] In formula (2), L cross-domain is the first loss, N is the number of samples, is the fourth voice data, is the fifth voice data, Y i is the first label, is the first similarity between the fourth voice data and the fifth voice data, is the second similarity between the fourth voice data and the first label, λ is an adjustment parameter, and fθ() is an alignment function used to align the dimensions of the voice data and the first label.

[0138] It should be noted that the second similarity is determined by the fourth voice data and the first label. In the actual implementation process, the second similarity can also be determined by the fifth voice data and the first label. That is, the value of the second similarity can be in, is the fifth voice data, Y i is the first label, and fθ() is the alignment function used to align the dimensions of the speech data and the first label.

[0139] As an example, the second loss can be determined by referring to the following formula:

[0140] L CTC =-log∑ π∈Align(X,Y) P(π|X) (3)

[0141] In formula (3), L CTC is the second loss, Align(X,Y) is the combination of the third text data and the speech sample with the same text label, and P(π|X) represents the probability of outputting the third text data.

[0142] In step 2014, the mapping layer and the conversion layer in the first network are trained based on the first loss and the second loss.

[0143] As an example, the loss function for training can be seen in the following formula:

[0144] L=L CTC +αL cross-domain (4)

[0145] In formula (4), L is the loss function for joint training, L CTC is the second loss function, L cross-domain is the first loss function, and α is the adjustment parameter.

[0146] In step 202, text feature extraction is performed on the third text data to obtain third text features.

[0147] In actual implementation, the first model may also have the following Figure 5 The text feature extraction network shown in , the third text data is input into the text feature extraction network in the first model, and the text feature extraction network outputs the third text feature.

[0148] Specifically, the text feature extraction network can first divide the input third text data into multiple sub-texts, perform feature extraction on each sub-text respectively, and obtain the text features of each sub-text; the distance between the text features of each sub-text and the text features of the adjacent sub-text can be determined. When the distance is lower than the fifth distance threshold, it indicates that the two sub-texts are highly similar, and the information included in the two sub-texts is highly repetitive. In order not to waste computing resources to obtain repeated information, the weight of the text feature of the sub-text can be reduced; on the contrary, if the distance is higher than the sixth distance threshold, it indicates that the two sub-texts are less similar, and the information included in the two sub-texts is less repetitive. In order to ensure that all the information of the third text data is obtained as much as possible, the weight of the sub-text can be increased at this time, and then the text features of each sub-text are fused based on the weight of each sub-text to output the third text feature.

[0149] In step 203 , object recognition is performed on the third text data to obtain a third object to be recommended, and feature extraction is performed on the multimodal data of the third object to obtain a second multimodal feature.

[0150] In actual implementation, object recognition can be performed on the recommended objects included in the third text data to obtain the third object through preliminary screening. For example, if the third text data is "Recommend a highly rated coat", object recognition can be performed on the third text to obtain the recommended object as a coat. The restriction condition of the recommended object is high rating, so all coats with high ratings in the database can be used as the third object. It should be noted that non-specific descriptions such as high ratings and good sales can be used to set corresponding thresholds based on experience in actual applications. For example, objects with a score higher than 8 points can be determined as highly rated objects, and objects with sales rankings in the top 10 can be determined as objects with good sales.

[0151] In actual implementation, the second multimodal data may include second visual data and fourth text data. The fourth text data may be input into a Figure 5 In the text feature extraction network of the first model shown in FIG, the fourth text feature is output by the text feature extraction network. The first model may also have the following features: Figure 5 The visual feature extraction network shown in the figure inputs the second visual data into the visual feature extraction network of the first model, and the visual network features output the fourth visual feature. It should be noted that the method of extracting text features from the fourth text data can be the same as the method of extracting text features from the third text data mentioned above.

[0152] In actual implementation, the process of training the text feature extraction network can be to obtain the label value of the fourth text data, construct a text feature loss function based on the fourth text feature and the label value of the fourth text data, wherein the text feature loss function can adopt a cross entropy loss function, a mean square error loss function and other loss functions, and finally train the text feature extraction network based on the text feature loss function.

[0153] In actual implementation, the process of training the visual feature extraction network can be to obtain the label value of the second visual data, construct a visual feature loss function based on the fourth visual feature and the label value of the second visual data, wherein the visual feature loss function can adopt loss functions such as the cross entropy loss function and the mean square error loss function, and finally train the visual feature extraction network based on the visual feature loss function.

[0154] In step 204 , the third text feature and the second multimodal feature are fused to obtain a third fused feature.

[0155] As an example, the second multi-model feature may include a fourth text feature, and the method of performing feature fusion on the third text feature and the fourth text feature may be to perform feature splicing on the third text feature and the fourth text feature to obtain a fourth fused feature.

[0156] In actual implementation, the attention mechanism can be used to determine the weight of the second visual feature and the weight of the fourth fused feature. Specifically, the weight of the second visual feature can be determined by using the fourth fused feature as the query feature of the attention mechanism, using the fourth fused feature as the key feature of the attention mechanism, performing a dot product on the second visual feature and the fourth fused feature, and obtaining an attention score of the second visual feature. Similarly, the weight of the fourth fused feature can be determined by using the fourth fused feature as the query feature of the attention mechanism, using the second visual feature as the key feature of the attention mechanism, performing a dot product on the fourth fused feature and the second visual feature, and obtaining an attention score of the fourth fused feature. The attention scores of the second visual feature and the fourth fused feature are normalized by a normalized exponential function to obtain the weights of the second visual feature and the fourth fused feature.

[0157] As an example, the first visual feature and the second fused feature are weightedly fused using the weight of the first visual feature and the weight of the second fused feature to obtain a first fused feature.

[0158] In step 205 , predicted recommendation information of the speech sample is determined based on the third fusion feature, and the first model is trained based on the difference between the predicted recommendation information and the actual recommendation information of the speech sample.

[0159] In some embodiments, the first model further includes a second network, and the speech sample is annotated with a third label and a fourth label, the third label being used to represent first recommendation information related to the speech sample, and the fourth label being used to represent second recommendation information unrelated to the speech sample. Determining the predicted recommendation information of the speech sample based on the third fused feature in step 205 can be implemented by the following technical solution: predicting the speech sample based on the third fused feature to obtain the predicted recommendation information of the speech sample. Determining a first distance in the semantic space between the third fused feature and the semantic feature of the first recommendation information, and determining a second distance in the semantic space between the third fused feature and the semantic feature of the second recommendation information; determining a third loss based on the first distance and the second distance, and training the second network based on the third loss.

[0160] In actual implementation, the first model may also include Figure 5 The second network shown inputs the third fusion feature into the second network of the first model, and the second network outputs the predicted recommendation information.

[0161] During implementation, the second network is used to perform information prediction based on the third fused image to obtain predicted recommendation information. The second network can be any neural network, such as a convolutional neural network, a deep neural network, etc. The embodiment of the present application does not limit the structure of the third network.

[0162] The process of training the second network can be, first, determining a first distance between the third fusion feature and the semantic feature of the third label, the semantic feature of the third label can be the semantic feature of the label of the first recommendation information, wherein the label of the first recommendation information is the recommendation information label for the first voice sample, and then determining a second distance between the third fusion feature and the semantic feature of the fourth label, the semantic feature of the fourth label can be the semantic feature of the second recommendation label, wherein the label of the second recommendation information is irrelevant recommendation information for the first voice sample, and finally, taking maximizing the first distance and minimizing the second distance as the training goal, constructing a third loss function, and training the second network by the third loss function.

[0163] In some embodiments, the first model includes a first network and a second network; the training of the first model based on the difference between the predicted recommendation information and the actual recommendation information of the voice sample in step 205 can be achieved by the following technical solution: determining the fourth loss based on the difference between the predicted recommendation information and the actual recommendation information of the voice sample; training the first model based on the fourth loss, the first loss and the second loss corresponding to the first network, and the third loss corresponding to the second network.

[0164] In actual implementation, the loss function for constructing the fourth loss can be referred to the following formula:

[0165]

[0166] In formula (5), L diff-contrast is the fourth loss, h i is with h j Predicted recommendation information with similar relationships, h j is with h i Predicted recommendation information with similar relationships, h k For other predicted recommendation information, sim() is the similarity determination formula, exp() is the natural exponential function, τ is the adjustment parameter, N is the number of predicted recommendation information, y i,j For actual recommended information.

[0167] It can be understood that during the training process of the first model, the text feature extraction network, the visual feature extraction network, the first network, and the second network can be trained independently, or they can be trained jointly by integrating the loss conditions of each network.

[0168] Through the above embodiments, multiple networks included in the first model can be trained simultaneously by adopting a joint training method, so that different networks and tasks in the model can share information and integrate their respective learning results, so that each network can promote each other during the training process. Moreover, by flexibly adjusting the weights and balances of different tasks, the relevance of different tasks can be fully utilized, and the performance and robustness of the model can be improved, so that the first model can more accurately understand the content input by the user in actual application and push accurate recommendation information to the user.

[0169] The following describes an exemplary application of the present invention in a practical application scenario. Figure 6 , Figure 6 This is a flowchart of the implementation of the embodiment of the present application in a practical application scenario.

[0170] In step 601, the server prepares data.

[0171] The server collects data in various document formats, including PDF, Word, and text files, with a focus on technical standards, customer requirements documents, and product manuals. It extracts data from videos and images related to telemarketing scenarios, especially visual information in customer interactions, such as product demonstrations and advertising materials. It also collects voice data from telephone conversations and remote meetings, covering different customer interaction scenarios. Through manual annotation, it determines the parsing breakpoints, visual focus points, and important context of each data type to create a training dataset.

[0172] In step 602, the server performs data preprocessing.

[0173] The server pre-processes the text by adaptively determining the document parsing breakpoints based on the similarity of embeddings to ensure that important information is not lost. The server also pre-processes the speech to clear the noise in the speech data.

[0174] In step 603, the server performs feature extraction.

[0175] The server uses a pre-trained large language model as the base model to extract preliminary features of the fourth text data and generate a basic embedding vector that captures the essential semantic information of the fourth text data. During the text feature extraction process, the generated low-rank matrix is ​​used to fine-tune the fourth text features. Specifically, the low-rank matrix adjustment mechanism introduces subtle changes in the embedding space based on the context. These changes enhance contextual similarity and generate a more context-sensitive embedding representation.

[0176] When a subtext similar to the context is identified, the weight of the corresponding subtext is adjusted to make the current subtext closer to the similar context. Specifically, the adjustment is to reduce the weight of the dimension of standard text similarity so that the extracted fourth text feature better reflects the common characteristics between texts; when the identified subtext is not similar to the current subtext, the weight of the dimension reflecting the difference in the fourth text feature is increased. It can better distinguish the same words and phrases in different contexts; combining the embedding vectors adjusted for context similarity and difference to generate the final context-sensitive embedding representation. The vector accurately reflects the semantics of the text in a specific context and is used for subsequent reasoning.

[0177] The server can extract visual features by selecting a contrastive language-image pre-training model (CLIP) as the base model of a high-resolution visual language model (VLM) to extract useful information from images and text. The HiRED model can then be integrated into the selected VLM model.

[0178] In the initial layer of the visual feature extraction network, HiRED uses an attention mechanism to partition the input image (i.e., the first visual data described above) into multiple visual regions. Each visual region is evaluated based on attention weights to determine the importance of the visual content in each region. Based on the attention mechanism calculations in the initial layer, each visual region is assigned a corresponding visual feature budget. This budget determines the importance of each region. This step ensures that the visual feature extraction network focuses on the most critical image components while ignoring less important ones. In the final layer of the visual feature extraction network, HiRED uses an attention mechanism to refine the higher-budget visual regions. In this layer, the model further extracts features from each visual region to generate a more refined visual representation. The attention mechanism in the final layer helps HiRED select the most important visual features from each visual region and discard unimportant ones. The visual features generated by the initial and final layers are fused to form a refined visual representation. These representations not only preserve the global information of the image but also, through the multi-layer attention mechanism, capture the nuances of key details within the image. Finally, HiRED generates high-resolution visual features as output.

[0179] In step 604, the server performs feature fusion.

[0180] The following describes two approaches to feature fusion. First, for text generation tasks, we use the Mamba model as a linearly complex visual backbone, combined with a large language model (LLM) to generate high-quality recommendations and parsing content. Mamba is a lightweight visual backbone network known for its linear complexity and high performance. Compared to traditional models, Mamba maintains low computational overhead when processing large amounts of visual data, making it ideal for applications requiring fast inference.

[0181] The features extracted from the visual data are combined with the text data. The visual embeddings extracted by Mamba (i.e., the first visual features mentioned above) are fused with the text embeddings in the LLM (i.e., the second text features mentioned above). This fusion is performed interactively through an attention mechanism. After fusion, the LLM is guided by contextual information, which includes customer needs, product information, and historical conversation records.

[0182] Another fusion method is for the server to first align the visual and language embeddings so that the model can process visual and language data in a shared feature space. This stage ensures that the visual features and language features are aligned in the same semantic space. Specifically, through contrastive learning, the visual features and corresponding language features are aligned. During training, the model attempts to bring feature vectors from the same image-text pair closer together and push feature vectors from different image-text pairs further apart. In contrastive learning, positive samples are features from the same image-text pair, and negative samples are features from different image-text pairs. This construction of samples allows the model to learn the semantic correspondence between vision and language.

[0183] Finally, the aligned visual and linguistic features are fused using feature concatenation. The fused features comprehensively reflect the semantic information of the image and text. Based on the context of the input question and the visual features, the model generates an answer relevant to the image content. LLM is used to generate the final text output.

[0184] In step 605, the server constructs a loss function.

[0185] The information recommendation model (i.e., the first model mentioned above) may include a first network and a second network, wherein the first network may be trained by the first loss and the second loss, the second network may be trained by the third loss, and the information recommendation model may be trained by the fourth loss.

[0186] Among them, the first loss function can be as follows:

[0187]

[0188] In formula (6), L cross-domain is the first loss, N is the number of samples, is the fourth voice data, is the fifth voice data, Y i is the first label, is the first similarity between the fourth voice data and the fifth voice data, is the second similarity between the fourth voice data and the first label, λ is an adjustment parameter, and fθ() is an alignment function used to align the dimensions of the voice data and the first label.

[0189] The second loss can be as follows:

[0190] L CTC =-log∑ π∈Align(X,Y) P(π|X) (7)

[0191] In formula (7), l CTCis the second loss, Align(X,Y) is the combination of the third text data and the speech sample with the same text label, and P(π|X) represents the probability of outputting the third text data.

[0192] The fourth loss can be as follows:

[0193]

[0194] In formula (8), L diff-contrast is the fourth loss, h i is with h j Predicted recommendation information with similar relationships, h j is with h i Predicted recommendation information with similar relationships, h k For other predicted recommendation information, sim() is the similarity determination formula, exp() is the natural exponential function, τ is the adjustment parameter, N is the number of predicted recommendation information, y i,j The label of the first recommended information.

[0195] The server can dynamically adjust the loss function during training to balance accuracy and computational efficiency. For subtle selection problems, a contrastive loss function is introduced to distinguish between commonalities and subtle differences. Commonalities refer to shared semantics or features across multiple options. During the selection process, the model must identify these commonalities to ensure basic semantic consistency. Subtle differences refer to small but important differences between options that determine the success of a selection. Using contrastive learning, similar options can be treated as positive samples and dissimilar options as negative samples in subtle selection tasks.

[0196] The following describes the process of applying the trained information recommendation model to perform information recommendation.

[0197] First, the server obtains input data: including the customer's basic information (age, income, occupation, location, etc.), historical purchase records, and search and browsing records related to financial products.

[0198] As an example:

[0199] Age: 35

[0200] Income: ￥150,000 / year

[0201] Occupation: Company management

[0202] Location: XX

[0203] Historical purchase records: Have you ever purchased funds or insurance products?

[0204] Browsing history: Recently paid attention to high-yield financial products

[0205] Call voice data: Recordings of telephone conversations between customers and sales staff, including customer questions and sales staff responses, in WAV format with a sampling rate of 16kHz.

[0206] As an example:

[0207] Customer question: "Can I understand the risks of this financial product?"

[0208] The salesperson replied: "The risk of this product is low to medium and is suitable for conservative investors."

[0209] Text data: Financial product introduction documents, including product name, yield, investment period, risk level, historical performance, etc.

[0210] As an example:

[0211] Product Name: Stable Financial Product A

[0212] Yield: 4.5% annualized

[0213] Investment period: 1 year

[0214] Risk level: Medium-low

[0215] Historical performance: Average annualized return over the past three years: 4.2%

[0216] The server can then annotate the acquired data, including customer demand annotation: identifying and annotating the customer's investment preferences and risk tolerance; voice transcription annotation: manually transcribing the call voice data and annotating important question and answer segments; and text information annotation: annotating the key attributes of financial products for use in the recommendation process.

[0217] The text and speech are then preprocessed, specifically by word segmentation and encoding: the financial product introduction document is segmented and encoded into word vectors; context processing: customer questions and product information are contextually encoded for fusion with other features; speech slicing: long call recordings are sliced ​​into short segments, each 5 seconds long; noise filtering: noise reduction algorithms are used to clean up background noise and enhance speech quality.

[0218] The server can perform feature extraction on the preprocessed data, specifically by performing multimodal fusion of the transcribed speech text and the financial product introduction text, specifically by first aligning the text features (i.e., the first text features and the second text features mentioned above) with the first visual features, and generating the final multimodal features (i.e., the first fusion features mentioned above).

[0219] Finally, the first fusion feature is input into the information recommendation model trained in the above method. The information recommendation model generates accurate product recommendations based on the customer's real-time questions by expanding the context window.

[0220] As an example:

[0221] Input: Client asks: "I want higher returns but lower risk."

[0222] Output: Recommends stable financial product A and generates detailed explanation text: "This product has an annualized yield of 4.5% and a medium-low risk level, making it ideal for customers who want stable returns and controllable risks."

[0223] Speech Recognition: Use information recommendation models to transcribe customer-sales calls in real time.

[0224] As an example:

[0225] Input voice: The content of the customer's call asking for product details.

[0226] Output text: Transcribed as "This product has a low to medium risk level and is suitable for conservative investors."

[0227] Final recommendation: Based on the client's risk appetite and return expectations, the system recommends the most suitable financial product and provides relevant explanations.

[0228] Example output:

[0229] Recommended product: Stable financial product A

[0230] Recommended reason: "This product has stable returns and low risks, which meets your investment needs."

[0231] Transcript: "This product has a low to medium risk profile and is suitable for conservative investors."

[0232] It is understandable that in the embodiments of the present application, when user information and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0233] The following continues to describe the exemplary structure of the information recommendation device 455A provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2A As shown, the software modules stored in the information recommendation device 455A of the memory 450 may include:

[0234] a data determining module 4551A, configured to determine first text data of the first voice data based on the received first voice data;

[0235] A feature extraction module 4552A is configured to extract text features from the first text data to obtain first text features;

[0236] The feature extraction module 4552A is further configured to perform object recognition on the first text data to obtain a first object to be recommended, and perform feature extraction on the multimodal data of the first object to obtain a first multimodal feature;

[0237] A feature fusion module 4553A is configured to fuse the first text feature and the first multimodal feature to obtain a first fused feature;

[0238] The information determination module 4554A is used to determine the recommendation information of the first voice data based on the first fusion feature.

[0239] In some embodiments, the data determination module 4551A is further configured to perform scene mapping on the first voice data to obtain second voice data that conforms to the first scene; perform text conversion on the second voice data to obtain first text data of the first voice data in the first scene;

[0240] In some embodiments, the feature extraction module 4552A is also used to segment the first text data to obtain multiple sub-texts; for each sub-text, perform text feature extraction on the sub-text to obtain the text feature of the sub-text; determine the distance between the text feature of the sub-text and the text feature of the adjacent sub-text, and adjust the weight of the text feature of the sub-text based on the distance; based on the adjusted weight of the text feature of the sub-text, perform feature fusion on the text features of each sub-text to obtain the first text feature.

[0241] In some embodiments, the feature extraction module 4552A is further configured to reduce the weight of the text feature of the sub-text if the distance is lower than a first distance threshold, and to increase the weight of the text feature of the sub-text if the distance is higher than a second distance threshold.

[0242] In some embodiments, the feature extraction module 4552A is also used to extract second text data and first visual data from the multimodal data of the first object; perform text feature extraction on the second text data to obtain a second text feature; perform visual feature extraction on the first visual data to obtain a first visual feature; and combine the second text feature and the first visual feature to obtain a first multimodal feature.

[0243] In some embodiments, the feature extraction module 4552A is also used to extract visual features from the first visual data to obtain a second visual feature; when the first visual data is an image, the image is partitioned to obtain multiple visual areas, and visual features are extracted from each of the visual areas to obtain a third visual feature of each of the visual areas; based on the second visual feature, weight estimation is performed on each of the visual areas to obtain a weight of each of the visual areas; based on the weight of each of the visual areas, feature fusion is performed on each of the third visual features and the second visual feature to obtain the first visual feature.

[0244] In some embodiments, the feature extraction module 4552A is also used to use the second visual feature as a query feature in the attention mechanism and the third visual feature of each visual area as a key feature in the attention mechanism; for each visual area, perform a dot product operation on the query feature and the key feature to obtain an attention score for each visual area; and normalize the attention score of each visual area to obtain a weight for each visual area.

[0245] In some embodiments, the feature fusion module 4553A is also used to perform feature fusion on the first text feature and the second text feature in the first multimodal feature to obtain a second fused feature; determine the weight of the first visual feature in the first multimodal feature based on the second fused feature, and determine the weight of the second fused feature based on the first visual feature; based on the weight of the first visual feature and the weight of the second fused feature, perform feature fusion on the second fused feature and the first visual feature to obtain a first fused feature.

[0246] In some embodiments, the information determination module 4554A is further used to determine, from the first object, a second object that matches the first voice data based on the first fusion feature; and output recommendation information for the second object.

[0247] The following continues to describe the exemplary structure of the model training device 455B provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2B As shown, the software modules stored in the model training device 455B of the memory 450 may include:

[0248] a data determination module 4551B, configured to determine third text data of the voice sample based on the received voice sample;

[0249] A feature extraction module 4552B is configured to perform text feature extraction on the third text data to obtain a third text feature;

[0250] The feature extraction module 4552B is further configured to perform object recognition on the third text data to obtain a third object to be recommended, and perform feature extraction on the multimodal data of the third object to obtain a second multimodal feature;

[0251] A feature fusion module 4553B is configured to fuse the third text feature and the second multimodal feature to obtain a third fused feature;

[0252] The model training module 4554B is used to determine the predicted recommendation information of the voice sample based on the third fusion feature, and train the first model based on the difference between the predicted recommendation information and the actual recommendation information of the voice sample.

[0253] In some embodiments, the first model includes a first network, which includes a mapping layer and a conversion layer. The data determination module 5551B is further used to perform scene mapping on the voice sample through the mapping layer to obtain third voice data that conforms to the second scene; perform text conversion on the third voice data through the conversion layer to obtain third text data of the voice sample; before performing scene mapping on the voice sample through the mapping layer, the method also includes: constructing a first loss of the mapping layer based on the third voice data and the first label of the voice sample, and constructing a second loss of the conversion layer based on the third text data and the second label of the voice sample; based on the first loss and the second loss, training the mapping layer and the conversion layer in the first network.

[0254] In some embodiments, the speech samples include a first speech sample and a second speech sample that are labeled with the same first label and belong to different scenes; the data determination module 5551B is also used to map the first speech sample and the second speech sample to the space of the second scene through the mapping layer, to obtain fourth speech data corresponding to the first speech sample in the second scene and fifth speech data corresponding to the second voice sample in the second scene; determine a first similarity between the fourth speech data and the fifth speech data, and determine a second similarity between the fourth speech data and the first label; and construct a first loss of the mapping layer based on the first similarity and the second similarity.

[0255] In some embodiments, the model training module 5554B is also used to predict the speech sample based on the third fusion feature through the second network of the first model; determine a first distance between the third fusion feature and the semantic feature of the first recommendation information in the semantic space, and determine a second distance between the third fusion feature and the semantic feature of the second recommendation information in the semantic space; determine a third loss based on the first distance and the second distance, and train the second network based on the third loss.

[0256] In some embodiments, the model training module 5554B is further used to determine a fourth loss based on the difference between the predicted recommendation information and the actual recommendation information of the first voice sample; and train the first model based on the fourth loss, the first loss and second loss corresponding to the first network, and the third loss corresponding to the second network.

[0257] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the information recommendation method or model training method described in the present invention.

[0258] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the information recommendation method or model training method provided by the embodiment of the present application, for example, Figure 3A The information shown is recommended or Figure 3B The model training method shown.

[0259] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0260] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0261] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0262] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0263] In summary, the embodiments of the present application can achieve the following beneficial effects:

[0264] Through the above embodiment, when recommending information to a user, the first text data of the first voice data and the multimodal data of the first object can be determined based on the first voice data input by the user. By first determining the recommended object that matches the voice data based on the voice data input by the user, the recommended objects are first screened to avoid subsequent data processing of all recommended objects, thereby improving the efficiency of data processing. After that, text feature extraction can be performed on the first text data to obtain a first text feature, text feature extraction can be performed on the second text data in the multimodal data to obtain a second text feature, and visual feature extraction can be performed on the first visual data in the multimodal data to obtain a first visual feature. By performing feature fusion on the first text feature, the second text feature and the first visual feature, a first fusion feature is obtained. Based on the first fusion feature, the recommended information for the first voice data is determined. In this way, when processing data of different modalities of the recommended object, the data of multiple modalities can be accurately combined, and the recommended information for the user can be generated by combining the data of multiple modalities, thereby obtaining recommended information that better meets the user's needs and improving the accuracy of the push information pushed to the user.

[0265] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An information recommendation method, characterized in that: The method comprises: Determining first text data of the first voice data based on the received first voice data; Performing text feature extraction on the first text data to obtain a first text feature; Performing object recognition on the first text data to obtain a first object to be recommended, and performing feature extraction on multimodal data of the first object to obtain a first multimodal feature; Performing feature fusion on the first text feature and the first multimodal feature to obtain a first fused feature; Determine recommendation information for the first speech data based on the first fusion feature.

2. The method according to claim 1, characterized in that The determining, based on the received first voice data, first text data of the first voice data includes: Performing scene mapping on the first voice data to obtain second voice data that conforms to the first scene; The second voice data is converted into text to obtain first text data of the first voice data.

3. The method according to claim 1, characterized in that The extracting text features from the first text data to obtain first text features includes: Dividing the first text data to obtain a plurality of subtexts; For each of the sub-texts, extract text features of the sub-text to obtain text features of the sub-text; determining a distance between a text feature of the subtext and a text feature of an adjacent subtext, and adjusting a weight of the text feature of the subtext based on the distance; Based on the adjusted weights of the text features of the sub-texts, feature fusion is performed on the text features of the sub-texts to obtain a first text feature.

4. The method according to claim 3, characterized in that The step of adjusting the weight of the text feature of the subtext according to the distance includes: If the distance is lower than a first distance threshold, reducing the weight of the text feature of the subtext; If the distance is higher than a second distance threshold, the weight of the text feature of the subtext is increased.

5. The method according to claim 1, wherein The extracting features from the multimodal data of the first object to obtain first multimodal features includes: extracting second text data and first visual data from the multimodal data of the first object; Performing text feature extraction on the second text data to obtain second text features; performing visual feature extraction on the first visual data to obtain a first visual feature; The second text feature and the first visual feature are combined to obtain a first multimodal feature.

6. The method according to claim 5, characterized in that The extracting visual features from the first visual data to obtain first visual features includes: performing visual feature extraction on the first visual data to obtain a second visual feature; In the case where the first visual data is an image, partitioning the image to obtain a plurality of visual areas, and extracting visual features from each of the visual areas to obtain a third visual feature of each of the visual areas; Based on the second visual feature, weight estimation is performed on each of the visual areas to obtain a weight of each of the visual areas; Based on the weight of each visual area, feature fusion is performed on each third visual feature and the second visual feature to obtain the first visual feature.

7. The method according to claim 6, characterized in that The step of estimating a weight of each visual area based on the second visual feature to obtain a weight of each visual area includes: Using the second visual feature as a query feature in an attention mechanism, and using the third visual feature of each visual region as a key feature in the attention mechanism; For each of the visual areas, performing a dot product operation on the query feature and the key feature to obtain an attention score for each of the visual areas; The attention scores of the visual areas are normalized to obtain the weights of the visual areas.

8. The method according to claim 5, characterized in that The step of fusing the first text feature and the first multimodal feature to obtain a first fused feature includes: Performing feature fusion on the first text feature and the second text feature in the first multimodal feature to obtain a second fused feature; determining a weight of the first visual feature in the first multimodal feature based on the second fused feature, and determining a weight of the second fused feature based on the first visual feature; Based on the weight of the first visual feature and the weight of the second fused feature, the second fused feature and the first visual feature are fused to obtain a first fused feature.

9. The method according to claim 1, characterized in that The determining, based on the first fusion feature, recommendation information of the first voice data includes: determining, based on the first fusion feature, a second object from the first object that matches the first speech data; Output recommendation information for the second object.

10. A model training method, characterized in that: The method comprises: Determining third text data of the voice sample based on the received voice sample; Performing text feature extraction on the third text data to obtain third text features; performing object recognition on the third text data to obtain a third object to be recommended, and performing feature extraction on the multimodal data of the third object to obtain a second multimodal feature; Performing feature fusion on the third text feature and the second multimodal feature to obtain a third fused feature; Predicted recommendation information of the voice sample is determined based on the third fusion feature, and the first model is trained based on a difference between the predicted recommendation information and actual recommendation information of the voice sample.

11. The method according to claim 10, characterized in that The first model includes a first network, and the first network includes a mapping layer and a conversion layer; The determining, based on the received voice sample, third text data of the voice sample includes: Performing scene mapping on the voice sample through the mapping layer to obtain third voice data that conforms to the second scene; Performing text conversion on the third voice data through the conversion layer to obtain third text data of the voice sample; After obtaining the third voice data that meets the second scenario, the method further includes: Constructing a first loss of the mapping layer based on the third voice data and the first label of the voice sample, and constructing a second loss of the conversion layer based on the third text data and the second label of the voice sample; The mapping layer and the conversion layer in the first network are trained based on the first loss and the second loss.

12. The method according to claim 11, characterized in that The speech samples include a first speech sample and a second speech sample that are marked with the same first label and belong to different scenes; The performing scene mapping on the voice sample by the mapping layer to obtain third voice data that conforms to the second scene includes: Mapping the first voice sample and the second voice sample to a space of a second scene through the mapping layer to obtain fourth voice data corresponding to the first voice sample in the second scene and fifth voice data corresponding to the second voice sample in the second scene; The constructing the first loss of the mapping layer based on the third voice data and the first label of the voice sample includes: Determining a first similarity between the fourth voice data and the fifth voice data, and determining a second similarity between the fourth voice data and the first label; A first loss of the mapping layer is constructed based on the first similarity and the second similarity.

13. The method according to claim 10, characterized in that The first model further includes a second network, the voice sample is annotated with a third label and a fourth label, the third label is used to represent first recommendation information related to the voice sample, and the fourth label is used to represent second recommendation information unrelated to the voice sample; The determining the predicted recommendation information of the speech sample based on the third fusion feature includes: Predicting the voice sample based on the third fusion feature using the second network of the first model to obtain predicted recommendation information of the voice sample; The method further comprises: Determining a first distance between the third fused feature and the semantic feature of the first recommendation information in a semantic space, and determining a second distance between the third fused feature and the semantic feature of the second recommendation information in a semantic space; A third loss is determined based on the first distance and the second distance, and the second network is trained based on the third loss.

14. The method according to claim 9, characterized in that The first model includes a first network and a second network; and training the first model based on the difference between the predicted recommendation information and the actual recommendation information of the voice sample includes: determining a fourth loss based on a difference between the predicted recommendation information and actual recommendation information of the voice sample; The first model is trained based on the fourth loss, the first loss and the second loss corresponding to the first network, and the third loss corresponding to the second network.

15. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 14 when executing computer-executable instructions or computer programs stored in the memory.

16. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

17. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images