User portrait construction method, device, equipment and medium based on multimodal data

Abstract

The present invention is applicable to the field of data processing technology, and in particular relates to a method, apparatus, device, and medium for constructing a user profile based on multimodal data. The present invention outputs the classification category of each character in a conversation text based on an encoder and a classification model, treats a number of consecutive characters belonging to the same classification category in the conversation text as a conversation text segment, and obtains N conversation text segments. When the maximum value of the similarity between an acquired image text segment and the N conversation text segments is within a preset threshold range, the image text segment and its target category are determined. A user profile is constructed by combining the N conversation text segments and their corresponding classification categories. The image text segment is then measured using the user profile information extracted from the conversation text to obtain the image text segment and its target category to supplement the user profile information. The user profile is fully portrayed by fusing the conversation text and image data, thereby improving the accuracy of the user profile construction result.

Description

Technical Field

[0001] The present invention is applicable to the field of data analysis technology, and in particular relates to a method, device, equipment and medium for constructing a user profile based on multimodal data. Background Art

[0002] User profiling is a method of labeling user information in a big data environment. That is, big data provides a sufficient data foundation, and by abstracting the label information, the user's full picture is fully presented, ultimately forming a full picture of a virtual user.

[0003] Traditional user profiling algorithms often rely on extensive feature engineering or rules, using single, low-frequency, basic attribute data to construct user profiles, and lack the ability to integrate data from different modalities. However, user data in the financial sector is sparse, high-dimensional, heterogeneous, and dynamically changing. Furthermore, agents often instruct users to upload relevant images for identity or qualification verification, which also contain user-related information. Traditional user profiling algorithms struggle to fully leverage multimodal data to automatically generate features, making it difficult to fully and dynamically characterize users and reducing the accuracy of user profiling.

[0004] Therefore, in the field of data processing technology, how to improve the accuracy of constructing user portraits under multimodal data has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method, apparatus, device and medium for constructing a user portrait based on multimodal data to solve the problem that traditional user portrait algorithms are unable to fully utilize multimodal data, resulting in low accuracy in constructing user portraits.

[0006] In a first aspect, an embodiment of the present invention provides a method for constructing a user profile based on multimodal data, the method comprising:

[0007] Obtaining the conversation text corresponding to the target user and the representation vector of each character in the conversation text, inputting the representation vectors of all characters into a preset encoder for encoding, and outputting the encoding vector of the corresponding character containing context information;

[0008] Input the encoding vectors of all characters into the preset classification model in sequence for classification, and output the classification category of the corresponding characters;

[0009] For the dialogue text, a plurality of characters belonging to the same classification category and being continuous in the dialogue text are regarded as a dialogue text segment, thereby obtaining N dialogue text segments, where N is an integer greater than 0;

[0010] Obtaining a picture text segment within a picture provided by the target user, calculating similarities between the picture text segment and N conversation text segments, and if a maximum value among the similarities is within a preset threshold range, determining that the classification category to which the conversation text segment corresponding to the maximum value among the similarities belongs is the target category;

[0011] The N conversation text segments and their corresponding classification categories, as well as the picture text segments and their corresponding target categories are used as the portrait information of the target user to construct a user portrait of the target user.

[0012] In a second aspect, an embodiment of the present invention provides a user portrait construction device based on multimodal data, the user portrait construction device comprising:

[0013] A character encoding module is used to obtain the conversation text corresponding to the target user and the representation vector of each character in the conversation text, input the representation vectors of all characters into a preset encoder for encoding, and output the encoding vector of the corresponding character containing context information;

[0014] The character classification module is used to input the encoding vectors of all characters into the preset classification model in sequence for classification and output the classification category of the corresponding characters;

[0015] a text segmentation module for classifying, for the conversation text, a plurality of consecutive characters belonging to the same classification category in the conversation text as a conversation text segment, thereby obtaining N conversation text segments, where N is an integer greater than 0;

[0016] a category determination module, configured to obtain a picture text segment within the picture provided by the target user, calculate similarities between the picture text segment and N conversation text segments, and if a maximum value among the similarities is within a preset threshold range, determine that the classification category to which the conversation text segment corresponding to the maximum value among the similarities belongs is the target category;

[0017] The portrait construction module is used to use the N dialogue text segments and their corresponding classification categories, as well as the picture text segments and their corresponding target categories as the portrait information of the target user, to construct a user portrait of the target user.

[0018] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the user portrait construction method as described in the first aspect when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the user portrait construction method as described in the first aspect is implemented.

[0020] Compared with the prior art, the embodiments of the present invention have the following advantages: by obtaining the representation vector of the conversation text corresponding to the target user and each character in the conversation text, inputting the representation vectors of all characters into a preset encoder for encoding, outputting the encoding vector containing context information of the corresponding characters, inputting the encoding vectors of all characters into a preset classification model in sequence for classification, outputting the classification category of the corresponding characters, treating several characters belonging to the same classification category and continuous in the conversation text as a conversation text segment, obtaining N conversation text segments, obtaining the picture text segment in the picture provided by the target user, calculating the similarity between the picture text segment and the N conversation text segments, and if the maximum value of the similarity is If it is within the preset threshold range, the classification category of the dialogue text segment corresponding to the maximum value in the similarity is determined as the target category, and N dialogue text segments and their corresponding classification categories, as well as picture text segments and their corresponding target categories are used as the portrait information of the target user to construct a user portrait of the target user. By extracting the corresponding user portrait text information and user portrait category information in the dialogue text, and measuring the information of the picture provided by the target user based on the user portrait information in the dialogue text, the corresponding picture text segment and its target category are obtained to supplement the user portrait information, so as to integrate the dialogue text and the picture to fully and dynamically portray the user portrait, thereby improving the accuracy of user portrait construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 This is a schematic diagram of an application environment of a method for constructing a user profile based on multimodal data provided in the first embodiment of the present invention;

[0023] Figure 2 This is a flowchart of a method for constructing a user profile based on multimodal data provided by the first embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a device for constructing a user profile based on multimodal data provided in the second embodiment of the present invention;

[0025] Figure 4This is a structural diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0026] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0028] It will also be understood that the term "and / or" used in the present description and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0030] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" in the present specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] Embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0033] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] It should be understood that the order of execution of the steps in the following embodiments does not necessarily mean the order in which they are executed. The order in which each process is executed should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0036] The first embodiment of the present invention provides a method for constructing a user profile based on multimodal data, which can be applied in the following situations: Figure 1 In an application environment, a client communicates with a server. Clients include, but are not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0037] See also Figure 2 , is a flow chart of a method for constructing a user portrait based on multimodal data provided by the first embodiment of the present invention. The above method for constructing a user portrait can be applied to Figure 1 In the client, the user portrait construction method may include the following steps:

[0038] Step S201 , obtaining the conversation text corresponding to the target user and the representation vector of each character in the conversation text, inputting the representation vectors of all characters into a preset encoder for encoding, and outputting the encoding vector of the corresponding character containing context information.

[0039] Among them, user portraits are usually a labeled user model abstracted from user information such as user demographic characteristics, web browsing content, online social activities and consumption behavior. It has important guiding significance in many fields such as precision marketing, user research, personalized services and business decision-making. For example, in the field of financial sales, personalized product recommendations can be made to customers by building customer user portraits, thereby improving sales performance by improving customer experience.

[0040] When constructing a user profile for a target user, traditional user profiling algorithms often rely on extensive feature engineering or rules, using a single, low-frequency set of basic attribute data to construct the target user's profile. These algorithms lack the ability to integrate data from different modalities. However, in real-world applications, target users often correspond to multiple types of user data, and these traditional user profiling algorithms struggle to fully leverage multimodal data to extract user profile information for the target user. This makes it difficult to fully and dynamically characterize the user, reducing the accuracy of user profiling.

[0041] For example, in the field of financial sales, it is often possible to obtain the target user's conversation audio from the conversation between the salesperson and the target user. This audio is then converted into corresponding text using speech-to-text technology, and the target user's user profile information is extracted from the text. Furthermore, the target user can provide relevant images for identity or qualification verification, and the target user's user-related information can be extracted from the images provided. That is, the target user typically corresponds to two types of user data: text and images. This embodiment can extract the target user's user-related information from both the text and images, respectively, and then fuse the text and image data to create a complete and dynamic portrayal of the user, thereby improving the accuracy of user profile construction.

[0042] Specifically, when constructing the user profile of the target user, the user's corresponding conversation text and the representation vector of each character in the conversation text are first obtained. The representation vector represents the content information of the corresponding character. Then, a pre-trained encoder is obtained and the representation vectors of all characters are input into the preset encoder for encoding. The encoder outputs the encoding vector of each character containing contextual information. This strengthens the correlation between different characters in the conversation text, improving the accuracy of the character encoding vector and the amount of user information.

[0043] Optionally, obtaining the conversation text corresponding to the target user and the representation vector of each character in the conversation text includes:

[0044] Obtain the conversation text corresponding to the target user and the preset word embedding vector table, query the vector corresponding to each character in the conversation text according to the preset word embedding vector table, and obtain the representation vector of each character in the conversation text.

[0045] The preset word embedding vector table is pre-set based on actual conditions and is used to represent the one-to-one mapping relationship between characters and vectors. This embodiment queries the vector corresponding to each character in the conversation text based on the preset word embedding vector table, and determines the vector found for each character as the representation vector of each character in the conversation text, which is used to represent the content information of the corresponding character.

[0046] This embodiment pre-sets a word embedding vector table to represent the one-to-one mapping relationship between characters and vectors, so as to query the vector corresponding to each character in the conversation text according to the preset word embedding vector table, thereby improving the data acquisition efficiency of the target user.

[0047] The above steps of obtaining the dialogue text corresponding to the target user and the representation vector of each character in the dialogue text, inputting the representation vectors of all characters into a preset encoder for encoding, and outputting the encoding vector containing context information of the corresponding character, extracts the encoding vector containing context information of each character in the dialogue text, strengthens the correlation between different characters in the dialogue text, and improves the accuracy of the character encoding vector and the amount of user information.

[0048] In step S202 , the encoding vectors of all characters are sequentially input into a preset classification model for classification, and the classification categories of the corresponding characters are output.

[0049] Among them, the classification category of the character can be used to represent the user portrait category information corresponding to the character. Since different application scenarios focus on different user characteristics, the corresponding classification categories are also different. This embodiment first needs to set the specific classification category according to the actual application scenario.

[0050] For example, in the Internet scenario, people usually focus on the demographic attributes of users, so the classification categories may include age, gender, place of origin, education level, marital status, fertility status and occupation; in content-based media, websites and search engine scenarios, people usually focus on the interests of users in browsing content, so the classification categories may include sports, entertainment, food, finance, travel, real estate, automobiles, etc.; in e-commerce shopping website scenarios, people usually focus on the online shopping interests and consumption capacity of users, so the classification categories may include clothing, luggage, home furnishings, mother and baby products, cleaning and care products, food and beverages, etc.

[0051] Specifically, in this embodiment, a specific classification category is set according to the actual application scenario, and then the encoding vectors of all characters are input into the preset classification model in turn for classification, and the classification category of the corresponding character is output to represent the user portrait category information corresponding to the corresponding character, which serves as the basis for constructing the user portrait of the target user.

[0052] Optionally, the preset classification model includes a preset bidirectional long short-term memory model;

[0053] The encoding vectors of all characters are input into the preset classification model in sequence for classification, and the classification categories of the corresponding characters are output, including:

[0054] According to the order of all characters in the dialogue text from front to back and from back to front, the encoding vectors of all characters are input into the preset bidirectional long short-term memory model for classification, and the classification category of the corresponding characters is output.

[0055] Among them, the long short-term memory model is a type of recurrent neural network that can utilize contextual information in the mapping process between input and output sequences, and has a good processing effect on sequence data. In a recurrent neural network, the current output of a sequence is related to the previous output. Specifically, the recurrent neural network will remember the previous information, store it in the internal state of the recurrent neural network, and apply it to the calculation of the current output. In other words, the nodes between the hidden layers in the recurrent neural network are connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment.

[0056] In practical applications, recurrent neural networks have problems such as gradient vanishing, gradient exploding, and poor ability to learn long-distance dependent information. Therefore, the long short-term memory model was introduced to solve the above problems by adding a forget gate, input gate, and output gate in the hidden layer, and adding a hidden state.

[0057] In the operation process of the long short-term memory model, there are two calculation methods: forward calculation and backward calculation. In this embodiment, in order to improve the recognition accuracy of the classification category of characters, a bidirectional long short-term memory model is used, so that the character encoding vector sequence corresponds to two long short-term memory models with the same output layer forward and backward respectively, so as to provide complete past and future context information for each point in the output layer input sequence of the bidirectional long short-term memory model, thereby improving the accuracy of the classification results.

[0058] Specifically, all characters in the dialogue text are sorted in order from front to back and from back to front, respectively, to obtain a forward coding vector sequence and a backward coding vector sequence, which correspond to the forward calculation and backward calculation methods in the bidirectional long short-term memory model, respectively. Then, the forward coding vector sequence and the backward coding vector sequence are input into the preset bidirectional long short-term memory model for classification, and the classification category of the corresponding character is output.

[0059] This embodiment inputs the encoding vectors of all characters into a preset bidirectional long short-term memory model in sequence for classification according to the order of all characters from front to back and from back to front in the dialogue text, and outputs the classification category of the corresponding characters. This provides complete past and future context information for each point in the input sequence of the output layer of the bidirectional long short-term memory model, thereby improving the accuracy of the classification category of the corresponding characters.

[0060] The above steps of inputting the encoding vectors of all characters into the preset classification model for classification in sequence and outputting the classification categories of the corresponding characters classify the encoding vectors containing context information through the preset classification model, thereby improving the accuracy of the classification results, and representing the user portrait category information corresponding to the corresponding characters according to the classification categories, thereby improving the accuracy of extracting user portrait information in the conversation text.

[0061] Step S203 : for the dialogue text, a plurality of characters belonging to the same classification category and being continuous in the dialogue text are regarded as a dialogue text segment, thereby obtaining N dialogue text segments.

[0062] Different classification categories represent different user profile information, and this embodiment segments the conversation text according to the classification categories of all characters. Since interfering characters may exist in the conversation text, in order to reduce the impact of interfering characters, this embodiment does not extract relevant information of individual characters belonging to a certain classification category. Instead, it treats several consecutive characters belonging to the same classification category in the conversation text as a conversation text segment, obtaining N conversation text segments. The classification category to which the characters in each conversation text segment belong is determined as the classification category corresponding to the conversation text segment. Based on each conversation text segment and its corresponding classification category, the corresponding user profile text information and user profile category information can be represented, thereby improving the accuracy of the user profile information extracted from the conversation text, wherein N is an integer greater than 0.

[0063] For example, in this embodiment, the classification categories are set as A, B, and C. According to the order of the characters in the dialogue text, the classification categories corresponding to all the characters [a, b, c, d, e, f, g, h] are obtained as [A, A, B, C, C, A, A, A]. Then, the characters a and b that belong to the classification category A and are continuous in the dialogue text are regarded as a dialogue text segment, and the classification category corresponding to the dialogue text segment is determined to be A; the characters d and e that belong to the classification category C and are continuous in the dialogue text are regarded as a dialogue text segment, and the classification category corresponding to the dialogue text segment is determined to be C; the characters f, g, and h that belong to the classification category A and are continuous in the dialogue text are regarded as a dialogue text segment, and the classification category corresponding to the dialogue text segment is determined to be A, so as to obtain all the dialogue text segments.

[0064] The above-mentioned step of treating a conversation text as a conversation text segment by taking several consecutive characters belonging to the same classification category in the conversation text to obtain N conversation text segments, dividing the conversation text according to the classification categories of all characters to obtain N conversation text segments and determining the classification category corresponding to each conversation text segment, so as to characterize the corresponding user portrait text information and user portrait category information according to each conversation text segment and its corresponding classification category, thereby improving the accuracy of the user portrait information extracted from the conversation text.

[0065] Step S204: Obtain the image text segment in the image provided by the target user, calculate the similarity between the image text segment and N conversation text segments, and if the maximum value of the similarity is within a preset threshold range, determine that the classification category to which the conversation text segment corresponding to the maximum value of the similarity belongs is the target category.

[0066] Among them, the conversation text and picture in this embodiment represent the user portrait information of the same target user. When constructing the user portrait of the target user, the user portrait information can be extracted from the picture provided by the target user based on the conversation text, so as to integrate the conversation text and picture data to fully and dynamically portray the target user, so as to improve the accuracy of constructing the user portrait.

[0067] Specifically, a picture text segment in a picture provided by a target user is obtained, similarities between the picture text segment and N conversation text segments are calculated, and the maximum value among the similarities and the classification category to which the conversation text segment corresponding to the maximum value belongs are obtained.

[0068] In this embodiment, when the maximum value of the similarity is small, indicating that the image text segment contains less user portrait information, the maximum value of the similarity is compared with a preset first similarity threshold to determine whether the image text segment can serve as a supplementary basis for the user portrait information in the conversation text; when the maximum value of the similarity is large, indicating that the image text segment is likely to contain redundant information of the corresponding conversation text segment, the maximum value of the similarity is compared with a preset second similarity threshold to determine whether deduplication processing is required for the image text segment. The preset second similarity threshold is greater than the preset first similarity threshold.

[0069] When the maximum value of the similarity is less than the preset first similarity threshold, the image text segment cannot be used as a supplementary basis for the user portrait information in the conversation text. When the maximum value of the similarity is greater than the preset second similarity threshold, the image text segment needs to be deduplicated, and there is no need to supplement the user portrait information represented by the conversation text based on the image text segment.

[0070] Therefore, this embodiment determines the preset threshold range based on the preset first similarity threshold and the preset second similarity threshold. When the maximum value in the similarity is within the preset threshold range, the image text segment is used to supplement the user portrait information represented by the conversation text, and the classification category to which the conversation text segment corresponding to the maximum value in the similarity belongs is determined as the target category, which serves as the basis for supplementing the user portrait information in the conversation text.

[0071] Optionally, obtaining the image text segment in the image provided by the target user includes:

[0072] Obtain the image provided by the target user, perform character recognition on the image based on the optical character recognition model, and obtain the image text of the image;

[0073] Convert image text into image text vectors based on word vector technology;

[0074] Obtain the memory vector of each character in the preset classification model, and form a dialogue memory vector based on the memory vectors of all characters;

[0075] The vector similarity between the conversation memory vector and the image text vector is calculated. If the vector similarity is greater than a preset vector similarity threshold, the image text is determined to be an image text segment within the image provided by the target user.

[0076] Among them, the character recognition is performed on the picture provided by the target user according to the optical character recognition model to obtain the corresponding picture text, and the picture text is converted into a picture text vector according to the word vector technology to facilitate similarity calculation.

[0077] After the encoding vectors of all characters are sequentially input into a preset bidirectional long-short-term memory model according to the order of all characters in the dialogue text from front to back and from back to front, the memory vector corresponding to each character is obtained according to the output layer. The dialogue memory vector is composed of the memory vectors of all characters, and the vector similarity between the dialogue memory vector and the picture text vector is calculated. When the vector similarity is small, the user portrait information contained in the picture provided by the target user is small and it is difficult to serve as a supplementary basis for the user portrait information in the dialogue text. Therefore, when the vector similarity is greater than the preset vector similarity threshold, this embodiment determines that the picture text is a picture text segment in the picture provided by the target user, which is used for subsequent target category judgment to reduce the impact of low-quality pictures on the user portrait construction results and improve the accuracy and efficiency of user portrait construction.

[0078] This embodiment judges the pictures provided by the target user and uses pictures containing more user portrait information as a supplementary basis for the user portrait information in the conversation text to prevent low-quality pictures from affecting the user portrait construction results, thereby improving the accuracy and efficiency of user portrait construction.

[0079] Optionally, calculating the similarity between the image text segment and the N conversation text segments includes:

[0080] Convert the image text segment into an image text segment vector based on the word vector technology, and convert each conversation text segment into a corresponding conversation text segment vector based on the word vector technology;

[0081] The similarity between the image text segment vector and each conversation text segment vector is calculated as the similarity between the image text segment and the corresponding conversation text segment.

[0082] To facilitate similarity calculation between the image text segments and each conversation text segment, this embodiment first converts the image text segment into an image text segment vector and each conversation text segment into a corresponding conversation text segment vector using word embedding technology. The similarity between the image text segment vector and each conversation text segment vector is then calculated as the similarity between the image text segment and the corresponding conversation text segment, improving computational efficiency and accuracy.

[0083] This embodiment improves the calculation efficiency and accuracy of similarity by converting image text segments into image text segment vectors, converting each dialogue text segment into a corresponding dialogue text segment vector, and converting the similarity calculation between text segments into the similarity calculation between vectors.

[0084] The above-mentioned step of obtaining the picture text segment in the picture provided by the target user, calculating the similarity between the picture text segment and N conversation text segments, and if the maximum value of the similarity is within the preset threshold range, determining the classification category to which the conversation text segment corresponding to the maximum value of the similarity belongs as the target category, measures the user portrait information contained in the picture text segment by the similarity between the picture text segment and the conversation text segment, and then determines whether the picture text segment can supplement the user portrait information represented by the conversation text, and obtains the picture text segment and its corresponding target category as the data basis for user portrait construction, thereby improving the accuracy of the user portrait construction result.

[0085] Step S205 , taking the N conversation text segments and their corresponding classification categories, as well as the picture text segments and their corresponding target categories as the portrait information of the target user, to construct a user portrait of the target user.

[0086] Among them, N conversation text segments and their corresponding classification categories can represent the user portrait information of the target user contained in the conversation text, and the picture text segments and their corresponding target categories can represent the supplementary information of the picture provided by the target user to the user portrait information. In this embodiment, N conversation text segments and their corresponding classification categories and picture text segments and their corresponding target categories are spliced ​​to construct a user portrait of the target user.

[0087] Optionally, N conversation text segments and their corresponding classification categories, as well as image text segments and their corresponding target categories, are used as profile information of the target user. Constructing a user profile of the target user includes:

[0088] Get the preset first and second delimiters;

[0089] Each conversation text segment and its corresponding classification category are spliced ​​according to the first delimiter, and the image text segment and its corresponding target category are spliced ​​according to the first delimiter to obtain corresponding N+1 user sub-portraits;

[0090] The N+1 user sub-portraits are concatenated according to the second delimiter to construct a user portrait of the target user.

[0091] Among them, N conversation text segments and their corresponding classification categories, as well as picture text segments and their corresponding target categories serve as the portrait information of the target user. When constructing a user portrait based on the above portrait information, the first delimiter and the second delimiter are first preset, and each conversation text segment and its corresponding classification category are spliced ​​according to the first delimiter, and the picture text segment and its corresponding target category are spliced ​​according to the first delimiter to obtain the corresponding N+1 user sub-portraits, and then the N+1 user sub-portraits are spliced ​​according to the second delimiter to construct a user portrait of the target user.

[0092] For example, the conversation text segment "male" and its corresponding classification category "gender", the conversation text segment "undergraduate" and its corresponding classification category "education level", and the picture text segment "unmarried" and its corresponding target category "marital status" are obtained as the portrait information of the target user. The preset first delimiter is set to ":", and the preset second delimiter is set to ";". Then, the user portrait of the target user is constructed as [gender: male; education level: undergraduate; marital status: unmarried].

[0093] The above-mentioned step of constructing a user portrait of the target user by using N conversation text segments and their corresponding classification categories, as well as image text segments and their corresponding target categories as the portrait information of the target user, comprehensively and dynamically portrays the user by fusing the user-related information of the target user extracted from the conversation text and the image, thereby improving the accuracy of constructing the user portrait.

[0094] An embodiment of the present invention obtains a conversation text corresponding to a target user and a representation vector of each character in the conversation text, inputs the representation vectors of all characters into a preset encoder for encoding, outputs encoding vectors containing contextual information for the corresponding characters, sequentially inputs the encoding vectors of all characters into a preset classification model for classification, outputs classification categories for the corresponding characters, and treats a number of consecutive characters belonging to the same classification category in the conversation text as a conversation text segment to obtain N conversation text segments. A picture text segment in a picture provided by the target user is obtained, and similarities between the picture text segment and the N conversation text segments are calculated. If the maximum value of the similarities falls within a preset threshold, the classification category of the conversation text segment corresponding to the maximum value of the similarity is determined to be the target category. The N conversation text segments and their corresponding classification categories, as well as the picture text segment and its corresponding target category, are used as profile information of the target user to construct a user profile of the target user. The user profile text information and user profile category information corresponding to the conversation text are extracted, and information of the picture provided by the target user is weighed based on the user profile information in the conversation text to obtain the corresponding picture text segment and its target category to supplement the user profile information. The user profile is fully and dynamically portrayed by integrating the conversation text and the picture, thereby improving the accuracy of user profile construction.

[0095] Corresponding to the user portrait construction method of the above embodiment, Figure 3 A structural block diagram of a user portrait construction device based on multimodal data provided in the second embodiment of the present invention is given. For the sake of convenience, only the parts related to the embodiment of the present invention are shown.

[0096] See also Figure 3 , the user portrait building device includes:

[0097] The character encoding module 31 is used to obtain the target user's corresponding conversation text and the representation vector of each character in the conversation text, input the representation vectors of all characters into a preset encoder for encoding, and output the encoding vector of the corresponding character containing context information;

[0098] The character classification module 32 is used to input the encoding vectors of all characters into a preset classification model for classification, and output the classification category of the corresponding characters;

[0099] A text segmentation module 33 is configured to classify a plurality of consecutive characters belonging to the same classification category in the conversation text as a conversation text segment, thereby obtaining N conversation text segments, where N is an integer greater than 0;

[0100] Category determination module 34 is configured to obtain a picture text segment within a picture provided by a target user, calculate similarities between the picture text segment and N conversation text segments, and if the maximum similarity falls within a preset threshold, determine that the classification category of the conversation text segment corresponding to the maximum similarity belongs to the target category;

[0101] The portrait construction module 35 is used to construct a user portrait of the target user by using N conversation text segments and their corresponding classification categories, as well as image text segments and their corresponding target categories as the portrait information of the target user.

[0102] Optionally, the character encoding module 31 includes:

[0103] The representation vector query submodule is used to obtain the conversation text corresponding to the target user and the preset word embedding vector table, query the vector corresponding to each character in the conversation text according to the preset word embedding vector table, and obtain the representation vector of each character in the conversation text.

[0104] Optionally, the preset classification model includes a preset bidirectional long short-term memory model, and the character classification module 32 includes:

[0105] The character classification submodule is used to input the encoding vectors of all characters into the preset bidirectional long short-term memory model for classification according to the order of all characters from front to back and from back to front in the dialogue text, and output the classification category of the corresponding characters.

[0106] Optionally, the category determination module 34 includes:

[0107] The character recognition submodule is used to obtain the picture provided by the target user, perform character recognition on the picture according to the optical character recognition model, and obtain the picture text;

[0108] A first vector conversion submodule, configured to convert the image text into an image text vector based on a word vector technology;

[0109] The memory vector acquisition submodule is used to obtain the memory vector of each character in the preset classification model and form a dialogue memory vector based on the memory vectors of all characters;

[0110] The image text segment determination submodule is used to calculate the vector similarity between the conversation memory vector and the image text vector. If the vector similarity is greater than a preset vector similarity threshold, the image text is determined to be an image text segment within the image provided by the target user.

[0111] Optionally, the category determination module 34 includes:

[0112] A second vector conversion submodule is used to convert the image text segment into an image text segment vector based on the word vector technology, and to convert each dialogue text segment into a corresponding dialogue text segment vector based on the word vector technology;

[0113] The similarity calculation submodule is used to calculate the similarity between the image text segment vector and each dialogue text segment vector as the similarity between the image text segment and the corresponding dialogue text segment.

[0114] Optionally, the portrait building module 35 includes:

[0115] A delimiter acquisition submodule is used to obtain a preset first delimiter and a second delimiter;

[0116] A sub-portrait construction sub-module is used to splice each conversation text segment and its corresponding classification category according to the first delimiter, and to splice the image text segment and its corresponding target category according to the first delimiter to obtain corresponding N+1 user sub-portraits;

[0117] The portrait construction submodule is used to splice N+1 user sub-portraits according to the second delimiter to construct a user portrait of the target user.

[0118] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0119] Figure 4 This is a schematic diagram of the structure of a computer device provided in the third embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps of any of the above-mentioned user portrait construction method embodiments are implemented.

[0120] The computer device may include, but is not limited to, a processor and a memory. It will be understood by those skilled in the art that Figure 4 The above is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0121] The processor may be a CPU, or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor, or any conventional processor.

[0122] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the computer device, and in other embodiments, it can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Furthermore, the memory can also include both the internal storage unit of the computer device and the external storage device. The memory is used to store the operating system, application programs, boot loaders (BootLoader), data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or is about to be output.

[0123] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned method embodiment. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0124] The present invention may implement all or part of the processes in the above-mentioned method embodiments, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0125] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0126] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0127] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for constructing a user portrait based on multimodal data, characterized in that: The user portrait construction method includes: Obtaining the conversation text corresponding to the target user and the representation vector of each character in the conversation text, inputting the representation vectors of all characters into a preset encoder for encoding, and outputting the encoding vector of the corresponding character containing context information; Input the encoding vectors of all characters into the preset classification model in sequence for classification, and output the classification category of the corresponding characters; For the dialogue text, a plurality of characters belonging to the same classification category and being continuous in the dialogue text are regarded as a dialogue text segment, thereby obtaining N dialogue text segments, where N is an integer greater than 0; Obtaining a picture text segment within a picture provided by the target user, calculating similarities between the picture text segment and N conversation text segments, and if a maximum value among the similarities is within a preset threshold range, determining that the classification category to which the conversation text segment corresponding to the maximum value among the similarities belongs is the target category; Using the N conversation text segments and their corresponding classification categories, and the image text segments and their corresponding target categories as the profile information of the target user, to construct a user profile of the target user; The step of using the N conversation text segments and their corresponding classification categories, and the image text segments and their corresponding target categories as the portrait information of the target user to construct a user portrait of the target user includes: Get the preset first and second delimiters; splicing each conversation text segment and its corresponding classification category according to the first delimiter, and splicing the image text segment and its corresponding target category according to the first delimiter to obtain corresponding N+1 user sub-portraits; The N+1 user sub-portraits are spliced ​​according to the second delimiter to construct a user portrait of the target user.

2. The user portrait construction method according to claim 1, characterized in that: The step of obtaining the conversation text corresponding to the target user and the representation vector of each character in the conversation text includes: Obtain the conversation text corresponding to the target user and a preset word embedding vector table, query the vector corresponding to each character in the conversation text according to the preset word embedding vector table, and obtain a representation vector for each character in the conversation text.

3. The user portrait construction method according to claim 1, characterized in that: The preset classification model includes a preset bidirectional long short-term memory model; The encoding vectors of all characters are sequentially input into a preset classification model for classification, and the classification categories of the corresponding characters are outputted, including: According to the order of all characters in the dialogue text from front to back and from back to front, the encoding vectors of all characters are sequentially input into a preset bidirectional long short-term memory model for classification, and the classification categories of the corresponding characters are output.

4. The user portrait construction method according to claim 1, characterized in that: The step of obtaining the image text segment in the image provided by the target user includes: Obtaining a picture provided by a target user, performing character recognition on the picture according to an optical character recognition model, and obtaining picture text of the picture; Converting the image text into an image text vector according to word vector technology; Obtaining a memory vector for each character in the preset classification model, and forming a dialogue memory vector based on the memory vectors of all characters; A vector similarity between the conversation memory vector and the picture text vector is calculated. If the vector similarity is greater than a preset vector similarity threshold, it is determined that the picture text is a picture text segment in the picture provided by the target user.

5. The user portrait construction method according to claim 1, characterized in that: Calculating the similarity between the image text segment and the N conversation text segments includes: Convert the image text segment into an image text segment vector according to the word vector technology, and convert each dialogue text segment into a corresponding dialogue text segment vector according to the word vector technology; The similarity between the image text segment vector and each conversation text segment vector is calculated as the similarity between the image text segment and the corresponding conversation text segment.

6. A user portrait construction device based on multimodal data, characterized in that: The user portrait building device includes: A character encoding module is used to obtain the conversation text corresponding to the target user and the representation vector of each character in the conversation text, input the representation vectors of all characters into a preset encoder for encoding, and output the encoding vector of the corresponding character containing context information; The character classification module is used to input the encoding vectors of all characters into the preset classification model in sequence for classification and output the classification category of the corresponding characters; a text segmentation module for classifying, for the conversation text, a plurality of consecutive characters belonging to the same classification category in the conversation text as a conversation text segment, thereby obtaining N conversation text segments, where N is an integer greater than 0; a category determination module, configured to obtain a picture text segment within the picture provided by the target user, calculate similarities between the picture text segment and N conversation text segments, and if a maximum value among the similarities is within a preset threshold range, determine that the classification category to which the conversation text segment corresponding to the maximum value among the similarities belongs is the target category; A portrait construction module is configured to use the N conversation text segments and their corresponding classification categories, and the image text segments and their corresponding target categories as portrait information of the target user to construct a user portrait of the target user; The portrait construction module includes: A delimiter acquisition submodule is used to obtain a preset first delimiter and a second delimiter; a sub-portrait construction sub-module, configured to concatenate each conversation text segment and its corresponding classification category according to the first delimiter, and to concatenate the image text segment and its corresponding target category according to the first delimiter, to obtain corresponding N+1 user sub-portraits; The portrait construction submodule is used to splice the N+1 user sub-portraits according to the second delimiter to construct a user portrait of the target user.

7. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the user portrait construction method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the user portrait construction method according to any one of claims 1 to 5 is implemented.

Images