User purchase intention prediction method and device based on multi-modal information

Abstract

The invention discloses a multi-modal information-based user purchase intention prediction method and apparatus. The method comprises the steps of obtaining a user purchase behavior information set; the user purchase behavior information set comprises a collected behavior information set related to a purchase decision in the commodity purchase process of the user and a purchase decision information set of the corresponding user; each behavior information set has corresponding purchase decision information; performing association processing on the behavior information set and the purchase decision information set to obtain a user purchase intention prediction model; and processing the acquired user behavior information by using the user purchase intention prediction model to obtain a user purchase intention prediction value. According to the method, the key factors for determining the purchase decision of the user are obtained through extraction, the user purchase intention prediction model is established, and a quantitative basis is provided for improving the offline commodity sales volume.

Description

Technical Field

[0001] The present invention relates to the fields of big data and artificial intelligence, and in particular to a method for predicting user purchase intention based on multimodal information. Background Art

[0002] At present, with the continuous improvement of the level of commercial informatization, a large amount of user purchase behavior information has been collected in commercial scenarios, including visual information, auditory information, motion trajectory and user product purchase information. The current mining and analysis of user purchase behavior information in commercial scenarios is mainly carried out on single-modal data, such as user motion trajectory information. Single-modal information is difficult to fully reflect the user's behavior rules and decision-making process, resulting in the purchase model obtained having limited scope of use and low accuracy. How to comprehensively extract and analyze the user's multimodal information, effectively mine the user's purchase behavior information in commercial scenarios, and extract the key factors that determine the user's purchase decision, with the user's permission, is a very valuable problem facing the current commercial scenarios. Summary of the invention

[0003] The present invention mainly solves the problem of how to comprehensively extract and analyze users' multimodal information, effectively mine users' purchasing behavior information in commercial scenarios, and extract key factors that determine users' purchasing decisions. The present invention discloses a method for predicting users' purchasing intention based on multimodal information.

[0004] In a first aspect of an embodiment of the present application, a method for predicting user purchase intention based on multimodal information is disclosed, comprising:

[0005] S1, obtaining a user purchase behavior information set; the user purchase behavior information set includes a behavior information set related to purchase decisions during the user's purchase of goods, and a corresponding user purchase decision information set; each behavior information set has corresponding purchase decision information;

[0006] S2, performing association processing on the behavior information set and the purchase decision information set to obtain a user purchase intention prediction model;

[0007] S3, using the user purchase intention prediction model to process the collected user behavior information set to obtain the user purchase intention prediction value

[0008] The acquisition of the user purchase behavior information set includes:

[0009] Using an optical acquisition device, acquiring a user's facial image information sequence and a user's body image information sequence;

[0010] Performing body extraction and trajectory tracking processing on the user body image information to obtain a user trajectory information sequence;

[0011] Using acoustic collection equipment to collect user voice information sequences;

[0012] Performing emotion recognition processing on the user facial image information sequence and the user voice information sequence to obtain a user emotion information sequence;

[0013] Performing gaze point recognition processing on the user's facial image information sequence to obtain a gaze point position information sequence;

[0014] Performing collection time alignment processing on the user trajectory information sequence, the user voice information sequence, the user emotion information sequence, and the gaze point position information sequence to obtain a behavior information set;

[0015] Acquire purchase decision information, wherein the purchase decision information includes actual purchase information and target product information; the target product information includes target product location information;

[0016] The associating process of the behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes:

[0017] Performing correlation extraction processing on the behavior information set and the purchase decision information set to obtain a related behavior information set;

[0018] Performing regression processing on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model;

[0019] The correlating extraction process of the behavior information set and the purchase decision information set to obtain the correlating behavior information set includes:

[0020] Extract all the associated information associated with the target product in the user voice information sequence, and construct a language associated information sequence using all the extracted associated information;

[0021] Performing position difference calculation processing on the gaze point position information sequence and the target commodity position information to obtain a gaze point difference value sequence;

[0022] Performing correlation analysis on the user trajectory information sequence and the target product to obtain a trajectory correlation sequence;

[0023] Using the language association information sequence, the gaze point difference value sequence, the trajectory related sequence and the user emotion information sequence, a related behavior information set is constructed;

[0024] The step of performing regression processing on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes:

[0025] Performing statistical analysis on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model for describing the causal relationship between the relevant behavior information set and the purchase decision information set;

[0026] The statistical analysis is a regression prediction analysis method, a time series prediction method, a support vector machine prediction method or a fuzzy prediction method, etc.;

[0027] The step of performing regression processing on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes:

[0028] A training data set is constructed by using the relevant behavior information set and the purchase decision information set; the training data in the training data set is the relevant behavior information set; the label data in the training data set is the purchase decision information corresponding to the relevant behavior information set;

[0029] Construct an initial user purchase intention prediction model;

[0030] Using the training data set to train the initial user purchase intention prediction model to obtain the user purchase intention prediction model;

[0031] The initial user purchase intention prediction model includes: an input module, a feature extraction module, a feature fusion module and a prediction module;

[0032] The input module is used to receive and obtain a relevant behavior information set;

[0033] The feature extraction module is connected to the input module and the feature fusion module respectively, and is used to extract features from the relevant behavior information set to obtain a feature signal set;

[0034] The feature fusion module is connected to the prediction module and is used to perform fusion processing on the feature signal set to obtain a fused feature signal set; perform time series correlation processing on the fused feature signal set to obtain a time series cross feature signal set;

[0035] The prediction module is used to perform prediction processing on the time series cross feature signal set to obtain a user purchase intention prediction value.

[0036] The feature extraction module includes N feature extractors corresponding to the signal modes; each feature extractor is used to perform feature extraction processing on the information sequence of the corresponding mode to obtain a corresponding feature signal;

[0037] The information sequence of the modality is a language-related information sequence, a gaze point difference value sequence, a trajectory-related sequence, or a user emotion information sequence;

[0038] The feature fusion module includes N multi-head attention submodules and N long short-term memory network submodules; each multi-head attention submodule is used to process the information sequence of the corresponding modality; the dimension of each multi-head attention submodule matches the dimension of the information sequence of the modality it processes; the input end of each multi-head attention submodule is connected to the output end of the corresponding feature extractor; each long short-term memory network submodule is used to process the information sequence of the corresponding modality; each long short-term memory network submodule is connected to the output end of the corresponding multi-head attention submodule;

[0039] The prediction module includes a fully connected network and an output end; the N input ends of the fully connected network are respectively connected to the output ends of the N long short-term memory network sub-modules; the output end of the prediction module is used to output the user purchase intention prediction value.

[0040] The training process of the initial user purchase intention prediction model using the training data set to obtain the user purchase intention prediction model includes:

[0041] Initialize the number of training iterations;

[0042] Input the training data in the training data set as input data into the initial user purchase intention prediction model;

[0043] Using the initial user purchase intention prediction model, the input data is processed to obtain a prediction value;

[0044] Performing difference calculation processing on the obtained predicted value and the label data corresponding to the input data to obtain a difference value;

[0045] Determine whether the difference value satisfies a convergence condition, and obtain a first determination result;

[0046] When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result;

[0047] When the second judgment result is no, determining that the model training state does not meet the training termination condition;

[0048] When the second judgment result is yes, determining that the model training state satisfies the training termination condition;

[0049] When the first judgment result is yes, determining that the model training state satisfies a training termination condition;

[0050] When the model training status does not meet the termination training condition, the parameters of the initial user purchase intention prediction model are updated using the parameter update model, the training iteration number value is increased by 1, and the execution is triggered to input the training data in the training data set as input data into the initial user purchase intention prediction model;

[0051] When the model training status satisfies the training termination condition, the training process of the initial user purchase intention prediction model is completed to obtain the user purchase intention prediction model.

[0052] The parameter updating model is:

[0053]

[0054] θ←θ+v;

[0055] In the formula, I i is the difference value calculated for the i-th training data in the training data set, v is the parameter update value, θ is the parameter of the module to be updated, η is the initial parameter learning rate, α is the momentum angle parameter, 0≤α≤π / 4, It means to find partial derivatives with respect to variable θ;

[0056] In a second aspect of the embodiment of the present application, a device for predicting user purchase intention based on multimodal information is disclosed, comprising:

[0057] A memory storing executable program code;

[0058] a processor coupled to the memory;

[0059] The processor calls the executable program code stored in the memory to execute the user purchase intention prediction method based on multimodal information.

[0060] In a third aspect of an embodiment of the present application, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the user purchase intention prediction method based on multimodal information.

[0061] In a fourth aspect of the embodiments of the present application, an information data processing terminal is disclosed, wherein the information data processing terminal is used to implement the user purchase intention prediction method based on multimodal information.

[0062] The beneficial effects of the present invention are:

[0063] With the authorization of the user, the present invention comprehensively collects various behavioral information that affects the user's purchase of goods. By comprehensively extracting and analyzing the user's multimodal information, it effectively mines the user's purchasing behavior information in commercial scenarios, extracts the key factors that determine the user's purchasing decision, and establishes a user purchase intention prediction model, providing a quantitative basis for increasing product sales. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 1 is a flow chart for implementing the method of the present invention;

[0065] Figure 2 It is a composition diagram of the sales reminder device of the present invention. DETAILED DESCRIPTION

[0066] In order to better understand the content of the present invention, an embodiment is given here.

[0067] Figure 1 1 is a flow chart for implementing the method of the present invention; Figure 2 It is a composition diagram of the sales reminder device of the present invention.

[0068] In a first aspect of an embodiment of the present application, a method for predicting user purchase intention based on multimodal information is disclosed, comprising:

[0069] S1, obtaining a user purchase behavior information set; the user purchase behavior information set includes a behavior information set related to purchase decisions that can be collected during the user's purchase of goods, and a corresponding user purchase decision information set; each behavior information set has corresponding purchase decision information;

[0070] S2, performing association processing on the behavior information set and the purchase decision information set to obtain a user purchase intention prediction model;

[0071] S3, using the user purchase intention prediction model to process the collected user behavior information to obtain the user purchase intention prediction value;

[0072] The acquisition of the user purchase behavior information set includes:

[0073] Using an optical acquisition device, acquiring a user's facial image information sequence and a user's body image information sequence;

[0074] Performing body extraction and trajectory tracking processing on the user body image information to obtain a user trajectory information sequence;

[0075] Using acoustic collection equipment to collect user voice information sequences;

[0076] Performing emotion recognition processing on the user facial image information sequence and the user voice information to obtain a user emotion information sequence;

[0077] Performing gaze point recognition processing on the user's facial image information sequence to obtain a gaze point position information sequence;

[0078] Performing collection time alignment processing on the user trajectory information sequence, the user voice information sequence, the user emotion information sequence, and the gaze point position information sequence to obtain a behavior information set;

[0079] Acquire purchase decision information, wherein the purchase decision information includes actual purchase information and target commodity information; the target commodity information includes target commodity location information and value information;

[0080] Information sequence is the value of various types of information at different discrete moments.

[0081] The emotion recognition processing is performed on the user facial image information sequence and the user voice information to obtain a user emotion information sequence. CNN can be used to recognize the user facial image information sequence and the user voice information respectively to obtain corresponding emotion recognition results, and the two results are weighted summed to obtain a final user emotion information sequence.

[0082] The gaze point recognition process can be implemented by using an eye tracker or a Pico Neo 3Pro Eye Enterprise Edition eye tracking system;

[0083] The human body extraction and trajectory tracking processing of the user's human body image information to obtain a user trajectory information sequence can be implemented by using the YoLo series model.

[0084] The associating process of the behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes:

[0085] Performing correlation extraction processing on the behavior information set and the purchase decision information set to obtain a related behavior information set;

[0086] Performing regression processing on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model;

[0087] The correlating extraction process of the behavior information set and the purchase decision information set to obtain the correlating behavior information set includes:

[0088] Extract all the associated information associated with the target product in the user voice information sequence, and construct a language associated information sequence using all the extracted associated information;

[0089] Performing difference calculation processing on the gaze point position information sequence and the target commodity position information to obtain a gaze point difference value sequence;

[0090] Performing correlation analysis on the user trajectory information sequence and the target product to obtain a trajectory correlation sequence;

[0091] Using the language association information sequence, the gaze point difference value sequence, the trajectory related sequence and the user emotion information sequence, a related behavior information set is constructed;

[0092] The step of performing regression processing on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes:

[0093] Performing statistical analysis on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model for describing the causal relationship between the relevant behavior information set and the purchase decision information set;

[0094] The statistical analysis is a regression prediction analysis method, a time series prediction method, a support vector machine prediction method or a fuzzy prediction method, etc.;

[0095] The step of performing regression processing on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes:

[0096] A training data set is constructed by using the relevant behavior information set and the purchase decision information set; the training data in the training data set is the relevant behavior information set; the label data in the training data set is the purchase decision information corresponding to the relevant behavior information set;

[0097] Construct an initial user purchase intention prediction model;

[0098] Using the training data set to train the initial user purchase intention prediction model to obtain the user purchase intention prediction model;

[0099] The initial user purchase intention prediction model includes: an input module, a feature extraction module, a feature fusion module and a prediction module;

[0100] The input module is used to receive and obtain a relevant behavior information set;

[0101] The feature extraction module is connected to the input module and the feature fusion module respectively, and is used to extract features from the relevant behavior information set to obtain a feature signal set;

[0102] The feature fusion module is connected to the prediction module and is used to perform fusion processing on the feature signal set to obtain a fused feature signal set; perform time series correlation processing on the fused feature signal set to obtain a time series cross feature signal set;

[0103] The prediction module is used to perform prediction processing on the time series cross feature signal set to obtain a user purchase intention prediction value.

[0104] The feature extraction module includes N feature extractors corresponding to the signal modes; each feature extractor is used to perform feature extraction processing on the information sequence of the corresponding mode to obtain a corresponding feature signal;

[0105] The information sequence of the modality is a language-related information sequence, a gaze point difference value sequence, a trajectory-related sequence, or a user emotion information sequence;

[0106] The feature fusion module includes N multi-head attention submodules and N long short-term memory network submodules; each multi-head attention submodule is used to process the information sequence of the corresponding modality; the dimension of each multi-head attention submodule matches the dimension of the information sequence of the modality it processes; the input end of each multi-head attention submodule is connected to the output end of the corresponding feature extractor; each long short-term memory network submodule is used to process the information sequence of the corresponding modality; each long short-term memory network submodule is connected to the output end of the corresponding multi-head attention submodule;

[0107] The prediction module includes a fully connected network and an output end; the N input ends of the fully connected network are respectively connected to the output ends of the N long short-term memory network sub-modules; the output end of the prediction module is used to output the user purchase intention prediction value.

[0108] The training process of the initial user purchase intention prediction model using the training data set to obtain the user purchase intention prediction model includes:

[0109] Initialize the number of training iterations;

[0110] Input the training data in the training data set as input data into the initial user purchase intention prediction model;

[0111] Using the initial user purchase intention prediction model, the input data is processed to obtain a prediction value;

[0112] Performing difference calculation processing on the obtained predicted value and the label data corresponding to the input data to obtain a difference value;

[0113] Determine whether the difference value satisfies a convergence condition, and obtain a first determination result;

[0114] When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result;

[0115] When the second judgment result is no, determining that the model training state does not meet the training termination condition;

[0116] When the second judgment result is yes, determining that the model training state satisfies the training termination condition;

[0117] When the first judgment result is yes, determining that the model training state satisfies a training termination condition;

[0118] When the model training status does not meet the termination training condition, the parameters of the initial user purchase intention prediction model are updated using the parameter update model, the training iteration number value is increased by 1, and the execution is triggered to input the training data in the training data set as input data into the initial user purchase intention prediction model;

[0119] When the model training status satisfies the training termination condition, the training process of the initial user purchase intention prediction model is completed to obtain the user purchase intention prediction model.

[0120] The parameter updating model is:

[0121]

[0122] θ←θ+v;

[0123] In the formula, I i is the difference value calculated for the i-th training data in the training data set, v is the parameter update value, θ is the parameter of the module to be updated, η is the initial parameter learning rate, α is the momentum angle parameter, 0≤α≤π / 4, It means to find partial derivatives with respect to variable θ;

[0124] The difference value satisfies the convergence condition, which means that the difference value is less than a preset convergence threshold; the difference value does not satisfy the convergence condition, which means that the difference value is not less than the preset convergence threshold.

[0125] The difference calculation process may be implemented using a loss function.

[0126] The loss function may adopt a cross entropy loss function.

[0127] The fully connected network is used to implement the Softmax function.

[0128] The user purchase behavior information set includes, but is not limited to, a user trajectory sequence, a user facial image, a user voice sequence, a user emotion sequence, and a user gaze point position;

[0129] The multi-head attention submodule is used to perform modal cross-feature calculation processing on the corresponding feature signal to obtain modal cross-features; the output of a multi-head attention submodule is N modal cross-feature vectors; each multi-head attention submodule includes n attention heads;

[0130] The modal cross-feature calculation process is calculated as follows:

[0131]

[0132] Among them, F l-u is the modal cross feature vector of the feature signal of the current multi-head attention submodule and the u-th feature signal, l is the serial number of the current multi-head attention submodule, l = 1, 2, ..., N, u = 1, 2, ..., N, h i is the output of the i-th attention head of the current multi-head attention submodule, are the key, value, and weighted value of the query vector of the i-th attention head of the current multi-head attention submodule, i = 1, 2, ..., n, q, k, and v are the key, value, and query vector input by the current multi-head attention submodule, respectively; the values ​​of v and k are the l-th feature signal, and the value of q is the u-th feature signal; f is the first calculation function, and its expression is:

[0133]

[0134] Among them, d is the preset calculation constant, and s(g2, q1) is the calculation intermediate quantity.

[0135] The feature extractor is used to perform bandpass filtering on the input signal in a predefined frequency band to obtain a frequency signal, then calculate the root mean square of each spectrum signal (which can be calculated using the built-in function rms in MATLAB), and then take the logarithm of the root mean square with base 10 to obtain a power spectrum density characteristic signal, and determine the power spectrum density characteristic signal as the output of the feature extractor.

[0136] The signal modality is language, gaze point, trajectory and emotion;

[0137] When performing statistical analysis, a purchase decision information set is selected as a target variable, and a related behavior information set is selected as an independent variable. The target variable and the independent variable are analyzed using a regression prediction analysis method, a time series prediction method, a support vector machine prediction method, or a fuzzy prediction method, etc., to obtain a mathematical model between the target variable and the independent variable, and determine that the mathematical model is a user purchase intention prediction model;

[0138] The acquiring of the user purchase behavior information set may be to generate the user purchase behavior information set based on historical data collected from commercial scenarios;

[0139] The value information may be commodity profit information or price information, etc.

[0140] The user purchase intention prediction model is used to perform prediction processing on the collected user behavior information to obtain the user purchase intention prediction value;

[0141] Specifically, the user purchase behavior information set may include user trajectory information, user voice information, user emotion information, and user purchase decision information;

[0142] The value of the user emotion information is a coded value corresponding to the user's emotion type; the user's emotion type includes calmness, surprise, joy, dissatisfaction, anger, etc., which can be flexibly defined according to needs.

[0143] The actual purchase information is the digital code information corresponding to whether the user purchases the product;

[0144] The digital code information corresponding to whether the user purchases the product is 1 when the user purchases the target product, and 0 when the user does not purchase the target product.

[0145] Performing joint coding processing on the user purchase behavior information set to obtain a user purchase behavior digital information set; the user purchase behavior digital information set includes a behavior digital information set and a decision digital information set;

[0146] The user purchase decision information can be obtained by performing semantic analysis on the user's voice information to extract the user's final purchase decision information, or by matching the extracted merchant's fund records with the collection time of the user's purchase behavior information;

[0147] The extracted merchant's financial record is matched with the collection time of the user's purchasing behavior information, that is, the user payment information in the merchant's financial record within the collection time of the user's purchasing behavior information is used as the user's purchasing decision information, that is, if the user payment information appears in the merchant's financial record within the collection time of the user's purchasing behavior information, then the user's purchase decision information is confirmed to be purchased; if not, then the user's purchase decision information is confirmed to be not purchased.

[0148] The step of performing regression processing on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes:

[0149] Using the relevant behavior information set, a behavior information matrix is ​​constructed; the row vector of the behavior information matrix is ​​the user trajectory information, user voice information, user emotion information and gaze point position information obtained at the same time;

[0150] Using the purchase decision information set, a decision vector is constructed; the elements of the decision vector are the purchase decision information corresponding to the row vector of the behavior information matrix;

[0151] Using the behavior information matrix and the decision vector, a target optimization model is constructed;

[0152] Solving the target optimization model to obtain a user purchase intention prediction model;

[0153] The target optimization model is expressed as follows:

[0154]

[0155] v T v = 1,

[0156] Among them, a k is the kth element of the decision vector, v is the prediction vector to be solved, r i k is the i-th row vector of the behavior information matrix corresponding to the k-th element of the decision vector, P is the number of elements contained in the decision vector, and N is the row dimension of the behavior information matrix;

[0157] The user purchase intention prediction model is expressed as follows:

[0158] a0=Rv,

[0159] Among them, a0 represents the estimated decision vector, R is the behavior information matrix, and v is the prediction vector.

[0160] The solving of the target optimization model may be achieved by using an ant colony algorithm.

[0161] The user trajectory information, user voice information, user emotion information and gaze point location information obtained at the same time are extracted from the user trajectory information sequence, user voice information sequence, user emotion information sequence and gaze point location information sequence;

[0162] The performing correlation analysis on the user trajectory information sequence and the target product to obtain a trajectory correlation sequence includes:

[0163] Calculate the distance between each trajectory point in the user trajectory information sequence and the target product, and determine whether the distance is less than a preset distance threshold. If so, determine that the trajectory point is a related trajectory point; use all the related trajectory points to construct a trajectory related sequence;

[0164] The step of calculating the position difference between the gaze point position information sequence and the target commodity position information to obtain a gaze point difference value sequence includes:

[0165] For each gaze point position in the gaze point position information sequence, a difference value with the target commodity position information is calculated, and a gaze point difference value sequence is constructed using all the obtained difference values;

[0166] The extracting all the associated information associated with the target product in the user voice information sequence, and constructing a language associated information sequence using all the extracted associated information, includes:

[0167] For each user voice information in the user voice information sequence, calculate the correlation value between the user voice information and the target product information, and determine whether the correlation value is greater than a preset correlation threshold. If so, determine that the user voice information is associated information; and construct a language associated information sequence using all the associated information;

[0168] The relevance value can be calculated using the text relevance calculation formula in the knowledge graph;

[0169] In a second aspect of the embodiment of the present application, a device for predicting user purchase intention based on multimodal information is disclosed, comprising:

[0170] A memory storing executable program code;

[0171] a processor coupled to the memory;

[0172] The processor calls the executable program code stored in the memory to execute the user purchase intention prediction method based on multimodal information.

[0173] In a third aspect of an embodiment of the present application, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the user purchase intention prediction method based on multimodal information.

[0174] According to a third aspect of an embodiment of the present application, a sales reminder device is disclosed, which includes a signal processing device and a signal output device. The signal processing device is pre-installed with a user purchase intention prediction model, and is used to use the user purchase intention prediction model to process the collected user behavior information to obtain a user purchase intention prediction value, and determine whether the user purchase intention prediction value exceeds a preset discrimination threshold. When it exceeds, a reminder signal is sent to the signal output device; the signal output device is used to output corresponding user purchase prediction information according to the reminder signal; the output corresponding user purchase prediction information can be an image showing "the user has a strong purchase intention" or a sound outputting "the user has a strong purchase intention", the purpose of which is to remind the shopping guide or salesperson to pay attention.

[0175] Specifically, in a commercial scenario, a shopping guide or salesperson wears the sales reminder device. When promoting related products to users, the signal processing device in the sales reminder device analyzes the collected user purchase behavior information set. When the user purchase intention prediction value exceeds a preset judgment threshold, indicating that the user has a strong desire to buy, a reminder signal is sent to the signal output device. The signal output device uses a transparent single-sided display screen or a Bluetooth headset to output corresponding user purchase prediction information to the shopping guide or salesperson. The information is invisible or inaudible to the user. After receiving the corresponding user purchase prediction information, the shopping guide or salesperson can take corresponding promotional actions.

[0176] Specifically, the signal output device is communicatively connected with the signal processing device, and the signal output device can be implemented using VR glasses.

[0177] In a fourth aspect of the embodiments of the present application, an information data processing terminal is disclosed, wherein the information data processing terminal is used to implement the user purchase intention prediction method based on multimodal information.

[0178] It should be noted that since the method of the embodiment of the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, time is actually time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data for processing by the computer device. The details will not be repeated here.

[0179] It should be noted that the artificial intelligence related technologies that may be involved in this application are briefly described. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0180] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0181] The image processing processes involved in the embodiments of the present invention can all be implemented using computer vision technology. Computer vision technology (Computer Vision, CV) is a technology that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0182] Unimodal information is data of only one type, such as text, image, audio, video, electromagnetic signal, etc. Multimodal information is data that includes at least two types of unimodal information. Furthermore, multimodal information is suitable for complex tasks that require the integration of multiple information sources, such as sentiment analysis, robot interaction, autonomous driving, etc. By integrating information from multiple modalities, higher performance and accuracy can usually be achieved on the task.

[0183] A large model refers to an artificial neural network model with a very large number of parameters. In the field of artificial intelligence, a large model generally refers to a model with hundreds of millions to trillions of parameters. Models usually need to be trained on large-scale data sets and require a large amount of computing resources to be optimized and adjusted. Large models are often used to solve complex tasks such as natural language processing, computer vision, and speech recognition. Generative AI is an AI that can create new content and ideas, including conversations, stories, images, videos, and music. In an embodiment of the present application, the large model may be ChatGPT, BERT, XLNet, Zhipu model, Claude, Moonshot AI model, ChatGLM model, Qianyi Tongwen model, MiniMax model, Spark model, Llama model, 360GPT model, Qwen model, Baichuan model, Skylark model, vivoLM model, and Wenxin Yiyan scale language models, which are not limited in the embodiments of the present application.

[0184] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for predicting user purchase intention based on multimodal information, characterized in that: include: S1, obtaining a user purchase behavior information set; the user purchase behavior information set includes a behavior information set related to purchase decisions during the user's purchase of goods, and a corresponding user purchase decision information set; each behavior information set has corresponding purchase decision information; S2, performing association processing on the behavior information set and the purchase decision information set to obtain a user purchase intention prediction model; S3, using the user purchase intention prediction model to process the collected user behavior information set to obtain a user purchase intention prediction value.

2. The method for predicting user purchase intention based on multimodal information according to claim 1, characterized in that: The acquisition of the user purchase behavior information set includes: Using an optical acquisition device, acquiring a user's facial image information sequence and a user's body image information sequence; Performing body extraction and trajectory tracking processing on the user body image information to obtain a user trajectory information sequence; Using acoustic collection equipment to collect user voice information sequences; Performing emotion recognition processing on the user facial image information sequence and the user voice information sequence to obtain a user emotion information sequence; Performing gaze point recognition processing on the user's facial image information sequence to obtain a gaze point position information sequence; Performing collection time alignment processing on the user trajectory information sequence, the user voice information sequence, the user emotion information sequence, and the gaze point position information sequence to obtain a behavior information set; Acquiring purchase decision information, wherein the purchase decision information includes actual purchase information and target product information; The target product information includes the target product location information.

3. The method for predicting user purchase intention based on multimodal information according to claim 1, characterized in that: The associating process of the behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes: Performing correlation extraction processing on the behavior information set and the purchase decision information set to obtain a related behavior information set; Regression processing is performed on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model.

4. The method for predicting user purchase intention based on multimodal information according to claim 3, characterized in that: The correlating extraction process of the behavior information set and the purchase decision information set to obtain the correlating behavior information set includes: Extract all the associated information associated with the target product in the user voice information sequence, and construct a language associated information sequence using all the extracted associated information; Performing position difference calculation processing on the gaze point position information sequence and the target commodity position information to obtain a gaze point difference value sequence; Performing correlation analysis on the user trajectory information sequence and the target product to obtain a trajectory correlation sequence; The language association information sequence, the gaze point difference value sequence, the trajectory related sequence and the user emotion information sequence are used to construct a related behavior information set.

5. The method for predicting user purchase intention based on multimodal information according to claim 3, characterized in that: The step of performing regression processing on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes: Statistical analysis is performed on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model for describing the causal relationship between the relevant behavior information set and the purchase decision information set.

6. The method for predicting user purchase intention based on multimodal information according to claim 3, characterized in that: The step of performing regression processing on the relevant behavior information set and the purchase decision information set to obtain a user purchase intention prediction model includes: A training data set is constructed by using the relevant behavior information set and the purchase decision information set; the training data in the training data set is the relevant behavior information set; the label data in the training data set is the purchase decision information corresponding to the relevant behavior information set; Construct an initial user purchase intention prediction model; Using the training data set to train the initial user purchase intention prediction model to obtain the user purchase intention prediction model; The initial user purchase intention prediction model includes: an input module, a feature extraction module, a feature fusion module and a prediction module; The input module is used to receive and obtain a relevant behavior information set; The feature extraction module is connected to the input module and the feature fusion module respectively, and is used to extract features from the relevant behavior information set to obtain a feature signal set; The feature fusion module is connected to the prediction module and is used to perform fusion processing on the feature signal set to obtain a fused feature signal set; perform time series correlation processing on the fused feature signal set to obtain a time series cross feature signal set; The prediction module is used to perform prediction processing on the time series cross feature signal set to obtain a user purchase intention prediction value; The feature extraction module includes N feature extractors corresponding to the signal modes; each feature extractor is used to perform feature extraction processing on the information sequence of the corresponding mode to obtain a corresponding feature signal; The information sequence of the modality is a language-related information sequence, a gaze point difference value sequence, a trajectory-related sequence, or a user emotion information sequence; The feature fusion module includes N multi-head attention submodules and N long short-term memory network submodules; each multi-head attention submodule is used to process the information sequence of the corresponding modality; the dimension of each multi-head attention submodule matches the dimension of the information sequence of the modality it processes; the input end of each multi-head attention submodule is connected to the output end of the corresponding feature extractor; each long short-term memory network submodule is used to process the information sequence of the corresponding modality; each long short-term memory network submodule is connected to the output end of the corresponding multi-head attention submodule; The prediction module includes a fully connected network and an output end; the N input ends of the fully connected network are respectively connected to the output ends of the N long short-term memory network sub-modules; the output end of the prediction module is used to output the user purchase intention prediction value.

7. The method for predicting user purchase intention based on multimodal information according to claim 6, characterized in that: The training process of the initial user purchase intention prediction model using the training data set to obtain the user purchase intention prediction model includes: Initialize the number of training iterations; Input the training data in the training data set as input data into the initial user purchase intention prediction model; Using the initial user purchase intention prediction model, the input data is processed to obtain a prediction value; Performing difference calculation processing on the obtained predicted value and the label data corresponding to the input data to obtain a difference value; Determine whether the difference value satisfies a convergence condition, and obtain a first determination result; When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result; When the second judgment result is no, determining that the model training state does not meet the training termination condition; When the second judgment result is yes, determining that the model training state satisfies the training termination condition; When the first judgment result is yes, determining that the model training state satisfies a training termination condition; When the model training status does not meet the termination training condition, the parameters of the initial user purchase intention prediction model are updated using the parameter update model, the training iteration number value is increased by 1, and the execution is triggered to input the training data in the training data set as input data into the initial user purchase intention prediction model; When the model training status satisfies the training termination condition, the training process of the initial user purchase intention prediction model is completed to obtain the user purchase intention prediction model.

8. A user purchase intention prediction device based on multimodal information, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the user purchase intention prediction method based on multimodal information according to any one of claims 1 to 7.

9. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the user purchase intention prediction method based on multimodal information according to any one of claims 1 to 7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the user purchase intention prediction method based on multimodal information as described in any one of claims 1 to 7.

Images