Multi-modal intelligent customer service interaction method and system combined with computer vision

Abstract

The invention discloses a multi-mode intelligent customer service interaction method and system combined with computer vision, and relates to the technical field of intelligent interaction.The method comprises the steps that data information of a user is obtained, and the data information comprises image information, voice information and text information; constructing a computer vision model and a natural language processing model; analyzing and processing the image information through a computer vision model, and extracting image features of the user; analyzing and processing the voice information and the text information through a natural language processing model, and extracting language features of the user; and carrying out interactive fusion on the image features and the language features by adopting a multi-modal fusion technology, and automatically generating reply information. According to the fused feature information, the reply for the user is automatically generated, and more intelligent customer service experience is provided; the cross-domain information processing mode not only improves the intelligent degree of the customer service system, but also enhances the interaction effect with the user, and better meets the user demand.

Description

Technical Field

[0001] The present invention relates to the field of intelligent interaction technology, and more specifically to a multimodal intelligent customer service interaction method and system combined with computer vision. Background Art

[0002] Multimodal interaction refers to information exchange and interaction through multiple sensory channels (such as text, voice, images, gestures, etc.). In the intelligent customer service interaction system, multimodal interaction allows users to communicate with customer service in a more natural and convenient way, improving the quality and efficiency of customer service.

[0003] Traditional customer service systems mainly rely on manual services, which have problems such as high costs and low efficiency. With the development of artificial intelligence technology, intelligent customer service systems have gradually emerged, and automated customer service has been achieved through technologies such as natural language processing and machine learning. However, traditional intelligent customer service systems mainly rely on text and voice interactions and cannot meet the diverse needs of users.

[0004] Therefore, how to provide a multimodal intelligent customer service interaction method and system combined with computer vision to communicate with customer service through gestures, expressions, etc., to achieve a more natural and convenient interaction method, and to improve the quality and efficiency of customer service is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a multimodal intelligent customer service interaction method and system combined with computer vision, which communicates with customer service through gestures, expressions, etc., to achieve a more natural and convenient interaction method, and improve the quality and efficiency of customer service.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a multimodal intelligent customer service interaction method combined with computer vision, comprising:

[0007] Acquiring user data information, the data information including image information, voice information and text information;

[0008] Build computer vision models and natural language processing models;

[0009] Analyzing and processing the image information through the computer vision model to extract image features of the user;

[0010] Analyzing and processing the voice information and the text information through the natural language processing model to extract the language features of the user;

[0011] The multimodal fusion technology is used to interactively fuse the image features and the language features to automatically generate reply information.

[0012] Preferably, analyzing and processing the image information by the computer vision model to extract the image features of the user includes:

[0013] Preprocessing the image information according to the user's image information to obtain a first training data set;

[0014] Build an image detection model, use image detection algorithms to identify and extract the user's body contour, and use key point detection algorithms to identify the user's facial key points and gesture key points;

[0015] Analyze the user's facial expressions and extract emotional state features based on the user's facial key points;

[0016] According to the key points of the user's gestures, analyze the user's actions and extract the gesture command features;

[0017] The emotional state features and the gesture command features are fused to obtain the image features of the user.

[0018] Preferably, analyzing and processing the voice information by the natural language processing model includes:

[0019] Preprocessing the voice information according to the user's voice information to obtain a second training data set;

[0020] Based on the second training data set, a speech analysis model is constructed to calculate the MFCC features of the speech information to capture the frequency and time domain features of the speech signal; the pitch, volume and rhythm of the speech information are analyzed to extract speech features.

[0021] Preferably, analyzing and processing the text information by the natural language processing model includes:

[0022] Preprocessing the text information according to the text information of the user;

[0023] Use word embedding technology to convert text information into word vector representation;

[0024] The sentence embedding model converts the entire sentence or paragraph of text information into a sentence vector representation;

[0025] Using the word vector representation and sentence vector representation to identify keywords and phrases in text information;

[0026] Determine the user's intention and sentiment based on the identified keywords and phrases, and generate text features;

[0027] The extracted speech features and text features are concatenated and fused to generate the user's language features.

[0028] Preferably, a multimodal fusion technology is used to interactively fuse the image features and the language features to automatically generate reply information, including:

[0029] Introduce the attention mechanism to enhance the weight distribution of information between different modalities;

[0030] Construct a two-stream neural network to process image features and language features separately by sharing weights to obtain fused features and improve fusion effect;

[0031] Build an automatic reply generation model and use the fused features as the input of the generation model to capture the correlation between multiple modalities.

[0032] Preferably, constructing an automatic reply generation model includes:

[0033] Collect and annotate multimodal training datasets;

[0034] The automatic reply generation model is trained by using the multimodal training data set, and the parameters of the automatic reply generation model are optimized so that the automatic reply generation model can generate appropriate replies under given image and text conditions;

[0035] The difference between the generated response and the true response is evaluated through the loss function, and backpropagation and optimization are performed.

[0036] Preferably, a multimodal intelligent customer service interaction system combined with computer vision includes:

[0037] An information acquisition module is used to acquire user data information, wherein the data information includes image information, voice information and text information;

[0038] Model building module, used to build computer vision models and natural language processing models;

[0039] An image feature extraction module, used to analyze and process the image information through the computer vision model to extract the image features of the user;

[0040] A language feature extraction module, used to analyze and process the voice information and the text information through the natural language processing model to extract the language features of the user;

[0041] The multimodal fusion module is used to adopt the multimodal fusion technology to interactively fuse the image features and the language features to automatically generate reply information.

[0042] Preferably, the image feature extraction module comprises: a first training data set acquisition unit, an image detection model construction unit, a key point recognition unit, a feature extraction unit and a feature fusion unit connected in sequence;

[0043] The first training data set acquisition unit is used to pre-process the image information according to the image information of the user to obtain the first training data set;

[0044] The image detection model building unit is used to build an image detection model;

[0045] The key point recognition unit is used to use an image detection algorithm to recognize and extract the user's body contour, and use a key point detection algorithm to recognize the user's facial key points and gesture key points;

[0046] The feature extraction unit is used to analyze the user's facial expression based on the user's facial key points and extract the emotional state features; analyze the user's actions based on the user's gesture key points and extract the gesture instruction features;

[0047] The feature fusion unit is used to perform feature fusion on the emotional state feature and the gesture instruction feature to obtain the image feature of the user.

[0048] Preferably, the language feature extraction module includes: a second training data set acquisition unit, a speech feature extraction unit, a text information preprocessing unit, a word vector representation unit, a sentence vector representation unit, a keyword and phrase recognition unit, a text feature generation unit and a language feature generation unit;

[0049] The second training data set acquisition unit is used to pre-process the voice information according to the user's voice information to obtain a second training data set;

[0050] The speech feature extraction unit is used to construct a speech analysis model based on the second training data set, calculate the MFCC features of the speech information to capture the frequency and time domain features of the speech signal; analyze the pitch, volume and rhythm of the speech information to extract speech features;

[0051] The text information preprocessing unit is used to preprocess the text information according to the text information of the user;

[0052] The word vector representation unit is used to convert text information into word vector representation using word embedding technology;

[0053] The sentence vector representation unit is used to convert the entire sentence or paragraph of text information into a sentence vector representation through a sentence embedding model;

[0054] The keyword and phrase recognition unit is used to recognize keywords and phrases in text information using the word vector representation and sentence vector representation;

[0055] The text feature generation unit is used to judge the user's intention and emotional tendency according to the recognized keywords and phrases, and generate text features;

[0056] The language feature generation unit is used to concatenate and fuse the extracted voice features and text features to generate the user's language features.

[0057] Preferably, the multimodal fusion module comprises a weight distribution unit, a fusion unit and an automatic reply unit connected in sequence:

[0058] The weight distribution unit is used to introduce an attention mechanism to enhance the weight distribution of information between different modalities;

[0059] The fusion unit is used to construct a two-stream neural network, and processes the image features and the language features respectively by sharing weights to obtain fused features;

[0060] The automatic reply unit is used to construct an automatic reply generation model, using the fused features as the input of the generation model to capture the association between multiple modalities.

[0061] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a multimodal intelligent customer service interaction method and system combined with computer vision, including: obtaining user data information, the data information includes image information, voice information and text information; constructing a computer vision model and a natural language processing model; analyzing and processing the image information through the computer vision model to extract the user's image features; analyzing and processing the voice information and the text information through the natural language processing model to extract the user's language features; using multimodal fusion technology to interactively fuse the image features and language features to automatically generate reply information. The present invention automatically generates replies to users based on the fused feature information, providing a more intelligent customer service experience; this cross-domain information processing method not only improves the intelligence of the customer service system, but also enhances the interaction effect with users, and better meets user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0063] Figure 1 A flowchart of a multimodal intelligent customer service interaction method combined with computer vision provided by the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The embodiment of the present invention discloses a multimodal intelligent customer service interaction method and system combined with computer vision, which automatically generates replies for users based on fused feature information to provide a more intelligent customer service experience; this cross-domain information processing method not only improves the intelligence level of the customer service system, but also enhances the interaction effect with users, thereby better meeting user needs.

[0066] The embodiment of the present invention discloses a multimodal intelligent customer service interaction method combined with computer vision, such as Figure 1 As shown, including:

[0067] Acquiring user data information, the data information including image information, voice information and text information;

[0068] Build computer vision models and natural language processing models;

[0069] Analyze and process the image information through the computer vision model to extract the image features of the user; identify the user's identity, facial expressions, gestures and other visual features;

[0070] Analyze and process the voice information and the text information through the natural language processing model to extract the user's language features; understand and process the user's voice and text information to extract language features such as semantics and intent;

[0071] The multimodal fusion technology is used to interactively fuse the image features and the language features to automatically generate reply information.

[0072] Based on the integrated feature information, replies to users are automatically generated to provide a more intelligent customer service experience. This cross-domain information processing method not only improves the intelligence of the customer service system, but also enhances the interaction with users and better meets user needs.

[0073] Specifically, the image information includes: photos or video streams of the user.

[0074] Voice information includes: user's voice input, such as audio signals acquired through voice recognition.

[0075] Text information includes: text entered by the user through the keyboard, such as chat information.

[0076] Specifically, analyzing and processing the image information by using the computer vision model to extract the image features of the user includes:

[0077] According to the user's image information, the image information is preprocessed to obtain a first training data set; the preprocessing of the image information includes: image cleaning: removing noise, adjusting brightness and contrast, and improving image quality; size adjustment: adjusting the image to the input size required by the model, maintaining the aspect ratio, and preventing distortion.

[0078] Build an image detection model, use image detection algorithms to identify and extract the user's body contour, and use key point detection algorithms to identify the user's facial key points and gesture key points;

[0079] Analyze the user's facial expressions and extract emotional state features based on the user's facial key points;

[0080] According to the key points of the user's gestures, analyze the user's actions and extract the gesture command features;

[0081] The emotional state features and the gesture command features are fused to obtain the image features of the user.

[0082] Specifically, analyzing and processing the voice information by using the natural language processing model includes:

[0083] The user's voice information is obtained through microphone recording or other audio input devices. The voice information is preprocessed to obtain a second training data set; the preprocessing of the voice information includes: denoising: using filters and other technologies to remove background noise and improve the quality of voice signals; segmentation: dividing long voice signals into small segments for segment-by-segment analysis (for example, sentences or phrases);

[0084] Based on the second training data set, a speech analysis model is constructed to calculate the MFCC features of the speech information to capture the frequency and time domain features of the speech signal; the pitch, volume and rhythm of the speech information are analyzed to extract features related to the sound, such as intonation changes, speech speed, stress, etc.; and speech features are extracted. To obtain a higher level of expression. The speech signal is converted into a text representation and transcribed using automatic speech recognition (ASR) technology. By analyzing speech features (such as intonation, stress, and specific vocabulary of emotions), the user's emotional state is identified.

[0085] Specifically, analyzing and processing the text information by using the natural language processing model includes:

[0086] Preprocess the text information according to the user's text information; the preprocessing of the text information includes: tokenization: splitting the sentence into words or phrases; removing stop words: deleting high-frequency but low-information words such as "de" (of), "zai" (in).

[0087] Use word embedding techniques (such as Word2Vec, GloVe, or BERT) to convert the text information into a word vector representation;

[0088] Convert the entire sentence or paragraph of the text information into a sentence vector representation through a sentence embedding model;

[0089] Identify keywords and phrases in the text information using the word vector representation and the sentence vector representation;

[0090] Judge the user's intention and sentiment tendency based on the identified keywords and phrases, and generate text features;

[0091] Concatenate and fuse the extracted speech features and text features to generate the user's language features.

[0092] The user's intention can be a query, a complaint, a request, etc.; the sentiment tendency is positive, negative, or neutral; at the same time, combine the context information to understand the context of the text in order to more accurately extract the text meaning and intention.

[0093] Use the extracted language features for various subsequent tasks, such as automatic reply generation, user sentiment analysis, user intention discrimination, etc., to improve the response quality and efficiency of the intelligent customer service system.

[0094] Comprehensively analyze the user's speech and text information through the above steps, extract rich language features, and thus better understand the user's needs.

[0095] Specifically, adopt a multi-modal fusion technology to interactively fuse the image features and the language features to automatically generate reply information, including:

[0096] Introduce an attention mechanism to enhance the weight distribution of information between different modalities; combine the image features and the language features through weighted combination;

[0097] Construct a two-stream neural network, and process the image features and the language features separately in a shared-weight manner to obtain the fused features; improve the fusion effect;

[0098] Construct an automatic reply generation model, and use the fused features as the input of the generation model to capture the correlations between different modalities.

[0099] Specifically, constructing an automatic reply generation model includes:

[0100] Collect and annotate multimodal training datasets; including images and corresponding text responses to ensure data diversity and representativeness;

[0101] The automatic reply generation model is trained by using the multimodal training data set, and the parameters of the automatic reply generation model are optimized so that the automatic reply generation model can generate appropriate replies under given image and text conditions;

[0102] The difference between the generated response and the true response is evaluated through a loss function (such as cross entropy loss), and backpropagation and optimization are performed.

[0103] Specifically, the response generation process is as follows:

[0104] Input samples: Input user-submitted images and text information into the trained generative model.

[0105] Generation process: The model generates response information related to the input image and text.

[0106] Post-processing: Check the grammar and logic of the generated responses to ensure the natural flow of the responses.

[0107] The embodiment of the present invention also discloses a multimodal intelligent customer service interaction system combined with computer vision, including:

[0108] An information acquisition module is used to acquire user data information, wherein the data information includes image information, voice information and text information;

[0109] Model building module, used to build computer vision models and natural language processing models;

[0110] An image feature extraction module, used to analyze and process the image information through the computer vision model to extract the image features of the user;

[0111] A language feature extraction module, used to analyze and process the voice information and the text information through the natural language processing model to extract the language features of the user;

[0112] The multimodal fusion module is used to adopt the multimodal fusion technology to interactively fuse the image features and the language features to automatically generate reply information.

[0113] A multimodal intelligent customer service interaction system combined with computer vision, comprising:

[0114] An information acquisition module is used to acquire user data information, wherein the data information includes image information, voice information and text information;

[0115] Model building module, used to build computer vision models and natural language processing models;

[0116] An image feature extraction module, used to analyze and process the image information through the computer vision model to extract the image features of the user;

[0117] A language feature extraction module, used to analyze and process the voice information and the text information through the natural language processing model to extract the language features of the user;

[0118] The multimodal fusion module is used to adopt the multimodal fusion technology to interactively fuse the image features and the language features to automatically generate reply information.

[0119] Specifically, the image feature extraction module includes: a first training data set acquisition unit, an image detection model construction unit, a key point recognition unit, a feature extraction unit and a feature fusion unit connected in sequence;

[0120] The first training data set acquisition unit is used to pre-process the image information according to the image information of the user to obtain the first training data set;

[0121] The image detection model building unit is used to build an image detection model;

[0122] The key point recognition unit is used to use an image detection algorithm to recognize and extract the user's body contour, and use a key point detection algorithm to recognize the user's facial key points and gesture key points;

[0123] The feature extraction unit is used to analyze the user's facial expression based on the user's facial key points and extract the emotional state features; analyze the user's actions based on the user's gesture key points and extract the gesture instruction features;

[0124] The feature fusion unit is used to perform feature fusion on the emotional state feature and the gesture instruction feature to obtain the image feature of the user.

[0125] Specifically, the language feature extraction module includes: a second training data set acquisition unit, a speech feature extraction unit, a text information preprocessing unit, a word vector representation unit, a sentence vector representation unit, a keyword and phrase recognition unit, a text feature generation unit and a language feature generation unit;

[0126] The second training data set acquisition unit is used to pre-process the voice information according to the user's voice information to obtain a second training data set;

[0127] The speech feature extraction unit is used to construct a speech analysis model based on the second training data set, calculate the MFCC features of the speech information to capture the frequency and time domain features of the speech signal; analyze the pitch, volume and rhythm of the speech information to extract speech features;

[0128] The text information preprocessing unit is used to preprocess the text information according to the text information of the user;

[0129] The word vector representation unit is used to convert text information into word vector representation using word embedding technology;

[0130] The sentence vector representation unit is used to convert the entire sentence or paragraph of text information into a sentence vector representation through a sentence embedding model;

[0131] The keyword and phrase recognition unit is used to recognize keywords and phrases in text information using the word vector representation and sentence vector representation;

[0132] The text feature generation unit is used to judge the user's intention and emotional tendency according to the recognized keywords and phrases, and generate text features;

[0133] The language feature generation unit is used to concatenate and fuse the extracted voice features and text features to generate the user's language features.

[0134] Specifically, the multimodal fusion module includes a weight distribution unit, a fusion unit and an automatic reply unit connected in sequence:

[0135] The weight distribution unit is used to introduce an attention mechanism to enhance the weight distribution of information between different modalities;

[0136] The fusion unit is used to construct a two-stream neural network, and processes the image features and the language features respectively by sharing weights to obtain fused features;

[0137] The automatic reply unit is used to construct an automatic reply generation model, using the fused features as the input of the generation model to capture the association between multiple modalities.

[0138] In a specific embodiment of the present invention, the intelligent customer service interaction system includes three parts: a user end, a server end, and a data storage end. The user end collects the user's image, voice and other information through devices such as cameras and microphones, and sends this information to the server end for processing. The methods described in the embodiments of the present invention are all completed on the server end. The server end obtains user information and analyzes and processes the user's information through computer vision, natural language processing and other technologies, and generates corresponding replies. The data storage end is used to store the user's historical interaction record data to provide support for the system's intelligent replies. The multimodal intelligent customer service interaction system combined with computer vision obtains the user's image, voice and other information through devices such as cameras and microphones to achieve a more natural and convenient interaction method. Users can communicate with customer service through gestures, expressions and other methods to improve the quality and efficiency of customer service.

[0139] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0140] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal intelligent customer service interaction method combined with computer vision, characterized in that: include: Acquiring user data information, the data information including image information, voice information and text information; Build computer vision models and natural language processing models; Analyzing and processing the image information through the computer vision model to extract image features of the user; Analyzing and processing the voice information and the text information through the natural language processing model to extract the language features of the user; The multimodal fusion technology is adopted to interactively fuse the image features and the language features to automatically generate reply information.

2. According to claim 1, a multimodal intelligent customer service interaction method combined with computer vision is characterized in that: Analyzing and processing the image information through the computer vision model to extract the image features of the user includes: Preprocessing the image information according to the user's image information to obtain a first training data set; Build an image detection model, use image detection algorithms to identify and extract the user's body contour, and use key point detection algorithms to identify the user's facial key points and gesture key points; Analyze the user's facial expressions and extract emotional state features based on the user's facial key points; According to the key points of the user's gestures, analyze the user's actions and extract the gesture command features; The emotional state features and the gesture command features are fused to obtain the image features of the user.

3. According to claim 1, a multimodal intelligent customer service interaction method combined with computer vision is characterized in that: Analyzing and processing the speech information by using the natural language processing model includes: Preprocessing the voice information according to the user's voice information to obtain a second training data set; Based on the second training data set, a speech analysis model is constructed to calculate the MFCC features of the speech information to capture the frequency and time domain features of the speech signal; the pitch, volume and rhythm of the speech information are analyzed to extract speech features.

4. According to claim 1, a multimodal intelligent customer service interaction method combined with computer vision is characterized in that: Analyzing and processing the text information by using the natural language processing model includes: Preprocessing the text information according to the text information of the user; Use word embedding technology to convert text information into word vector representation; The sentence embedding model converts the entire sentence or paragraph of text information into a sentence vector representation; Using the word vector representation and sentence vector representation to identify keywords and phrases in text information; Determine the user's intention and sentiment based on the identified keywords and phrases, and generate text features; The extracted speech features and text features are concatenated and fused to generate the user's language features.

5. According to claim 1, a multimodal intelligent customer service interaction method combined with computer vision is characterized in that: The multimodal fusion technology is used to interactively fuse the image features and the language features to automatically generate reply information, including: Introduce the attention mechanism to enhance the weight distribution of information between different modalities; Construct a two-stream neural network to process image features and language features separately by sharing weights to obtain fused features; Build an automatic reply generation model and use the fused features as the input of the generation model to capture the correlation between multiple modalities.

6. The multimodal intelligent customer service interaction method combined with computer vision according to claim 5, characterized in that: Build an automatic reply generation model, including: Collect and annotate multimodal training datasets; The automatic reply generation model is trained by using the multimodal training data set, and the parameters of the automatic reply generation model are optimized so that the automatic reply generation model can generate replies under given image and text conditions; The difference between the generated response and the true response is evaluated through the loss function, and backpropagation and optimization are performed.

7. A multimodal intelligent customer service interaction system combined with computer vision, applying a multimodal intelligent customer service interaction method combined with computer vision as described in any one of claims 1 to 6, characterized in that: include: An information acquisition module is used to acquire user data information, wherein the data information includes image information, voice information and text information; Model building module, used to build computer vision models and natural language processing models; An image feature extraction module, used to analyze and process the image information through the computer vision model to extract the image features of the user; A language feature extraction module, used to analyze and process the voice information and the text information through the natural language processing model to extract the language features of the user; The multimodal fusion module is used to adopt the multimodal fusion technology to interactively fuse the image features and the language features to automatically generate reply information.

8. The multimodal intelligent customer service interaction system combined with computer vision according to claim 7, characterized in that: The image feature extraction module includes: a first training data set acquisition unit, an image detection model construction unit, a key point recognition unit, a feature extraction unit and a feature fusion unit connected in sequence; The first training data set acquisition unit is used to pre-process the image information according to the image information of the user to obtain the first training data set; The image detection model building unit is used to build an image detection model; The key point recognition unit is used to use an image detection algorithm to recognize and extract the user's body contour, and use a key point detection algorithm to recognize the user's facial key points and gesture key points; The feature extraction unit is used to analyze the user's facial expression according to the user's facial key points and extract the emotional state features; analyze the user's actions according to the user's gesture key points and extract the gesture instruction features; The feature fusion unit is used to perform feature fusion on the emotional state feature and the gesture instruction feature to obtain the image feature of the user.

9. The multimodal intelligent customer service interaction method combined with computer vision according to claim 7, characterized in that: The language feature extraction module includes: a second training data set acquisition unit, a speech feature extraction unit, a text information preprocessing unit, a word vector representation unit, a sentence vector representation unit, a keyword and phrase recognition unit, a text feature generation unit and a language feature generation unit; The second training data set acquisition unit is used to pre-process the voice information according to the user's voice information to obtain a second training data set; The speech feature extraction unit is used to construct a speech analysis model based on the second training data set, calculate the MFCC features of the speech information to capture the frequency and time domain features of the speech signal; analyze the pitch, volume and rhythm of the speech information to extract speech features; The text information preprocessing unit is used to preprocess the text information according to the text information of the user; The word vector representation unit is used to convert text information into word vector representation using word embedding technology; The sentence vector representation unit is used to convert the entire sentence or paragraph of text information into a sentence vector representation through a sentence embedding model; The keyword and phrase recognition unit is used to recognize keywords and phrases in text information using the word vector representation and sentence vector representation; The text feature generation unit is used to judge the user's intention and emotional tendency according to the recognized keywords and phrases, and generate text features; The language feature generation unit is used to concatenate and fuse the extracted voice features and text features to generate the user's language features.

10. The multimodal intelligent customer service interaction method combined with computer vision according to claim 7, characterized in that: The multimodal fusion module includes a weight distribution unit, a fusion unit and an automatic reply unit connected in sequence: The weight distribution unit is used to introduce an attention mechanism to enhance the weight distribution of information between different modalities; The fusion unit is used to construct a two-stream neural network, and processes the image features and the language features respectively by sharing weights to obtain fused features; The automatic reply unit is used to construct an automatic reply generation model, using the fused features as the input of the generation model to capture the association between multiple modalities.

Images