Multi-modal interaction method, device and equipment of intelligent customer service system and storage medium

Abstract

The invention provides a multi-modal interaction method and device of an intelligent customer service system, equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining multi-modal vehicle consultation data input by a target user; performing feature extraction on the vehicle consultation data of the multiple modes to obtain multiple feature vectors corresponding to the vehicle consultation data of the multiple modes; according to the types of the multiple modes and the current interaction scene of the target user, obtaining weights of the multiple modes in the current interaction scene; performing feature fusion on the multiple feature vectors according to the weights of the multiple modes to obtain a fused feature vector; and generating a service scheme for the target user according to the fused feature vector. According to the embodiment of the invention, standard processing and deep fusion of multi-modal data are realized, the problems of low response efficiency and inaccurate recommendation of traditional customer service are solved, and finally, the purposes of accurately insight into car purchase demands of users and efficiently outputting personalized marketing services are achieved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a multimodal interaction method, apparatus, device, and storage medium for an intelligent customer service system. Background Technology

[0002] In the field of intelligent customer service, with the continuous growth in demand for digital services, voice interaction and text interaction have become the mainstream customer communication methods. Among them, voice interaction, based on Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) technologies, enables natural spoken dialogue between customers and the system; while text interaction relies on Instant Messaging (IM) technology, allowing customers to convey their needs through text input, making it irreplaceable in scenarios requiring accurate recording or complex problem descriptions.

[0003] However, as customer needs evolve towards personalization and experiential approaches, especially in marketing scenarios requiring the delivery of complex information such as automotive product consultations, personalized configuration options, and test drive appointments, traditional modal interaction models are increasingly revealing significant limitations. They are failing to meet customers' demands for service accuracy and interaction efficiency. Specific problems include: existing multimodal interaction attempts are largely "simple overlays," lacking a deep collaborative mechanism. While some intelligent customer service systems attempt to support image uploads or structured menu selections, they fail to deeply integrate these new modalities with voice and text information—image data is stored only as independent attachments, and structured selection results are disconnected from the demand descriptions in voice / text. This still relies on human agents manually determining the relevance of each modal, failing to achieve automated demand insight and personalized recommendation loops. Consequently, the advantages of multimodal interaction are difficult to realize, and the system remains limited to a single-modal logical response framework.

[0004] Therefore, there is an urgent need for an intelligent customer service interaction solution that can achieve multimodal data collaborative collection, deep feature fusion and accurate intent recognition, in order to overcome the limitations of traditional single-modal interaction, meet customers' needs for complex information transmission and personalized services, and improve the service capabilities and conversion efficiency of intelligent customer service in marketing scenarios. Summary of the Invention

[0005] This application addresses the shortcomings of the prior art by providing a multimodal interaction method, apparatus, device, and storage medium for an intelligent customer service system, in order to solve the problems existing in the prior art.

[0006] The technical solution adopted in the embodiments of this application is as follows: In a first aspect, embodiments of this application provide a multimodal interaction method for an intelligent customer service system, including: Acquire vehicle consultation data from multiple modalities input by the target user; Feature extraction is performed on the vehicle consultation data of the multiple modalities to obtain multiple feature vectors corresponding to the vehicle consultation data of the multiple modalities; Based on the types of the multiple modalities and the current interaction scenario of the target user, obtain the weights of the multiple modalities in the current interaction scenario; Based on the weights of the multiple modalities, the multiple feature vectors are fused to obtain a fused feature vector; Based on the fused feature vector, a service plan is generated for the target user.

[0007] In one embodiment, the vehicle consultation data of the multiple modalities includes: call voice data and vehicle image data; the multiple feature vectors include: voice feature vectors and image feature vectors; The process of extracting features from the vehicle consultation data of the multiple modalities yields various feature vectors corresponding to the vehicle consultation data of the multiple modalities, including: The acoustic features of the call voice data are extracted, and the text recognition of the call voice data is performed to obtain text semantic features, resulting in a voice feature vector that includes the acoustic features and the text semantic features. A pre-defined convolutional neural network is used to extract the vehicle visual features from the vehicle image data, and the vehicle subject in the vehicle image data is enhanced based on the vehicle visual features to obtain an image feature vector.

[0008] In one embodiment, the vehicle consultation data of the multiple modalities further includes: structured form data and text data; the multiple feature vectors further include: structured feature vectors and text feature vectors; The step of extracting features from the vehicle consultation data of the multiple modalities to obtain multiple feature vectors corresponding to the vehicle consultation data of the multiple modalities also includes: The discrete data in the structured form data is converted into dense vectors using the embedding layer in a pre-defined large language model; The continuous data in the structured form data is normalized to obtain normalized data; the structured feature vector includes the dense vector and the normalized data; Keyword extraction is performed on the text data to obtain text feature vectors.

[0009] In one embodiment, obtaining the weights of the multiple modalities in the current interaction scenario based on the types of the multiple modalities and the current interaction scenario of the target user includes: Based on the types of the multiple modalities, the current interaction scenario of the target user, and the historical interaction behavior of the target user, a preset weight model is used to obtain the weights of the multiple modalities in the current interaction scenario.

[0010] In one embodiment, before fusing the multiple feature vectors according to the weights of the multiple modalities to obtain a fused feature vector, the method further includes: The features of the various feature vectors are batch normalized.

[0011] In one embodiment, generating a service plan for the target user based on the fused feature vector includes: Based on the fused feature vector, a preset intent recognition model is used to obtain the intent label of the target user in the current interaction scenario; Based on the intent tags, obtain the target key information in the current interaction scenario from the vehicle consultation data of the multiple modalities; Based on the target key information, a marketing service plan for the current interaction scenario is generated.

[0012] In one embodiment, if the current interaction scenario is a vehicle inquiry scenario, the target key information includes: multiple vehicle inquiry messages, and generating a marketing service plan for the current interaction scenario based on the target key information includes: Based on the multiple vehicle inquiry messages, a vehicle model recommendation report is generated as a marketing service solution for the vehicle inquiry scenario. or, If the current interaction scenario is a vehicle comparison scenario, the target key information includes: multiple vehicle comparison information entries. The step of generating a marketing service plan for the current interaction scenario based on the target key information includes: Based on the multiple vehicle comparison information, a vehicle comparison report is generated as a marketing service solution for the vehicle comparison scenario. or, If the current interaction scenario is a test drive reservation scenario, the target key information includes: the reserved test drive vehicle model and the user's location information. Generating a marketing service plan for the current interaction scenario based on the target key information includes: Based on the reserved test drive vehicle model and user location information, generate a vehicle test drive location that is closest to the target user and has the reserved test drive vehicle model, and generate a test drive invitation based on the information of the vehicle test drive location as a marketing service plan in the test drive reservation scenario.

[0013] Secondly, embodiments of this application provide a multimodal interaction device for an intelligent customer service system, comprising: The acquisition module is used to acquire vehicle information data in multiple modalities input by the target user; The extraction module is used to extract features from the vehicle consultation data of the multiple modalities to obtain multiple feature vectors corresponding to the vehicle consultation data of the multiple modalities. The weighting module is used to obtain the weights of the multiple modalities in the current interaction scenario based on the types of the multiple modalities and the current interaction scenario of the target user. The fusion module is used to fuse the multiple feature vectors according to the weights of the multiple modalities to obtain a fused feature vector; The generation module is used to generate a service plan for the target user based on the fused feature vector.

[0014] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to implement the multimodal interaction method of the intelligent customer service system described in any of the above embodiments.

[0015] Fourthly, embodiments of this application provide a readable storage medium storing program instructions, which, when executed by a processor, implement the multimodal interaction method of the intelligent customer service system described in any of the above embodiments.

[0016] The beneficial effects of this application are: This application provides a multimodal interaction method for an intelligent customer service system, which realizes the standardized processing and deep integration of multimodal data, solves the problems of inefficient response and inaccurate recommendations in traditional customer service, and ultimately achieves the goal of accurately understanding users' car purchase needs and efficiently outputting personalized marketing services, providing a feasible core solution for the digital upgrade of intelligent customer service in the automotive industry. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 One of the flowcharts illustrating the multimodal interaction method of the intelligent customer service system provided in this application embodiment; Figure 2 A second flowchart illustrating the multimodal interaction method of the intelligent customer service system provided in this application embodiment; Figure 3 The third flowchart illustrating the multimodal interaction method of the intelligent customer service system provided in this application embodiment; Figure 4 The fourth flowchart illustrating the multimodal interaction method of the intelligent customer service system provided in this application embodiment; Figure 5 A schematic diagram of the structure of the multimodal interaction device of the intelligent customer service system provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0020] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0023] This application provides a multimodal interaction method for an intelligent customer service system. This method can be generated by any electronic device with computing and processing capabilities. The electronic device can be, for example, a terminal-facing computer device or a backend server.

[0024] The following examples, in conjunction with the accompanying drawings, provide specific illustrations of the multimodal interaction method of the intelligent customer service system provided in this application.

[0025] Figure 1 This is one of the flowcharts illustrating the multimodal interaction method of the intelligent customer service system provided in the embodiments of this application, such as... Figure 1 As shown, the method includes: S101. Obtain vehicle consultation data in multiple modalities input by the target user.

[0026] A unified multimodal interaction portal is used to collect vehicle inquiry data from target users across multiple modalities. These portals include hotline customer service, online customer service (e.g., brand website / APP / H5 page), and mobile-related portals (e.g., official brand APP, WeChat mini-program).

[0027] The hotline customer service app only supports voice input. However, when users consult through the hotline, they can upload one or more of the following via the mobile app's entry point: vehicle image data, structured form data, and text data. The system automatically associates this with the target user's current session ID, ensuring that voice data is bound to other data. Structured form data refers to data entered through preset forms, drop-down menus, or configurators (such as vehicle configurators). Users can quickly select their budget range, preferred vehicle model, and key configurations (such as engine type, color, and seat material) using structured forms.

[0028] The online customer service client and mobile app support user submission of one or more of the following: call voice data, vehicle image data, structured form data, and text data. All collected data is automatically associated with the target user's current session ID and tagged according to the rule of "session ID + modality type + timestamp" to ensure data traceability and relevance.

[0029] S102. Perform feature extraction on vehicle consultation data of multiple modalities to obtain multiple feature vectors corresponding to vehicle consultation data of multiple modalities.

[0030] Feature extraction is performed on the collected vehicle consultation data of multiple modalities to generate multiple feature vectors corresponding to the multiple modalities.

[0031] For example, feature extraction is performed on speech data to obtain the corresponding speech feature vector; feature extraction is performed on image data to obtain the corresponding image feature vector; feature extraction is performed on structured form data to obtain the corresponding structured form feature vector; and feature extraction is performed on text data to obtain the corresponding text feature vector. Multiple feature vectors corresponding to multiple modalities include: speech feature vectors, image feature vectors, structured feature vectors, and text feature vectors.

[0032] Optionally, before extracting features from vehicle consultation data across multiple modalities to obtain various feature vectors corresponding to the vehicle consultation data across multiple modalities, the vehicle consultation data across multiple modalities can also be preprocessed. For example, background noise of the speech data can be processed using noise reduction algorithms (such as spectral subtraction), converted to PCM format at a sampling rate of 16k, and bound to the session ID; adaptive compression algorithms (such as WebP format) can be used to reduce the resolution of image data (while maintaining the clarity of key details), and vehicle features (such as vehicle model, color, and exterior parts) can be extracted using an image recognition model; structured form data can be parsed and converted into a unified feature vector, for example, converting "Budget: 200,000-300,000" into a numerical range feature and "Color preference: Sky gray" into a classification feature vector.

[0033] S103. Based on the types of multiple modalities and the target user's current interaction scenario, obtain the weights of multiple modalities in the current interaction scenario.

[0034] Based on a preset weight model, the input includes the types of multiple modalities, the current interaction scenario of the target user (such as vehicle consultation scenario, model comparison scenario, test drive appointment scenario), and the target user's historical interaction behavior (such as whether they repeatedly upload pictures of a certain type of vehicle or whether they use the configurator multiple times). The output is the weight coefficient of each modality in the current interaction scenario.

[0035] The weight model is generated through training on historical marketing interaction data. For example, in a car model comparison scenario, the weight of images is 0.5, the weight of structured data is 0.3, and the weight of voice is 0.2, ensuring that the weight allocation fits the needs of the scenario. In practical applications, the weights of various types of data can be adjusted according to user behavior. For example, if a user uploads a car picture and provides a detailed voice description, the weights of image data and voice data will be increased. If the user mainly uses the configurator, the weight of structured form data will be increased.

[0036] S104. Based on the weights of multiple modalities, perform feature fusion on multiple feature vectors to obtain a fused feature vector.

[0037] First, the various feature vectors obtained in S102 are subjected to batch normalization to reduce the differences in data distribution between modalities and avoid overfitting of single modal features. Then, based on the modal weights obtained in S103, the normalized feature vectors are weighted. Finally, all weighted modal feature vectors are integrated by feature concatenation to generate a unified fusion feature vector.

[0038] S105. Generate a service plan for the target user based on the fused feature vector.

[0039] Based on the fusion feature vector, targeted marketing service solutions are generated through preset intent recognition and service matching logic.

[0040] In summary, this embodiment provides a multimodal interaction method for an intelligent customer service system, which realizes standardized processing and deep integration of multimodal data, solves the problems of inefficient response and inaccurate recommendations in traditional customer service, and ultimately achieves the goal of accurately understanding users' car purchase needs and efficiently outputting personalized marketing services. It provides a feasible core solution for the digital upgrade of intelligent customer service in the automotive industry.

[0041] When vehicle consultation data in multiple modalities includes call voice data and vehicle image data, multiple feature vectors are used, including voice feature vectors and image feature vectors. Figure 2 The second flowchart illustrates the multimodal interaction method of the intelligent customer service system provided in this application embodiment. Figure 2 As shown, in this case, step S102 involves feature extraction from the vehicle consultation data across multiple modalities to obtain various feature vectors corresponding to the vehicle consultation data across multiple modalities, including: S201. Extract the acoustic features of the call speech data and perform text recognition on the call speech data to obtain text semantic features, resulting in a speech feature vector that includes acoustic features and text semantic features.

[0042] The Mel Frequency Cepstral Coefficient (MFCC) algorithm is used to extract acoustic features such as fundamental frequency, speech rate, and emotional tendency from call speech data. Then, ASR (Automatic Speech Recognition) technology is used to perform text recognition on the call speech data, and the recognition results are converted into text semantic features.

[0043] The acoustic features are combined with the semantic features of the text to form a complete speech feature vector.

[0044] S202. Using a pre-defined convolutional neural network, extract the vehicle visual features from the vehicle image data, and enhance the features of the vehicle subject in the vehicle image data based on the vehicle visual features to obtain the image feature vector.

[0045] A lightweight convolutional neural network (such as MobileNet) is selected as the preset model to process vehicle image data and extract vehicle visual features such as vehicle type, color, body lines, and grille style. Based on the vehicle visual features, key features are enhanced for the main vehicle area in the image to highlight key visual information, and finally, an image feature vector is generated.

[0046] When vehicle consultation data in multiple modalities includes structured form data and text data, multiple feature vectors also include structured feature vectors and text feature vectors. Figure 3 The third flowchart illustrates the multimodal interaction method of the intelligent customer service system provided in this application embodiment. Figure 3 As shown, in this case, step S102, which involves extracting features from vehicle consultation data across multiple modalities to obtain various feature vectors corresponding to the vehicle consultation data across multiple modalities, further includes: S301. The embedded layer in the preset large language model is used to convert the discrete data in the structured form data into dense vectors.

[0047] By using the embedding layer in the pre-defined large language model, discrete category data such as "vehicle type", "power type" and "seat material" in the structured form data are converted into dense vector form.

[0048] S302. Normalize the continuous data in the structured form data to obtain normalized data.

[0049] Continuous data such as "budget range" and "car purchase cycle" in structured form data are normalized to eliminate differences in data units and obtain normalized data.

[0050] Dense vectors are combined with normalized data to form structured feature vectors.

[0051] S303. Extract keywords from the text data to obtain text feature vectors.

[0052] Keyword extraction algorithms (such as TF-IDF) are used to process user-submitted text data (such as inquiry messages and requirement descriptions) to extract core keywords such as "SUV", "hybrid", and "panoramic sunroof", and text feature vectors are generated based on the keywords.

[0053] Figure 4 The fourth flowchart illustrates the multimodal interaction method of the intelligent customer service system provided in this application embodiment. Figure 4 As shown in step S105, generating a service plan for the target user based on the fused feature vector includes: S401. Based on the fused feature vector, a preset intent recognition model is used to obtain the intent label of the target user in the current interaction scenario.

[0054] The fused feature vector obtained from S104 is input into the pre-trained pre-set intent recognition model (such as a BERT+CNN hybrid model). The model calculates and matches the intent tags in the marketing knowledge base to determine the core needs of the target user in the current interaction scenario. Intent tags can be, for example, "consultation on mid-to-high-end hybrid SUVs", "model comparison", "test drive appointment", etc.

[0055] S402. Based on the intent tags, obtain the target key information in the current interaction scenario from vehicle consultation data of multiple modalities.

[0056] Based on the identified intent tags, core information in the current interaction scenario is filtered from vehicle consultation data across all modalities as the target key information.

[0057] For example, if the intent tag is vehicle consultation, extract vehicle consultation information such as budget range, model preference, and configuration requirements; if it is model comparison, extract model comparison information such as comparison models and focus dimensions; if it is test drive appointment, extract key information such as target model and user's geographical location.

[0058] S403. Based on the key target information, generate a marketing service plan for the current interaction scenario.

[0059] Finally, a marketing service plan for the current interaction scenario can be generated based on the target key information. Specifically, if the current interaction scenario is a vehicle inquiry scenario, the target key information includes multiple vehicle inquiry messages. Based on these multiple vehicle inquiry messages, a vehicle model recommendation report is generated as the marketing service plan for the vehicle inquiry scenario. For example, based on multiple vehicle inquiry messages, matching vehicle models are selected from the product database, and a vehicle model recommendation report containing vehicle images, core parameters, configuration details, and preferential information is generated.

[0060] Alternatively, if the current interaction scenario is a car model comparison scenario, the target key information includes multiple car model comparison information. Based on the multiple car model comparison information, the differences between different models in terms of power, space, configuration, price, etc. are sorted out, and a structured car model comparison report is generated as a marketing service solution in the car model comparison scenario. The car model comparison report includes the advantages and disadvantages analysis of each model and adaptation suggestions.

[0061] Alternatively, if the current interaction scenario is a test drive reservation scenario, the target key information includes the vehicle model to be reserved for test drive and the user's location information. Based on the vehicle model to be reserved for test drive and the user's location information, generate a test drive location that is closest to the target user and has the vehicle model to be reserved for test drive. Generate a test drive invitation containing the store address, contact number and available time slot as a marketing service solution in the test drive reservation scenario, and provide an online reservation portal.

[0062] Optionally, the method of this application further includes: if the data of a certain modality is incomplete (e.g., an image is not uploaded), the system automatically reduces the weight of that modality, continues to identify based on the remaining modality features, and guides the user to supplement it (e.g., reminding the user via voice or text, "Would it be convenient to upload a picture of a car model you are interested in so that we can recommend one to you more accurately?"); if the confidence of the intent matched by the fused features is lower than the threshold, the system generates a "candidate intent list" (e.g., "Do you want to know the latest offers for XX car model, or do you want to book a test drive?"), and further confirms it in combination with multimodal information; if the interaction is interrupted (e.g., network fluctuations, APP crash), the system restores the historical multimodal data based on the session ID, and the user can continue from the breakpoint after re-entering without repeating the operation.

[0063] The following will continue to explain the apparatus, device and storage medium for implementing the multimodal interaction method of the intelligent customer service system provided in any of the above embodiments of this application. The specific implementation process and the resulting technical effects are the same as those in the corresponding method embodiments. For the sake of brevity, the parts not mentioned in the following embodiments can be referred to the corresponding content in the method embodiments.

[0064] Figure 5 This is a schematic diagram of the structure of the multimodal interaction device of the intelligent customer service system provided in the embodiments of this application, as shown below. Figure 5 As shown, this application also provides a multimodal interaction device for an intelligent customer service system, comprising: The acquisition module 10 is used to acquire vehicle consultation data in multiple modalities input by the target user.

[0065] The extraction module 20 is used to extract features from the vehicle consultation data of the multiple modalities to obtain multiple feature vectors corresponding to the vehicle consultation data of the multiple modalities.

[0066] The weighting module 30 is used to obtain the weights of the multiple modalities in the current interaction scenario based on the types of the multiple modalities and the current interaction scenario of the target user.

[0067] The fusion module 40 is used to perform feature fusion on the multiple feature vectors according to the weights of the multiple modalities to obtain a fused feature vector.

[0068] The generation module 50 is used to generate a service plan for the target user based on the fused feature vector.

[0069] Optionally, the multiple modalities of vehicle consultation data include: call voice data and vehicle image data; the multiple feature vectors include: voice feature vectors and image feature vectors. The extraction module 20 is further configured to extract the acoustic features of the call voice data, perform text recognition on the call voice data to obtain text semantic features, and obtain a voice feature vector including the acoustic features and the text semantic features; and use a preset convolutional neural network to extract the vehicle visual features of the vehicle image data, and perform feature enhancement on the vehicle subject in the vehicle image data based on the vehicle visual features to obtain an image feature vector.

[0070] Optionally, the vehicle consultation data of multiple modalities further includes: structured form data and text data; the multiple feature vectors further include: structured feature vectors and text feature vectors. The extraction module 20 is further configured to convert the discrete data in the structured form data into dense vectors using an embedding layer in a preset large language model; normalize the continuous data in the structured form data to obtain normalized data; the structured feature vectors include the dense vectors and the normalized data; and extract keywords from the text data to obtain text feature vectors.

[0071] Optionally, the weighting module 30 is further configured to obtain the weights of the multiple modalities in the current interaction scenario by using a preset weighting model based on the types of the multiple modalities, the current interaction scenario of the target user, and the historical interaction behavior of the target user.

[0072] Optionally, the fusion module 40 is also used to perform batch normalization of the features of the multiple feature vectors.

[0073] Optionally, the generation module 50 is further configured to obtain the target user's intent tag in the current interaction scenario by using a preset intent recognition model based on the fused feature vector; obtain target key information in the current interaction scenario from the vehicle consultation data of the multiple modalities based on the intent tag; and generate a marketing service plan in the current interaction scenario based on the target key information.

[0074] Optionally, if the current interaction scenario is a vehicle consultation scenario, the target key information includes: multiple vehicle consultation information. The generation module 50 is also used to generate a vehicle model recommendation report based on the multiple vehicle consultation information as a marketing service plan for the vehicle consultation scenario.

[0075] If the current interaction scenario is a vehicle model comparison scenario, the target key information includes: multiple vehicle model comparison information. The generation module 50 is also used to generate a vehicle model comparison report as a marketing service plan under the vehicle model comparison scenario based on the multiple vehicle model comparison information.

[0076] If the current interaction scenario is a test drive reservation scenario, the target key information includes: the reserved test drive vehicle model and the user's location information. The generation module 50 is also used to generate a vehicle test drive location that is closest to the target user and has the reserved test drive vehicle model based on the reserved test drive vehicle model and the user's location information, and generate a test drive invitation based on the information of the vehicle test drive location as a marketing service plan under the test drive reservation scenario.

[0077] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0078] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0079] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown in the figure, this application embodiment also provides an electronic device, including a processor 100, a storage medium 200 and a bus 300. The storage medium stores program instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the program instructions to realize the multimodal interaction method of the intelligent customer service system described in any of the above embodiments.

[0080] This application also provides a readable storage medium storing program instructions, which, when executed by a processor, implement the multimodal interaction method of the intelligent customer service system described in any of the above embodiments.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0084] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal interaction method for an intelligent customer service system, characterized in that, include: Acquire vehicle consultation data from multiple modalities input by the target user; Feature extraction is performed on the vehicle consultation data of the multiple modalities to obtain multiple feature vectors corresponding to the vehicle consultation data of the multiple modalities; Based on the types of the multiple modalities and the current interaction scenario of the target user, obtain the weights of the multiple modalities in the current interaction scenario; Based on the weights of the multiple modalities, the multiple feature vectors are fused to obtain a fused feature vector; Based on the fused feature vector, a service plan is generated for the target user.

2. The method according to claim 1, characterized in that, The vehicle consultation data in multiple modalities includes: call voice data and vehicle image data; the multiple feature vectors include: voice feature vectors and image feature vectors; The process of extracting features from the vehicle consultation data of the multiple modalities yields various feature vectors corresponding to the vehicle consultation data of the multiple modalities, including: The acoustic features of the call voice data are extracted, and the text recognition of the call voice data is performed to obtain text semantic features, resulting in a voice feature vector that includes the acoustic features and the text semantic features. A pre-defined convolutional neural network is used to extract the vehicle visual features from the vehicle image data, and the vehicle subject in the vehicle image data is enhanced based on the vehicle visual features to obtain an image feature vector.

3. The method according to claim 2, characterized in that, The vehicle consultation data in the multiple modalities also includes: structured form data and text data; the multiple feature vectors also include: structured feature vectors and text feature vectors; The step of extracting features from the vehicle consultation data of the multiple modalities to obtain multiple feature vectors corresponding to the vehicle consultation data of the multiple modalities also includes: The discrete data in the structured form data is converted into dense vectors using the embedding layer in a pre-defined large language model; The continuous data in the structured form data is normalized to obtain normalized data; the structured feature vector includes the dense vector and the normalized data; Keyword extraction is performed on the text data to obtain text feature vectors.

4. The method according to claim 1, characterized in that, The step of obtaining the weights of the multiple modalities in the current interaction scenario based on the types of the multiple modalities and the current interaction scenario of the target user includes: Based on the types of the multiple modalities, the current interaction scenario of the target user, and the historical interaction behavior of the target user, a preset weight model is used to obtain the weights of the multiple modalities in the current interaction scenario.

5. The method according to claim 1, characterized in that, Before fusing the multiple feature vectors according to the weights of the multiple modalities to obtain the fused feature vector, the method further includes: The features of the various feature vectors are batch normalized.

6. The method according to claim 1, characterized in that, The step of generating a service plan for the target user based on the fused feature vector includes: Based on the fused feature vector, a preset intent recognition model is used to obtain the intent label of the target user in the current interaction scenario; Based on the intent tags, obtain the target key information in the current interaction scenario from the vehicle consultation data of the multiple modalities; Based on the target key information, a marketing service plan for the current interaction scenario is generated.

7. The method according to claim 6, characterized in that, If the current interaction scenario is a vehicle inquiry scenario, the target key information includes: multiple vehicle inquiry messages. The step of generating a marketing service plan for the current interaction scenario based on the target key information includes: Based on the multiple vehicle inquiry messages, a vehicle model recommendation report is generated as a marketing service solution for the vehicle inquiry scenario. or, If the current interaction scenario is a vehicle comparison scenario, the target key information includes: multiple vehicle comparison information entries. The step of generating a marketing service plan for the current interaction scenario based on the target key information includes: Based on the multiple vehicle comparison information, a vehicle comparison report is generated as a marketing service solution for the vehicle comparison scenario. or, If the current interaction scenario is a test drive reservation scenario, the target key information includes: the reserved test drive vehicle model and the user's location information. Generating a marketing service plan for the current interaction scenario based on the target key information includes: Based on the reserved test drive vehicle model and user location information, generate a vehicle test drive location that is closest to the target user and has the reserved test drive vehicle model, and generate a test drive invitation based on the information of the vehicle test drive location as a marketing service plan in the test drive reservation scenario.

8. A multimodal interaction device for an intelligent customer service system, characterized in that, include: The acquisition module is used to acquire vehicle information data in multiple modalities input by the target user; The extraction module is used to extract features from the vehicle consultation data of the multiple modalities to obtain multiple feature vectors corresponding to the vehicle consultation data of the multiple modalities. The weighting module is used to obtain the weights of the multiple modalities in the current interaction scenario based on the types of the multiple modalities and the current interaction scenario of the target user. The fusion module is used to fuse the multiple feature vectors according to the weights of the multiple modalities to obtain a fused feature vector; The generation module is used to generate a service plan for the target user based on the fused feature vector.

9. An electronic device, characterized in that, include: The system includes a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to implement the multimodal interaction method of the intelligent customer service system according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores program instructions, which, when executed by a processor, implement the multimodal interaction method of the intelligent customer service system according to any one of claims 1 to 7.

Images