Multi-modal product service application method, system, equipment and medium

Abstract

The invention discloses a multi-modal product service application method, system, equipment and medium, belongs to the technical field of software application, and aims to solve the technical problem of how to enhance the processing efficiency of a product service system in a multi-information scene and enhance the interactive experience and satisfaction of a user. According to the technical scheme, Llama3 is adopted as a basic large language base model, the Llama3 is finely adjusted through the LLaVA technology to adapt to multi-modal tasks, further fine adjustment is conducted according to professional knowledge in the product service field, and diagnosis is conducted according to pictures; the method comprises the following steps: data acquisition; processing data; evaluating and classifying; and training the Llama3 model.

Description

Technical Field

[0001] The present invention relates to the field of software application technology, and in particular to a method, system, device and medium based on multimodal product service application. Background Art

[0002] Traditional product service models generally rely on manual customer service operations. When the user base is large, service request overloads are frequently encountered, which creates a huge workload for customer service personnel and significantly reduces the user experience quality due to extended waiting time. In order to effectively solve the current situation of heavy manual customer service work orders, intelligent customer service technology is experiencing rapid development. Despite this, most existing technologies are limited to text interaction and fail to fully respond to complex requirements involving images and videos, such as applications in automatic quality inspection, resulting in these tasks still relying on manual review.

[0003] For multimodal interaction scenarios containing image information, several innovative attempts have been made to incorporate image processing technology to build an intelligent customer service system with multimodal processing capabilities. However, existing technologies have shown limitations when processing image information, especially relying on OCR technology to only extract text while ignoring key visual information such as the location of objects; the process of converting images into text also leads to the loss of detailed information such as the order of objects, light and shadow effects, etc. In application scenarios such as product service quality control, these details are crucial for identifying surface defects of objects, and the current level of technology is not sufficient to fully meet such refined processing needs.

[0004] Therefore, how to enhance the processing efficiency of product service systems in multiple information scenarios and enhance users' interactive experience and satisfaction is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical task of the present invention is to provide a method, system, device and medium based on multimodal product service application to solve the problem of how to enhance the processing efficiency of the product service system in multiple information scenarios and enhance the user's interactive experience and satisfaction.

[0006] The technical task of the present invention is achieved in the following way: a method based on multimodal product service application, which adopts Llama3 as the large language base model, and uses LLaVA technology to fine-tune Llama3 to adapt to multimodal tasks, and further fine-tune it according to professional knowledge in the product service field to achieve diagnosis based on pictures; the details are as follows:

[0007] Data collection;

[0008] Data processing;

[0009] Assessment classification;

[0010] Llama3 model training.

[0011] As a preferred method, the data collection method is as follows:

[0012] Collect diverse image resources through readily available large datasets;

[0013] Through the real records of the current customer service support system, we focus on pictures of typical scenarios of optical modem quality inspection and supporting document review.

[0014] As a preference, the data processing is specifically as follows:

[0015] Perform pre-processing operations on the collected data, including data format understanding, data segmentation, data enhancement, data loader creation, script writing, and label mapping, and save the pre-processed text data;

[0016] Llama3's named entity recognition technology is used to extract key information from text data. The key information includes city, problem category, terminal type, fault cause, and treatment plan, and defines image ID, category ID, bounding box coordinates, and key point information.

[0017] As a preference, the evaluation categories are as follows:

[0018] The processed data is divided into training set, validation set and test set. The training set is used to train the Llama3 model, the validation set is used to adjust the parameters of the Llama3 model, and the test set is used to evaluate the final performance of the Llama3 model.

[0019] Correct the errors in the data based on the training, validation and testing results; errors in the data include inaccurate labels, missing details or illogical images;

[0020] or,

[0021] Ensure the comprehensiveness and balance of data through multi-angle review.

[0022] Preferably, the Llama3 model training is as follows:

[0023] For the input image X v , using the pre-trained visual encoder ViT-L / 14, the visual encoder ViT-L / 14 provides visual features Z v =g(X v );

[0024] Use a simple linear layer to map image features into word vector space;

[0025] Use a trainable projection matrix W to transform Z v Converted into language-encoded token H v ;

[0026] For each input image, generate multiple rounds of dialogue data Among them, the instructions in the tth round of dialogue It is expressed as:

[0027]

[0028] Target Answer X a To be answered with the highest possible probability, the following probability formula is optimized during training:

[0029]

[0030] Where L is the sequence length, X instruct,<i ,X a,<i are the instructions and answers before the current prediction token, respectively, and θ is the model parameter.

[0031] Better yet, the Llama3 model training process integrates the initial learning and specialized optimization of the Llama3 model, covering the entire process from generalized pre-training to refined fine-tuning;

[0032] In the pre-training phase, we created concise picture description question-answering examples to lay a solid cognitive foundation for the Llama3 model;

[0033] After fine-tuning the dataset, we used actual interaction data from the customer service support system to conduct domain adaptability training, expand and diversify the training data, and further enhance the generalization ability and deep understanding of the Llama3 model.

[0034] A multimodal product service application system, the system comprising:

[0035] The data collection module is used to collect diverse image resources through existing large data sets and real records from the current customer service support system, focusing on pictures of typical scenes for optical modem quality inspection and supporting document review;

[0036] The data processing module is used to perform pre-processing operations on the collected data, including data format understanding, data partitioning, data enhancement, data loader creation, script writing, and label mapping. It saves the pre-processed text data and uses Llama3's named entity recognition technology to extract key information from the text data. The key information includes cities, problem categories, terminal types, fault reporting reasons, and treatment plans. It also defines image IDs, category IDs, bounding box coordinates, and key point information.

[0037] The evaluation classification module is used to divide the processed data into training set, validation set and test set. The training set is used to train the Llama3 model, the validation set is used to adjust the parameters of the Llama3 model, and the test set is used to evaluate the final performance of the Llama3 model. According to the training, validation and test results, the errors in the data are corrected. The errors in the data include inaccurate labels, missing details or illogical pictures. Or the comprehensiveness and balance of the data are ensured through multi-angle review.

[0038] Training module, used to train the Llama3 model.

[0039] As a preferred method, the process of training the model is as follows:

[0040] ①For the input image X v , using the pre-trained visual encoder ViT-L / 14, the visual encoder ViT-L / 14 provides visual features Z v =g(X v );

[0041] ② Use a simple linear layer to map image features into word vector space;

[0042] ③ Use a trainable projection matrix W to transform Z v Converted into language-encoded token H v ;

[0043] ④For each input image, generate multiple rounds of dialogue data Among them, the instructions in the tth round of dialogue It is expressed as:

[0044]

[0045] ⑤Target answer X a To be answered with the highest possible probability, the following probability formula is optimized during training:

[0046]

[0047] Where L is the sequence length, X instruct,<i ,X a,<i are the instructions and answers before the current prediction token, respectively, and θ is the model parameter;

[0048] The model training process integrates the initial learning and special optimization of the Llama3 model, covering the entire process from general pre-training to refined fine-tuning. In the pre-training stage, a solid cognitive foundation is laid for the Llama3 model by creating concise picture description question-and-answer examples. After completing the fine-tuning of the data set, the actual interaction data from the customer service support system is used for domain adaptability training, and the training data is expanded and diversified to further enhance the generalization ability and deep understanding of the Llama3 model.

[0049] Among them, the Llama3 model has an 8K context processing capacity and reasoning capabilities close to GPT4, providing a strong foundation for the intelligent customer service system to more accurately parse and respond to complex user interactions and ensure efficient personalization of the service experience.

[0050] Large multimodal models are a cutting-edge technology in the current field of artificial intelligence. By integrating data from different modalities (such as text, images, audio and video), they can achieve more complex and accurate information processing capabilities. Such models use deep learning technology and learn how to interpret and generate cross-modal content through large-scale data training. For example, multimodal models can analyze the content in an image and generate corresponding text descriptions, or understand and respond to questions about the content of an image raised in natural language.

[0051] At the application level, large multimodal models have been widely used in a variety of scenarios, including but not limited to automatic image annotation, virtual assistants, and specific applications in education and medicine. These models not only improve processing efficiency, but also enhance the machine's ability to understand human language and visual information. With the advancement of technology, such models are expected to play a key role in more industries in the future, such as providing safer and more effective technical support in areas such as autonomous driving and remote medical diagnosis.

[0052] An electronic device comprising: a memory and at least one processor;

[0053] Wherein, the memory stores a computer program;

[0054] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the multimodal product service application method as described above.

[0055] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the multimodal product service application method as described above.

[0056] The multimodal product service application method, system, device and medium of the present invention have the following advantages:

[0057] (i) The present invention significantly reduces the number of manual work orders generated by image recognition requirements, especially in basic operations such as quality inspection, showing an accuracy of up to 97%;

[0058] (ii) The present invention successfully achieved a slight increase in model performance by adopting a data enhancement strategy for error cases and performing additional fine-tuning on the model accordingly. This not only greatly reduced the burden on customer service personnel in image analysis, but also effectively relieved the pressure on the customer service system, marking a major leap forward in customer service efficiency and automated processing capabilities;

[0059] (III) The present invention constructs a suitable multimodal processing framework by fine-tuning the visual instructions of the high-performance model Llama3, seamlessly integrating image and text information processing capabilities. In the example of quality inspection of optical modem patches for home customer service, the accuracy rate on the test set is as high as 97%, which verifies its excellent performance in image processing tasks and marks an important step towards an efficient and automated customer service system.

[0060] (iv) The present invention aims to enhance the flexibility and adaptability of the model in multimodal interaction scenarios and simultaneously improve the overall quality of experience of end users;

[0061] (V) The present invention uses the Llama3 model, which is known for its excellent 8K context processing capacity and reasoning performance comparable to GPT4, and implements a visual instruction fine-tuning strategy to precisely adjust Llama3 so that it can effectively parse image data, and then in specific product service fields, implement deep domain adaptive fine-tuning on the multimodal model that has completed preliminary fine-tuning, so that it can accurately analyze and judge the characteristics of each product, thereby enhancing the processing performance of the product service system in multiple information scenarios, and significantly enhancing the user's interactive experience and satisfaction;

[0062] (VI) The present invention uses the Llama3 large language model, combined with visual instruction fine-tuning technology, to accurately process multimodal information involving images, meet user quality inspection and other requirements, and for the first time applies a general large multimodal model to the field of intelligent customer service, and finds adversarial samples that make the model less effective. The idea of ​​further optimizing the customer service system should be protected;

[0063] (VII) The present invention fine-tunes Llama3’s visual instructions to achieve efficient processing of image data by the model, and through the ingenious combination of the visual encoder and the linear mapping layer, the image features are cleverly converted into word vectors that are compatible with the language model, simplifying the processing flow of image information;

[0064] (VIII) The present invention innovatively integrates the top large language model technology and advanced visual command fine-tuning strategy, bringing unprecedented multimodal dialogue processing capabilities to the field of intelligent customer service; the core adopts Llama3, a flagship large language model, which is pre-trained based on a huge data set of 15 trillion bytes, and the pre-training scale is seven times that of its predecessor open source model Llama2; Llama3 has been specially optimized and designed for dialogue interaction. In standard tests of multiple industries, its performance has surpassed many current open source chat models, especially when dealing with complex and long dialogue scenarios. Its excellent fine-tuning adaptability and generalization capabilities are undoubtedly demonstrated, providing customized solutions for the product service field. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention is further described below in conjunction with the accompanying drawings.

[0066] Attached Figure 1 Schematic diagram of the Llama3 model framework. DETAILED DESCRIPTION

[0067] The following detailed description is made of the multimodal product service application method, system, device and medium of the present invention with reference to the accompanying drawings and specific embodiments.

[0068] Embodiment 1:

[0069] This embodiment provides a multimodal product service application method, which uses Llama3 as a large language base model, and uses LLaVA technology to fine-tune Llama3 to adapt to multimodal tasks, and further fine-tune it according to professional knowledge in the product service field to achieve diagnosis based on pictures; the details are as follows:

[0070] S1, data collection;

[0071] S2, data processing;

[0072] S3, evaluation classification;

[0073] S4. Llama3 model training.

[0074] The data collection method in step S1 of this embodiment is as follows:

[0075] Collect diverse image resources through readily available large datasets;

[0076] Through the real records of the current customer service support system, we focus on pictures of typical scenarios of optical modem quality inspection and supporting document review.

[0077] The data processing in step S2 of this embodiment is specifically as follows:

[0078] S201, performing pre-processing data operations such as data format understanding, data partitioning, data enhancement, data loader creation, script writing, and label mapping on the collected data, and saving the pre-processed text data;

[0079] S202. Use Llama3's named entity recognition technology to extract key information from text data. The key information includes city, problem category, terminal type, cause of fault reporting, and treatment plan. Define image ID, category ID, bounding box coordinates, and key point information.

[0080] The evaluation classification in step S3 of this embodiment is specifically as follows:

[0081] S301, dividing the processed data into a training set, a validation set and a test set, wherein the training set is used for training the Llama3 model, the validation set is used for adjusting the parameters of the Llama3 model, and the test set is used for evaluating the final performance of the Llama3 model;

[0082] S302, correcting errors in the data based on the training, verification and testing results; wherein the errors in the data are inaccurate labels, missing details or illogical pictures;

[0083] or,

[0084] Ensure the comprehensiveness and balance of data through multi-angle review.

[0085] The Llama3 model training in step S4 of this embodiment is specifically as follows:

[0086] S401: For the input image X v , using the pre-trained visual encoder ViT-L / 14, the visual encoder ViT-L / 14 provides visual features Z v =g(X v );

[0087] S402, use a simple linear layer to map the image features into the word vector space;

[0088] S403, use a trainable projection matrix W to transform Z v Converted into language-encoded token H v ;

[0089] S404: Generate multiple rounds of dialogue data for each input image Among them, the instructions in the tth round of dialogue It is expressed as:

[0090]

[0091] S405, target answer Xa To be answered with the highest possible probability, the following probability formula is optimized during training:

[0092]

[0093] Where L is the sequence length, X instruct,<i ,X a,<i are the instructions and answers before the current prediction token, respectively, and θ is the model parameter.

[0094] The Llama3 model training process in this embodiment integrates the initial learning and special optimization of the Llama3 model, covering the entire process from generalized pre-training to refined fine-tuning;

[0095] In the pre-training phase, we created concise picture description question-answering examples to lay a solid cognitive foundation for the Llama3 model;

[0096] After fine-tuning the dataset, we used actual interaction data from the customer service support system to conduct domain adaptability training, expand and diversify the training data, and further enhance the generalization ability and deep understanding of the Llama3 model.

[0097] Embodiment 2:

[0098] This embodiment provides a multi-modal product service application system, which includes:

[0099] The data collection module is used to collect diverse image resources through existing large data sets and real records from the current customer service support system, focusing on pictures of typical scenes for optical modem quality inspection and supporting document review;

[0100] The data processing module is used to perform pre-processing operations on the collected data, including data format understanding, data partitioning, data enhancement, data loader creation, script writing, and label mapping. It saves the pre-processed text data and uses Llama3's named entity recognition technology to extract key information from the text data. The key information includes cities, problem categories, terminal types, fault reporting reasons, and treatment plans. It also defines image IDs, category IDs, bounding box coordinates, and key point information.

[0101] The evaluation classification module is used to divide the processed data into training set, validation set and test set. The training set is used to train the Llama3 model, the validation set is used to adjust the parameters of the Llama3 model, and the test set is used to evaluate the final performance of the Llama3 model. According to the training, validation and test results, the errors in the data are corrected. The errors in the data include inaccurate labels, missing details or illogical pictures. Or the comprehensiveness and balance of the data are ensured through multi-angle review.

[0102] Training module, used to train the Llama3 model.

[0103] As attached Figure 1 As shown, the process of training the model in this embodiment is as follows:

[0104] ①For the input image X v , using the pre-trained visual encoder ViT-L / 14, the visual encoder ViT-L / 14 provides visual features Z v =g(X v );

[0105] ② Use a simple linear layer to map image features into word vector space;

[0106] ③ Use a trainable projection matrix W to transform Z v Converted into language-encoded token H v ;

[0107] ④For each input image, generate multiple rounds of dialogue data Among them, the instructions in the tth round of dialogue It is expressed as:

[0108]

[0109] ⑤Target answer X a To be answered with the highest possible probability, the following probability formula is optimized during training:

[0110]

[0111] Where L is the sequence length, X instruct,<i ,X a,<i are the instructions and answers before the current prediction token, respectively, and θ is the model parameter;

[0112] The model training process integrates the initial learning and special optimization of the Llama3 model, covering the entire process from general pre-training to refined fine-tuning. In the pre-training stage, a solid cognitive foundation is laid for the Llama3 model by creating concise picture description question-and-answer examples. After completing the fine-tuning of the data set, the actual interaction data from the customer service support system is used for domain adaptability training, and the training data is expanded and diversified to further enhance the generalization ability and deep understanding of the Llama3 model.

[0113] Embodiment 3:

[0114] This embodiment also provides an electronic device, including: a memory and a processor;

[0115] Wherein, the memory stores computer-executable instructions;

[0116] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the multimodal product service application method in any embodiment of the present invention.

[0117] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0118] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0119] Embodiment 4:

[0120] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the multimodal product service application method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0121] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0122] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0123] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0124] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for applying multimodal product services, characterized in that: This method uses Llama3 as the basic large language base model, and uses LLaVA technology to fine-tune Llama3 to adapt to multimodal tasks, and further fine-tune it based on product service domain expertise to achieve diagnosis based on images; the details are as follows: Data collection; Data processing; Assessment classification; Llama3 model training.

2. The multimodal product service application method according to claim 1, characterized in that: The data collection methods are as follows: Collect diverse image resources through readily available large datasets; Through the real records of the current customer service support system, we focus on pictures of typical scenarios of optical modem quality inspection and supporting document review.

3. The multimodal product service application method according to claim 1, characterized in that: The data processing is as follows: Perform pre-processing operations on the collected data, including data format understanding, data segmentation, data enhancement, data loader creation, script writing, and label mapping, and save the pre-processed text data; Llama3's named entity recognition technology is used to extract key information from text data. The key information includes city, problem category, terminal type, fault cause, and treatment plan, and defines image ID, category ID, bounding box coordinates, and key point information.

4. The multimodal product service application method according to claim 1, characterized in that: The assessment categories are as follows: The processed data is divided into training set, validation set and test set. The training set is used to train the Llama3 model, the validation set is used to adjust the parameters of the Llama3 model, and the test set is used to evaluate the final performance of the Llama3 model. Correct the errors in the data based on the training, validation and testing results; errors in the data include inaccurate labels, missing details or illogical images; or, Ensure the comprehensiveness and balance of data through multi-angle review.

5. The multimodal product service application method according to any one of claims 1 to 4, characterized in that: Llama3 model training is as follows: For the input image X v , using the pre-trained visual encoder ViT-L / 14, the visual encoder ViT-L / 14 provides visual features Z v =g(X v ); Use a simple linear layer to map image features into word vector space; Use a trainable projection matrix W to transform Z v Converted into language-encoded token H v ; For each input image, generate multiple rounds of dialogue data Among them, the instructions in the tth round of dialogue It is expressed as: Target Answer X a To be answered with the highest possible probability, the following probability formula is optimized during training: Where L is the sequence length, X instruct，<i , X a，＜i are the instructions and answers before the current prediction token, respectively, and θ is the model parameter.

6. The multimodal product service application method according to claim 5, characterized in that: The Llama3 model training process integrates the initial learning and specialized optimization of the Llama3 model, covering the entire process from generalized pre-training to refined fine-tuning; In the pre-training phase, we created concise picture description question-answering examples to lay a solid cognitive foundation for the Llama3 model; After fine-tuning the dataset, we used actual interaction data from the customer service support system to conduct domain adaptability training, expand and diversify the training data, and further enhance the generalization ability and deep understanding of the Llama3 model.

7. A multimodal product service application system, characterized in that: The system includes: The data collection module is used to collect diverse image resources through existing large data sets and real records from the current customer service support system, focusing on pictures of typical scenes of optical modem quality inspection and supporting document review; The data processing module is used to perform pre-processing operations on the collected data, including data format understanding, data partitioning, data enhancement, data loader creation, script writing, and label mapping. It saves the pre-processed text data and uses Llama3's named entity recognition technology to extract key information from the text data. The key information includes cities, problem categories, terminal types, fault reporting reasons, and processing solutions. It also defines image IDs, category IDs, bounding box coordinates, and key point information. The evaluation classification module is used to divide the processed data into training set, validation set and test set. The training set is used to train the Llama3 model, the validation set is used to adjust the parameters of the Llama3 model, and the test set is used to evaluate the final performance of the Llama3 model. According to the training, validation and test results, the errors in the data are corrected. The errors in the data include inaccurate labels, missing details or illogical pictures. Or the comprehensiveness and balance of the data are ensured through multi-angle review. Training module, used to train the Llama3 model.

8. The multimodal product service application system according to claim 7, characterized in that: The process of training the model is as follows: ①For the input image X v , using the pre-trained visual encoder ViT-L / 14, the visual encoder ViT-L / 14 provides visual features Z v =g(X v ); ② Use a simple linear layer to map image features into word vector space; ③ Use a trainable projection matrix W to transform Z v Converted into language-encoded token H v ; ④For each input image, generate multiple rounds of dialogue data Among them, the instructions in the tth round of dialogue It is expressed as: ⑤Target answer X a To be answered with the highest possible probability, the following probability formula is optimized during training: Where L is the sequence length, X instruct，<i , X a，<i are the instructions and answers before the current prediction token, respectively, and θ is the model parameter; The model training process integrates the initial learning and special optimization of the Llama3 model, covering the entire process from general pre-training to refined fine-tuning. In the pre-training stage, a solid cognitive foundation is laid for the Llama3 model by creating concise picture description question-and-answer examples. After completing the fine-tuning of the data set, the actual interaction data from the customer service support system is used for domain adaptability training, and the training data is expanded and diversified to further enhance the generalization ability and deep understanding of the Llama3 model.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the multimodal product service application method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the multimodal product service application method as described in any one of claims 1 to 6.

Images