Question and answer information generation method and device, computer equipment and readable storage medium

Abstract

The invention provides a question and answer information generation method and device, computer equipment and a readable storage medium, and the method comprises the steps: obtaining question information to be answered, the question information at least comprising first image information; performing image processing on the first image information to obtain second image information corresponding to the first image information; the resolution of the second image information is lower than that of the first image information; inputting the first image information to a pre-trained first encoder, and obtaining feature map information corresponding to the first image information; inputting the second image information into a pre-trained second encoder, and obtaining visual embedding information corresponding to the second image information; querying in the feature map information according to the visual embedding information so as to obtain related information corresponding to the visual embedding information in the feature map information based on a self-attention mechanism; and answer information corresponding to the question information is generated according to the related information, and the question information is answered.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a method, apparatus, computer device and readable storage medium for generating question and answer information. Background Art

[0002] With the rapid development of Internet technology, the application of intelligent customer service systems in various industries has gradually become popular. Traditional intelligent customer service systems mainly rely on the processing of text information, using natural language processing (NLP) technology to understand users' questions and provide corresponding answers. Nowadays, when users interact with customer service systems, they often upload multimodal data such as images and videos to express their needs or questions more intuitively.

[0003] However, traditional intelligent customer service still has some limitations, such as single function, limited automation level, insufficient ability to respond to emergencies, limited personalized service capabilities, user experience needs to be improved, and high labor costs. These problems limit the performance of intelligent customer service in handling complex businesses and meeting diverse customer needs.

[0004] Therefore, there is an urgent need for a question-and-answer information generation method for multimodal data to improve the performance of the intelligent customer service to which the method is applied. Summary of the invention

[0005] This application provides a question-answer information generation method, device, computer equipment and readable storage medium, which aims to solve some limitations of traditional intelligent customer service, such as single function, limited automation level, insufficient ability to deal with emergencies, limited personalized service capabilities, user experience needs to be improved, and high labor costs. These problems limit the performance of intelligent customer service in handling complex business and meeting diverse customer needs.

[0006] In a first aspect, the present application provides a method for generating question and answer information, comprising:

[0007] Acquire question information to be answered, where the question information at least includes first image information;

[0008] Performing image processing on the first image information to obtain second image information corresponding to the first image information; the resolution of the second image information is lower than that of the first image information;

[0009] Inputting the first image information into a pre-trained first encoder to obtain feature map information corresponding to the first image information;

[0010] Inputting the second image information into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information;

[0011] Query the feature map information according to the visual embedding information to obtain relevant information corresponding to the visual embedding information in the feature map information based on the self-attention mechanism;

[0012] Generate answer information corresponding to the question information based on relevant information to complete the answer to the question information.

[0013] In some embodiments, the resolution of the first image information is higher than a preset resolution; performing image processing on the first image information to obtain second image information corresponding to the first image information includes: performing bilinear interpolation on the first image information to obtain the second image information.

[0014] In some embodiments, the first encoder is a convolutional neural network encoder; the first image information is input into a pre-trained first encoder to obtain feature map information corresponding to the first image information, including: inputting the first image information into a convolutional neural network encoder, the convolutional neural network encoder performs feature extraction on the first image information, obtains feature information of multiple convolution stages, and upsamples multiple feature information at a preset scale to obtain feature map information.

[0015] In some embodiments, the second encoder is a visual transformation encoder; the second image information is input into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information, including: inputting the second image information into the visual transformation encoder, the visual change encoder divides the second image information into multiple first regions and generates position embedding information corresponding to each first region, so as to generate visual embedding information according to each first region and the corresponding position embedding information.

[0016] In some embodiments, a query is performed in the feature map information according to the visual embedding information to obtain relevant information corresponding to the visual embedding information in the feature map information based on a self-attention mechanism, including: obtaining the region dimension corresponding to the visual embedding information; expanding the feature map according to the region dimension so that the feature map dimension of the expanded feature map matches the region dimension; querying each second region in the feature map information according to each first region in the visual embedding information respectively to query relevant regions corresponding to each first region in multiple second regions; and generating relevant information based on multiple relevant regions.

[0017] In some embodiments, the question information also includes text information corresponding to the first image information; generating answer information corresponding to the question information based on relevant information includes: inputting the text information and the question information into a preset autoregressive model, the autoregressive model parses the text information and the question information, and generates answer information.

[0018] In some embodiments, after generating answer information corresponding to the question information based on relevant information and completing the answer to the question information, it also includes: obtaining user information corresponding to the question information; the user information includes at least basic information and historical interaction information; generating service information based on the user information and the answer information; the service information includes at least any one of product recommendation information and service recommendation information; sending the service information to the user terminal corresponding to the question information to complete the service recommendation to the user terminal.

[0019] In a second aspect, the present application provides a question-answer information generating device, comprising:

[0020] An information acquisition module, used to acquire question information to be answered, wherein the question information at least includes first image information;

[0021] An image processing module is used to perform image processing on the first image information to obtain second image information corresponding to the first image information; the resolution of the second image information is lower than that of the first image information;

[0022] A first input module, used to input the first image information into a pre-trained first encoder to obtain feature map information corresponding to the first image information;

[0023] A second input module, used to input the second image information into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information;

[0024] An information query module, configured to query the feature map information according to the visual embedding information, so as to obtain relevant information corresponding to the visual embedding information in the feature map information based on a self-attention mechanism;

[0025] The information generation module is used to generate answer information corresponding to the question information according to the relevant information, and complete the answer to the question information.

[0026] In a third aspect, the present application further provides a computer device, including:

[0027] Memory and processor;

[0028] The memory is used to store computer programs;

[0029] The processor is used to execute the computer program and implement the steps of the question and answer information generating method as described in the first aspect above when executing the computer program.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the question and answer information generating method as described in the first aspect above.

[0031] The embodiments of the present application provide a question and answer information generation method, apparatus, computer device and readable storage medium, which are intended to optimize the performance of an intelligent customer service system and the user experience. By combining image processing technology and deep learning models, the method solves several key problems existing in traditional intelligent customer service systems, such as single functionality, limited automation level, insufficient ability to respond to emergencies, lack of personalized services and poor user experience.

[0032] First, the method receives questions that need to be answered from users or systems. The question information here includes not only traditional text form, but also image information (called first image information). This can make the customer service system more diversified. When handling user questions, it not only relies on text information, but also can combine image information.

[0033] Then, the acquired first image information is processed to generate second image information with lower resolution. This step may be to reduce the complexity of subsequent processing and reduce the consumption of computing resources while ensuring the recognition effect of image information. By reducing the resolution of the image, the amount of data processing can be reduced without significantly affecting the accuracy of image content recognition.

[0034] Then, the original first image information is input into the pre-trained first encoder (first encoder), and the feature map information corresponding to the image is output. This step extracts the multi-level features of the image through the deep learning model for subsequent information matching.

[0035] Furthermore, the processed second image information is input into a pre-trained second encoder (second encoder) to obtain visual embedding information of the second image information. The visual embedding information can be understood as an abstract representation of the image information, which is used to find relevant information in the feature map later.

[0036] Furthermore, based on the self-attention mechanism, the generated visual embedding information is used to query the feature map information to find the content most relevant to the visual embedding information. The self-attention mechanism allows the model to focus on some content while ignoring other parts when processing input data, improving the efficiency and accuracy of information retrieval.

[0037] Finally, an answer to the question is generated based on the matched relevant information. This may involve text generation technology, which generates natural and accurate answer content based on the queried image feature information and combined with context understanding.

[0038] The method provided thereby has at least the following beneficial effects:

[0039] 1. Diversified information processing capabilities: Compared with traditional intelligent customer service systems that can only process text information, this method can more comprehensively understand users' questions by introducing image processing capabilities, especially when the questions involve non-text information such as images, providing users with more accurate and relevant answers.

[0040] 2. Improve the level of automation: Use pre-trained encoders for image information processing and feature extraction, combined with the self-attention mechanism, to automatically generate answers, reducing the need for human intervention and improving the system's level of automation.

[0041] 3. Optimize resource utilization: By reducing the resolution of the original image, the amount of data required for calculation is reduced, the consumption of computing resources is reduced, and the processing speed is also accelerated.

[0042] 4. Enhanced personalized services: The addition of image information enables the system to better adapt to the needs of different users, such as handling graphical interface issues, product appearance consultation, etc., and improves the level of personalization of services.

[0043] 5. Improve user experience: Through the comprehensive application of the above technologies, the system can respond to user needs more accurately and quickly, provide more satisfactory services, and thus improve user experience.

[0044] 6. Reduce labor costs: Intelligent handling of user issues reduces reliance on manual customer service, which helps reduce the company's labor costs in the long run.

[0045] In summary, this method achieves efficient and accurate processing of user problems in complex business scenarios by integrating advanced technologies such as image processing, deep learning, and self-attention mechanism, and provides a new technical path for improving the comprehensive service capabilities of intelligent customer service systems. In summary, the method provided provides a new problem-solving approach for intelligent customer service systems by integrating image processing and deep learning technologies, which not only solves the shortcomings of traditional intelligent customer service, but also opens up new possibilities for the future development of intelligent customer service.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1This is a flowchart showing the steps of a method for generating question and answer information provided by an embodiment of the present application;

[0049] Figure 2 This is a schematic flow chart of the steps of a service recommendation method provided by an embodiment of the present application;

[0050] Figure 3 It is a structural diagram of a question and answer information generating device provided by an embodiment of the present application;

[0051] Figure 4 It is a schematic block diagram of the structure of a computer device provided in one embodiment of the present application.

[0052] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0055] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the difference.

[0056] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0057] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0058] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0059] With the rapid development of Internet technology, the application of intelligent customer service systems in various industries has gradually become popular. Traditional intelligent customer service systems mainly rely on the processing of text information, using natural language processing (NLP) technology to understand users' questions and provide corresponding answers. Nowadays, when users interact with customer service systems, they often upload multimodal data such as images and videos to express their needs or questions more intuitively.

[0060] However, traditional intelligent customer service still has some limitations, such as single function, limited automation level, insufficient ability to respond to emergencies, limited personalized service capabilities, user experience needs to be improved, and high labor costs. These problems limit the performance of intelligent customer service in handling complex businesses and meeting diverse customer needs.

[0061] Therefore, there is an urgent need for a question-and-answer information generation method for multimodal data to improve the performance of the intelligent customer service to which the method is applied.

[0062] To solve the above problems, please refer to Figure 1 , Figure 1 This is a schematic flow chart of a question-answer information generation method provided by an embodiment of the present application. The question-answer information generation method can be implemented by a computer device, which can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a laptop computer, a wearable device, or a robot.

[0063] It should be noted that the acquisition of any information mentioned in the provided method is in compliance with relevant regulations and is carried out with the user's consent, and will not infringe on the user's privacy or violate relevant laws and regulations.

[0064] At the same time, the method provided by this application can be applied to scenarios such as the financial field where users need to communicate frequently with customer service, and can also achieve good application effects in other scenarios that require a large number of intelligent customer service applications, such as medical and education fields. Therefore, the financial field is only a specific example of this application, and the specific application scope of the method provided by this application is not limited.

[0065] To solve the above problems, please refer to Figure 1 Specifically, Figure 1 As shown, the provided question-answer information generating method includes steps S101 to S106. The details are as follows:

[0066] Step S101: Obtain question information to be answered, where the question information at least includes first image information.

[0067] Specifically, the method receives the user's question information through a front-end interface. The question information may include text information and image information, or may only include image information. The user uploads relevant pictures through channels such as mobile phone APPs and web pages. The computer device receives these data through an API interface to answer after analysis. For example, in an insurance scenario, the text information may be a user's text description, such as "My car was damaged in an accident and I need to pay compensation"; the image information may be photos uploaded by the user, such as photos of the accident scene, photos of damaged parts of the car, etc.

[0068] Step S102: Perform image processing on the first image information to obtain second image information corresponding to the first image information; the resolution of the second image information is lower than that of the first image information.

[0069] Specifically, by performing image processing on the first image information, second image information corresponding to the first image information is obtained; the resolution of the second image information is lower than that of the first image information. The uploaded high-resolution image is preprocessed to generate a low-resolution version. This can reduce the complexity and processing time of subsequent calculations while retaining the key features of the image. For example, an image processing library (such as OpenCV) is used to scale the high-resolution image. For example, a 1080p image is scaled to 480p or lower. Other resolution adjustment methods can also be used, and the embodiments of the present application are not limited to this.

[0070] In some embodiments, the resolution of the first image information is higher than a preset resolution; performing image processing on the first image information to obtain second image information corresponding to the first image information includes: performing bilinear interpolation on the first image information to obtain the second image information.

[0071] By downsampling the high-resolution first image information into low-resolution second image information through the bilinear interpolation method, the amount of data of the second image information can be significantly reduced. This not only reduces the computational complexity of subsequent processing, but also improves the processing speed, allowing the intelligent customer service system to respond to user requests more quickly.

[0072] At the same time, by processing high-resolution images and low-resolution images separately with two encoders, more detailed high-level features can be extracted from high-resolution images, and more basic low-level features can be extracted from low-resolution images. This multi-resolution feature extraction method can more comprehensively understand the image content and improve the system's ability to process complex images.

[0073] The above embodiment processes high-resolution images through bilinear interpolation method, combines multi-resolution feature extraction and self-attention mechanism, and significantly improves the processing efficiency, feature extraction capability and query accuracy of the intelligent customer service system, thereby optimizing user experience and reducing labor costs. These improvements make the intelligent customer service system built by the method perform better in handling complex business and meeting diverse customer needs.

[0074] Step S103: Input the first image information into a pre-trained first encoder to obtain feature map information corresponding to the first image information.

[0075] Specifically, the encoder can be a model based on architectures such as ResNet, VGG, or MobileNet, which can extract high-level features of images. The high-resolution image is input into the pre-trained first encoder, which outputs a multi-dimensional feature map containing image features. For example, the first encoder can output a 1024-dimensional feature map.

[0076] In some embodiments, the first encoder is a convolutional neural network encoder; the first image information is input into a pre-trained first encoder to obtain feature map information corresponding to the first image information, including: inputting the first image information into a convolutional neural network encoder, the convolutional neural network encoder performs feature extraction on the first image information, obtains feature information of multiple convolution stages, and upsamples multiple feature information at a preset scale to obtain feature map information.

[0077] The first encoder is a convolutional neural network (CNN) encoder. The first image information is input into the pre-trained convolutional neural network encoder, and the convolutional neural network performs multi-stage feature extraction on the first image information to obtain feature information of multiple convolution stages. The convolutional neural network encoder extracts feature information in multiple convolution stages (such as Conv1, Conv2, Conv3, etc.), and the feature information contains image features at different levels, from low-level edge and texture information to high-level object and scene information.

[0078] Then, the feature information of multiple convolution stages is upsampled at a preset scale so that the feature information is restored to a consistent resolution in the spatial dimension. The feature map information finally generated contains the comprehensive features from multiple convolution stages, and the feature map information can more comprehensively describe the content of the first image information.

[0079] In summary, the convolutional neural network encoder is used to perform multi-stage feature extraction on the first image information, and the extracted feature information is upsampled at a preset scale to generate comprehensive feature map information. This method not only improves the richness and representation ability of feature extraction, but also optimizes the accuracy of the system's query and generated answers, making the intelligent customer service system perform better when processing complex images and diverse user needs. At the same time, the application of pre-trained models further improves the performance and adaptability of the system.

[0080] Step S104: Input the second image information into the pre-trained second encoder to obtain visual embedding information corresponding to the second image information.

[0081] Specifically, another pre-trained second encoder is used to process the low-resolution image to generate visual embedding information. Visual embedding is a technology that converts image content into a fixed-length vector to facilitate subsequent query and matching. For example, a low-resolution image is input into the pre-trained second encoder to generate a fixed-length vector.

[0082] In some embodiments, the second encoder is a visual transformation encoder; the second image information is input into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information, including: inputting the second image information into the visual transformation encoder, the visual change encoder divides the second image information into multiple first regions and generates position embedding information corresponding to each first region, so as to generate visual embedding information according to each first region and the corresponding position embedding information.

[0083] The second encoder is a visual transform encoder (Vi si on Transformer, Vi T). The second image information is input into the pre-trained visual transform encoder, and the visual transform encoder divides the second image information into multiple first regions (or image blocks, Patches). The visual transform encoder generates corresponding position embedding information for each first region. The position embedding information is used to represent the position of each image block in the original image, so that the model can understand the spatial relationship between image blocks. Based on the image content of each first region and its corresponding position embedding information, the visual transform encoder generates visual embedding information corresponding to the second image information. The visual embedding information is a high-level feature representation of the second image information, which contains the content information and position information of the image block.

[0084] In summary, by using the visual transform encoder to process the second image information, the image is divided into multiple image blocks and the position embedding information corresponding to each image block is generated, and finally the visual embedding information is generated. This method not only improves the global understanding and representation capabilities of features, but also improves the processing efficiency and generalization ability of the model. The application of the visual transform encoder enables the intelligent customer service system to perform better in processing complex images and diverse user needs, thereby optimizing the overall performance of the system and user experience.

[0085] Step S105. Query the feature map information according to the visual embedding information to obtain relevant information corresponding to the visual embedding information in the feature map information based on the self-attention mechanism.

[0086] Specifically, the self-attention mechanism is used to find content related to the visual embedding information in the feature map. The self-attention mechanism can help the system identify which parts of the features are most important for answering questions. For example, by inputting the visual embedding information and the feature map into a self-attention model (such as Transformer), the model calculates the similarity between each position in the feature map and the visual embedding and outputs a weighted feature map. In the weighted feature map, more relevant content will be given a higher weight.

[0087] In some embodiments, a query is performed in the feature map information according to the visual embedding information to obtain relevant information corresponding to the visual embedding information in the feature map information based on a self-attention mechanism, including: obtaining the region dimension corresponding to the visual embedding information; expanding the feature map according to the region dimension so that the feature map dimension of the expanded feature map matches the region dimension; querying each second region in the feature map information according to each first region in the visual embedding information respectively to query relevant regions corresponding to each first region in multiple second regions; and generating relevant information based on multiple relevant regions.

[0088] First, the region dimension corresponding to the visual embedding information is obtained, which generally refers to the feature dimension of each image block (first region) in the visual embedding information, for example, the length of the feature vector of each image block.

[0089] Then, the feature map information (the feature map obtained from the first encoder) is expanded so that the dimension of the expanded feature map matches the regional dimension of the visual embedding information. This usually involves converting the feature map from a two-dimensional or three-dimensional structure to a one-dimensional structure to facilitate subsequent query operations. Ensure that the feature dimension of the expanded feature map is consistent with the regional dimension of the visual embedding information for effective query and matching.

[0090] Then, according to each first region in the visual embedding information, each second region in the feature map information is queried respectively to query the relevant region corresponding to each first region in the multiple second regions. The self-attention mechanism is used to query in the feature map information. The self-attention mechanism dynamically adjusts the weights of different feature parts by calculating the similarity or correlation between each first region and each second region in the feature map, so as to find the relevant region more accurately.

[0091] Finally, based on the query results, multiple relevant regions in the feature map are determined, which are highly relevant to the first region in the visual embedding information. Related information is generated based on these relevant regions. This information is usually a further description or explanation of the image content, which is used to generate answer information later.

[0092] Then, by obtaining the regional dimension of the visual embedding information, expanding the feature map to match the regional dimension, and using the self-attention mechanism to query in the feature map, relevant information is finally generated. This method not only improves the accuracy of the query and the richness of the feature representation, but also improves the processing efficiency and the adaptability of the system. These improvements enable the intelligent customer service system to perform better when processing complex images and diverse user needs, thereby optimizing the user experience and the quality of the answer information.

[0093] Step S106: Generate answer information corresponding to the question information according to the relevant information, and complete the answer to the question information.

[0094] Specifically, based on the generated relevant information, natural language processing (NLP) or autoregression techniques can be used to generate answers. Answers can include text descriptions, image explanations, link guides, and other forms. For example, the weighted feature map and the user's text question information are input into a pre-trained generation model (such as BERT, T5, etc.) to generate detailed answer information. In insurance scenarios, the answer information can include a preliminary estimate of the compensation amount, a description of the claims process, a list of required materials, and so on.

[0095] Then, through image processing and self-attention mechanism, steps S101 to S106 enable the system to understand the user's questions more accurately, provide intuitive and detailed answers, and improve user satisfaction and trust. At the same time, it reduces dependence on manual customer service and improves processing efficiency. Users can get feedback in a short time, which speeds up problem solving. And the application of self-attention mechanism enables the system to better understand problems and customer needs and provide personalized services. For example, based on the specific circumstances of the accident, the system can recommend the most appropriate claims settlement plan. Ultimately, it reduces human resource costs, while improving the accuracy and speed of problem solving, and can achieve good results in scenarios that require a large amount of manual customer service.

[0096] In some embodiments, the question information also includes text information corresponding to the first image information; generating answer information corresponding to the question information based on relevant information includes: inputting the text information and the question information into a preset autoregressive model, the autoregressive model parses the text information and the question information, and generates answer information.

[0097] In the present application, the question information includes not only the first image information, but also the text information corresponding to the first image information. The text information can be the user's description of the image, the specific content of the question, context information, etc. By inputting the text information and question information into a preset autoregressive model (Autoregressive Model l). The autoregressive model parses the text information and question information to generate answer information. This process includes: Text parsing: understanding the semantics and context in the text information. Image parsing: combining visual embedding information to understand the image content. Comprehensive generation: based on the comprehensive understanding of text and image, generate answer information.

[0098] The autoregressive model generates answer information word by word, that is, it generates one word at a time, and then uses the generated word as input to continue generating the next word until a complete answer is generated. At the same time, the autoregressive model can gradually generate answer information that conforms to semantics and logic based on the input context information.

[0099] In summary, by inputting the text information and image information (visual embedding information) in the question information into the preset autoregressive model, the parsing ability of the autoregressive model is used to generate the answer information. This method not only improves the accuracy and relevance of the answer, but also enhances the flexibility and intelligence level of the system, and improves the user experience and processing efficiency. Through the fusion of multimodal information, the intelligent customer service system can provide higher quality and more comprehensive services in a variety of different scenarios.

[0100] In some embodiments, Figure 2 As shown, after generating answer information corresponding to the question information according to the relevant information and completing the answer to the question information, it also includes steps S107a to S107c.

[0101] Step S107a. Obtain user information corresponding to the question information; the user information at least includes basic information and historical interaction information.

[0102] Step S107b. Generate service information based on the user information and the answer information; the service information includes at least one of product recommendation information and service recommendation information.

[0103] Step S107c: Send the service information to the user terminal corresponding to the question information to complete the service recommendation to the user terminal.

[0104] After generating answer information corresponding to the question information based on relevant information and completing the answer to the question information, it is also necessary to obtain the user information corresponding to the question information. User information includes at least basic information (such as user ID, age, gender, etc.) and historical interaction information (such as the user's past question records, purchase records, browsing records, etc.). Based on the acquired user information and the generated answer information, service information is further generated. The service information includes at least one of product recommendation information and service recommendation information. Product recommendation information can be a product recommendation related to the user's historical purchase or browsing records, and service recommendation information can be a service recommendation related to the user's historical interaction records, such as technical support, after-sales service, etc. Finally, the generated service information is sent to the user terminal corresponding to the question information, thereby completing the service recommendation for the user terminal.

[0105] In summary, after answering the user's questions, the user's personalized information is further obtained, and service information is generated based on the user information and answer information, and finally the service information is sent to the user terminal. This method not only improves the level of personalized service, enhances user stickiness and loyalty, but also improves service efficiency and user experience, and increases the commercial value of the system. Through this multi-step processing and recommendation mechanism, the intelligent customer service system built by the method can provide deeper and higher quality services while meeting the basic needs of users.

[0106] Next, an example is used to illustrate the method provided by this application:

[0107] 1. Starting from the high-resolution image XH∈RH×W×3, the corresponding low-resolution image XL∈RH′×W′×3 is generated by bilinear interpolation. They are then processed and encoded into multi-grid visual embeddings in two parallel image streams. For the LR stream, CLIP pre-trained ViT is used to encode the visual embedding X′L. For the HR stream, a CNN-based encoder is used for adaptive and efficient HR image processing. The HR feature map X′H is obtained by upsampling the features of different convolution stages and concatenating them to 1 / 4 input scale.

[0108] 2. Using the LR embedding X′L and HR features X′H generated above, block information mining is proposed to expand the potential of VLM with enhanced visual tags. In order to maintain the number of final visual tags in LLM for efficiency, the low-resolution visual embedding X′L is used as the query Q, aiming to retrieve relevant visual clues from HR candidates. At the same time, the HR feature map X′H is regarded as the key K and value V. Here, the low-resolution patch in Q is related to the corresponding high-resolution patch. The resolution sub-regions in K and V contain M2 pixel-level features.

[0109] 3. Through the mined visual tags TV and input text tags TT, they are concatenated as the input of LLM for autoregressive generation to obtain the output.

[0110] The method provided in this application focuses on comprehensive data collection and preprocessing. The method uses advanced data collection technology to widely collect users' basic information, historical interaction records, user feedback, and multivariate data such as images and text attachments. Subsequently, these data are preprocessed, including text segmentation, annotation, and image format standardization to ensure that the data can meet the input requirements of the model.

[0111] At the same time, in terms of multimodal data fusion and deep model training, the method conducts comprehensive correlation analysis on text, image and numerical data, uses the dual visual encoder of the multimodal model to process image data of different resolutions, and combines text information for in-depth understanding. At the same time, the model is deeply trained using large-scale customer service field labeled data, and through continuous interactive data collection and feedback loops, the model performance is continuously optimized to improve its ability to understand user queries and the accuracy of answers.

[0112] In key functional modules such as intelligent question and answer, personalized service, and user intention prediction, the method fully utilizes the natural language processing and generation capabilities of the model to realize automatic question and answer functions, provide fast and accurate answers, and support multiple rounds of dialogue and context retention. At the same time, personalized services and product recommendations are provided based on users' historical data and real-time interactions. In addition, the method also uses big data analysis and machine learning algorithms to predict user needs and behavior patterns to provide support for proactive services. To ensure the integrity and practicality of the system, modules such as knowledge management and updating, system integration and API management, user feedback collection and evaluation, security and privacy protection, and monitoring and reporting are also designed to comprehensively improve the performance and user experience of the intelligent customer service system.

[0113] In summary, the method proposed in this application also has the following advantages:

[0114] 1. The model's dual encoder architecture is used to process visual inputs of different resolutions and combine them with text information, enabling the model to understand and respond to complex queries containing images and text, providing a richer interactive experience.

[0115] 2. The method can analyze a large amount of historical data to gain insight into and predict user needs and behavior patterns, and provide proactive services. In addition, the system has the ability to self-learn and continuously optimize by collecting user feedback and interaction data in real time, ensuring that the quality and relevance of services continue to improve over time.

[0116] 3. The method can identify customers' personalized needs and preferences through in-depth analysis of the data provided by users, thereby providing customized service suggestions. This personalized service not only improves user satisfaction, but also reduces dependence on traditional manual customer service, effectively reducing the company's human resource costs and management burden.

[0117] See also Figure 3 As shown, Figure 3 1 is a schematic diagram of the structure of the question and answer information generating device 200 provided in an embodiment of the present application. The question and answer information generating device 200 is used to execute the steps of the question and answer information generating method shown in the above embodiments. The question and answer information generating device 200 can be a single server or a server cluster, or the question and answer information generating device 200 can be a terminal, which can be a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0118] like Figure 3 As shown, the question and answer information generating device 200 includes:

[0119] The information acquisition module 201 is used to acquire question information to be answered, wherein the question information at least includes first image information;

[0120] An image processing module 202 is used to perform image processing on the first image information to obtain second image information corresponding to the first image information; the resolution of the second image information is lower than that of the first image information;

[0121] A first input module 203, used to input the first image information into a pre-trained first encoder to obtain feature map information corresponding to the first image information;

[0122] A second input module 204, configured to input the second image information into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information;

[0123] An information query module 205, configured to query the feature map information according to the visual embedding information, so as to obtain relevant information corresponding to the visual embedding information in the feature map information based on a self-attention mechanism;

[0124] The information generation module 206 is used to generate answer information corresponding to the question information according to the relevant information, and complete the answer to the question information.

[0125] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the question and answer information generating device and each module described above can refer to the corresponding process in the question and answer information generating method embodiments described in the above embodiments, and will not be repeated here.

[0126] The above-mentioned question-answer information generating method can be implemented in the form of a computer program. The computer program can be used in Figure 3 Run on the device shown.

[0127] See also Figure 4 , Figure 4 1 is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device includes a processor, a memory and a network interface connected via a device bus, wherein the memory may include a storage medium and an internal memory.

[0128] The storage medium can store an operating device and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the question-answer information generating methods.

[0129] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0130] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any one of the question and answer information generating methods.

[0131] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0133] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0134] Acquire question information to be answered, where the question information at least includes first image information;

[0135] Performing image processing on the first image information to obtain second image information corresponding to the first image information; the resolution of the second image information is lower than that of the first image information;

[0136] Inputting the first image information into a pre-trained first encoder to obtain feature map information corresponding to the first image information;

[0137] Inputting the second image information into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information;

[0138] Query the feature map information according to the visual embedding information to obtain relevant information corresponding to the visual embedding information in the feature map information based on the self-attention mechanism;

[0139] Generate answer information corresponding to the question information based on relevant information to complete the answer to the question information.

[0140] In some embodiments, the resolution of the first image information is higher than a preset resolution; performing image processing on the first image information to obtain second image information corresponding to the first image information includes: performing bilinear interpolation on the first image information to obtain the second image information.

[0141] In some embodiments, the first encoder is a convolutional neural network encoder; the first image information is input into a pre-trained first encoder to obtain feature map information corresponding to the first image information, including: inputting the first image information into a convolutional neural network encoder, the convolutional neural network encoder performs feature extraction on the first image information, obtains feature information of multiple convolution stages, and upsamples multiple feature information at a preset scale to obtain feature map information.

[0142] In some embodiments, the second encoder is a visual transformation encoder; the second image information is input into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information, including: inputting the second image information into the visual transformation encoder, the visual change encoder divides the second image information into multiple first regions and generates position embedding information corresponding to each first region, so as to generate visual embedding information according to each first region and the corresponding position embedding information.

[0143] In some embodiments, a query is performed in the feature map information according to the visual embedding information to obtain relevant information corresponding to the visual embedding information in the feature map information based on a self-attention mechanism, including: obtaining the region dimension corresponding to the visual embedding information; expanding the feature map according to the region dimension so that the feature map dimension of the expanded feature map matches the region dimension; querying each second region in the feature map information according to each first region in the visual embedding information respectively to query relevant regions corresponding to each first region in multiple second regions; and generating relevant information based on multiple relevant regions.

[0144] In some embodiments, the question information also includes text information corresponding to the first image information; generating answer information corresponding to the question information based on relevant information includes: inputting the text information and the question information into a preset autoregressive model, the autoregressive model parses the text information and the question information, and generates answer information.

[0145] In some embodiments, after generating answer information corresponding to the question information based on relevant information and completing the answer to the question information, it also includes: obtaining user information corresponding to the question information; the user information includes at least basic information and historical interaction information; generating service information based on the user information and the answer information; the service information includes at least any one of product recommendation information and service recommendation information; sending the service information to the user terminal corresponding to the question information to complete the service recommendation to the user terminal.

[0146] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the question and answer information generating method as described in the first aspect above.

[0147] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0148] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating question and answer information, characterized in that: include: Acquire question information to be answered, wherein the question information at least includes first image information; performing image processing on the first image information to obtain second image information corresponding to the first image information; the resolution of the second image information is lower than that of the first image information; Inputting the first image information into a pre-trained first encoder to obtain feature map information corresponding to the first image information; Inputting the second image information into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information; Searching the feature graph information according to the visual embedding information to obtain relevant information corresponding to the visual embedding information in the feature graph information based on a self-attention mechanism; Generate answer information corresponding to the question information according to the relevant information, and complete the answer to the question information.

2. The method according to claim 1, characterized in that: The resolution of the first image information is higher than a preset resolution; and performing image processing on the first image information to obtain second image information corresponding to the first image information includes: Perform bilinear interpolation on the first image information to obtain the second image information.

3. The method according to claim 1, characterized in that The first encoder is a convolutional neural network encoder; the step of inputting the first image information into a pre-trained first encoder to obtain feature map information corresponding to the first image information includes: The first image information is input into the convolutional neural network encoder, and the convolutional neural network encoder performs feature extraction on the first image information to obtain feature information of multiple convolution stages, and upsamples the multiple feature information at a preset scale to obtain the feature map information.

4. The method according to claim 1, characterized in that: The second encoder is a visual transformation encoder; the step of inputting the second image information into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information includes: The second image information is input into the visual change encoder, and the visual change encoder divides the second image information into multiple first regions and generates position embedding information corresponding to each first region, so as to generate the visual embedding information according to each first region and the corresponding position embedding information.

5. The method according to claim 1, characterized in that: The querying in the feature graph information according to the visual embedding information to obtain relevant information corresponding to the visual embedding information in the feature graph information based on a self-attention mechanism includes: Obtaining a region dimension corresponding to the visual embedding information; Expanding the feature map according to the region dimension so that the feature map dimension of the expanded feature map matches the region dimension; According to each first area in the visual embedding information, query each second area in the feature map information respectively, so as to query the relevant area corresponding to each first area in the plurality of second areas; The relevant information is generated according to the plurality of relevant areas.

6. The method according to claim 1, characterized in that The question information also includes text information corresponding to the first image information; and generating answer information corresponding to the question information according to the relevant information includes: The text information and the question information are input into a preset autoregressive model, and the autoregressive model analyzes the text information and the question information to generate the answer information.

7. The method according to claim 1, characterized in that After generating answer information corresponding to the question information according to the relevant information and completing the answer to the question information, the method further includes: Acquire user information corresponding to the question information; the user information at least includes basic information and historical interaction information; Generate service information according to the user information and the answer information; the service information at least includes any one of product recommendation information and service recommendation information; The service information is sent to the user terminal corresponding to the question information to complete the service recommendation for the user terminal.

8. A question and answer information generating device, characterized in that: include: An information acquisition module, used to acquire question information to be answered, wherein the question information at least includes first image information; An image processing module is used to perform image processing on the first image information to obtain second image information corresponding to the first image information; the resolution of the second image information is lower than that of the first image information; A first input module, used to input the first image information into a pre-trained first encoder to obtain feature map information corresponding to the first image information; A second input module, used to input the second image information into a pre-trained second encoder to obtain visual embedding information corresponding to the second image information; An information query module, configured to query the feature map information according to the visual embedding information, so as to obtain relevant information corresponding to the visual embedding information in the feature map information based on a self-attention mechanism; The information generation module is used to generate answer information corresponding to the question information according to the relevant information, and complete the answer to the question information.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 7.

Images