A visual question answering processing method and system

Abstract

The present invention discloses a visual question-answering processing method and system, including a knowledge graph of a preset product, analyzing the dialogue information to determine the user's problem demands for the product, extracting a series of frames most relevant to the problem demands from a longer product problem video based on the problem demands and the product's knowledge graph, and then using the above frames to generate a short video with a shorter duration, and synchronously displaying the short video and the problem demands. When the present invention is applied, when an e-commerce customer service receives a longer product problem video, it does not need to watch the product problem video. By watching the corresponding short video, the user's demand for the product problem can be accurately grasped, thereby providing accurate data support for accurate and timely replies to users and winning valuable time, greatly improving the timeliness and quality of the e-commerce customer service's response.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a visual question answering processing method and system. Background Art

[0002] With the development of Internet technology, e-commerce has also developed rapidly in the past decade. In the rapidly developing e-commerce environment, providing efficient and accurate customer service is the key to improving user experience and maintaining a good brand reputation. However, e-commerce customer service also faces many challenges, especially in ensuring the response time. First of all, the e-commerce platform has high requirements for the response time:

[0003] In order to improve user satisfaction and enhance competitiveness, modern e-commerce platforms generally set extremely high standards for customer service response. Many platforms require customer service to respond to user inquiries within 30 seconds. Although this immediacy requirement improves user experience, it also puts tremendous pressure on the customer service team. During peak hours, customer service staff need to quickly handle a large influx of information to ensure that each inquiry can be responded to in a timely manner, which is a great test for the response speed of manual customer service and the system processing capacity.

[0004] Secondly, many customers do not describe the problem clearly and specifically: when seeking help, users often lack professional knowledge or are eager to solve the problem, which leads to vague, incomplete, and even misunderstandings in the description of the problem. Customer service staff need to spend extra time to clarify the details of the problem, which not only prolongs the processing time of a single service, but may also affect the waiting time of subsequent customers. In many cases, users will send video messages, but watching and analyzing video content is much more time-consuming than text messages. Customer service staff need to watch the video in full before identifying the problem, and this process often far exceeds the 30-second response time limit. In addition, the video may involve multiple dimensions of problems such as the specific usage scenarios of the product and the performance of the failure. Therefore, it is difficult for customer service staff to accurately determine the cause of the problem and provide solutions.

[0005] Therefore, how to solve the challenges faced by e-commerce customer service, such as high response time requirements, vague customer descriptions, video content processing, and problem point analysis, has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0006] In view of the above problems, the present application provides a visual question-answering processing method for assisting e-commerce customer service in answering questions to improve the timeliness and accuracy of question answers.

[0007] To achieve the above objectives, the present application provides a visual question answering processing method for assisting e-commerce customer service in answering product questions. The visual question answering processing method includes the following steps:

[0008] A knowledge graph of preset products, wherein the knowledge graph of the products includes the structure, function, operation, common problems and solutions of each product in the e-commerce trading platform, wherein each common problem is associated with an operation corresponding to the common problem, and the operation and the common problem are both configured with corresponding images, and the knowledge graph is also provided with a feature vector set for the image, wherein the feature vector set stores feature vectors extracted from image encoding of each of the images using a CNN model;

[0009] Obtain product problem videos uploaded by users and conversation information between customer service and users from the e-commerce customer service platform; and obtain product order information of the user from the e-commerce transaction platform;

[0010] Selecting corresponding product data from the knowledge graph according to the product order information; and predicting the user's problem demands for the product by combining the common questions and the current conversation information;

[0011] According to the problem demand, corresponding common problems and the associated operations are locked from the data of the product in the knowledge graph, and a first feature vector corresponding to the locked common problem and a second feature vector corresponding to the operation are extracted from the feature vector set;

[0012] Decompose the product problem video into several frames, delete duplicate frames and use the CNN model to extract feature vectors for each of the remaining frames to obtain a third feature vector; use a cosine similarity algorithm to calculate the similarity between the third feature vector and the first feature vector and the second feature vector, filter out frames whose similarity is greater than a preset value, and mark them as target frames;

[0013] Arranging the target frames in time sequence and performing smooth transition processing to synthesize a coherent short video that is shorter than a preset length;

[0014] The short video is played and the question and demand are displayed simultaneously.

[0015] Furthermore, the steps are also included: continuously monitoring the conversation information between the customer service and the user in the e-commerce customer service platform to obtain new conversation information; judging whether the problem demands have changed and / or increased based on the new conversation information; if so, extracting the corresponding feature vector from the feature vector set based on the changed problem demands and / or increased problem demands, and marking it as a supplementary feature vector; and extracting frames from the product problem video whose similarity to the supplementary feature vector is greater than a preset value, and marking them as updated frames; updating the short video with the updated frames so that the short video includes frames corresponding to the changed problem demands and the increased problem demands, or marking the short video as the first short video, and generating a second short video based on the updated frames.

[0016] Furthermore, the playing of the short video and the synchronous display of the question and demand include the steps of:

[0017] Generate a comparison table of the question, the target frame, and the dialogue information, wherein the comparison table records the target frame corresponding to the question, and the position of the dialogue content most relevant to the question in the dialogue information;

[0018] The question and the position pointer are inserted into the target frame in the short video, and the position pointer is used to point to the position in the conversation information; when the short video is played, the position pointer corresponding to the current target frame is obtained according to the comparison table, and the conversation interface between the customer service and the user in the e-commerce customer service platform is jumped to the location of the most relevant conversation content through the position pointer.

[0019] Furthermore, the method further comprises the steps of:

[0020] Filter out the relevant solutions from the knowledge graph based on the problem requirements;

[0021] And according to the confirmation operation of the customer service personnel, a final solution is determined from the solutions and sent to the user through the e-commerce customer service platform.

[0022] Furthermore, the CNN model uses a self-attention mechanism to capture feature vectors in the image.

[0023] In order to solve the above technical problems, this application also provides another technical solution:

[0024] A visual question answering system is used to assist e-commerce customer service in answering product questions, and the visual question answering system includes:

[0025] A preset module, used for presetting a knowledge graph of a product, wherein the knowledge graph of the product includes the structure, function, operation, common problems and solutions to the problems of each product in the e-commerce trading platform, wherein each common problem is associated with an operation that causes the common problem, and the operation and the common problem are both configured with corresponding images, and the knowledge graph is also provided with a feature vector set for the image, wherein the feature vector set stores feature vectors extracted from image encoding of each of the images using a CNN model;

[0026] A prediction module is used to obtain product problem videos uploaded by users and conversation information between customer service and users from an e-commerce customer service platform; and obtain the user's product order information from an e-commerce transaction platform; select corresponding product data from the knowledge graph according to the product order information; and predict the user's problem demands for the product by combining the common problems and the current conversation information;

[0027] A locking module, configured to lock corresponding common problems and the associated operations from the data of the product in the knowledge graph according to the problem demand, and extract a first feature vector corresponding to the locked common problem and a second feature vector corresponding to the operation from the feature vector set;

[0028] The short video module is used to decompose the product problem video into several frames, delete duplicate frames and use the CNN model to extract feature vectors for each remaining frame to obtain a third feature vector; use the cosine similarity algorithm to calculate the similarity between the third feature vector and the first feature vector and the second feature vector, filter out frames with similarities greater than a preset value, and mark them as target frames; arrange the target frames in chronological order and perform smooth transition processing to synthesize a coherent short video with a duration less than a preset length;

[0029] The display module is used to play the short video and simultaneously display the question and demand.

[0030] Furthermore, the short video module is also used to: continuously monitor the conversation information between the customer service and the user in the e-commerce customer service platform to obtain new conversation information; determine whether the problem demands have changed and / or increased based on the new conversation information; if so, extract the corresponding feature vector from the feature vector set based on the changed problem demands and / or increased problem demands, and mark it as a supplementary feature vector; and extract frames from the product problem video whose similarity to the supplementary feature vector is greater than a preset value, and mark them as updated frames; update the short video with the updated frames so that the short video includes frames corresponding to the changed problem demands and the increased problem demands, or mark the short video as the first short video, and generate a second short video based on the updated frames.

[0031] Furthermore, the display module is further used to: generate a comparison table of the question demand, the target frame and the dialogue information, wherein the comparison table records the target frame corresponding to the question demand and the position of the dialogue content most relevant to the question demand in the dialogue information;

[0032] The question and the position pointer are inserted into the target frame in the short video, and the position pointer is used to point to the position in the conversation information; when the short video is played, the position pointer corresponding to the current target frame is obtained according to the comparison table, and the conversation interface between the customer service and the user in the e-commerce customer service platform is jumped to the location of the most relevant conversation content through the position pointer.

[0033] Furthermore, it also includes:

[0034] The reply module is used to filter out the relevant solutions from the knowledge graph based on the problem demand; and determine the final solution from the solutions according to the confirmation operation of the customer service personnel, and send it to the user through the e-commerce customer service platform.

[0035] Furthermore, the CNN model uses a self-attention mechanism to capture feature vectors in the image.

[0036] Different from the prior art, the visual question-answering processing method of the above technical solution is used to assist e-commerce customer service in answering product questions, which involves knowledge graphs, image recognition and other technologies. This method can determine the user's problem demands for the product (i.e., the product problem) by analyzing the dialogue information, and then, based on the problem demands and the product's knowledge graph, a series of frames most relevant to the problem demands can be extracted from a longer product problem video (these frames include the problem frames of the problem image and the operation frames that cause the problem), and then the above frames are used to generate a short video with a shorter duration, and the short video is displayed synchronously with the problem demands. Therefore, when an e-commerce customer service receives a longer product problem video, he or she does not need to watch the product problem video, but only needs to watch the corresponding short video to accurately grasp the user's demand for the product problem, thereby providing accurate data support and winning valuable time for accurate and timely responses to users, greatly improving the timeliness and quality of e-commerce customer service responses.

[0037] The above-mentioned records related to the invention content are only an overview of the technical solution of the present application. In order to enable ordinary technicians in the field to more clearly understand the technical solution of the present application, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purpose and other purposes, features and advantages of the present application easier to understand, the following is an explanation in combination with the specific implementation mode and drawings of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of the present invention and other related contents, and shall not be considered as limiting the present application.

[0039] In the drawings of the specification:

[0040] Figure 1is a flowchart of the visual question answering processing method described in the specific implementation method;

[0041] Figure 2 A flowchart of the dynamic update of short videos described in the specific implementation method;

[0042] Figure 3 A flowchart showing the simultaneous display of short videos and questions and demands as described in the specific implementation method;

[0043] Figure 4 A block diagram of a visual question answering system for a specific implementation method;

[0044] The reference numerals in the above drawings are described as follows:

[0045] 400, visual question answering system; 401, preset module; 402, prediction module; 403, locking module; 404, short video module; 405, display module; DETAILED DESCRIPTION

[0046] In order to explain in detail the possible application scenarios, technical principles, specific schemes that can be implemented, and the purposes and effects that can be achieved, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0047] Reference to "embodiment" herein means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The term "embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or association with other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the various technical features mentioned in the embodiments can be combined in any way to form a corresponding implementable technical solution.

[0048] Unless otherwise defined, the technical terms used in this document have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms in this document is only for describing specific embodiments and is not intended to limit this application.

[0049] In the description of this application, the term "and / or" is an expression used to describe the logical relationship between objects, indicating that three relationships may exist, for example, A and / or B, which means: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" logical relationship.

[0050] In the present application, terms such as “first” and “second” are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship of quantity, priority or sequence between these entities or operations.

[0051] Without further limitations, in this application, the words "include", "comprises", "has" or other similar open-ended expressions used in sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those limited elements, but also other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0052] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than", "less than", "exceed" and the like are understood to exclude the number itself; expressions such as "above", "below", "within" and the like are understood to include the number itself. In addition, in the description of the embodiments of this application, "multiple" means more than two (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups", "multiple times", etc., unless otherwise clearly and specifically limited.

[0053] In the description of the embodiments of the present application, space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the referred device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0054] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms such as "install", "connect", "connect", "fix", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For technicians in the technical field to which the present application belongs, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0055] This embodiment provides a visual question-answering processing method. The visual question-answering processing method is used to assist e-commerce customer service in answering product questions, thereby improving the timeliness and accuracy of e-commerce customer service's responses. The visual question-answering processing method can be applied to customer service of various e-commerce platforms such as Taobao, JD.com, and Douyin.

[0056] See also Figure 1 In this embodiment, the visual question answering processing method includes the following steps:

[0057] S101, a knowledge graph of a preset product, wherein the knowledge graph of the product includes the structure, function, operation, common problems and solutions of each product in the e-commerce trading platform, wherein each common problem is associated with an operation corresponding to the common problem, and the operation and the common problem are both configured with corresponding images, and the knowledge graph is further provided with a feature vector set for the image, wherein the feature vector set stores feature vectors extracted from image encoding of each image using a CNN model;

[0058] S102, obtaining a product problem video uploaded by a user and conversation information between the customer service and the user from an e-commerce customer service platform; and obtaining the product order information of the user from an e-commerce transaction platform;

[0059] S103, selecting corresponding product data from the knowledge graph according to the product order information; and predicting the user's problem demands for the product by combining the common questions and the current conversation information;

[0060] S104, locking corresponding common problems and the associated operations from the data of the product in the knowledge graph according to the problem demand, and extracting a first feature vector corresponding to the locked common problem and a second feature vector corresponding to the operation from the feature vector set;

[0061] S105, decomposing the product problem video into several frames, deleting duplicate frames and extracting feature vectors from each of the remaining frames using the CNN model to obtain a third feature vector; using a cosine similarity algorithm to calculate the similarity between the third feature vector and the first feature vector and the second feature vector, screening out frames whose similarity is greater than a preset value, and marking them as target frames;

[0062] S106. Arrange the target frames in chronological order and perform smooth transition processing to synthesize a coherent short video that is shorter than a preset length; play the short video and simultaneously display the question demand.

[0063] In the knowledge graph of the product in step S101, entities, attributes and relationships are included, wherein the product is an entity, the structure, function, operation, etc. of the product are attributes, and the common problems of the product and their solutions are entities or attributes associated with the product entity, and the association between the attributes such as structure, function and operation is the relationship. Each common problem has a relationship with the operation that causes the common problem. For example, operation 1 (improper operation) of the product may cause problem 1, so there is a causal relationship between operation 1 and problem 1, and the causal relationship is recorded as relationship 1 in the knowledge graph.

[0064] The knowledge graph of the preset product includes: collecting data related to the product structure, function, operation, common problems and solutions. This data may come from multiple channels such as product documentation, user feedback, customer service records, etc. Clean the data to ensure the accuracy and consistency of the data. Based on the product's structure, function, operation and other attributes, define entities, attributes and relationships, and extract entities, attributes and relationships from the collected data. Integrate the extracted knowledge to eliminate contradictions and ambiguities. For example, there may be multiple solutions to the same problem, which need to be screened and integrated. Build a complete knowledge system through steps such as knowledge reasoning and quality assessment. For example, you can use reasoning to discover the relationship between different functions of the product, or evaluate the effectiveness of the solution.

[0065] Therefore, in this embodiment, the knowledge graph of the preset product can comprehensively and accurately grasp the product-related knowledge from the product's performance, problems, solutions, etc. The difference from the knowledge graph in the related art is that, in this embodiment, each of the operations and the common problems is configured with a corresponding image, that is, operation 1 is configured with operation 1 image, and common problem 1 is configured with problem image 1. And the knowledge graph is also provided with a feature vector set for storing the feature vectors of each image. The feature vectors are obtained after these images are encoded by the CNN model (i.e., convolutional neural network model). In this embodiment, the above-mentioned images and the feature vectors of the images are set in the knowledge graph in order to more accurately and quickly filter out the target frames from the product problem video according to the knowledge graph in step S105. In the above step S101, the CNN model uses a self-attention mechanism to capture the feature vectors in the image.

[0066] In step S102, the e-commerce customer service platform is an information interaction platform provided by the e-commerce transaction platform for e-commerce customer service and users, through which users can send various types of information such as video, text, picture, voice, etc. Among them, the product problem video is a video uploaded by the user through the e-commerce customer service platform to illustrate the product problem.

[0067] In step S103, it is possible to determine which product the user purchased based on the product order information, so that only the data related to the product is selected in the knowledge graph, thereby reducing the scope and data volume of the knowledge graph, and improving the efficiency and accuracy of locking the corresponding common problems and the associated operations in the subsequent step S104. In addition, natural speech processing technology can be used in step S103 to accurately predict the user's problem demands from the context of the dialogue information. In many cases, the user's language expression is not clear enough, which makes it easy for the e-commerce customer service to misunderstand the user's meaning. In this embodiment, natural speech processing technology can assist the e-commerce customer service to understand the user's demands more quickly and accurately.

[0068] In step S104, the common problems and the associated operations related to the user's problem demands can be determined based on the attributes and relationships of the product in the knowledge graph. Thus, the first feature vector corresponding to the common problem and the second feature vector corresponding to the operation can be locked from the feature vector set. There can be multiple first feature vectors and second feature vectors, that is, the first feature vector sequence corresponding to the common problem and the second feature vector sequence corresponding to the operation can be locked from the feature vector set.

[0069] In step S105, the cosine similarity algorithm can be used to calculate a target frame with a high similarity to the first feature vector and the second feature vector. The target frame has a high similarity to the image of the operation and the image of the common problem in the knowledge graph. Through the above steps S101 to S105, the target frame related to the user's problem demand can be obtained, thereby filtering out frames in the video that are irrelevant or not very related to the problem demand, so that the length of the short video obtained in step S106 can be greatly reduced, while retaining useful information. In step S106, the preset length of the short video can be determined according to the response time requirements of each e-commerce platform. For example, if the time requirement of a certain e-commerce platform is 30 seconds, then the preset length is preferably set to within 10 seconds.

[0070] It can be seen that in the above embodiment, the user's problem demands for the product (i.e., the problem of the product) can be determined by analyzing the conversation information, and then based on the problem demands and the product knowledge graph, a series of frames most relevant to the problem demands can be extracted from the longer product problem video (these frames include the problem frames containing the problem image and the operation frames that cause the problem), and then the above frames are used to generate a short video with a shorter duration, and the short video is displayed synchronously with the problem demands. Therefore, when the e-commerce customer service receives a longer product problem video, it does not need to watch the product problem video. By watching the corresponding short video, it can accurately grasp the user's demand for the product problem, thereby providing accurate data support for accurate and timely replies to users and winning valuable time, greatly improving the timeliness and quality of the e-commerce customer service's response.

[0071] In some cases, during the communication between e-commerce customer service and users, users may raise new questions and demands. For example, a user raises question demand 1 at time point 1, and the above embodiment can generate a corresponding short video 1 for demand 1; but when the user raises question demand 2 at a subsequent time point 2, the content related to question demand 2 is not included in short video 1. The e-commerce customer service may still need to watch a longer product problem video to understand the content related to question demand 2, which may also cause a response timeout. Therefore, Figure 2 As shown, in order to solve this problem, in one embodiment, the visual question answering processing method further includes the steps of:

[0072] S201, continuously monitoring the conversation information between the customer service and the user in the e-commerce customer service platform to obtain new conversation information;

[0073] S202, judging whether the problem demand has been changed and / or increased according to the newly added dialogue information; if so, executing step S203, extracting corresponding feature vectors from the feature vector set according to the changed problem demand and / or the increased problem demand, and marking them as supplementary feature vectors; and extracting frames from the product problem video whose similarity to the supplementary feature vector is greater than a preset value, and marking them as updated frames;

[0074] S204, updating the short video with the updated frame so that the short video includes frames corresponding to the changed question appeal and the added question appeal, or marking the short video as a first short video, and generating a second short video based on the updated frame.

[0075] That is, in this embodiment, the visual question answering processing method can monitor the user's conversation information in real time, and identify whether the user's question demands have changed or increased, and update the generated short video according to the change or increase of the question demands (applicable when the question demands have changed), or generate a second short video for the increased question demands. Therefore, in this embodiment, the short video can be dynamically adjusted in real time according to the changes in the user's question demands for the product.

[0076] like Figure 3 As shown, in one embodiment, playing the short video and synchronously displaying the question request include the steps of:

[0077] S301, generating a comparison table of the question demand, the target frame and the dialogue information, wherein the comparison table records the target frame corresponding to the question demand and the position of the dialogue content most relevant to the question demand in the dialogue information;

[0078] S302, inserting the question and the position pointer into the target frame in the short video, where the position pointer is used to point to the position in the dialogue information;

[0079] S303. When the short video is played, a position pointer corresponding to the current target frame is obtained according to the comparison table, and the conversation interface between the customer service and the user in the e-commerce customer service platform is jumped to the location of the most relevant conversation content through the position pointer.

[0080] In this embodiment, the short video, the problem and the conversation content related to the problem can be displayed synchronously and dynamically. Among them, the location pointer is used to adjust the content displayed in the chat interface of the e-commerce customer service platform where the e-commerce customer service communicates with the user (its function is similar to the "reference" function in WeChat), and the chat interface can be jumped to the location of the specified content through the location pointer.

[0081] For example, the user's question 1 is obtained based on the conversation content at the 5th line in the conversation information, and question 2 is obtained based on the conversation content at the 20th line in the conversation information; in this case, when playing the generated short video, when the target frame related to question 1 is played, question 1 is displayed on the short video, and the chat interface in the e-commerce customer service platform is jumped to the 5th line through the position pointer; and when the target frame related to question 2 is played, question 2 is displayed on the short video, and the chat interface in the e-commerce customer service platform is jumped to the 20th line through the position pointer. Through this embodiment, the e-commerce customer service can efficiently and synchronously watch the short questions, question demands and key conversation content, so as to have a comprehensive grasp of the entire customer service communication process, especially when the e-commerce customer service personnel are replaced, so that the new customer service personnel can grasp the previous customer service communication process and content in the shortest time.

[0082] In the above embodiment, the visual question answering processing method further includes the steps of:

[0083] Filter out the relevant solutions from the knowledge graph based on the problem requirements;

[0084] And according to the confirmation operation of the customer service personnel, a final solution is determined from the solutions and sent to the user through the e-commerce customer service platform.

[0085] like Figure 4 As shown, in another embodiment, a visual question answering system 400 is provided. The visual question answering system 400 is used to assist e-commerce customer service in answering product questions, thereby improving the timeliness and accuracy of the e-commerce customer service's responses. The visual question answering processing method can be applied to customer service of various e-commerce platforms such as Taobao, JD.com, and Douyin.

[0086] like Figure 4As shown, the visual question answering system 400 includes: a preset module 401, a prediction module 402, a locking module 403, a short video module 404 and a display module 405.

[0087] A preset module 401, a knowledge graph for preset products, wherein the knowledge graph of the products includes the structure, function, operation, common problems and solutions of each product in the e-commerce trading platform, wherein each common problem is associated with an operation that causes the common problem, and the operation and the common problem are both configured with corresponding images, and the knowledge graph is also provided with a feature vector set for the image, wherein the feature vector set stores feature vectors extracted from image encoding of each image using a CNN model;

[0088] The prediction module 402 is used to obtain product problem videos uploaded by users and conversation information between customer service and users from the e-commerce customer service platform; and obtain the product order information of the user from the e-commerce transaction platform; select the corresponding product data from the knowledge graph according to the product order information; and predict the user's problem demands for the product by combining the common problems and the current conversation information;

[0089] The locking module 403 is used to lock the corresponding common problems and the associated operations from the data of the product in the knowledge graph according to the problem demand, and extract the first feature vector corresponding to the locked common problems and the second feature vector corresponding to the operation from the feature vector set;

[0090] The short video module 404 is used to decompose the product problem video into several frames, delete duplicate frames and use the CNN model to extract feature vectors for each remaining frame to obtain a third feature vector; use the cosine similarity algorithm to calculate the similarity between the third feature vector and the first feature vector and the second feature vector, filter out frames with similarities greater than a preset value, and mark them as target frames; arrange the target frames in chronological order and perform smooth transition processing to synthesize a coherent short video with a duration less than a preset length;

[0091] The display module 405 is used to play the short video and simultaneously display the question and demand.

[0092] The CNN model uses a self-attention mechanism to capture feature vectors in the image. In this embodiment, the description of each technical feature and the description of the technical effect in the visual question answering system follow the description in the visual question answering method in the above embodiment, and will not be repeated here.

[0093] In one embodiment, the short video module is also used to: continuously monitor the conversation information between the customer service and the user in the e-commerce customer service platform to obtain new conversation information; determine whether the problem demands have changed and / or increased based on the new conversation information; if so, extract the corresponding feature vector from the feature vector set based on the changed problem demands and / or increased problem demands, and mark it as a supplementary feature vector; and extract frames from the product problem video whose similarity to the supplementary feature vector is greater than a preset value, and mark them as updated frames; update the short video with the updated frames so that the short video includes frames corresponding to the changed problem demands and the increased problem demands, or mark the short video as the first short video, and generate a second short video based on the updated frames.

[0094] In one embodiment, the display module is further used to: generate a comparison table of the question demand, the target frame and the dialogue information, wherein the comparison table records the target frame corresponding to the question demand and the position of the dialogue content most relevant to the question demand in the dialogue information;

[0095] The question and the position pointer are inserted into the target frame in the short video, and the position pointer is used to point to the position in the conversation information; when the short video is played, the position pointer corresponding to the current target frame is obtained according to the comparison table, and the conversation interface between the customer service and the user in the e-commerce customer service platform is jumped to the location of the most relevant conversation content through the position pointer.

[0096] In one embodiment, the visual question answering system 400 further includes:

[0097] The reply module is used to filter out the relevant solutions from the knowledge graph based on the problem demand; and determine the final solution from the solutions according to the confirmation operation of the customer service personnel, and send it to the user through the e-commerce customer service platform.

[0098] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concept of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A visual question answering method, characterized in that: To assist e-commerce customer service in answering product questions, the visual question answering method includes the following steps: A knowledge graph of preset products, wherein the knowledge graph of the products includes the structure, function, operation, common problems and solutions of each product in the e-commerce trading platform, wherein each common problem is associated with an operation corresponding to the common problem, and the operation and the common problem are both configured with corresponding images, and the knowledge graph is also provided with a feature vector set for the image, wherein the feature vector set stores feature vectors extracted from image encoding of each of the images using a CNN model; Obtain product problem videos uploaded by users and conversation information between customer service and users from the e-commerce customer service platform; and obtain product order information of the user from the e-commerce transaction platform; Selecting corresponding product data from the knowledge graph according to the product order information; and predicting the user's problem demands for the product by combining the common questions and the current conversation information; According to the problem demand, corresponding common problems and the associated operations are locked from the data of the product in the knowledge graph, and a first feature vector corresponding to the locked common problem and a second feature vector corresponding to the operation are extracted from the feature vector set; Decompose the product problem video into several frames, delete duplicate frames and use the CNN model to extract feature vectors for each of the remaining frames to obtain a third feature vector; use a cosine similarity algorithm to calculate the similarity between the third feature vector and the first feature vector and the second feature vector, filter out frames whose similarity is greater than a preset value, and mark them as target frames; Arranging the target frames in time sequence and performing smooth transition processing to synthesize a coherent short video that is shorter than a preset length; Play the short video and simultaneously display the question and demand; It also includes the steps of: continuously monitoring the conversation information between the customer service and the user in the e-commerce customer service platform to obtain new conversation information; judging whether the problem demands have changed and / or increased according to the new conversation information; if so, extracting the corresponding feature vector from the feature vector set according to the changed problem demands and / or increased problem demands, and marking it as a supplementary feature vector; and extracting a frame whose similarity with the supplementary feature vector is greater than a preset value from the product problem video, and marking it as an updated frame; updating the short video with the updated frame so that the short video includes frames corresponding to the changed problem demands and the increased problem demands, or marking the short video as a first short video, and generating a second short video based on the updated frame; The playing of the short video and the synchronous display of the question and demand include the steps of: Generate a comparison table of the question, the target frame, and the dialogue information, wherein the comparison table records the target frame corresponding to the question, and the position of the dialogue content most relevant to the question in the dialogue information; The question and the position pointer are inserted into the target frame in the short video, and the position pointer is used to point to the position in the conversation information; when the short video is played, the position pointer corresponding to the current target frame is obtained according to the comparison table, and the conversation interface between the customer service and the user in the e-commerce customer service platform is jumped to the location of the most relevant conversation content through the position pointer.

2. The visual question answering method according to claim 1, characterized in that: Also includes the steps: Filter out relevant solutions from the knowledge graph based on the problem requirements; And according to the confirmation operation of the customer service personnel, a final solution is determined from the solutions and sent to the user through the e-commerce customer service platform.

3. The visual question answering method according to claim 1, characterized in that: The CNN model uses a self-attention mechanism to capture feature vectors in the image.

4. A visual question answering system, characterized in that: Used to assist e-commerce customer service in answering product questions, the visual question answering system includes: A preset module, used for presetting a knowledge graph of a product, wherein the knowledge graph of the product includes the structure, function, operation, common problems and solutions to the problems of each product in the e-commerce trading platform, wherein each common problem is associated with an operation that causes the common problem, and the operation and the common problem are both configured with corresponding images, and the knowledge graph is also provided with a feature vector set for the image, wherein the feature vector set stores feature vectors extracted from image encoding of each of the images using a CNN model; A prediction module is used to obtain product problem videos uploaded by users and conversation information between customer service and users from an e-commerce customer service platform; and obtain the user's product order information from an e-commerce transaction platform; select corresponding product data from the knowledge graph according to the product order information; and predict the user's problem demands for the product by combining the common problems and the current conversation information; A locking module, configured to lock corresponding common problems and the associated operations from the data of the product in the knowledge graph according to the problem demand, and extract a first feature vector corresponding to the locked common problem and a second feature vector corresponding to the operation from the feature vector set; The short video module is used to decompose the product problem video into several frames, delete duplicate frames and use the CNN model to extract feature vectors for each remaining frame to obtain a third feature vector; use the cosine similarity algorithm to calculate the similarity between the third feature vector and the first feature vector and the second feature vector, filter out frames with similarities greater than a preset value, and mark them as target frames; arrange the target frames in chronological order and perform smooth transition processing to synthesize a coherent short video with a duration less than a preset length; A display module, used to play the short video and simultaneously display the question and demand; The short video module is also used to: continuously monitor the conversation information between the customer service and the user in the e-commerce customer service platform to obtain new conversation information; determine whether the problem demands have changed and / or increased according to the new conversation information; if so, extract the corresponding feature vector from the feature vector set according to the changed problem demands and / or increased problem demands, and mark it as a supplementary feature vector; and extract frames with a similarity to the supplementary feature vector greater than a preset value from the product problem video, and mark them as updated frames; update the short video with the updated frames so that the short video includes frames corresponding to the changed problem demands and the increased problem demands, or mark the short video as the first short video, and generate a second short video based on the updated frames; The display module is further used to: generate a comparison table of the question demand, the target frame and the dialogue information, wherein the comparison table records the target frame corresponding to the question demand and the position of the dialogue content most relevant to the question demand in the dialogue information; The question and the position pointer are inserted into the target frame in the short video, and the position pointer is used to point to the position in the conversation information; when the short video is played, the position pointer corresponding to the current target frame is obtained according to the comparison table, and the conversation interface between the customer service and the user in the e-commerce customer service platform is jumped to the location of the most relevant conversation content through the position pointer.

5. The visual question answering system according to claim 4, characterized in that: Also includes: A reply module, used for filtering out relevant solutions from the knowledge graph based on the problem demand; And according to the confirmation operation of the customer service personnel, a final solution is determined from the solutions and sent to the user through the e-commerce customer service platform.

6. The visual question answering system according to claim 4, characterized in that: The CNN model uses a self-attention mechanism to capture feature vectors in the image.

Images