Method, device and storage medium for human-machine solution consistency detection in customer service field

Abstract

The present disclosure relates to a method, device and storage medium for detecting consistency between human-machine solutions in the field of customer service. The method comprises: extracting a machine conversation from a first conversation record between a robot customer service and a customer, and extracting a human conversation from a second conversation record between a human customer service and a customer; performing data preprocessing on the machine conversation and the human conversation; determining a first core solution of the machine conversation that has undergone data preprocessing and a second core solution of the human conversation that has undergone data preprocessing through a deep learning model; calculating a consistency score between the first core solution and the second core solution through a deep learning model; and determining a consistency rate of core solutions corresponding to the machine conversation and the human conversation according to the consistency score.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition, and in particular to a method, device and storage medium for detecting consistency between human-machine solutions in the customer service field. Background Art

[0002] In the field of e-commerce, using human or robot customer service to receive customers, answer customers' relevant questions in the pre-sales and after-sales fields, help customers successfully complete the entire shopping process, and improve customers' shopping experience is a relatively common solution in the industry. In the process of robot customer service receiving customers, there will inevitably be some problems that robot customer service cannot solve. In this case, the user will actively request to transfer to human customer service. The data of robot customer service transferring to human customer service can just reflect the shortcomings of the robot, so the analysis and optimization of this part of the data can well improve the robot's response ability, which is also particularly important. At the same time, the solutions given by robot customer service and human customer service are different, and the solution of human customer service will be better than that of robot customer service. By comparing whether the two solutions are consistent, it can be evaluated whether the solution given by robot customer service is excellent and whether there are points for improvement in robot customer service. The existing technology lacks a method to compare whether the two solutions are consistent.

[0003] In the process of implementing the concept of the present disclosure, the inventors discovered that there are at least the following technical problems in the related technology: the prior art lacks a method for comparing whether the solutions of robot customer service and manual customer service are consistent. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a method, device and storage medium for detecting the consistency of human-machine solutions in the customer service field, so as to at least solve the problem in the prior art that there is a lack of a method for comparing whether the solutions of robot customer service and manual customer service are consistent.

[0005] The purpose of this disclosure is achieved through the following technical solutions:

[0006] In a first aspect, an embodiment of the present disclosure provides a method for detecting consistency between human-machine solutions in the field of customer service, including: extracting a machine conversation from a first conversation record between a robot customer service and a customer, and extracting a manual conversation from a second conversation record between a manual customer service and the customer; performing data preprocessing on the machine conversation and the manual conversation; determining a first core solution of the machine conversation after the data preprocessing and a second core solution of the manual conversation after the data preprocessing through a deep learning model; calculating a consistency score between the first core solution and the second core solution through the deep learning model, wherein the deep learning model has learned through deep learning the correspondence between the machine conversation and the first core solution, the correspondence between the manual conversation and the second core solution, and the correspondence between the first core solution and the second core solution and the consistency score; and determining the consistency rate of the core solutions corresponding to the machine conversation and the manual conversation according to the consistency score.

[0007] In an exemplary embodiment, determining a first core solution for a machine conversation that has undergone data preprocessing and a second core solution for a manual conversation that has undergone data preprocessing through a deep learning model includes: extracting first conversation features for the machine conversation that has undergone data preprocessing and second conversation features for the manual conversation that has undergone data preprocessing; determining the first core solution based on the first conversation features, and determining the second core solution based on the second conversation features.

[0008] In an exemplary embodiment, determining the first core solution according to the first session feature and determining the second core solution according to the second session feature includes: acquiring a core solution set; when the similarity between the first session feature and the core solution set is greater than a first preset threshold, determining a first segmented session corresponding to the first session feature as the first core solution, wherein the first segmented session is obtained by performing the data preprocessing on the machine session; when the similarity between the second session feature and the core solution set is greater than the first preset threshold, determining a second segmented session corresponding to the second session feature as the second core solution, wherein the second segmented session is obtained by performing the data preprocessing on the manual session.

[0009] In an exemplary embodiment, the first session feature and the second session feature each include at least one of the following: a first feature for indicating the business meaning of a split session, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: a first split session and a second split session; a second feature for indicating whether the customer service answer in the split session is a question sentence; a third feature for indicating key information of the customer service answer in the split session; a fourth feature for indicating the sentence length of the split session; a fifth feature for indicating the sequence number of the split session in the machine session or the manual session; and a sixth feature for indicating the business meaning of the previous split session of the split session.

[0010] In an exemplary embodiment, the core solution set includes: a first object, used to indicate that a split session has business meaning, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: the first split session and the second split session; a second object, used to indicate that the customer service answer in the split session is not a question sentence; a third object, used to indicate that the customer service answer in the split session contains key information; a fourth object, used to indicate that the sentence length of the split session is in a first preset interval; a fifth object, used to indicate that the sequence number of the split session in the machine session or the manual session is in a second preset interval; and a sixth object, used to indicate that the previous split session of the split session has business meaning.

[0011] In an exemplary embodiment, the method includes: obtaining historical conversation data, wherein the historical conversation data includes: machine conversation history data and manual conversation history data; extracting conversation features of the historical conversation data; and performing deep learning training on the deep learning model through the historical conversation data and the conversation features.

[0012] In an exemplary embodiment, the data preprocessing of the machine conversation and the manual conversation includes: filtering the machine conversation and the manual conversation according to a noise reduction strategy; and / or segmenting the machine conversation and the manual conversation respectively according to a segmentation logic to obtain a plurality of segmented conversations respectively.

[0013] In an exemplary embodiment, segmenting the machine conversation and the manual conversation respectively according to the segmentation logic to obtain a plurality of segmented conversations respectively includes: determining a segmentation code according to the segmentation logic, wherein the segmentation logic includes: one question followed by one answer, multiple questions followed by one answer, one question followed by multiple answers, and multiple questions followed by multiple answers; segmenting the machine conversation and the manual conversation respectively according to the segmentation code to obtain the plurality of segmented conversations respectively.

[0014] In a second aspect, an embodiment of the present disclosure provides a device for detecting consistency between human-machine solutions in the field of customer service, comprising: an extraction module, used to extract a machine conversation from a first conversation record between a robot customer service and a customer, and to extract a manual conversation from a second conversation record between a manual customer service and the customer; a preprocessing module, used to perform data preprocessing on the machine conversation and the manual conversation; a first determination module, used to determine a first core solution of the machine conversation after the data preprocessing, a second core solution of the manual conversation after the data preprocessing, and a consistency score between the first core solution and the second core solution through a deep learning model; including: extracting a first conversation feature of the machine conversation after the data preprocessing and a second conversation feature of the manual conversation after the data preprocessing; determining a first core solution according to the similarity between the first conversation feature and the acquired core solution set; determining a second core solution according to the similarity between the second conversation feature and the core solution set; wherein the deep learning model has learned the correspondence between the machine conversation and the first core solution, the correspondence between the manual conversation and the second core solution, and the correspondence between the first core solution and the second core solution and the consistency score through deep learning; and a second determination module, used to determine the consistency rate of the core solutions corresponding to the machine conversation and the manual conversation according to the consistency score.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the above-mentioned method for detecting consistency between human-machine solutions in the customer service field or the method for image processing when executing the program stored in the memory.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting consistency between human-machine solutions in the customer service field or the method for image processing as described above.

[0017] The above-mentioned technical scheme provided by the embodiment of the present disclosure has at least some or all of the following advantages compared with the prior art: extracting machine conversations from the first conversation record between the robot customer service and the customer, and extracting manual conversations from the second conversation record between the manual customer service and the customer; performing data preprocessing on the machine conversations and the manual conversations; determining a first core scheme of the machine conversation after the data preprocessing and a second core scheme of the manual conversation after the data preprocessing through a deep learning model; calculating the consistency score of the first core scheme and the second core scheme through the deep learning model, wherein the deep learning model has learned the correspondence between the machine conversation and the first core scheme, the correspondence between the manual conversation and the second core scheme, and the correspondence between the first core scheme and the second core scheme and the consistency score through deep learning; determining the consistency rate of the core schemes corresponding to the machine conversation and the manual conversation according to the consistency score. Because the disclosed embodiment can determine the first core solution of the machine conversation and the second core solution of the manual conversation through the deep learning model, and can also calculate the consistency score of the first core solution and the second core solution through the deep learning model. Therefore, the above-mentioned technical means can solve the problem in the prior art that there is a lack of methods to compare whether the solutions of robot customer service and manual customer service are consistent, and further provide a solution for calculating the consistency rate of the solutions of robot customer service and manual customer service. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A hardware structure block diagram of a computer terminal for a method for detecting consistency between a human-machine solution in the customer service field according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 2 A flowchart schematically shows a method for detecting consistency between human-machine solutions in the customer service field according to an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of a process flow of a method for detecting consistency between a human-machine solution in the customer service field according to an embodiment of the present disclosure is shown;

[0023] Figure 4 A structural block diagram schematically shows a device for detecting consistency between human-machine solutions in the customer service field according to an embodiment of the present disclosure;

[0024] Figure 5 The structural block diagram of an electronic device provided by an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present disclosure may be combined with each other without conflict.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0027] The method embodiments provided in the embodiments of the present disclosure can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 The hardware structure block diagram of a computer terminal of a method for detecting consistency between human-machine solutions in the customer service field according to an embodiment of the present disclosure is schematically shown. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (the processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD) and other processing devices and a memory 104 for storing data. Optionally, the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for detecting the consistency of human-machine solutions in the customer service field in the embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In an embodiment of the present disclosure, a method for detecting consistency between human-machine solutions in the customer service field is provided. Figure 2 A flowchart of a method for detecting consistency between a human-machine solution in the customer service field according to an embodiment of the present disclosure is schematically shown. Figure 2 As shown, the process includes the following steps:

[0031] Step S202, extracting a machine conversation from a first conversation record between a robot customer service and a customer, and extracting a human conversation from a second conversation record between a human customer service and the customer;

[0032] Step S204, performing data preprocessing on the machine conversation and the manual conversation;

[0033] Step S206, determining a first core solution of the machine conversation after the data preprocessing and a second core solution of the manual conversation after the data preprocessing through a deep learning model;

[0034] Step S208, calculating the consistency score between the first core solution and the second core solution through the deep learning model, wherein the deep learning model has learned the correspondence between the machine conversation and the first core solution, the correspondence between the manual conversation and the second core solution, and the correspondence between the first core solution, the second core solution and the consistency score through deep learning;

[0035] Step S210: determining the consistency rate of the core solutions corresponding to the machine conversation and the manual conversation according to the consistency score.

[0036] Through the present disclosure, a machine conversation is extracted from a first conversation record between a robot customer service and a customer, and a human conversation is extracted from a second conversation record between a human customer service and the customer; data preprocessing is performed on the machine conversation and the human conversation; a first core solution of the machine conversation after the data preprocessing and a second core solution of the human conversation after the data preprocessing are determined by a deep learning model; a consistency score of the first core solution and the second core solution is calculated by the deep learning model, wherein the deep learning model has learned the correspondence between the machine conversation and the first core solution, the correspondence between the human conversation and the second core solution, and the correspondence between the first core solution and the second core solution and the consistency score through deep learning; and the consistency rate of the core solutions corresponding to the machine conversation and the human conversation is determined according to the consistency score. Because the embodiment of the present disclosure can determine the first core solution of the machine conversation and the second core solution of the human conversation through a deep learning model, and can also calculate the consistency score of the first core solution and the second core solution through the deep learning model, therefore, the above technical means can be used to solve the problem of the lack of a method for comparing whether the solutions of the robot customer service and the human customer service are consistent in the prior art, and then provide a solution for calculating the consistency rate of the solutions of the robot customer service and the human customer service.

[0037] In step S206, a first core solution of the machine conversation after the data preprocessing and a second core solution of the manual conversation after the data preprocessing are determined by a deep learning model, including: extracting first conversation features of the machine conversation after the data preprocessing and second conversation features of the manual conversation after the data preprocessing; determining the first core solution according to the first conversation features, and determining the second core solution according to the second conversation features.

[0038] Because whether it is a machine conversation or a manual conversation, there will be a lot of useless information, such as the conversation between customer service and customers greeting each other, which is useless information. The disclosed embodiment determines the first core solution of the machine conversation and the second core solution of the manual conversation through a deep learning model, and then calculates the consistency score of the first core solution and the second core solution, which can reduce the interference of unnecessary useless information. It should be noted that there are differences in the extraction of the core solutions of machine conversations and manual conversations. When extracting the core solution of machine conversations, the filtering conditions are different from those of the extraction of the core solution of manual conversations during data preprocessing, and some keyword filtering of invalid machine answers needs to be added.

[0039] For example, machine answers containing the following keywords can be filtered out: 'In order to better serve you', 'Hi~ Nice to meet you', 'Dear PLUS member', 'Good afternoon', 'Good afternoon', 'Good morning', 'Good evening', 'I have questions to consult', 'If you encounter problems when shopping, you can try to click the button below to solve it', 'Hi~ Enter the shopping problem you encountered', 'Please describe the problem you want to consult in detail', 'Please click the button below to transfer to manual customer service for you', 'Hi~, buy good things on JD.com, JIMI will solve your problems', 'After receiving your feedback, Jimi will go back and study hard', 'Contact manual', 'Consult other questions', 'How can I help you', 'None of the above', 'How can I serve you', 'The inconvenience caused to you', 'I hope you can calm down', 'What questions do you want to consult', 'Guess you are also interested in the following questions'……

[0040] Determining the first core solution according to the first session feature and determining the second core solution according to the second session feature includes: acquiring a core solution set; when the similarity between the first session feature and the core solution set is greater than a first preset threshold, determining a first segmented session corresponding to the first session feature as the first core solution, wherein the first segmented session is obtained by performing the data preprocessing on the machine session; when the similarity between the second session feature and the core solution set is greater than the first preset threshold, determining a second segmented session corresponding to the second session feature as the second core solution, wherein the second segmented session is obtained by performing the data preprocessing on the manual session.

[0041] The disclosed embodiment adopts a divide-and-conquer concept to break down a relatively complex problem into multiple relatively easy problems. Specifically, the machine conversation and the manual conversation are first broken down into a number of segmented conversations, and then the segmented conversation is judged from the dimension of the segmented conversation to determine whether it is a core solution. Among them, the machine conversation and the manual conversation are broken down into a number of first segmented conversations, and the manual conversation and the manual conversation are broken down into a number of second segmented conversations.

[0042] In an optional embodiment, a non-core solution set is obtained; when the similarity between the first session feature and the non-core solution set is greater than a second preset threshold, a first segmented session corresponding to the first session feature is determined as the first non-core solution, wherein the first segmented session is obtained by performing the data preprocessing on the machine session; when the similarity between the second session feature and the non-core solution set is greater than the second preset threshold, a segmented session corresponding to the second session feature is determined as the second non-core solution, wherein the second segmented session is obtained by performing the data preprocessing on the manual session.

[0043] The first session feature and the second session feature each include at least one of the following: a first feature, used to indicate the business meaning of a split session, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: a first split session and a second split session; a second feature, used to indicate whether a customer service answer in the split session is a question sentence; a third feature, used to indicate key information of a customer service answer in the split session; a fourth feature, used to indicate a sentence length of the split session; a fifth feature, used to indicate a sequence number of the split session in the machine session or the manual session; and a sixth feature, used to indicate the business meaning of a previous split session of the split session.

[0044] The segmented conversations of the core solution often have the following characteristics: have specific business meanings; the customer service answers in the segmented conversations are affirmative sentences; have key information: such as key action words, such as "review, apply, handle, process, follow up, implement, call back", and key business words, such as "after-sales, price guarantee, return, exchange, repair, refund, logistics, compensation"; the sentence length of the segmented conversation is in the first preset interval, such as the sentence length is in the interval of 7-25 words; the sequence number of the segmented conversation in the machine conversation or the manual conversation is in the second preset interval, such as the 1st to 5th or the last 3 segmented conversation intervals; the previous segmented conversation of the segmented conversation has business meanings. The disclosed embodiment can determine whether the segmented conversation corresponding to the conversation feature is a core solution by calculating the similarity between the conversation feature and the core solution set.

[0045] The core solution set includes: a first object, used to indicate that the split session has business meaning, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: the first split session and the second split session; a second object, used to indicate that the customer service answer in the split session is not a question sentence; a third object, used to indicate that the customer service answer in the split session contains key information; a fourth object, used to indicate that the sentence length of the split session is in a first preset interval; a fifth object, used to indicate that the sequence number of the split session in the machine session or the manual session is in a second preset interval; and a sixth object, used to indicate that the previous split session of the split session has business meaning.

[0046] In an optional embodiment, the non-core solution set includes: a first object, used to indicate that the split session has no business meaning, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: the first split session and the second split session; a second object, used to indicate that the customer service answer in the split session is a question sentence; a third object, used to indicate that the customer service answer in the split session does not contain key information; a fourth object, used to indicate that the sentence length of the split session is not in a first preset interval; a fifth object, used to indicate that the sequence number of the split session in the machine session or the manual session is not in a second preset interval; a sixth object, used to indicate that the previous split session of the split session has no business meaning.

[0047] Training the deep learning model includes: acquiring historical conversation data, wherein the historical conversation data includes: machine conversation history data and manual conversation history data; extracting conversation features of the historical conversation data; and performing deep learning training on the deep learning model through the historical conversation data and the conversation features.

[0048] The embodiment of the present disclosure adds a layer of LSTM encoding network to the deep learning model. The input of the LSTM encoding network is a session, and the output is the sentence vector of the session. The deep learning model is trained through the historical session data and the session features, which is actually the deep learning model trained through the sentence vector of the historical session data and the session features. The trained deep learning model can process the sentence vector and the session features uniformly, and at the same time has the ability to find the core solution corresponding to the session and calculate the consistency score of the first core solution and the second core solution. Among them, the LSTM encoding network is a long short-term memory network. The deep learning model has learned the correspondence between the machine session and the first core solution, and the correspondence between the artificial session and the second core solution through deep learning. In fact, the deep learning model has learned the correspondence between the machine session and the first session feature, and the correspondence between the artificial session and the second session feature through deep learning. Among them, it can be judged whether the first segmented session of the machine session is the first core solution according to the first session feature, and whether the second segmented session of the artificial session is the second core solution according to the second session feature.

[0049] Sentence vectors can also be obtained by first using word2vec to obtain the word vectors of the conversation, and then a layer of LSTM encoding is performed based on the word vectors to obtain the sentence vector information. Based on the above results, the cosine similarity of the two vectors can be directly calculated to obtain the consistency score. Alternatively, BERT can be used for matching, and the sentences of the manual and machine core solutions can be directly input to let the model give a consistency score.

[0050] In step 204, data preprocessing is performed on the machine conversation and the manual conversation, including: filtering the machine conversation and the manual conversation according to a noise reduction strategy; and / or segmenting the machine conversation and the manual conversation according to a segmentation logic to obtain a plurality of segmented conversations.

[0051] The noise reduction strategy filters the machine conversation and the manual conversation, which actually filters out noise information, reduces the complexity of the problem, and facilitates the subsequent core solution extraction. The noise reduction strategy means that if any one of the following conditions is met, filtering will be performed: the length of the customer's question is less than or equal to 3 characters; the customer's question does not contain Chinese; the customer's question and any sentence in the following set have an LSC common subsequence score greater than 0.8. The set mainly includes some meaningless words said by the user, including "OK, thank you, OK, please, please, please, please, please, please, please, please, please". The length of the customer service answer is less than or equal to 3; the customer service answer does not contain Chinese; the customer service answer and any sentence in the following set have an LSC score greater than 0.8. The set mainly includes some meaningless words said by the customer service, including "what can I do for you, it's okay, yes, please provide your order, please provide your order, please wait, please wait patiently, checking, processing, my sister will handle it for you, please take a look, please take a look, do you think it's okay, do you think it's okay".

[0052] The machine conversation and the manual conversation are respectively divided according to the division logic to obtain a plurality of divided conversations, including: determining a division code according to the division logic, wherein the division logic includes: one question followed by one answer, multiple questions followed by one answer, one question followed by multiple answers, and multiple questions followed by multiple answers; and the machine conversation and the manual conversation are respectively divided according to the division code to obtain the plurality of divided conversations.

[0053] The segmentation logic uses the idea of ​​divide and conquer or something similar to divide and conquer to simplify complex problems. It is difficult to find the core solution in a long conversation, but it is relatively easy to find the key answer in a question-answer pair. The segmentation logic can be segmented according to the following rules: one question followed by one answer, multiple questions followed by one answer, one question followed by multiple answers, and multiple questions followed by multiple answers.

[0054] Table 1 is a long conversation, which includes multiple rounds of questions and answers, as shown in the following table:

[0055]

[0056] "0" is the mark of the non-core solution, and "1" is the mark of the core solution.

[0057] Table 1 is a segmented session obtained after segmenting a long session, as shown in the following table:

[0058]

[0059] Pair indicates a split session, and Pair1 indicates the split session with sequence number 1 in a long session.

[0060] It should be noted that before data preprocessing is performed on the machine conversation and the manual conversation, desensitization processing may also be performed on the machine conversation and the manual conversation.

[0061] In order to better understand the above technical solution, the embodiments of the present disclosure also provide an optional embodiment for explaining the above technical solution.

[0062] Figure 3 The following is a schematic diagram showing a flow chart of a method for detecting consistency between human-machine solutions in the customer service field according to an embodiment of the present disclosure. Figure 3 As shown:

[0063] S302: extracting a machine conversation from a first conversation record between a robot customer service and a customer, and extracting a human conversation from a second conversation record between a human customer service and the customer;

[0064] S304: Desensitizing the machine conversation and the manual conversation;

[0065] S306: Filtering the machine conversation and the manual conversation according to a noise reduction strategy;

[0066] S308: Segment the machine conversation and the manual conversation respectively according to the segmentation logic to obtain a plurality of segmentation conversation results, wherein the machine conversation is segmented to obtain a plurality of first segmentation conversations, and the manual conversation is segmented to obtain a plurality of second segmentation conversations;

[0067] S310: Determine, by using a deep learning model, whether the plurality of first segmentation sessions are the first core solution and whether the plurality of second segmentation sessions are the second core solution;

[0068] S312: Calculating the consistency score between the first core solution and the second core solution through the deep learning model;

[0069] S314: Determine the consistency rate of the core solutions corresponding to the machine conversation and the manual conversation according to the consistency score.

[0070] Through the present disclosure, a machine conversation is extracted from a first conversation record between a robot customer service and a customer, and a human conversation is extracted from a second conversation record between a human customer service and the customer; data preprocessing is performed on the machine conversation and the human conversation; a first core solution of the machine conversation after the data preprocessing and a second core solution of the human conversation after the data preprocessing are determined by a deep learning model; a consistency score of the first core solution and the second core solution is calculated by the deep learning model, wherein the deep learning model has learned the correspondence between the machine conversation and the first core solution, the correspondence between the human conversation and the second core solution, and the correspondence between the first core solution and the second core solution and the consistency score through deep learning; and the consistency rate of the core solutions corresponding to the machine conversation and the human conversation is determined according to the consistency score. Because the embodiment of the present disclosure can determine the first core solution of the machine conversation and the second core solution of the human conversation through a deep learning model, and can also calculate the consistency score of the first core solution and the second core solution through the deep learning model, therefore, the above technical means can be used to solve the problem of the lack of a method for comparing whether the solutions of the robot customer service and the human customer service are consistent in the prior art, and then provide a solution for calculating the consistency rate of the solutions of the robot customer service and the human customer service.

[0071] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk), including a number of instructions for a terminal device (which can be a mobile phone, a computer, a component server, or a network device, etc.) to execute the methods of each embodiment of the present disclosure.

[0072] In the present embodiment, a device for detecting the consistency of human-machine solutions in the field of customer service is also provided. The device for detecting the consistency of human-machine solutions in the field of customer service is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0073] Figure 4 The structure block diagram of a device for detecting consistency between human-machine solutions in the customer service field according to an optional embodiment of the present disclosure is schematically shown. Figure 4 As shown, the device comprises:

[0074] An extraction module 402 is used to extract a machine conversation from a first conversation record between a robot customer service and a customer, and to extract a human conversation from a second conversation record between a human customer service and the customer;

[0075] A preprocessing module 404, configured to perform data preprocessing on the machine conversation and the manual conversation;

[0076] A first determination module 406, configured to determine, by a deep learning model, a first core solution for the machine conversation after the data preprocessing and a second core solution for the human conversation after the data preprocessing;

[0077] A calculation module 408 is used to calculate the consistency score of the first core solution and the second core solution through the deep learning model, wherein the deep learning model has learned the correspondence between the machine conversation and the first core solution, the correspondence between the manual conversation and the second core solution, and the correspondence between the first core solution, the second core solution and the consistency score through deep learning;

[0078] The second determination module 410 is configured to determine a consistency rate between the core solutions corresponding to the machine conversation and the manual conversation according to the consistency score.

[0079] Through the present disclosure, a machine conversation is extracted from a first conversation record between a robot customer service and a customer, and a human conversation is extracted from a second conversation record between a human customer service and the customer; data preprocessing is performed on the machine conversation and the human conversation; a first core solution of the machine conversation after the data preprocessing and a second core solution of the human conversation after the data preprocessing are determined by a deep learning model; a consistency score of the first core solution and the second core solution is calculated by the deep learning model, wherein the deep learning model has learned the correspondence between the machine conversation and the first core solution, the correspondence between the human conversation and the second core solution, and the correspondence between the first core solution and the second core solution and the consistency score through deep learning; and the consistency rate of the core solutions corresponding to the machine conversation and the human conversation is determined according to the consistency score. Because the embodiment of the present disclosure can determine the first core solution of the machine conversation and the second core solution of the human conversation through a deep learning model, and can also calculate the consistency score of the first core solution and the second core solution through the deep learning model, therefore, the above technical means can be used to solve the problem of the lack of a method for comparing whether the solutions of the robot customer service and the human customer service are consistent in the prior art, and then provide a solution for calculating the consistency rate of the solutions of the robot customer service and the human customer service.

[0080] Optionally, the first determination module 406 is further used to extract a first session feature of the machine session after the data preprocessing and a second session feature of the manual session after the data preprocessing; determine the first core solution according to the first session feature, and determine the second core solution according to the second session feature.

[0081] Because whether it is a machine conversation or a manual conversation, there will be a lot of useless information, such as the conversation between customer service and customers greeting each other, which is useless information. The disclosed embodiment determines the first core solution of the machine conversation and the second core solution of the manual conversation through a deep learning model, and then calculates the consistency score of the first core solution and the second core solution, which can reduce the interference of unnecessary useless information. It should be noted that there are differences in the extraction of the core solutions of machine conversations and manual conversations. When extracting the core solution of machine conversations, the filtering conditions are different from those of the extraction of the core solution of manual conversations during data preprocessing, and some keyword filtering of invalid machine answers needs to be added.

[0082] For example, machine answers containing the following keywords can be filtered out: 'In order to better serve you', 'Hi~ Nice to meet you', 'Dear PLUS member', 'Good afternoon', 'Good afternoon', 'Good morning', 'Good evening', 'I have questions to consult', 'If you encounter problems when shopping, you can try to click the button below to solve it', 'Hi~ Enter the shopping problem you encountered', 'Please describe the problem you want to consult in detail', 'Please click the button below to transfer to manual customer service for you', 'Hi~, buy good things on JD.com, JIMI will solve your problems', 'After receiving your feedback, Jimi will go back and study hard', 'Contact manual', 'Consult other questions', 'How can I help you', 'None of the above', 'How can I serve you', 'The inconvenience caused to you', 'I hope you can calm down', 'What questions do you want to consult', 'Guess you are also interested in the following questions'……

[0083] Optionally, the first determination module 406 is further configured to obtain a core solution set; when the similarity between the first session feature and the core solution set is greater than a first preset threshold, determine a first segmented session corresponding to the first session feature as the first core solution, wherein the first segmented session is obtained by performing the data preprocessing on the machine session; when the similarity between the second session feature and the core solution set is greater than the first preset threshold, determine a second segmented session corresponding to the second session feature as the second core solution, wherein the second segmented session is obtained by performing the data preprocessing on the manual session.

[0084] The disclosed embodiment adopts a divide-and-conquer concept to break down a relatively complex problem into multiple relatively easy problems. Specifically, the machine conversation and the manual conversation are first broken down into a number of segmented conversations, and then the segmented conversation is judged from the dimension of the segmented conversation to determine whether it is a core solution. Among them, the machine conversation and the manual conversation are broken down into a number of first segmented conversations, and the manual conversation and the manual conversation are broken down into a number of second segmented conversations.

[0085] Optionally, the first determination module 406 is further configured to obtain a set of non-core solutions; when the similarity between the first session feature and the set of non-core solutions is greater than a second preset threshold, determine a first split session corresponding to the first session feature as the first non-core solution, wherein the first split session is obtained by performing the data preprocessing on the machine session; when the similarity between the second session feature and the set of non-core solutions is greater than the second preset threshold, determine a split session corresponding to the second session feature as the second non-core solution, wherein the second split session is obtained by performing the data preprocessing on the manual session.

[0086] The first session feature and the second session feature each include at least one of the following: a first feature, used to indicate the business meaning of a split session, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: a first split session and a second split session; a second feature, used to indicate whether a customer service answer in the split session is a question sentence; a third feature, used to indicate key information of a customer service answer in the split session; a fourth feature, used to indicate a sentence length of the split session; a fifth feature, used to indicate a sequence number of the split session in the machine session or the manual session; and a sixth feature, used to indicate the business meaning of a previous split session of the split session.

[0087] The segmented conversations of the core solution often have the following characteristics: have specific business meanings; the customer service answers in the segmented conversations are affirmative sentences; have key information: such as key action words, such as "review, apply, handle, process, follow up, implement, call back", and key business words, such as "after-sales, price guarantee, return, exchange, repair, refund, logistics, compensation"; the sentence length of the segmented conversation is in the first preset interval, such as the sentence length is in the interval of 7-25 words; the sequence number of the segmented conversation in the machine conversation or the manual conversation is in the second preset interval, such as the 1st to 5th or the last 3 segmented conversation intervals; the previous segmented conversation of the segmented conversation has business meanings. The disclosed embodiment can determine whether the segmented conversation corresponding to the conversation feature is a core solution by calculating the similarity between the conversation feature and the core solution set.

[0088] The core solution set includes: a first object, used to indicate that the split session has business meaning, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: the first split session and the second split session; a second object, used to indicate that the customer service answer in the split session is not a question sentence; a third object, used to indicate that the customer service answer in the split session contains key information; a fourth object, used to indicate that the sentence length of the split session is in a first preset interval; a fifth object, used to indicate that the sequence number of the split session in the machine session or the manual session is in a second preset interval; and a sixth object, used to indicate that the previous split session of the split session has business meaning.

[0089] In an optional embodiment, the non-core solution set includes: a first object, used to indicate that the split session has no business meaning, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: the first split session and the second split session; a second object, used to indicate that the customer service answer in the split session is a question sentence; a third object, used to indicate that the customer service answer in the split session does not contain key information; a fourth object, used to indicate that the sentence length of the split session is not in a first preset interval; a fifth object, used to indicate that the sequence number of the split session in the machine session or the manual session is not in a second preset interval; a sixth object, used to indicate that the previous split session of the split session has no business meaning.

[0090] Optionally, the computing module 408 is also used to obtain historical conversation data, wherein the historical conversation data includes: machine conversation history data and manual conversation history data; extract conversation features of the historical conversation data; and perform deep learning training on the deep learning model through the historical conversation data and the conversation features.

[0091] The embodiment of the present disclosure adds a layer of LSTM encoding network to the deep learning model. The input of the LSTM encoding network is a session, and the output is the sentence vector of the session. The deep learning model is trained through the historical session data and the session features, which is actually the deep learning model trained through the sentence vector of the historical session data and the session features. The trained deep learning model can process the sentence vector and the session features uniformly, and at the same time has the ability to find the core solution corresponding to the session and calculate the consistency score of the first core solution and the second core solution. Among them, the LSTM encoding network is a long short-term memory network. The deep learning model has learned the correspondence between the machine session and the first core solution, and the correspondence between the artificial session and the second core solution through deep learning. In fact, the deep learning model has learned the correspondence between the machine session and the first session feature, and the correspondence between the artificial session and the second session feature through deep learning. Among them, it can be judged whether the first segmented session of the machine session is the first core solution according to the first session feature, and whether the second segmented session of the artificial session is the second core solution according to the second session feature.

[0092] Optionally, the preprocessing module 404 is further configured to filter the machine conversation and the manual conversation according to a noise reduction strategy; and / or segment the machine conversation and the manual conversation respectively according to a segmentation logic to obtain a plurality of segmented conversations respectively.

[0093] The noise reduction strategy filters the machine conversation and the manual conversation, which actually filters out noise information, reduces the complexity of the problem, and facilitates the subsequent core solution extraction. The noise reduction strategy means that if any one of the following conditions is met, filtering will be performed: the length of the customer's question is less than or equal to 3 characters; the customer's question does not contain Chinese; the customer's question and any sentence in the following set have an LSC common subsequence score greater than 0.8. The set mainly includes some meaningless words said by the user, including "OK, thank you, OK, please, please, please, please, please, please, please, please, please". The length of the customer service answer is less than or equal to 3; the customer service answer does not contain Chinese; the customer service answer and any sentence in the following set have an LSC score greater than 0.8. The set mainly includes some meaningless words said by the customer service, including "what can I do for you, it's okay, yes, please provide your order, please provide your order, please wait, please wait patiently, checking, processing, my sister will handle it for you, please take a look, please take a look, do you think it's okay, do you think it's okay".

[0094] Optionally, the preprocessing module 404 is also used to determine a segmentation code according to the segmentation logic, wherein the segmentation logic includes: one question followed by one answer, multiple questions followed by one answer, one question followed by multiple answers, and multiple questions followed by multiple answers; the machine conversation and the manual conversation are segmented respectively according to the segmentation code to obtain the multiple segmented conversations respectively.

[0095] The segmentation logic uses the idea of ​​divide and conquer or something similar to divide and conquer to simplify complex problems. It is difficult to find the core solution in a long conversation, but it is relatively easy to find the key answer in a question-answer pair. The segmentation logic can be segmented according to the following rules: one question followed by one answer, multiple questions followed by one answer, one question followed by multiple answers, and multiple questions followed by multiple answers.

[0096] Table 1 is a long conversation, which includes multiple rounds of questions and answers, as shown in the following table:

[0097]

[0098] "0" is the mark of the non-core solution, and "1" is the mark of the core solution.

[0099] Table 1 is a segmented session obtained after segmenting a long session, as shown in the following table:

[0100]

[0101] Pair indicates a split session, and Pair1 indicates the split session with sequence number 1 in a long session.

[0102] It should be noted that before data preprocessing is performed on the machine conversation and the manual conversation, desensitization processing may also be performed on the machine conversation and the manual conversation.

[0103] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0104] An embodiment of the present disclosure provides an electronic device.

[0105] Figure 5 The structural block diagram of an electronic device provided by an embodiment of the present disclosure is schematically shown.

[0106] Reference Figure 5 As shown, the electronic device 500 provided by the embodiment of the present disclosure includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 communicate with each other through the communication bus 504; the memory 503 is used to store computer programs; the processor 501 is used to implement the steps in any of the above method embodiments when executing the program stored in the memory.

[0107] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the input / output device is connected to the processor.

[0108] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0109] S1, extracting a machine conversation from a first conversation record between a robot customer service and a customer, and extracting a human conversation from a second conversation record between a human customer service and the customer;

[0110] S2, performing data preprocessing on the machine conversation and the manual conversation;

[0111] S3, determining a first core solution for machine conversation after the data preprocessing and a second core solution for human conversation after the data preprocessing through a deep learning model;

[0112] S4, calculating the consistency score between the first core solution and the second core solution through the deep learning model, wherein the deep learning model has learned the correspondence between the machine conversation and the first core solution, the correspondence between the manual conversation and the second core solution, and the correspondence between the first core solution, the second core solution and the consistency score through deep learning;

[0113] S5, determining a consistency rate between the core solutions corresponding to the machine conversation and the manual conversation according to the consistency score.

[0114] The embodiments of the present disclosure further provide a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0115] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0116] S1, extracting a machine conversation from a first conversation record between a robot customer service and a customer, and extracting a human conversation from a second conversation record between a human customer service and the customer;

[0117] S2, performing data preprocessing on the machine conversation and the manual conversation;

[0118] S3, determining a first core solution for machine conversation after the data preprocessing and a second core solution for human conversation after the data preprocessing through a deep learning model;

[0119] S4, calculating the consistency score between the first core solution and the second core solution through the deep learning model, wherein the deep learning model has learned the correspondence between the machine conversation and the first core solution, the correspondence between the manual conversation and the second core solution, and the correspondence between the first core solution, the second core solution and the consistency score through deep learning;

[0120] S5, determining a consistency rate between the core solutions corresponding to the machine conversation and the manual conversation according to the consistency score.

[0121] The computer-readable storage medium may be included in the device / apparatus described in the above embodiment; or it may exist independently without being assembled into the device / apparatus. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0122] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0123] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0124] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present disclosure can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present disclosure is not limited to any specific combination of hardware and software.

[0125] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for detecting consistency between human-machine solutions in the customer service field, characterized in that: include: Extracting a machine conversation from a first conversation record between a robot customer service and a customer, and extracting a manual conversation from a second conversation record between a manual customer service and the customer; the machine conversation and the manual conversation occur during a process of providing customer service to the same customer, and a jump operation from the robot customer service to the manual customer service occurs during this process; Performing data preprocessing on the machine conversation and the manual conversation; Determine, by a deep learning model, a first core solution of the machine conversation after the data preprocessing, a second core solution of the manual conversation after the data preprocessing, and a consistency score between the first core solution and the second core solution; The method includes: extracting a first conversation feature of a machine conversation that has undergone data preprocessing and a second conversation feature of a manual conversation that has undergone data preprocessing; determining a first core solution according to the similarity between the first conversation feature and the acquired core solution set; and determining a second core solution according to the similarity between the second conversation feature and the core solution set; wherein the deep learning model has learned, through deep learning, the correspondence between the machine conversation and the first core solution, the correspondence between the manual conversation and the second core solution, and the correspondence between the first core solution, the second core solution and the consistency score; The consistency rate of the core solutions corresponding to the machine conversation and the manual conversation is determined according to the consistency score.

2. The method according to claim 1, characterized in that determining the first core solution according to the similarity between the first session feature and the acquired core solution set; Determining a second core solution according to the similarity between the second session feature and the core solution set includes: When the similarity between the first session feature and the core solution set is greater than a first preset threshold, determining a first segmented session corresponding to the first session feature as the first core solution, wherein the first segmented session is obtained by performing the data preprocessing on the machine session; When the similarity between the second session feature and the core solution set is greater than the first preset threshold, a second segmented session corresponding to the second session feature is determined as the second core solution, wherein the second segmented session is obtained by performing the data preprocessing on the artificial session.

3. The method according to claim 1, characterized in that The first session feature and the second session feature each include at least one of the following: A first feature is used to represent the business meaning of a split session, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: a first split session and a second split session; The second feature is used to indicate whether the customer service answer in the segmented conversation is a question sentence; The third feature is used to represent key information of the customer service answer in the segmented session; A fourth feature is used to represent the sentence length of the segmented conversation; A fifth feature is used to indicate the sequence number of the split session in the machine session or the manual session; The sixth feature is used to indicate the business meaning of the previous split session of the split session.

4. The method according to claim 2, characterized in that: The core solution set includes: A first object is used to indicate that the split session has business meaning, wherein the split session is obtained by performing the data preprocessing on the machine session and the manual session, and the split session includes: the first split session and the second split session; The second object is used to indicate that the customer service answer in the segmented conversation is not a question sentence; A third object is used to indicate that the customer service answer in the split session contains key information; A fourth object, used to indicate that the length of the sentence of the segmented conversation is within a first preset interval; A fifth object, used to indicate that the sequence number of the split session in the machine session or the manual session is in a second preset interval; The sixth object is used to indicate that the previous split session of the split session has business meaning.

5. The method according to claim 1, characterized in that include: Acquire historical conversation data, wherein the historical conversation data includes: machine conversation history data and manual conversation history data; Extracting session features of the historical session data; The deep learning model is trained by using the historical conversation data and the conversation features.

6. The method according to claim 1, characterized in that The performing data preprocessing on the machine conversation and the manual conversation includes: Filtering the machine conversation and the manual conversation according to a noise reduction strategy; and / or The machine conversation and the manual conversation are respectively divided according to the division logic to obtain a plurality of divided conversations.

7. The method according to claim 6, characterized in that The machine conversation and the manual conversation are divided according to the division logic to obtain a plurality of divided conversations, including: Determine a segmentation code according to the segmentation logic, wherein the segmentation logic includes: one question followed by one answer, multiple questions followed by one answer, one question followed by multiple answers, and multiple questions followed by multiple answers; The machine conversation and the manual conversation are respectively divided according to the division code to obtain the plurality of divided conversations.

8. A device for detecting consistency between human-machine solutions in the customer service field, characterized in that: include: An extraction module is used to extract a machine conversation from a first conversation record between a robot customer service and a customer, and to extract a manual conversation from a second conversation record between a manual customer service and the customer; the machine conversation and the manual conversation occur during a process of providing customer service to the same customer, and a jump operation from the robot customer service to the manual customer service occurs during this process; A preprocessing module, used for performing data preprocessing on the machine conversation and the manual conversation; A first determination module is used to determine, through a deep learning model, a first core solution of the machine conversation after the data preprocessing, a second core solution of the manual conversation after the data preprocessing, and a consistency score between the first core solution and the second core solution; The method includes: extracting a first conversation feature of a machine conversation that has undergone data preprocessing and a second conversation feature of a manual conversation that has undergone data preprocessing; determining a first core solution according to the similarity between the first conversation feature and the acquired core solution set; and determining a second core solution according to the similarity between the second conversation feature and the core solution set; wherein the deep learning model has learned, through deep learning, the correspondence between the machine conversation and the first core solution, the correspondence between the manual conversation and the second core solution, and the correspondence between the first core solution, the second core solution and the consistency score; The second determination module is used to determine the consistency rate of the core solutions corresponding to the machine conversation and the manual conversation according to the consistency score.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images