Popying recognition method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a reply recognition method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: extracting training video data from first audio and video data, and extracting training audio data from second audio and video data; merging the training video data and the training audio data to obtain preliminary audio and video data; performing audio interval adjustment on the initial audio and video data according to a preset random threshold to obtain training audio and video data; training a preset original reply recognition model according to the training audio and video data to obtain a target reply recognition model; obtaining target audio and video data; and carrying out reply recognition on the target audio and video data through the target reply recognition model. The method and the device can be applied to business systems needing a large amount of data, such as financial science and technology and health medical treatment, can automatically identify the reply-on-behalf condition, and reduce manpower in the reply-on-behalf identification process.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applied to the fields of fintech and healthcare, particularly to a method and apparatus for identifying proxy responses, computer equipment, and storage medium. Background Technology

[0002] AI-powered question answering is a technology or system that uses artificial intelligence to enable computers to automatically understand and answer natural language questions posed by humans. However, currently, AI-powered question answering is being used in various fields for online interviews, replacing real users in completing the process. For example, in fintech scenarios, insurance claims require insurance agents or intelligent voice assistants to conduct online interviews with policyholders, asking questions and receiving responses. During these interviews, if a policyholder intends to conceal information or commit insurance fraud, they may use AI to automatically answer questions, affecting the accuracy of the claim. Similarly, in healthcare scenarios, to more accurately assess a patient's condition, online interviews are necessary beforehand. If some patients intend to conceal their condition and use AI to answer questions from medical staff, it will affect the accuracy of treatment and could easily lead to medical accidents.

[0003] In related technologies, identifying whether a speaker during an online interview is an AI assistant or a real user typically requires manual verification by interviewers, which consumes significant time and effort. Furthermore, if some online interviews are conducted using intelligent voice assistants, the authenticity of the respondent cannot be verified. Therefore, how to accurately identify the authenticity of the speaker while minimizing the human cost of the verification process has become a pressing technical problem. Summary of the Invention

[0004] The main objective of this application is to propose a method and apparatus, computer equipment and storage medium for identifying proxy responses, aiming to achieve automated identification of proxy responses and save on human resources costs in the identification process.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for identifying substitute responses, the method comprising:

[0006] Training video data is extracted from the first audio and video data, and training audio data is extracted from the second audio and video data; wherein the first audio and video data and the second audio and video data are different;

[0007] The training video data and the training audio data are merged and processed to obtain preliminary audio and video data;

[0008] The preliminary audio and video data are adjusted in terms of audio range according to a preset random threshold to obtain training audio and video data;

[0009] The preset original proxy response recognition model is trained based on the training audio and video data to obtain the target proxy response recognition model;

[0010] Acquire target audio and video data;

[0011] The target audio and video data is identified by using the target proxy response identification model.

[0012] In some embodiments, merging the training video data and the training audio data to obtain preliminary audio and video data includes:

[0013] Obtain the raw audio data of the training video data;

[0014] The original audio data is subjected to content recognition to obtain the first speech content;

[0015] The training audio data is subjected to content recognition to obtain the second speech content;

[0016] The pronunciation similarity of the first speech content and the second speech content is evaluated to obtain pronunciation similarity evaluation data;

[0017] Merging indication information is determined based on the pronunciation similarity assessment data and the preset merging assessment threshold;

[0018] The training video data and the training audio data are merged according to the merging instruction information to obtain the preliminary audio and video data.

[0019] In some embodiments, the step of evaluating the pronunciation similarity of the first and second speech content to obtain pronunciation similarity evaluation data includes:

[0020] Convert the first speech content into a first pinyin sequence;

[0021] The second speech content is converted into a second pinyin sequence; wherein the pinyin in the first pinyin sequence and the second pinyin sequence are sorted by time.

[0022] The pronunciation similarity evaluation data is obtained by calculating the edit distance based on the first pinyin sequence and the second pinyin sequence.

[0023] In some embodiments, determining the merging indication information based on the pronunciation similarity assessment data and a preset similarity threshold includes:

[0024] If the pronunciation similarity evaluation data is less than or equal to the preset merging evaluation threshold, the merging indication information is determined to be a suggested merging;

[0025] If the pronunciation similarity assessment data is greater than the preset merging assessment threshold, the merging indication information is determined to be a rejection of merging.

[0026] In some embodiments, adjusting the audio range of the initial audio and video data according to a preset random threshold to obtain training audio and video data includes:

[0027] Obtain the current speaking interval information of the preliminary audio and video data;

[0028] Random numbers are generated based on the preliminary audio and video data;

[0029] The current speaking interval information is adjusted based on the random number and a preset random threshold to obtain the target speaking interval information;

[0030] The preliminary audio and video data is processed by blacking out the screen based on the target speaking interval information to obtain the training audio and video data.

[0031] In some embodiments, adjusting the current speaking interval information based on the random number and a preset random threshold to obtain the target speaking interval information includes:

[0032] If the random number is greater than the random threshold, the current speaking interval information is used as the target speaking interval information;

[0033] If the random number is less than the random threshold, the current speaking interval information is extended to obtain the target speaking interval information.

[0034] In some embodiments, the step of identifying proxy responses to the target audio and video data using the target proxy response identification model to obtain proxy response identification information includes:

[0035] Audio data is extracted from the target audio and video data to obtain target audio data;

[0036] Convert the target audio data into a target spectrogram;

[0037] Audio features are extracted from the target spectrogram to obtain the target audio features;

[0038] Lip features are extracted from the target audio and video data to obtain the target lip features;

[0039] The target audio features and the target lip features are mapped to obtain audio lip feature mapping information;

[0040] The audio lip feature mapping information is used to identify the proxy response, and the proxy response identification information is obtained.

[0041] To achieve the above objectives, a second aspect of this application provides a proxy response identification device, the device comprising:

[0042] The device includes:

[0043] The data extraction module is used to extract training video data from the first audio and video data and to extract training audio data from the second audio and video data; wherein the first audio and video data and the second audio and video data are different.

[0044] The merging module is used to merge the training video data and the training audio data to obtain preliminary audio and video data;

[0045] The audio adjustment module is used to adjust the audio range of the preliminary audio and video data according to a preset random threshold to obtain training audio and video data.

[0046] The training module is used to train the preset original proxy response recognition model based on the training audio and video data to obtain the target proxy response recognition model.

[0047] The data acquisition module is used to acquire target audio and video data;

[0048] The proxy response identification module is used to identify proxy responses in the target audio and video data using the target proxy response identification model.

[0049] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0050] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0051] The proposed method, apparatus, computer equipment, and storage medium for identifying proxy responses extract training video data and training audio data from different audio and video data sets. These are then combined to form preliminary audio and video data. The audio ranges of the preliminary audio and video data are adjusted to match the training audio and video data, thus constructing training audio and video data for the proxy response identification model. The proxy response identification model is trained using this training audio and video data. The trained model then automatically identifies whether the speaker in the target audio and video data is genuinely speaking, automating the proxy response identification process and eliminating the need for manual identification of proxy responses in the audio and video data, thereby reducing manpower costs. Attached Figure Description

[0052] Figure 1 This is a flowchart of the proxy response identification method provided in the embodiments of this application;

[0053] Figure 2 This is a schematic diagram illustrating the application of the proxy response identification method provided in this application to a fintech scenario;

[0054] Figure 3 This is a schematic diagram illustrating the application of the proxy response identification method provided in this application to a health and medical scenario;

[0055] Figure 4 yes Figure 1 The flowchart of step S102 in the document;

[0056] Figure 5 yes Figure 4 The flowchart of step S204 in the process;

[0057] Figure 6 yes Figure 4 The flowchart of step S205 in the document;

[0058] Figure 7 yes Figure 1 The flowchart of step S103 in the process;

[0059] Figure 8 yes Figure 7 The flowchart of step S503 in the process;

[0060] Figure 9 yes Figure 1 The flowchart of step S106 in the process;

[0061] Figure 10 This is a schematic diagram of the structure of the proxy response identification device provided in the embodiments of this application;

[0062] Figure 11 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0066] First, let's analyze some of the terms used in this application:

[0067] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0068] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0069] MPEG is a powerful open-source multimedia processing tool that supports transcoding, editing, merging, extraction, and streaming media processing of video, audio, and subtitle formats.

[0070] Neural networks are machine learning models inspired by biological nervous systems (especially the way neurons connect in the human brain). Their core is to automatically learn features and patterns from data through the connection and collaboration of multiple layers of "neurons" to model and predict complex problems.

[0071] MFCC (Mel Frequency Ancestral Coefficients) is a feature extraction technique widely used in speech and audio processing to convert audio signals into numerical vectors representing sound features. It simulates the differences in how the human auditory system perceives different frequencies, making it particularly suitable for tasks such as speech recognition, speaker recognition, and audio classification.

[0072] Liechtenstein distance, also known as edit distance, is a metric that measures the difference between two strings. It is defined as the minimum number of single-character edit operations (insertion, deletion, or replacement) required to transform one string into another.

[0073] With the development of internet technology, online interviews have gradually replaced in-person interviews due to their convenience and speed, greatly improving work efficiency. However, the widespread adoption of online interviews has also brought some security risks, such as the difficulty in detecting cheating by interviewees. It's worth noting that a common form of cheating in interviews is proxy answering, where the user's mouth moves, but the voice is answered by someone else, i.e., proxy responses. Taking fintech scenarios as an example, during insurance claims, intelligent interviewers conduct a series of questions and answers with the insured. If the insured is found to have breached the contract and lacks sufficient evidence, AI-generated voice data may be used to complete the question-and-answer process, allowing the insured to commit insurance fraud simply by moving their mouth. In healthcare scenarios, some patients, in order to conceal infectious diseases or have difficulty disclosing their medical history to doctors, may set up intelligent assistants to answer questions on their behalf during interviews with medical staff or intelligent medical assistants. This can affect the accuracy of medical staff's analysis of the condition and lead to medical accidents.

[0074] In related technologies, to reduce cheating during face-to-face interviews, manual review of the audio and video data after the interview is required. If a large amount of audio and video data needs to be reviewed, it consumes a significant amount of manpower. How to achieve automated response identification and save on manpower costs has become an urgent technical problem to be solved.

[0075] Based on this, embodiments of this application provide a method and apparatus for identifying proxy responses, a computer device and a storage medium, which aim to achieve automated identification of proxy responses and save manpower for review.

[0076] The proxy response identification method, apparatus, computer equipment, and storage medium provided in this application are specifically described through the following embodiments. First, the proxy response identification method in this application is described.

[0077] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0078] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0079] The proxy response identification method provided in this application relates to the field of artificial intelligence technology. This proxy response identification method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the proxy response identification method, but is not limited to the above forms.

[0080] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0081] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0082] Figure 1 This is an optional flowchart of the proxy response identification method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0083] Step S101: Extract training video data from the first audio and video data, and extract training audio data from the second audio and video data; wherein the first audio and video data and the second audio and video data are different;

[0084] Step S102: Merge the training video data and training audio data to obtain preliminary audio and video data;

[0085] Step S103: Adjust the audio range of the preliminary audio and video data according to the preset random threshold to obtain training audio and video data;

[0086] Step S104: Train the preset original proxy response recognition model based on the training audio and video data to obtain the target proxy response recognition model;

[0087] Step S105: Obtain target audio and video data;

[0088] Step S106: Use the target proxy response recognition model to identify proxy responses in the target audio and video data.

[0089] Steps S101 to S106, as illustrated in this embodiment, involve merging training video data and training audio data from different audio and video datasets into preliminary audio and video data. Then, the audio range of the preliminary audio and video data is adjusted according to a preset random threshold to obtain training audio and video data. Therefore, by automatically creating the training audio and video data, manual creation is eliminated, saving manpower. The created training audio and video data is then used to train a target proxy response recognition model, which performs proxy response recognition on the audio and video data, eliminating the need for manual recognition and further saving manpower.

[0090] Taking fintech scenarios as an example, such as Figure 2 As shown, during the in-person interview, the intelligent interview assistant asks questions, which the applicant then answers. Throughout the process, the target proxy answer recognition model continuously analyzes the applicant's responses. If the applicant's lip movements and the content of their response do not match, a "suspected proxy answer" message is displayed, and a notification is sent to backend personnel for intervention. Therefore, by setting up the target proxy answer recognition model to continuously monitor the applicant's audio and video responses, no manual review is required, saving manpower and improving the accuracy of the interview.

[0091] Taking the health and medical scenario as an example, such as Figure 3 As shown, when online medical diagnostic platforms analyze patients' conditions, patients need to answer questions posed by intelligent medical assistants or medical staff. If, during the patient's response, the target proxy response recognition model identifies a mismatch between the patient's lip movements and audio, a "suspected proxy response" warning pops up, and the warning information is sent to relevant medical staff for further verification. Therefore, by setting up a target proxy response recognition model on the medical diagnostic platform to automatically identify proxy responses, manual intervention is reduced.

[0092] In step S101 of some embodiments, the first audio / video data and the second audio / video data can be generated in real time, or they can be extracted from a database. The first audio / video data and the second audio / video data can also be obtained through other means, and are not limited to these.

[0093] It should be noted that the training video data contains video data of the speaker performing speaking actions, and the training audio data contains audio data of the speaker speaking. The training video data is obtained by removing the audio from the first audio-visual data, and the training audio data is obtained by extracting the audio from the second audio-visual data separately. The first and second audio-visual data are not identical. It should also be noted that the first and second audio-visual data can be from the same speaker or from different speakers. Specifically, an audio-visual processing tool is used to remove the audio from the first audio-visual data to obtain the training video data, which is then stored in a video library. Similarly, the audio is extracted from the second audio-visual data using the same tool and stored in an audio library. When training the original response recognition model is required, the training video data is directly extracted from the video library, and the training audio data is extracted from the audio library. In this embodiment, the audio-visual processing tool used is the MPEG tool, an open-source toolkit for processing multimedia files such as video and audio. Therefore, this embodiment uses MPEG tools to process the first and second audio and video data, which is simple to operate. Other types of audio and video processing tools can also be used, and this embodiment does not impose any specific restrictions.

[0094] As previously disclosed, automatically generating training audio and video data can save manpower. However, the challenges of generating training audio and video data for long and short conversations differ. Generating training audio and video data for long conversations simply requires replacing the audio in the video with the audio of a long conversation. For example, select two normal speaking audio and video data sets, A and B. Cut the video content of A and the audio of B into equal length segments and combine them, thus creating a training audio and video data set for a long conversation. However, generating training audio and video data for short conversations presents two problems: the speaking time is very short, and randomly selected A and B audio and video data sets may contain matching audio content and lip movements, which will affect the training effect of the proxy response recognition model.

[0095] Please see Figure 4 In some embodiments, step S102 may include, but is not limited to, steps S201 to S206:

[0096] Step S201: Obtain the raw audio data of the training video data;

[0097] Step S202: Perform content recognition on the original audio data to obtain the first speech content;

[0098] Step S203: Perform content recognition on the training audio data to obtain the second speech content;

[0099] Step S204: Evaluate the pronunciation similarity between the first speech content and the second speech content to obtain pronunciation similarity evaluation data;

[0100] Step S205: Determine merging instruction information based on pronunciation similarity assessment data and preset merging assessment threshold;

[0101] Step S206: Merge the training video data and training audio data according to the merging instruction information to obtain preliminary audio and video data.

[0102] In step S201 of some embodiments, the original audio data is extracted from the first audio-visual data, and the speech content in the original audio data is the same as the lip movements of the speaking object in the training video data. It should be noted that the training video data only contains the video content of the speaking object performing the speaking action, and the original audio data only contains the audio segment from the start to the end of the speaking sound.

[0103] In steps S202 to S203 of some embodiments, a speech content recognition model is used to perform content recognition on the original audio data to obtain the first speech content, and the same model is used to perform content recognition on the training audio data to obtain the second speech content. It should be noted that when the audio and video data of a short speech contain content with the same pronunciation, it will affect the accuracy of the target response recognition model. For example, if the first speech content corresponding to the original audio data is "clear," and the second speech content corresponding to the training audio data is "clear," these two speech contents have completely different meanings but the same pronunciation, making it difficult for the target response recognition model to recognize them, thus affecting the training effect of the target response recognition model.

[0104] In step S204 of some embodiments, the pronunciation similarity evaluation data represents the pronunciation similarity between the first speech content and the second speech content. It should be noted that if both the first and second speech content are extracted from the audio / video data of a long speech, the pronunciation similarity evaluation data is low, so there is no need to consider the pronunciation similarity issue during the generation of the training audio / video data for the long speech. If the first and second speech content are extracted from the audio / video data of a short speech, the pronunciation similarity evaluation data is high. Therefore, the generation of the training audio / video data for the short speech requires calculating pronunciation similarity.

[0105] In step S205 of some embodiments, the pronunciation similarity evaluation data is compared with a preset merging evaluation threshold to obtain a comparison result, and merging instruction information is determined based on the comparison result.

[0106] In step S206 of some embodiments, the merging instruction information can be characterized as a suggested merge or a rejected merge. If the merging instruction information is characterized as a suggested merge, the training video data and the training audio data are merged into preliminary audio and video data; if the merging instruction information is characterized as a rejected merge, the training video data and the training audio data are not merged.

[0107] In steps S201 to S206 of this embodiment, pronunciation similarity evaluation data is obtained by calculating the pronunciation similarity between the original audio data of the training video data and the speech content of the training audio data. Based on the pronunciation similarity evaluation data and the preset merging evaluation threshold, the training video data and the training audio data are merged into preliminary audio-visual data to produce preliminary audio-visual data in which the mouth shape and pronunciation do not match, and a target proxy response recognition model that can accurately identify proxy response situations is trained.

[0108] Please see Figure 5 In some embodiments, step S204 may include, but is not limited to, steps S301 to S303:

[0109] Step S301: Convert the first speech content into a first pinyin sequence;

[0110] Step S302: Convert the second speech content into a second pinyin sequence; wherein the pinyin in the first pinyin sequence and the second pinyin sequence are sorted by time.

[0111] Step S303: Calculate the edit distance based on the first and second pinyin sequences to obtain pronunciation similarity evaluation data.

[0112] In steps S301 to S302 of some embodiments, in order to evaluate the pronunciation similarity between the first speech content and the second speech content, it is necessary to first convert the first speech content into a first pinyin sequence and convert the second speech content into a second pinyin sequence. It should be noted that the pinyin in the first and second pinyin sequences are ordered in chronological order, mainly judging whether the pronunciations at the same time are similar, while the similarity of pronunciations at different times is not considered.

[0113] In step S303 of some embodiments, the edit distance of the pinyin in the first pinyin sequence and the second pinyin sequence is calculated in chronological order, and the edit distance between the pinyin is used as the pronunciation similarity evaluation data. It should be noted that the edit distance is an indicator that measures the degree of difference between two pinyin strings, defined as the minimum number of edits required to convert one pinyin string into another.

[0114] In steps S301 to S303 of this embodiment, the first speech content is converted into a first pinyin sequence, the second speech content is converted into a second pinyin sequence, and the edit distance between the first pinyin sequence and the second pinyin sequence is calculated as the pronunciation similarity evaluation data, so that the pronunciation similarity evaluation between the training audio data and the original audio data is accurate and simple.

[0115] Please see Figure 6 In some embodiments, step S205 may include, but is not limited to, steps S401 to S402:

[0116] Step S401: If the pronunciation similarity evaluation data is less than or equal to the preset merging evaluation threshold, the merging indication information is determined to be a suggested merging.

[0117] Step S402: If the pronunciation similarity evaluation data is greater than the preset merging evaluation threshold, the merging indication information is determined to be rejected for merging.

[0118] In step S401 of some embodiments, if the pronunciation similarity evaluation data is less than a preset merging evaluation threshold, it indicates that the pronunciations of the first speech content and the second speech content are not similar, and the merging indication information is determined to be a suggested merge. It should be noted that when the merging indication information is a suggested merge, it indicates that the original audio data corresponding to the first speech content and the training audio data corresponding to the second speech content can be merged to generate preliminary audio and video data.

[0119] In step S402 of some embodiments, if the pronunciation similarity evaluation data is greater than or equal to a preset merging evaluation threshold, it indicates that the pronunciations of the first speech content and the second speech content are very similar. For example, if the first speech content is "clear" and the second speech content is "clear," or if the first speech content is "agree" and the second speech content is "synonymous," they are all considered to have similar pronunciations. It should be noted that the preset merging evaluation threshold is pre-set and may also be manually adjusted based on the generated preliminary audio and video data. Specifically, adjustments are made by manually reviewing the similarity of mouth shapes and pronunciations in the preliminary audio and video data. Therefore, when the pronunciation similarity evaluation data is greater than or equal to the preset merging evaluation threshold, the merging indication information is determined to be rejected, and the training video data and training audio data are not merged.

[0120] For example, the pronunciation similarity evaluation data is the edit distance, which is defined as the Liechtenstein distance, and the similarity threshold is defined as F. If the Liechtenstein distance is greater than F, it indicates that the pronunciation similarity between the first and second speech content is too high, and the training video data and training audio data are not merged. If the Liechtenstein distance is less than or equal to F, it indicates that the pronunciation of the first and second speech content is not similar, and the training video data and training audio data are merged into preliminary audio-video data.

[0121] In steps S401 to S402 of this embodiment, the pronunciation similarity evaluation data is compared with a preset merging evaluation threshold to determine whether the merging indication information is characterized as a suggested merging or a rejected merging. When the pronunciation similarity evaluation data is less than the preset merging evaluation threshold, the merging indication information is determined to be a suggested merging, and the training audio data and training video data are merged into preliminary audio-visual data with dissimilar mouth shapes and pronunciations. The target substitute response recognition model trained on the preliminary audio-visual data can accurately identify whether there is a substitute response in the audio-visual data.

[0122] In some embodiments, after generating preliminary audio and video data, if the timing of the video and audio in the preliminary audio and video data is not aligned, for example, if training audio data C only shows the speaking time interval, and if it is mapped to the scene of training video data D, the timing may not match, that is, the start time of speaking in the training audio data and the start time of mouth movement in the training video data D are difficult to coincide. If the original proxy response recognition model is directly trained with the non-overlapping preliminary audio and video data, the original proxy response recognition model is prone to learning shortcuts, only focusing on the inconsistency of the start time of the preliminary audio and video data, and will not truly learn the inconsistency of mouth shape in the preliminary audio and video data, which is prone to overfitting. Therefore, the audio interval of the preliminary audio and video data is adjusted, and the start of speaking in the training audio data and the mouth movement scene of the training video data are merged together to obtain the training audio and video data.

[0123] Please see Figure 7 In some embodiments, step S103 includes, but is not limited to, steps S501 to S504:

[0124] Step S501: Obtain the current speaking interval information of the preliminary audio and video data;

[0125] Step S502: Generate random numbers based on the preliminary audio and video data;

[0126] Step S503: Adjust the current speaking interval information according to the random number and the preset random threshold to obtain the target speaking interval information;

[0127] Step S504: Perform black screen processing on the preliminary audio and video data according to the target speaking interval information to obtain training audio and video data.

[0128] In step S501 of some embodiments, speaking time interval information is extracted from the preliminary audio and video data. The start and end times of audio playback can be determined through the speaking time interval information. For example, the speaking time interval information is defined as [start, end], where start represents the time when speaking begins and end represents the time when speaking ends.

[0129] In steps S502 to S503 of some embodiments, random numbers are generated based on the preliminary audio and video data, specifically based on the training video data of the preliminary audio and video data. It should be noted that these random numbers are sampled from a uniform distribution of 0 to 1, and the random numbers are used to determine whether to adjust the speaking interval in the preliminary audio and video data, indicating the matching operation between speaking and mouth movements in the preliminary audio and video data. Therefore, the current speaking interval information is adjusted based on the random number and a random threshold, that is, whether the start of speaking and mouth speaking actions match in the preliminary audio and video data.

[0130] In step S504 of some embodiments, the preliminary audio-visual data is processed by blacking out the screen according to the target speaking interval information. That is, the intervals in the preliminary audio-visual data other than those corresponding to the target speaking interval information are blacked out to obtain training audio-visual data. Therefore, by retaining or removing the intervals other than speaking in the preliminary audio-visual data, the randomly generated training audio-visual data includes both audio and mouth movement matching and cases where audio and mouth movement do not match. By training the audio-visual data, the original proxy response recognition model can learn the differences in the start time of the proxy response video and the differences in mouth shape, so as to build a target proxy response recognition model that can accurately identify proxy responses.

[0131] In steps S501 to S504 of this embodiment, the current speaking interval of the preliminary audio-visual data is adjusted according to random numbers and random thresholds. This randomly adjusts the matching between the speaking audio and mouth movements in the preliminary audio-visual data, creating training audio-visual data where the speaking audio and mouth movements partially match and partially do not. Therefore, the target proxy response recognition model trained on the training audio-visual data can identify different proxy response situations, improving the accuracy of proxy response recognition.

[0132] Please see Figure 8 In some embodiments, step S503 includes, but is not limited to, steps S601 to S602:

[0133] Step S601: If the random number is greater than the preset random threshold, the current speaking interval information is used as the target speaking interval information.

[0134] Step S602: If the random number is less than the preset random threshold, expand the current speaking interval information to obtain the target speaking interval information.

[0135] In steps S601 to S602 of some embodiments, if the random number is greater than a random threshold, the current speaking interval is taken as the target speaking interval, and the other content positions in the preliminary audio-visual data are blacked out according to the target speaking interval information to preserve the mouth movement image matching the speaking audio, thus obtaining the target audio-visual data. If the random number is less than the random threshold, the current speaking interval information is expanded outward to obtain the target speaking interval information. The expanded target speaking interval information includes not only the interval where the speaking audio is played, but also the interval where no audio is played. Therefore, by blacking out the preliminary audio-visual data according to the target speaking interval information, the video image other than the audio is preserved to obtain the training audio-visual data.

[0136] For example, let r be the random number and threshold be the random threshold. If r > threshold, use MPEG tools to black out the video content outside the [start, end] range of the training video data v in the initial audio and video data, obtaining new video data v_new. Then, merge the training audio data a and the new video data v_new in the initial audio and video data using MPEG to obtain the training audio and video data. If r < threshold, expand the current speech interval information [start, end] outward to obtain start_new and end_new. It should be noted that start_new <= start, end_new >= end. Use MPEG tools to black out the video content outside the [start_new, end_new] range of the training video data v in the initial audio and video data, obtaining new video data v_new. Then, merge the training audio data a and the new video data v_new using MPEG tools to obtain the training video data.

[0137] In steps S601 to S602 of this embodiment, by comparing a random number and a random threshold, the video footage other than the spoken audio in the initial audio-video data is randomly blacked out, while some of the video footage other than the spoken audio is retained, in order to construct training audio-video data with richer content. Therefore, by training the original proxy response recognition model using the audio-video data, the original proxy response recognition model can learn the differences in start time and lip shape in the proxy response video, thereby training a more accurate target proxy response recognition model.

[0138] In step S104 of some embodiments, the original proxy response recognition model is trained using sample audio-visual data of long conversations. This sample audio-visual data is obtained by replacing the audio in the long conversation audio-visual data with the audio of another long conversation. Therefore, the original proxy response recognition model can accurately identify proxy responses in long conversation scenarios. The training audio-visual data consists of audio-visual data of short conversations, including audio-visual data with different mouth movements and audio start positions, as well as audio-visual data with different lip shapes and speech content. The original proxy response recognition model is trained using this training audio-visual data to construct a target proxy response recognition model that can accurately identify both long and short conversations.

[0139] It should be noted that the original proxy response identification model is similar to a discriminator, used to determine whether the training audio and video data is a proxy response, outputting yes or no. The training audio and video data is input into the original proxy response identification model to obtain training proxy response identification information. Based on the training and validation proxy response identification information, a recognition loss function is constructed. The parameters of the original proxy response identification model are then adjusted according to the recognition loss function to obtain the target proxy response identification model.

[0140] In step S105 of some embodiments, the target audio and video data is the audio and video generated directly during the user's video communication. It can be generated in real time or stored in a database and then retrieved after the communication ends. The method of collecting the target audio and video data varies depending on the application for identifying proxy responses. For example, if it is necessary to monitor whether a user is answering on behalf of another during an insurance claims interview, the target audio and video data is generated after each user's response, meaning one target audio and video data point is generated each time the user completes a question. If it is necessary to monitor the claims interview audio and video data directly uploaded by the user, the uploaded claims interview audio and video data is directly used as the target audio and video data.

[0141] Please see Figure 9 In some embodiments, step S106 may include, but is not limited to, steps S701 to S706:

[0142] Step S701: Extract audio from the target audio and video data to obtain the target audio data;

[0143] Step S702: Convert the target audio data into a target spectrogram;

[0144] Step S703: Extract audio features from the target spectrogram to obtain the target audio features;

[0145] Step S704: Extract lip features from the target audio and video data to obtain the target lip features;

[0146] Step S705: Map the target audio features and target lip features to obtain audio lip feature mapping information;

[0147] Step S706: Use the target proxy response recognition model to perform proxy response recognition on the audio lip feature mapping information.

[0148] It should be noted that the target-based proxy response recognition model performs proxy response recognition on target audio and video data, that is, it identifies whether the speech content and lip movements in the target audio and video data match, and the target-based proxy response recognition model is a neural network model. The following details the process of the target-based proxy response recognition model in performing proxy response recognition.

[0149] In steps S701 to S703 of some embodiments, audio features are first extracted from the target audio and video data as target audio features. Specifically, target audio data is first extracted from the target audio and video data, then the target audio data is converted into a target spectrogram, and audio features are extracted from the target spectrogram as target audio features. It should be noted that the target spectrogram can be an MFCC graph or a Mel spectrogram; this embodiment does not impose specific limitations on the target spectrogram.

[0150] In step S704 of some embodiments, lip features are extracted from the target audio and video data. Specifically, the target video data is extracted from the target audio and video data, then the target video data is processed into frames to obtain target video frame images, then the lip features in the target video frame images are extracted, and then the lip features are combined into target lip features in chronological order.

[0151] In step S705 of some embodiments, the target audio features and target lip features are associated and mapped in chronological order to obtain audio lip feature mapping information, that is, to establish the correspondence between audio and speech, so as to analyze whether the lip movements and speech content match.

[0152] In step S706 of some embodiments, the target proxy response recognition model is equivalent to a class class predictor. The class class predictor determines whether the audio lip feature mapping information is lip-synced, so as to realize proxy response recognition of the target audio and video data.

[0153] For example, if the voice content of the target audio / video data is "agree," and the video shows lip movements conveying the message "okay," then the target audio / video data is identified as a proxy response, meaning the user has provided a proxy response. It's important to note that after the proxy response identification model identifies a proxy response, if such a situation is found, a proxy response notification will be generated and sent to the platform's administrators. The administrators will then further confirm whether the target audio / video data contains a proxy response, reducing the likelihood of misjudgments by the proxy response identification model. Furthermore, the administrators will generate feedback based on the review results and optimize the proxy response identification model accordingly to improve its accuracy.

[0154] In steps S701 to S706 of this embodiment, audio features and lip features are extracted from the target audio and video data. The correspondence between the audio features and lip features is used to determine whether the sound and lips are synchronized, so as to realize the proxy response recognition and make the proxy response recognition operation simple.

[0155] This application uses an insurance claims scenario as an example. Audio and video data collected from a user's face-to-face interview process on an insurance platform are defined as first audio and video data and second audio and video data. MPEG tools are used to extract training video data and original audio data from the first audio and video data, and the same tools are used to extract training audio data from the second audio and video data. The speech content in the original audio data is identified as the first speech content, and the speech content in the training audio data is identified as the second speech content. The first speech content is converted into a first pinyin sequence, and the second speech content is converted into a second pinyin sequence. The edit distance between the first and second pinyin sequences is calculated. If the edit distance is greater than a preset merging evaluation threshold, indicating that the pronunciations of the first and second speech content are similar, the training audio data and training video data are not merged. If the edit distance is less than or equal to the preset merging evaluation threshold, the training audio data and training video data are merged into preliminary audio and video data. After merging, it is necessary to further extract the current speaking interval information from the training audio data. If the random number is greater than a preset random threshold, the video content other than the audio in the initial audio-video data is blacked out to obtain the training audio-video data; if the random number is less than the preset random threshold, the video content other than the audio in the initial audio-video data is retained to obtain the training audio-video data. Therefore, the original proxy response recognition model is trained into a target proxy response recognition model using the training audio-video data. When the target proxy response recognition model is loaded into each face-to-face interview process on the insurance platform, each time the policyholder answers a question during the interview, the recorded audio-video data is used as the target audio-video data. The target proxy response recognition model judges whether the audio features and lip features in the target audio-video data match, thereby realizing proxy response recognition during the policyholder's face-to-face interview process. This does not require manual supervision, saves manpower, improves the effectiveness of insurance services, and reduces the probability of insurance fraud.

[0156] Please see Figure 10 This application also provides a proxy response identification device, which can implement the above-described proxy response identification method. The device includes:

[0157] The data extraction module 1001 is used to extract training video data from the first audio and video data and to extract training audio data from the second audio and video data; wherein the first audio and video data and the second audio and video data are different.

[0158] The merging module 1002 is used to merge the training video data and training audio data to obtain preliminary audio and video data.

[0159] The audio adjustment module 1003 is used to adjust the audio range of the preliminary audio and video data according to a preset random threshold to obtain training audio and video data.

[0160] Training module 1004 is used to train the preset original proxy response recognition model based on training audio and video data to obtain the target proxy response recognition model;

[0161] Data acquisition module 1005 is used to acquire target audio and video data;

[0162] The proxy response identification module 1006 is used to identify proxy responses in target audio and video data through the target proxy response identification model.

[0163] The specific implementation of this proxy response identification device is basically the same as the specific implementation of the proxy response identification method described above, and will not be repeated here.

[0164] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described proxy response identification method. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0165] Please see Figure 11 , Figure 11 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes:

[0166] The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0167] The memory 1102 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called and executed by the processor 1101 using the proxy response identification method of the embodiments of this application.

[0168] Input / output interface 1103 is used to implement information input and output;

[0169] The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0170] Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104);

[0171] The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0172] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described proxy response identification method.

[0173] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0174] The proxy response identification method, apparatus, computer device, and storage medium provided in this application extract training video data and training audio data from different audio and video data. They then synthesize the training video data and training audio data into preliminary audio and video data, and adjust the audio ranges of the preliminary audio and video data to match the training audio and video data, thereby constructing audio and video data for training the proxy response identification model. The proxy response identification model is trained using this training audio and video data, and then automatically identifies whether the speaker in the target audio and video data is genuinely speaking, automating the proxy response identification process and eliminating the need for manual identification of proxy responses in audio and video data, thus reducing manpower costs.

[0175] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0176] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0177] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0179] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0180] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0182] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0184] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for identifying proxy responses, characterized in that, The method includes: Training video data is extracted from the first audio and video data, and training audio data is extracted from the second audio and video data; wherein the first audio and video data and the second audio and video data are different. The training video data and the training audio data are merged and processed to obtain preliminary audio and video data; The preliminary audio and video data are adjusted in terms of audio range according to a preset random threshold to obtain training audio and video data; The preset original proxy response recognition model is trained based on the training audio and video data to obtain the target proxy response recognition model; Acquire target audio and video data; The target audio and video data is identified by using the target proxy response identification model.

2. The method according to claim 1, characterized in that, The step of merging the training video data and the training audio data to obtain preliminary audio and video data includes: Obtain the raw audio data of the training video data; The original audio data is subjected to content recognition to obtain the first speech content; The training audio data is subjected to content recognition to obtain the second speech content; The pronunciation similarity of the first speech content and the second speech content is evaluated to obtain pronunciation similarity evaluation data; Merging indication information is determined based on the pronunciation similarity assessment data and the preset merging assessment threshold; The training video data and the training audio data are merged according to the merging instruction information to obtain the preliminary audio and video data.

3. The method according to claim 2, characterized in that, The step of evaluating the pronunciation similarity of the first and second speech content to obtain pronunciation similarity evaluation data includes: Convert the first speech content into a first pinyin sequence; The second speech content is converted into a second pinyin sequence; wherein the pinyin in the first pinyin sequence and the second pinyin sequence are sorted by time. The pronunciation similarity evaluation data is obtained by calculating the edit distance based on the first pinyin sequence and the second pinyin sequence.

4. The method according to claim 2, characterized in that, The step of determining the merging indication information based on the pronunciation similarity evaluation data and a preset similarity threshold includes: If the pronunciation similarity evaluation data is less than or equal to the preset merging evaluation threshold, the merging indication information is determined to be a suggested merging; If the pronunciation similarity assessment data is greater than the preset merging assessment threshold, the merging indication information is determined to be a rejection of merging.

5. The method according to any one of claims 1 to 4, characterized in that, The step of adjusting the audio range of the preliminary audio and video data according to a preset random threshold to obtain training audio and video data includes: Obtain the current speaking interval information of the preliminary audio and video data; Random numbers are generated based on the preliminary audio and video data; The current speaking interval information is adjusted based on the random number and a preset random threshold to obtain the target speaking interval information; The preliminary audio and video data is processed by blacking out the screen based on the target speaking interval information to obtain the training audio and video data.

6. The method according to claim 5, characterized in that, The step of adjusting the current speaking interval information based on the random number and a preset random threshold to obtain the target speaking interval information includes: If the random number is greater than the preset random threshold, the current speaking interval information is used as the target speaking interval information; If the random number is less than the preset random threshold, the current speaking interval information is expanded to obtain the target speaking interval information.

7. The method according to any one of claims 1 to 4, characterized in that, The step of identifying proxy responses to the target audio and video data using the target proxy response identification model includes: Audio data is extracted from the target audio and video data to obtain target audio data; Convert the target audio data into a target spectrogram; Audio features are extracted from the target spectrogram to obtain the target audio features; Lip features are extracted from the target audio and video data to obtain the target lip features; The target audio features and the target lip features are mapped to obtain audio lip feature mapping information; The target-based proxy response recognition model is used to identify proxy responses based on the audio lip feature mapping information.

8. A proxy response identification device, characterized in that, The device includes: The data extraction module is used to extract training video data from the first audio and video data and to extract training audio data from the second audio and video data; wherein the first audio and video data and the second audio and video data are different. The merging module is used to merge the training video data and the training audio data to obtain preliminary audio and video data; The audio adjustment module is used to adjust the audio range of the preliminary audio and video data according to a preset random threshold to obtain training audio and video data. The training module is used to train the preset original proxy response recognition model based on the training audio and video data to obtain the target proxy response recognition model. The data acquisition module is used to acquire target audio and video data; The proxy response identification module is used to identify proxy responses in the target audio and video data using the target proxy response identification model.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the proxy response identification method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the proxy response identification method according to any one of claims 1 to 7.

Images