Audio and video data processing method and device, program product and electronic equipment

Abstract

The invention discloses an audio and video data processing method and device, a program product and electronic equipment, and relates to the field of artificial intelligence or other related fields, and the processing method comprises the steps: collecting first audio and video data of a target object when it is detected that the target object handles a financial service and obtains the authorization of the target object, the first audio and video data carries a timestamp; based on the first audio and video data, determining a target state of the target object at each time point and a target state probability value corresponding to the target state; based on the target state at each time point and the target state probability value corresponding to the target state, constructing a state change curve of the target object; and triggering state prompt information before monitoring that the state change curve reaches the first probability threshold line or the second probability threshold line. According to the method and the device, the technical problem of relatively low accuracy of determining the emotional state of the customer in related technologies is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for processing audio and video data, a device thereof, a program product, and an electronic device. Background Art

[0002] With the rapid development of financial technology, financial institutions, as key providers of financial services, are facing the problem of improving customer satisfaction. As customer expectations continue to rise, the current satisfaction monitoring methods are gradually showing their shortcomings. At present, most financial institutions rely on manual selection to collect customer satisfaction feedback at the counter, usually through paper questionnaires or electronic devices set up at the counter, allowing customers to make choices after handling business. Although this method is simple, there are the following problems in the implementation process:

[0003] (1) Highly subjective and lacking objectivity: The manual selection method of satisfaction mainly relies on the subjective feelings of customers. Customers may be influenced by many factors when selecting, such as personal emotions, loyalty to the brand of financial institutions, etc., resulting in a certain degree of subjectivity and uncertainty in the satisfaction results.

[0004] (2) Low data collection efficiency and long feedback cycle: The manual satisfaction selection method usually requires customers to provide feedback through paper or electronic questionnaires after completing the business. This method not only has low data collection efficiency, but also has a long feedback cycle, making it difficult to timely reflect the actual situation of counter services at financial institutions.

[0005] (3) Inability to monitor in real time and lack of timeliness: The manual satisfaction selection method cannot monitor the changes in customer satisfaction during the business process in real time. Feedback can only be provided after the business is completed. This results in financial institution branches being unable to promptly discover and resolve problems in the service process, affecting the continuous improvement of service quality.

[0006] (4) The information dimension is single, making it difficult to conduct a comprehensive assessment: The manual satisfaction rating method usually only focuses on the customer's evaluation of the overall service satisfaction, lacks in-depth analysis of multiple dimensions such as customer behavior characteristics, emotional changes, and transaction results, and is difficult to comprehensively and accurately assess the quality of counter services at financial institutions.

[0007] To the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0008] The embodiments of the present invention provide a method for processing audio and video data and a device, a program product, and an electronic device thereof, so as to at least solve the technical problem of low accuracy in determining the emotional state of a customer in the related art.

[0009] According to one aspect of an embodiment of the present invention, a method for processing audio and video data is provided, comprising: upon detecting that a target object conducts financial business and obtaining authorization from the target object, collecting first audio and video data of the target object, wherein the first audio and video data carries a timestamp; based on the first audio and video data, determining a target state of the target object at each time point and a target state probability value corresponding to the target state; constructing a state change curve of the target object based on the target state at each time point and the target state probability value corresponding to the target state; and triggering a state prompt message before monitoring that the state change curve reaches a first probability threshold line or a second probability threshold line.

[0010] Furthermore, based on the first audio and video data, the step of determining the target state of the target object at each time point and the target state probability value corresponding to the target state includes: separating the first audio and video data to obtain video data and audio data; processing the video data to obtain target video data; extracting facial feature data and action feature data of the target object based on the target video data; processing the facial feature data, the action feature data and the audio data respectively to obtain a first state result, a second state result and a third state result; based on the first state result, the second state result and the third state result, determining the target state of the target object at each time point and the target state probability value corresponding to the target state.

[0011] Furthermore, the step of processing the video data to obtain target video data includes: marking each frame image in the video data to obtain a rectangular frame on each frame image; and cropping each frame image based on the rectangular frame on each frame image to obtain the target video data.

[0012] Furthermore, the steps of processing the facial feature data, the action feature data and the audio data respectively to obtain a first state result, a second state result and a third state result include: using a first model to process the facial feature data to obtain the first state result, wherein the first model is a model trained using first historical data, and the first historical data includes: a historical facial feature data set and a state labeling result of labeling each historical facial feature data in the historical facial feature data set; using a second model to process the action feature data to obtain the second state result, wherein the second model is a model trained using second historical data, and the second historical data includes: a historical action feature data set and a state labeling result of labeling each historical action feature data in the historical action feature data set; using a third model to process the audio data to obtain the third state result, wherein the third model is a model trained using third historical data, and the second historical data includes: a historical audio data set and a state labeling result of labeling each historical audio data in the historical audio data set.

[0013] Furthermore, before using the third model to process the audio data to obtain the third state result, it also includes: filtering the audio data to obtain target audio data; and extracting features of the target audio data to obtain spectrum features, pitch features and volume features.

[0014] Further, the first state result, the second state result and the third state result all include at least: a positive state and a positive probability value corresponding to the positive state, a negative state and a negative probability value corresponding to the negative state. Based on the first state result, the second state result and the third state result, the step of determining the target state of the target object at each time point and the target state probability value corresponding to the target state includes: comparing the positive probability value with the negative probability value in the state result, and determining that the state corresponding to the state result is the positive state when the positive probability value is greater than or equal to the negative probability value, or determining that the state corresponding to the state result is the negative state when the positive probability value is less than the negative probability value, wherein the state result is the first state result, the second state result or the third state result; determining the target state of the target object at the current time point based on the states corresponding to all the state results; and determining the target state probability value corresponding to the target state based on the probability values ​​corresponding to the states in all the state results consistent with the target state.

[0015] Furthermore, the processing method also includes: collecting second audio and video data of the target teller, wherein the target teller is the teller who handles the financial business for the target object, and the second audio and video data carries a timestamp; processing the second audio and video data to obtain the teller status of the target teller at each time point and the teller status probability value corresponding to the teller status; constructing a teller status change curve of the target teller based on the teller status at each time point and the teller status probability value corresponding to the teller status; and displaying the time points when the teller status is a negative state in a preset form on the teller status change curve.

[0016] Furthermore, after collecting the second audio and video data of the target teller, it also includes: integrating the audio data in the first audio and video data and the audio data of the second audio and video data according to the timestamp to obtain a conversation text; based on the teller state change curve, determining in the conversation text the conversation content corresponding to all time points when the target teller is in the negative state; inputting the conversation text into a preset language model and outputting summary content, wherein the preset language model is a model trained using a historical conversation text set and annotated summary content annotated for each historical conversation text in the historical conversation text set.

[0017] Furthermore, before monitoring the state change curve reaching the first probability threshold line or the second probability threshold line, the step of triggering the status prompt information includes: before monitoring the state change curve reaching the first probability threshold line or the second probability threshold line, based on the conversation content and the summary content before the current time point, triggering the status prompt information.

[0018] According to another aspect of an embodiment of the present invention, there is also provided an audio and video data processing device, including: an acquisition unit, for acquiring first audio and video data of the target object when it is detected that the target object conducts financial business and obtains authorization from the target object, wherein the first audio and video data carries a timestamp; a determination unit, for determining, based on the first audio and video data, the target state of the target object at each time point and the target state probability value corresponding to the target state; a construction unit, for constructing a state change curve of the target object based on the target state at each time point and the target state probability value corresponding to the target state; and a trigger unit, for triggering a state prompt message before monitoring that the state change curve reaches a first probability threshold line or a second probability threshold line.

[0019] Furthermore, the determination unit includes: a first separation module, used to separate the first audio and video data to obtain video data and audio data; a first processing module, used to process the video data to obtain target video data; a first extraction module, used to extract facial feature data and action feature data of the target object based on the target video data; a second processing module, used to process the facial feature data, the action feature data and the audio data respectively to obtain a first state result, a second state result and a third state result; a first determination module, used to determine the target state of the target object at each time point and the target state probability value corresponding to the target state based on the first state result, the second state result and the third state result.

[0020] Furthermore, the first processing module includes: a first labeling submodule, used to label each frame image in the video data to obtain a rectangular frame on each frame image; a first cropping submodule, used to crop each frame image based on the rectangular frame on each frame image to obtain the target video data.

[0021] Furthermore, the second processing module includes: a first processing submodule, used to process the facial feature data using a first model to obtain the first state result, wherein the first model is a model trained using first historical data, and the first historical data includes: a historical facial feature data set and a state annotation result of annotating each historical facial feature data in the historical facial feature data set; a second processing submodule, used to process the action feature data using a second model to obtain the second state result, wherein the second model is a model trained using second historical data, and the second historical data includes: a historical action feature data set and a state annotation result of annotating each historical action feature data in the historical action feature data set; a third processing submodule, used to process the audio data using a third model to obtain the third state result, wherein the third model is a model trained using third historical data, and the second historical data includes: a historical audio data set and a state annotation result of annotating each historical audio data in the historical audio data set.

[0022] Furthermore, the processing device also includes: a first filtering module, used to filter the audio data to obtain target audio data before using the third model to process the audio data to obtain the third state result; a second extraction module, used to extract features of the target audio data to obtain spectrum features, pitch features and volume features.

[0023] Further, the first state result, the second state result and the third state result all include at least: a positive state and a positive probability value corresponding to the positive state, a negative state and a negative probability value corresponding to the negative state, and the first determination module includes: a first comparison submodule, used to compare the positive probability value with the negative probability value in the state result, and when the positive probability value is greater than or equal to the negative probability value, determine that the state corresponding to the state result is the positive state, or when the positive probability value is less than the negative probability value, determine that the state corresponding to the state result is the negative state, wherein the state result is the first state result, the second state result or the third state result; a first determination submodule, used to determine the target state of the target object at the current time point based on the states corresponding to all the state results respectively; a second determination submodule, used to determine the target state probability value corresponding to the target state based on the probability values ​​corresponding to the states in all the state results consistent with the target state.

[0024] Furthermore, the processing device also includes: a first acquisition module, used to collect second audio and video data of the target teller, wherein the target teller is the teller who handles the financial business for the target object, and the second audio and video data carries a timestamp; a third processing module, used to process the second audio and video data to obtain the teller status of the target teller at each time point and the teller status probability value corresponding to the teller status; a first construction module, used to construct a teller status change curve of the target teller based on the teller status at each time point and the teller status probability value corresponding to the teller status; a first display module, used to display the time points when the teller status is a negative state in a preset form on the teller status change curve.

[0025] Furthermore, the processing device also includes: a first integration module, which is used to integrate the audio data in the first audio and video data and the audio data in the second audio and video data according to the timestamp after collecting the second audio and video data of the target teller, so as to obtain a conversation text; a second determination module, which is used to determine the conversation content corresponding to all time points when the target teller is in the negative state in the conversation text based on the teller state change curve; a first input module, which is used to input the conversation text into a preset language model and output summary content, wherein the preset language model is a model trained using a historical conversation text set and annotated summary content annotated for each historical conversation text in the historical conversation text set.

[0026] Furthermore, the trigger unit includes: a first trigger module, which is used to trigger the status prompt information based on the conversation content and the summary content before the current time point before monitoring that the status change curve reaches the first probability threshold line or the second probability threshold line.

[0027] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the above-mentioned methods for processing audio and video data is implemented.

[0028] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned methods for processing audio and video data.

[0029] In the present invention, when it is detected that the target object conducts financial business and the target object's authorization is obtained, the first audio and video data of the target object is collected, wherein the first audio and video data carries a timestamp; based on the first audio and video data, the target state of the target object at each time point and the target state probability value corresponding to the target state are determined; based on the target state at each time point and the target state probability value corresponding to the target state, a state change curve of the target object is constructed; before it is monitored that the state change curve reaches the first probability threshold line or the second probability threshold line, the state prompt information is triggered, thereby solving the technical problem of low accuracy in determining the customer's emotional state in the related art.

[0030] In the present invention, by performing real-time analysis on the collected audio and video data of the target object, the target state of the target object at each time point and the target state probability value corresponding to the target state can be obtained, and then a state change curve that changes with time is constructed, so as to trigger the state prompt information in time before the state change curve is monitored to reach the first probability threshold line or the second probability threshold line. In this way, not only can the customer's emotional state be determined in real time and accurately, but also timely intervention can be made before the customer's emotional state further deteriorates, which is conducive to improving customer satisfaction and achieves the technical effect of real-time monitoring of customer emotional state and improving customer satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0032] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing audio and video data is shown;

[0033] Figure 2 is a flowchart of a method for processing audio and video data according to embodiment 1 of the present invention;

[0034] Figure 3 is a schematic diagram of a coordinate system of a state change curve according to Example 1 of the present invention;

[0035] Figure 4 is a schematic diagram of the audio and video processing flow of a user according to Embodiment 1 of the present invention;

[0036] Figure 5 is a schematic diagram of an optional apparatus for processing audio and video data according to an embodiment of the present invention;

[0037] Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) collected and involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set between the system and the relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0041] In the present invention, a method for early warning of the emotional state of users at the counters of financial institutions is proposed, which overcomes the problem that the current method cannot monitor the changes in the user's status in real time during the business handling process, only by manually selecting the satisfaction through the interactive device after the business handling is completed by the user. By real-time monitoring the changes in the emotional state of the user during the counter business handling process and giving early warning prompts in time, timely intervention can be made before the user's emotional state further deteriorates, thereby improving user satisfaction. At the same time, by monitoring the state changes of the relevant tellers and the content of the conversation between the two, it is convenient for relevant personnel to analyze the reasons for the changes in the user's emotional state, and then the subsequent service quality can be improved in a targeted manner, thereby improving the user's business handling experience.

[0042] The present invention is described in detail below in conjunction with various embodiments.

[0043] Example 1

[0044] According to an embodiment of the present application, an embodiment of a method for processing audio and video data is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for processing audio and video data is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1102a, 102b, ..., 102n are used to illustrate) processor 102 (processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera, wherein the network interface may be connected to a wired and / or wireless network. A person of ordinary skill in the art will understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0046] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0047] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for processing audio and video data in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned method for processing audio and video data is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0050] Under the above operating environment, this application provides Figure 2 The method for processing audio and video data shown. Figure 2 is a flowchart of a method for processing audio and video data according to Embodiment 1 of the present invention. Figure 2 As shown, the method comprises the following steps:

[0051] Step S201, when it is detected that the target object handles financial business and the target object's authorization is obtained, first audio and video data of the target object is collected, wherein the first audio and video data carries a timestamp.

[0052] Optionally, an emotion warning system may be used to execute the method for processing audio and video data.

[0053] In the embodiment of the present invention, the emotion warning system includes: an off-counter collection device, an on-counter collection device, and a processing device.

[0054] Among them, the off-counter collection device is installed outside the counter. Through this off-counter collection device, the audio and video of the user who is currently handling business can be collected in real time. In addition, in order to protect the privacy of the user, a corresponding authorization button can be set on the off-counter collection device to collect the audio and video of the user with the user's authorization.

[0055] The in-counter collection device is installed inside the counter and is bound one-to-one with the corresponding out-counter collection device. Through the in-counter collection device, the audio and video of the teller who is currently handling business can be collected in real time.

[0056] The processing device is connected to the collection device outside the counter and the collection device inside the counter. When the collection device outside the counter collects that a user comes to the corresponding counter, it starts to analyze the audio and video of the user and the audio and video of the teller to monitor the user's emotional state in real time.

[0057] In an embodiment of the present invention, after detecting that the target object (i.e., the user who needs to handle financial services at the counter) is handling financial services and obtaining authorization from the target object, the emotional warning system can use an off-counter collection device to collect the first audio and video data of the target object, and the collected first audio and video data carries a timestamp, which means that the emotional warning system can accurately record the moment when the user's emotions change, as well as the timestamp of every sentence communicated with the teller. The existence of timestamps ensures the real-time and traceability of the data, and provides accurate timing information for subsequent emotional analysis and dialogue text integration. This helps to instantly identify the moment of emotional fluctuations, so that responses and interventions can be made more quickly.

[0058] Step S202: determining the target state of the target object at each time point and the target state probability value corresponding to the target state based on the first audio and video data.

[0059] In an embodiment of the present invention, a processing device in an emotion warning system may be used to process the first audio and video data to determine the target state (positive emotion or negative emotion) of the target object at each time point and the target probability value corresponding to the target emotion state.

[0060] In the embodiment of the present invention, positive emotion and negative emotion are two basic categories used to describe the emotional state of an individual, and are usually used to reflect a person's inner feelings and reactions in a specific situation. In the emotional early warning system, these two concepts are used to evaluate the user's emotional state in real time so that financial institution branches can respond quickly and optimize user experience.

[0061] Among them, positive emotions refer to a positive, pleasant or satisfactory emotional experience. In the service scenario of financial institution branches, positive emotions may include user satisfaction with the service, gratitude for the friendly attitude of the staff, and the relaxed and pleasant feeling of the business handling process. When the system recognizes that the user expresses positive facial expressions, body language or voice features such as smiling, nodding, and relaxed tone, it will be classified as positive emotion.

[0062] Negative emotions are the opposite of positive emotions, and refer to negative, unpleasant, or unsatisfactory emotional experiences. In the service scenario of financial institution branches, negative emotions may come from users' dissatisfaction with the service, annoyance with the long waiting time, anger at the attitude of the staff, or disappointment with the results of certain transactions. The system determines whether the user is in a negative emotional state by identifying negative non-verbal and verbal signals such as frowning, shaking head, faster speech, higher volume, or negative words in speech.

[0063] In this embodiment, by real-time monitoring of changes in positive and negative emotions, immediate feedback on counter services can be obtained, service strategies can be adjusted in a timely manner, customer experience can be optimized, customer dissatisfaction can be prevented and reduced, and customer trust and satisfaction with financial institution outlets can be further enhanced.

[0064] Step S203: construct a state change curve of the target object based on the target state at each time point and the target state probability value corresponding to the target state.

[0065] In the embodiment of the present invention, a memory is correspondingly configured in the processing device. After the processing device extracts the user's emotional state (i.e., target state, including: positive emotion or negative emotion) at different time points and its corresponding state probability value from the collected first audio and video data, these data can be recorded and saved in the memory in chronological order, providing a basis for constructing a state change curve.

[0066] In the embodiment of the present invention, the state change curve (i.e., emotion change curve) of the target object can be generated with time as the horizontal coordinate and the emotion state and its corresponding probability value as the vertical coordinate. The high and low probability values ​​reflect the confidence in the user's emotion state. By connecting these data points to form a curve, it is possible to intuitively see how the user's emotions fluctuate over time.

[0067] Step S204: before the state change curve is monitored to reach the first probability threshold line or the second probability threshold line, triggering state prompt information.

[0068] In the embodiment of the present invention, two probability thresholds may be preset, namely a first probability threshold and a second probability threshold. The first probability threshold is used to determine whether a transition from positive emotion to negative emotion is about to occur, and the second probability threshold is used to monitor whether the negative emotion has reached a severity level that requires immediate intervention.

[0069] In an embodiment of the present invention, a first probability threshold line can be drawn on the coordinate axis where the emotion change curve is located according to the first probability threshold (that is, the value of the horizontal axis changes with time, and the value of the vertical axis is a fixed first probability threshold), and a second probability threshold line can be drawn on the coordinate axis where the emotion change curve is located according to the second probability threshold (that is, the value of the horizontal axis changes with time, and the value of the vertical axis is a fixed second probability threshold).

[0070] In an embodiment of the present invention, the emotion change curve can be continuously monitored. When it is monitored that the curve is about to reach the first probability threshold line (i.e., the probability of positive emotion drops to a certain level, indicating that the emotion may change from positive to negative) or the second probability threshold line (i.e., the probability of negative emotion rises to a certain level, indicating that the emotion has reached a level that requires emergency intervention), the emotion warning system will trigger the status prompt information in advance. In this way, by triggering the warning before the emotion change curve reaches the threshold line, it can provide financial institution outlets with sufficient time windows to take measures to avoid further deterioration of user emotional problems, which enables financial institution outlets to more proactively manage customers' emotional experiences rather than passively respond.

[0071] Figure 3 is a schematic diagram of a coordinate system of a state change curve according to Embodiment 1 of the present invention, such as Figure 3 As shown, with time as the horizontal axis and the probability value of emotion as the vertical axis, a coordinate system of the emotion change curve is constructed. For example, the probability value of positive emotion can be represented above the time axis, and the probability value increases from bottom to top; the probability value of negative emotion can be represented below the time axis, and the probability value increases from top to bottom. In addition, a dotted line representing the first probability threshold is drawn above the time axis, and a dotted line representing the second probability threshold is drawn below the time axis.

[0072] In this embodiment, by triggering status prompt information in advance, financial institution branches can take action before user emotional problems occur or deteriorate, which not only improves customer satisfaction but also promotes continuous optimization of service processes.

[0073] In summary, by performing real-time analysis on the collected audio and video data of the target object, the target state of the target object at each time point and the target state probability value corresponding to the target state can be obtained, and then a state change curve that changes over time can be constructed, so as to trigger the state prompt information in time before the state change curve is monitored to reach the first probability threshold line or the second probability threshold line. In this way, not only can the customer's emotional state be determined in real time and accurately, but also timely intervention can be made before the customer's emotional state further deteriorates, which is conducive to improving customer satisfaction and achieving the technical effect of real-time monitoring of customer emotional state and improving customer satisfaction, thereby solving the technical problem of low accuracy in determining customer emotional state in related technologies.

[0074] In order to improve the accuracy of determining the target state of the target object at each time point and the target state probability value corresponding to the target state, in the audio and video data processing method provided in Example 1 of the present application, the first audio and video data are separated to obtain video data and audio data; the video data is processed to obtain target video data; based on the target video data, facial feature data and action feature data of the target object are extracted; the facial feature data, action feature data and audio data are processed respectively to obtain a first state result, a second state result and a third state result; based on the first state result, the second state result and the third state result, the target state of the target object at each time point and the target state probability value corresponding to the target state are determined.

[0075] In an embodiment of the present invention, the originally collected first audio and video data can be separated into two independent data streams of video and audio. This step is based on the need of multimodal emotion recognition, because video data can provide information on facial expressions and body language, while audio data contains voice features. The separated video data is then processed to obtain target video data. Thereafter, facial feature data is extracted from the target video data, including but not limited to: morphological changes in key parts such as eyes, mouth, eyebrows, etc., which can reflect the emotional state of the user. At the same time, action feature data is extracted from the target video data, including: feature data such as the user's body movements and postures, to further supplement the emotional information, because some emotions may be conveyed through body language rather than facial expressions.

[0076] In an embodiment of the present invention, corresponding models are used to process facial feature data, motion feature data and audio data, respectively, to obtain a first state result, a second state result and a third state result based on different information sources, and these three results correspond to the user's emotion analysis in video and audio modalities, respectively.

[0077] In an embodiment of the present invention, taking into account that users may suppress their emotions in public, such as controlling facial expressions, or inconsistency between facial and action expressions, or inconsistency in emotional expression among different users, the first emotion, the second emotion, and the third emotion may be used to comprehensively consider the user's current emotions to determine the target emotional state (positive or negative) of the target object at each time point and the confidence level corresponding to this state (i.e., the target state probability value).

[0078] In order to improve the accuracy of determining the target video data, in the audio and video data processing method provided in Example 1 of the present application, each frame of the video data is marked to obtain a rectangular frame on each frame of the image; based on the rectangular frame on each frame of the image, each frame of the image is cropped to obtain the target video data.

[0079] In an embodiment of the present invention, each frame of the video stream can be processed, and the upper body of the target object can be located and labeled using computer vision technology, such as a target detection algorithm. For example, using pre-trained models such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector), these models can identify and frame the human body parts in the image, especially the upper body, because facial expressions and body language are important bases for emotion recognition. In this way, when extracting features (i.e., facial features and action features) later, the user's upper body area can be focused on, reducing the influence of background interference and other irrelevant information, thereby improving the accuracy of subsequent facial feature and action feature extraction.

[0080] Furthermore, if the rectangular frames on each frame are determined, each frame can be cropped based on these frames to remove irrelevant areas and retain only the image information of the user's upper body. The image data processed in this way is named "target video data".

[0081] In this embodiment, the cropped target video data not only reduces the amount of data and speeds up subsequent processing, but also ensures that the focus of the emotion recognition algorithm is on the key emotion expression area of ​​the user, namely the face and upper body movements. This step improves the efficiency and accuracy of emotion recognition because the algorithm does not have to look for key features in a large amount of background information, but can directly process image data directly related to emotional expression.

[0082] In order to improve the accuracy of determining multiple state results, in the audio and video data processing method provided in Example 1 of the present application, a first model is used to process facial feature data to obtain a first state result, wherein the first model is a model trained using first historical data, and the first historical data includes: a historical facial feature data set and a state annotation result of annotating each historical facial feature data in the historical facial feature data set; a second model is used to process action feature data to obtain a second state result, wherein the second model is a model trained using second historical data, and the second historical data includes: a historical action feature data set and a state annotation result of annotating each historical action feature data in the historical action feature data set; a third model is used to process audio data to obtain a third state result, wherein the third model is a model trained using third historical data, and the second historical data includes: a historical audio data set and a state annotation result of annotating each historical audio data in the historical audio data set.

[0083] In an embodiment of the present invention, a first model can be constructed to process facial feature data and identify emotional states. The model is trained using a large amount of first historical data, which includes a set of historical facial feature data and corresponding emotion annotation results. The emotion annotation results are based on the labeling of the emotional states (such as happiness, sadness, anger, etc.) of the historical facial feature data by experts or supervised learning algorithms, which provides a reference standard for learning the model. Through deep learning techniques, such as convolutional neural networks, the model can learn the association between facial features and emotional states, so that in practical applications, it can output emotion recognition results based on newly collected facial feature data, i.e., first state results.

[0084] In addition, a second model can be constructed to process the motion feature data. The second model is trained based on a large amount of second historical data, and the data set includes historical motion feature data and its corresponding emotion labeling results. Through a recurrent neural network or a long short-term memory network, the second model can capture the dynamic changes of body movements over time, as well as the connection between these changes and emotional states, thereby outputting a second state result.

[0085] A third model can also be constructed to process the audio data. The third model identifies emotions by analyzing the pitch, volume, speech speed and other characteristics of the sound. The training of the third model depends on the third historical data, which includes the historical audio data and its emotion annotation results. The model can use deep learning techniques, such as recursive neural networks or gated recurrent units, to learn the complex relationship between sound features and emotional states, thereby outputting the third state result.

[0086] In order to facilitate the subsequent processing of the audio data, in the audio and video data processing method provided in Example 1 of the present application, before using the third model to process the audio data to obtain the third state result, the audio data is filtered to obtain the target audio data; and feature extraction is performed on the target audio data to obtain spectrum features, pitch features and volume features.

[0087] In the embodiment of the present invention, the user audio data is extracted, the audio of other people is filtered out, and only the user's audio is retained, thereby obtaining the target audio data.

[0088] In this embodiment, the filtering of audio data is intended to remove irrelevant or interfering background sounds, ensuring that the third model can focus on the user's voice signal, thereby improving the accuracy and efficiency of subsequent emotion recognition. Filtering usually uses noise suppression algorithms or voice activity detection technology, which can identify and separate the user's voice while reducing or completely removing interfering signals such as other customers' conversations and ambient noise at the counter. For example, detecting whether there is voice activity in the audio can distinguish between the user's voice and background noise.

[0089] In an embodiment of the present invention, if target audio data (ie, a clear voice signal of a user) is acquired, feature extraction may then be performed on the audio data to obtain audio features (ie, spectral features, pitch features, and volume features) that can represent the emotional state.

[0090] In this embodiment, the spectral features in the target audio data can be extracted. The spectral features reflect the frequency composition of the audio signal. In emotion recognition, an important spectral feature is the Mel-frequency cepstral coefficient, which can capture the frequency information in the speech signal that is closely related to human perception. The Mel-frequency cepstral coefficient is usually calculated by fast Fourier transform and Mel-frequency filter bank, which can reflect the timbre characteristics of the speech and is the key to identifying different emotional states. In addition, the pitch features in the target audio data can be extracted. The pitch feature refers to the fundamental frequency or main frequency of the sound, which is related to the intonation of the speech and can reflect the changes in the speaker's tone. In emotion recognition, the rise or fall of the pitch can imply the speaker's emotional state, such as the pitch will change differently when happy or angry. In addition, the volume feature in the target audio data can be extracted. The volume feature is the size of the sound. The emotional state is related to the volume. For example, when angry or excited, the volume of the sound will increase; while when sad or tired, the volume may decrease. The extraction of the volume feature helps the model to identify the intensity of the user's emotions.

[0091] Optionally, the first state result, the second state result, and the third state result all include at least: a positive state and a positive probability value corresponding to the positive state, a negative state and a negative probability value corresponding to the negative state. In order to further improve the accuracy of determining the target state of the target object at each time point and the target state probability value corresponding to the target state, in the method for processing audio and video data provided in Example 1 of the present application, the positive probability value and the negative probability value in the state result are compared, and when the positive probability value is greater than or equal to the negative probability value, the state corresponding to the state result is determined to be a positive state, or, when the positive probability value is less than the negative probability value, the state corresponding to the state result is determined to be a negative state, wherein the state result is the first state result, the second state result, or the third state result; based on the states corresponding to all the state results, respectively, the target state of the target object at the current time point is determined; based on the probability values ​​corresponding to the states in all the state results consistent with the target state, the target state probability value corresponding to the target state is determined.

[0092] In an embodiment of the present invention, the first state result, the second state result and the third state result all include at least: a positive emotional state and a positive probability value corresponding to the positive emotional state, a negative emotional state and a negative probability value corresponding to the negative emotional state, and the sum of the positive probability value and the negative probability value in the same state result is 1. For example, the probability of outputting a positive emotion is 0.7, and the probability of outputting a negative emotion is 0.3.

[0093] In an embodiment of the present invention, for each emotion recognition result (first state result, second state result, or third state result), the probability value of positive emotion (positive probability value) and the probability value of negative emotion (negative probability value) can be compared. If the positive probability value is greater than or equal to the negative probability value, the emotional state corresponding to the state result is determined to be a positive emotional state; conversely, if the positive probability value is less than the negative probability value, the emotional state is determined to be a negative emotional state. This process ensures that the system can make a clear emotional classification based on the probability value, that is, positive or negative.

[0094] In an embodiment of the present invention, after obtaining the emotional states of all emotion recognition results, the system will perform a comprehensive analysis to determine the target emotional state of the target object at the current time point. If the emotional states of most emotion recognition results are positive, the target emotional state is determined to be positive; conversely, if most emotional states are negative, the target emotional state is determined to be negative. For example, if the results of the first and second emotion recognition are positive emotions, and the result of the third emotion recognition is a negative emotion, then the final emotion recognition result is considered to be a positive emotion. This decision-making mechanism of majority voting or weighted average makes full use of the complementarity of multimodal information, avoids errors caused by emotion recognition of a single method, and improves the robustness and accuracy of emotion recognition.

[0095] In an embodiment of the present invention, a target state probability value corresponding to a target emotional state can be calculated, and this probability value is determined based on the state probability values ​​in all emotion recognition results that are consistent with the target emotional state. If the target emotional state is positive, the system will calculate the final positive state probability value based on the positive probability values ​​of all positive state results; if the target emotional state is negative, the final negative state probability value is calculated based on the negative probability values ​​of all negative state results. This process usually involves a weighted average of probability values ​​to reflect the contribution of different modal information to emotion recognition. For example, the first emotion is a positive emotion and the first probability value is 0.7, the second emotion is a positive emotion and the first probability value is 0.6. The third emotion is a negative emotion and the second probability value is 0.6, then the final emotion recognition result is a positive emotion, and its corresponding probability value is (0.7+0.6) / 2=0.65.

[0096] In order to improve the accuracy of constructing the teller status change curve, in the audio and video data processing method provided in Example 1 of the present application, second audio and video data of the target teller is collected, wherein the target teller is a teller who handles financial services for the target object, and the second audio and video data carries a timestamp; the second audio and video data is processed to obtain the teller status of the target teller at each time point and the teller status probability value corresponding to the teller status; based on the teller status at each time point and the teller status probability value corresponding to the teller status, a teller status change curve of the target teller is constructed; and the time points when the teller status is a negative state are displayed in a preset form on the teller status change curve.

[0097] In the embodiment of the present invention, the in-counter acquisition device in the emotion early warning system can collect the audio and video data (i.e., the second audio and video data) of the target teller (i.e., the teller who handles financial business for the target object) in real time, and add a time stamp to these data. The addition of the time stamp makes the teller's audio and video data consistent with the user's audio and video data in time, which is convenient for subsequent emotional synchronization analysis.

[0098] In the embodiment of the present invention, similar to the process of identifying user emotions, the system separates the audio and video data of the target teller into two parts: video and audio. The video data is used to extract facial features and action features, and the audio data is used to extract sound features. These features are then input into the corresponding model to identify the teller's emotional state (positive or negative) and its probability value at each time point. This step makes full use of the system's multimodal analysis capabilities for emotion recognition, ensuring the accuracy and real-time recognition of the teller's emotional state.

[0099] Then, based on the teller's emotional state at each time point and its corresponding teller state probability value, the target teller's emotional change curve (i.e., teller emotional change curve) is constructed. This curve uses time as the horizontal axis and the emotional state and its probability value as the vertical axis, which intuitively shows the fluctuation of the teller's emotions over time. In addition, the system can also set a warning threshold for negative emotions. When the teller's emotional state reaches or exceeds this threshold, the system will trigger the warning mechanism.

[0100] In an embodiment of the present invention, on the teller's emotion change curve, the system will highlight the time points when the teller's emotional state is in a negative emotional state in a preset form (such as different colors, bold, flashing, etc.). This design makes it easy for managers to quickly identify abnormal periods of teller emotional fluctuations, so that corresponding intervention measures can be taken in a timely manner.

[0101] In this embodiment, the emotional warning system can not only monitor the emotional changes of users, but also pay attention to the emotional state of tellers, thereby comprehensively improving the emotional management level of counter services and providing a comprehensive, real-time emotional monitoring and early warning platform for financial institutions. This not only helps financial institutions to deal with emotional problems in a timely manner, but also promotes the optimization of service processes and improves the overall service quality.

[0102] In order to facilitate the subsequent determination of early warning prompt information, in the audio and video data processing method provided in Example 1 of the present application, after collecting the second audio and video data of the target teller, the audio data in the first audio and video data and the audio data of the second audio and video data are integrated according to the timestamp to obtain a conversation text; based on the teller state change curve, the conversation content corresponding to all time points when the target teller is in a negative state is determined in the conversation text; the conversation text is input into a preset language model, and the summary content is output, wherein the preset language model is a model trained using a historical conversation text set and annotated summary content annotated for each historical conversation text in the historical conversation text set.

[0103] In an embodiment of the present invention, the audio of the user (first audio and video data) and the teller (second audio and video data) can be aligned and integrated according to the timestamp on each audio data to obtain the conversation text. This integration process ensures that the context information of the conversation between the user and the teller is consistent, and the generated conversation text can accurately reflect the communication content of the two parties at different time points. Through speech recognition technology, the audio is converted into text so that the content of the conversation can be further analyzed.

[0104] Then, based on the teller's emotion change curve, all time points when the target teller was in a negative emotional state were screened out. For these time points, the system will extract the corresponding conversation content from the integrated conversation text, that is, the conversation record between the teller and the user in a negative emotional state. This analysis process helps to deeply understand the specific reasons for the emotional changes and provide more targeted data support for subsequent emotional management.

[0105] In an embodiment of the present invention, since the conversation text may be relatively long and contain some routine conversation content, viewing the complete conversation text may consume too much time and may result in the inability to intervene in time. Therefore, in order to more efficiently understand the key information in the conversation text, the conversation text can be input into a preset language model, which is trained through a collection of historical conversation texts and corresponding annotated summary content. The model can automatically identify the topic, emotional trigger points, and important details in the conversation, and output a concise conversation summary content. This summary content not only contains the main information of the conversation, but also particularly highlights the specific conversations when the teller is in a negative emotional state, making it easy to quickly understand the details of emotional problems.

[0106] In order to improve the accuracy of determining status prompt information, in the audio and video data processing method provided in Example 1 of the present application, before monitoring the state change curve reaching the first probability threshold line or the second probability threshold line, the status prompt information is triggered based on the conversation content and summary content before the current time point.

[0107] In an embodiment of the present invention, the emotion change curves of users and tellers can be continuously monitored. These curves reflect the fluctuation of emotional states over time and are accompanied by probability values ​​of emotional states. The first probability threshold line and the second probability threshold line on the emotion change curve represent warning lines for the transition from positive emotions to negative emotions and the increase in the intensity of negative emotions, respectively. When the system detects that the emotion change curve is about to touch these warning lines, it will not wait until the threshold is actually crossed before taking action, but will trigger status prompt information in advance based on the conversation content before the current time point and the summary content generated by the preset language model. The design of this early warning mechanism aims to capture signs of emotional changes in advance and provide branch staff with sufficient time to take preventive measures to avoid the deterioration of emotional problems.

[0108] In the embodiment of the present invention, when an early warning is triggered, the system will provide all relevant conversation content and summary content before the current time point. This information includes the background and specific situation of the triggering early warning, helping the staff to understand the root cause of the emotional problem, so that they can intervene more accurately. For example, the staff can see which topics or service links in the conversation may have triggered the user's negative emotions, and what emotional states the teller showed in the process.

[0109] In the embodiment of the present invention, if the first intervention opportunity is missed (i.e., before the emotion change curve reaches the first probability threshold line), or if it is subjectively judged that intervention is not necessary at the moment, a backup can be provided at the second intervention opportunity (i.e., before the emotion change curve reaches the second probability threshold line). In addition, since the early warning occurs before the first probability threshold or the second probability threshold is actually reached, it also gives relevant personnel sufficient preparation time.

[0110] In this embodiment, by triggering status prompt information before the emotional change reaches the warning line, the emotional warning system can not only monitor emotional fluctuations in real time, but also actively warn, provide decision-making basis for staff, and achieve prevention and management of emotional problems, thereby effectively improving the quality of counter services and user satisfaction.

[0111] Figure 4 is a schematic diagram of the audio and video processing flow of a user according to Embodiment 1 of the present invention, such as Figure 4As shown, when the user handles business at the counter, the acquisition device outside the counter can obtain the user's audio and video data in real time. Then, the user's audio and video data are separated to obtain user video and user audio. The user video is processed by annotating a rectangular frame to generate a cropped video. The video focuses on the user's upper body, especially the face, to improve the accuracy of emotion recognition. Subsequently, based on the cropped video, facial features and action features are extracted, and the facial features are input into the first emotion recognition model (i.e., the first model) to obtain the first emotion, and the action features are input into the second emotion recognition model (i.e., the second model) to obtain the second emotion. At the same time, the user audio is filtered to remove background noise to ensure that only the user's voice is retained, and the filtered audio is obtained. The filtered audio data is input into the third emotion recognition model (i.e., the third model) to obtain the third emotion. In addition, the filtered audio is converted to obtain text with timestamps, which is saved in the form of text + timestamp.

[0112] The method for processing audio and video data provided in the embodiment of the present application can obtain the target emotional state of the target object at each time point and the target state probability value corresponding to the target emotional state by performing real-time analysis on the collected audio and video data of the target object, and then construct an emotion change curve that changes over time, so as to timely trigger status prompt information before monitoring the emotion change curve to reach the first probability threshold line or the second probability threshold line. In this way, not only can the customer's emotional state be determined in real time and accurately, but also timely intervention can be made before the customer's emotion further deteriorates, which is conducive to improving customer satisfaction and achieving the technical effect of real-time monitoring of customer emotions and improving customer satisfaction, thereby solving the technical problem of low accuracy in determining the customer's emotional state in related technologies.

[0113] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0114] Example 2

[0115] The embodiment of the present application also provides a device for processing audio and video data. It should be noted that the device for processing audio and video data in the embodiment of the present application can be used to execute the method for processing audio and video data provided in the embodiment of the present application. The following introduces the device for processing audio and video data provided in the embodiment of the present application.

[0116] According to an embodiment of the present application, a device for implementing the above-mentioned audio and video data processing method is also provided. Figure 5 is a schematic diagram of an optional apparatus for processing audio and video data according to an embodiment of the present invention, such as Figure 5 As shown, the processing device may include: a collecting unit 50 , a determining unit 51 , a constructing unit 52 , and a triggering unit 53 .

[0117] The collection unit 50 is used to collect first audio and video data of the target object when it is detected that the target object handles financial business and obtains authorization from the target object, wherein the first audio and video data carries a timestamp;

[0118] A determination unit 51 is used to determine a target state of a target object at each time point and a target state probability value corresponding to the target state based on the first audio and video data;

[0119] A construction unit 52, configured to construct a state change curve of the target object based on the target state at each time point and the target state probability value corresponding to the target state;

[0120] The trigger unit 53 is used to trigger the status prompt information before monitoring the status change curve reaching the first probability threshold line or the second probability threshold line.

[0121] The audio and video data processing device provided in the embodiment of the present application can obtain the target state of the target object at each time point and the target state probability value corresponding to the target state by performing real-time analysis on the collected audio and video data of the target object, and then construct a state change curve that changes with time, so as to timely trigger the state prompt information before monitoring the state change curve to reach the first probability threshold line or the second probability threshold line. In this way, not only can the customer's emotional state be determined in real time and accurately, but also timely intervention can be made before the customer's emotional state further deteriorates, which is conducive to improving customer satisfaction and achieving the technical effect of real-time monitoring of customer emotional state and improving customer satisfaction, thereby solving the technical problem of low accuracy in determining customer emotional state in related technologies.

[0122] Optionally, the determination unit includes: a first separation module, used to separate the first audio and video data to obtain video data and audio data; a first processing module, used to process the video data to obtain target video data; a first extraction module, used to extract facial feature data and motion feature data of the target object based on the target video data; a second processing module, used to process the facial feature data, motion feature data and audio data respectively to obtain a first state result, a second state result and a third state result; a first determination module, used to determine the target state of the target object at each time point and the target state probability value corresponding to the target state based on the first state result, the second state result and the third state result.

[0123] Optionally, the first processing module includes: a first labeling submodule, used to label each frame image in the video data to obtain a rectangular frame on each frame image; a first cropping submodule, used to crop each frame image based on the rectangular frame on each frame image to obtain target video data.

[0124] Optionally, the second processing module includes: a first processing submodule, used to process facial feature data using a first model to obtain a first state result, wherein the first model is a model trained using first historical data, and the first historical data includes: a historical facial feature data set and a state annotation result of annotating each historical facial feature data in the historical facial feature data set; a second processing submodule, used to process action feature data using a second model to obtain a second state result, wherein the second model is a model trained using second historical data, and the second historical data includes: a historical action feature data set and a state annotation result of annotating each historical action feature data in the historical action feature data set; a third processing submodule, used to process audio data using a third model to obtain a third state result, wherein the third model is a model trained using third historical data, and the second historical data includes: a historical audio data set and a state annotation result of annotating each historical audio data in the historical audio data set.

[0125] Optionally, the processing device also includes: a first filtering module, used to filter the audio data to obtain target audio data before using the third model to process the audio data to obtain a third state result; a second extraction module, used to extract features from the target audio data to obtain spectral features, pitch features and volume features.

[0126] Optionally, the first state result, the second state result and the third state result all include at least: a positive state and a positive probability value corresponding to the positive state, a negative state and a negative probability value corresponding to the negative state, and the first determination module includes: a first comparison submodule, used to compare the positive probability value and the negative probability value in the state result, and when the positive probability value is greater than or equal to the negative probability value, determine that the state corresponding to the state result is a positive state, or, when the positive probability value is less than the negative probability value, determine that the state corresponding to the state result is a negative state, wherein the state result is the first state result, the second state result or the third state result; the first determination submodule, used to determine the target state of the target object at the current time point based on the states corresponding to all state results respectively; the second determination submodule, used to determine the target state probability value corresponding to the target state based on the probability values ​​corresponding to the states in all state results consistent with the target state.

[0127] Optionally, the processing device also includes: a first acquisition module, used to collect second audio and video data of the target teller, wherein the target teller is a teller who handles financial services for the target object, and the second audio and video data carries a timestamp; a third processing module, used to process the second audio and video data to obtain the teller status of the target teller at each time point and the teller status probability value corresponding to the teller status; a first construction module, used to construct a teller status change curve of the target teller based on the teller status at each time point and the teller status probability value corresponding to the teller status; a first display module, used to display the time point when the teller status is a negative state in a preset form on the teller status change curve.

[0128] Optionally, the processing device also includes: a first integration module, which is used to integrate the audio data in the first audio and video data and the audio data in the second audio and video data according to the timestamp after collecting the second audio and video data of the target teller, so as to obtain a conversation text; a second determination module, which is used to determine the conversation content corresponding to all time points when the target teller is in a negative state in the conversation text based on the teller state change curve; a first input module, which is used to input the conversation text into a preset language model and output summary content, wherein the preset language model is a model trained using a historical conversation text set and annotated summary content annotated for each historical conversation text in the historical conversation text set.

[0129] Optionally, the trigger unit includes: a first trigger module, used to trigger status prompt information based on the conversation content and summary content before the current time point before monitoring the status change curve reaching the first probability threshold line or the second probability threshold line.

[0130] The above-mentioned processing device may also include a processor and a memory. The above-mentioned acquisition unit 50, determination unit 51, construction unit 52, trigger unit 53, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0131] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set to trigger the status prompt information before the state change curve reaches the first probability threshold line or the second probability threshold line by adjusting the kernel parameters.

[0132] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (fl ash RAM), and the memory includes at least one storage chip.

[0133] It should be noted that the above-mentioned acquisition unit 50, determination unit 51, construction unit 52, and trigger unit 53 correspond to steps S201 to S204 in Example 1, and the above-mentioned units and corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned units can also be run in the computer terminal 10 provided in Example 1 as part of the device.

[0134] Example 3

[0135] An embodiment of the present application may provide an electronic device, Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present invention. Figure 6 As shown, the electronic device may include: one or more ( Figure 6 (only one is shown) processor 602, memory 604, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0136] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for processing audio and video data in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for processing audio and video data. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: when it is detected that the target object is conducting financial business and the target object is authorized, collect the first audio and video data of the target object, wherein the first audio and video data carries a timestamp; based on the first audio and video data, determine the target state of the target object at each time point and the target state probability value corresponding to the target state; based on the target state at each time point and the target state probability value corresponding to the target state, construct a state change curve of the target object; before monitoring that the state change curve reaches the first probability threshold line or the second probability threshold line, trigger the state prompt information.

[0138] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: separate the first audio and video data to obtain video data and audio data; process the video data to obtain target video data; extract facial feature data and motion feature data of the target object based on the target video data; process the facial feature data, motion feature data and audio data respectively to obtain a first state result, a second state result and a third state result; based on the first state result, the second state result and the third state result, determine the target state of the target object at each time point and the target state probability value corresponding to the target state.

[0139] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: annotate each frame of the video data to obtain a rectangular frame on each frame; based on the rectangular frame on each frame, crop each frame to obtain the target video data.

[0140] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: using a first model to process facial feature data to obtain a first state result, wherein the first model is a model trained using first historical data, and the first historical data includes: a historical facial feature data set and a state labeling result of labeling each historical facial feature data in the historical facial feature data set; using a second model to process action feature data to obtain a second state result, wherein the second model is a model trained using second historical data, and the second historical data includes: a historical action feature data set and a state labeling result of labeling each historical action feature data in the historical action feature data set; using a third model to process audio data to obtain a third state result, wherein the third model is a model trained using third historical data, and the second historical data includes: a historical audio data set and a state labeling result of labeling each historical audio data in the historical audio data set.

[0141] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: filtering the audio data to obtain target audio data; extracting features of the target audio data to obtain spectrum features, pitch features and volume features.

[0142] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: compare the positive probability value and the negative probability value in the state result, and when the positive probability value is greater than or equal to the negative probability value, determine that the state corresponding to the state result is a positive state, or, when the positive probability value is less than the negative probability value, determine that the state corresponding to the state result is a negative state, wherein the state result is a first state result, a second state result or a third state result; based on the states corresponding to all state results, determine the target state of the target object at the current time point; based on the probability values ​​corresponding to the states in all state results consistent with the target state, determine the target state probability value corresponding to the target state.

[0143] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: collect the second audio and video data of the target teller, wherein the target teller is a teller who handles financial services for the target object, and the second audio and video data carries a timestamp; process the second audio and video data to obtain the teller status of the target teller at each time point and the teller status probability value corresponding to the teller status; based on the teller status at each time point and the teller status probability value corresponding to the teller status, construct a teller status change curve of the target teller; and display the time point when the teller status is a negative state in a preset form on the teller status change curve.

[0144] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: integrating the audio data in the first audio and video data and the audio data in the second audio and video data according to the timestamp to obtain a conversation text; determining the conversation content corresponding to all time points when the target teller is in a negative state in the conversation text based on the teller state change curve; inputting the conversation text into a preset language model and outputting summary content, wherein the preset language model is a model trained using a historical conversation text set and annotated summary content annotated for each historical conversation text in the historical conversation text set.

[0145] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: before monitoring the state change curve reaching the first probability threshold line or the second probability threshold line, trigger the status prompt information based on the conversation content and summary content before the current time point.

[0146] By adopting the embodiment of the present application, a solution for processing audio and video data is provided. By performing real-time analysis on the collected audio and video data of the target object, the target state of the target object at each time point and the target state probability value corresponding to the target state can be obtained, and then a state change curve that changes over time is constructed, so as to timely trigger the state prompt information before the state change curve is monitored to reach the first probability threshold line or the second probability threshold line. In this way, not only can the customer's emotional state be determined in real time and accurately, but also timely intervention can be made before the customer's emotional state further deteriorates, which is conducive to improving customer satisfaction, achieving the technical effect of real-time monitoring of customer emotional state and improving customer satisfaction, and thus solving the technical problem of low accuracy in determining the customer's emotional state in related technologies.

[0147] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, and a Mobile Internet Device (MID). Figure 6 The structure of the electronic device is not limited. Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 6 Different configurations are shown.

[0148] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0149] Example 4

[0150] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for processing audio and video data provided in the first embodiment.

[0151] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0152] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the method for processing audio and video data.

[0153] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0154] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0156] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0159] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing audio and video data, characterized in that: include: When it is detected that the target object conducts financial business and the target object authorizes the target object, first audio and video data of the target object is collected, wherein the first audio and video data carries a timestamp; Based on the first audio and video data, determining a target state of the target object at each time point and a target state probability value corresponding to the target state; Based on the target state at each of the time points and the target state probability value corresponding to the target state, constructing a state change curve of the target object; Before it is monitored that the state change curve reaches the first probability threshold line or the second probability threshold line, the state prompt information is triggered.

2. The processing method according to claim 1, characterized in that: The step of determining the target state of the target object at each time point and the target state probability value corresponding to the target state based on the first audio and video data includes: Separating the first audio and video data to obtain video data and audio data; Processing the video data to obtain target video data; Based on the target video data, extracting facial feature data and action feature data of the target object; Processing the facial feature data, the action feature data and the audio data respectively to obtain a first state result, a second state result and a third state result; Based on the first state result, the second state result, and the third state result, the target state of the target object at each time point and the target state probability value corresponding to the target state are determined.

3. The processing method according to claim 2, characterized in that: The step of processing the video data to obtain target video data includes: Annotate each frame of the video data to obtain a rectangular frame on each frame of the video data; Based on the rectangular frame on each frame of image, each frame of image is cropped to obtain the target video data.

4. The processing method according to claim 2, characterized in that: The steps of processing the facial feature data, the action feature data and the audio data respectively to obtain a first state result, a second state result and a third state result include: Processing the facial feature data using a first model to obtain the first state result, wherein the first model is a model trained using first historical data, and the first historical data includes: a historical facial feature data set and a state labeling result of labeling each historical facial feature data in the historical facial feature data set; Processing the action feature data using a second model to obtain the second state result, wherein the second model is a model trained using second historical data, and the second historical data includes: a historical action feature data set and a state labeling result of labeling each historical action feature data in the historical action feature data set; The audio data is processed using a third model to obtain the third state result, wherein the third model is a model trained using third historical data, and the second historical data includes: a historical audio data set and a state labeling result for labeling each historical audio data in the historical audio data set.

5. The processing method according to claim 4, characterized in that: Before using the third model to process the audio data to obtain the third state result, the method further includes: Filtering the audio data to obtain target audio data; Feature extraction is performed on the target audio data to obtain spectrum features, pitch features and volume features.

6. The processing method according to claim 2, characterized in that: The first state result, the second state result, and the third state result all include at least: a positive state and a positive probability value corresponding to the positive state, a negative state and a negative probability value corresponding to the negative state. Based on the first state result, the second state result, and the third state result, the step of determining the target state of the target object at each time point and the target state probability value corresponding to the target state includes: Comparing the positive probability value and the negative probability value in the state result, and determining that the state corresponding to the state result is the positive state when the positive probability value is greater than or equal to the negative probability value, or determining that the state corresponding to the state result is the negative state when the positive probability value is less than the negative probability value, wherein the state result is the first state result, the second state result or the third state result; Determining the target state of the target object at the current time point based on the states respectively corresponding to all the state results; The target state probability value corresponding to the target state is determined based on the probability values ​​corresponding to the state in all the state results consistent with the target state.

7. The processing method according to claim 1, characterized in that: The processing method also includes: Collecting second audio and video data of a target teller, wherein the target teller is a teller who handles the financial business for the target object, and the second audio and video data carries a timestamp; Processing the second audio and video data to obtain the teller state of the target teller at each time point and the teller state probability value corresponding to the teller state; Based on the teller state at each of the time points and the teller state probability value corresponding to the teller state, constructing a teller state change curve of the target teller; The time point when the teller status is in a negative state is displayed on the teller status change curve in a preset form.

8. The processing method according to claim 7, characterized in that: After collecting the second audio and video data of the target teller, the method further includes: Integrate the audio data in the first audio and video data and the audio data in the second audio and video data according to the timestamp to obtain a conversation text; Based on the teller status change curve, determining in the conversation text the conversation contents corresponding to all time points when the target teller is in the negative state; The dialogue text is input into a preset language model, and summary content is output, wherein the preset language model is a model trained using a historical dialogue text set and annotated summary content annotated for each historical dialogue text in the historical dialogue text set.

9. The processing method according to claim 8, characterized in that: Before monitoring that the state change curve reaches the first probability threshold line or the second probability threshold line, the step of triggering the state prompt information includes: Before it is monitored that the state change curve reaches the first probability threshold line or the second probability threshold line, the state prompt information is triggered based on the conversation content and the summary content before the current time point.

10. A device for processing audio and video data, characterized in that: include: A collection unit, configured to collect first audio and video data of the target object when detecting that the target object handles financial business and obtaining authorization from the target object, wherein the first audio and video data carries a timestamp; a determination unit, configured to determine, based on the first audio and video data, a target state of the target object at each time point and a target state probability value corresponding to the target state; A construction unit, configured to construct a state change curve of the target object based on the target state at each of the time points and the target state probability value corresponding to the target state; The trigger unit is used to trigger the status prompt information before monitoring that the status change curve reaches the first probability threshold line or the second probability threshold line.

11. A computer program product, characterized in that It comprises a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for processing audio and video data according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for processing audio and video data as described in any one of claims 1 to 9.

Images