Call processing method and device, electronic equipment and storage medium

Abstract

The invention provides a call processing method, which comprises the following steps of: acquiring image data and voice data of two parties in a video call process of the two parties in the call; determining a visual call state and a voice call state of the two call parties based on the image data and the voice data of the two call parties; and hanging up or maintaining the video call based on the visual call state and the voice call state of the two parties of the call. According to the video call processing method and device, in the video call process, whether hanging-up processing or maintaining processing needs to be carried out is jointly judged according to the visual call states and the voice call states of the two parties of the call, and the visual call states and the voice call states of the two parties of the call are determined through the image data and the voice data of the two parties of the call; according to the invention, intentions of both parties of a call can be determined more accurately, the error rate of intention judgment is reduced, hang-up processing or maintenance processing is performed more accurately, and the video call experience of a user is improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a call processing method, device, electronic device and storage medium. Background Art

[0002] With the development of communication technology, video calls have entered people's lives. People can communicate visually and vocally through video calls, making contact more convenient and vivid. In real life, when using communication software for video calls, users may need to leave the screen for some reason. If the user forgets to turn off the video call, the video call will be maintained, which will lead to some privacy leaks on the one hand and waste power on the other. In this regard, the existing technology controls the automatic hang-up of the video call by detecting that there is no one in front of the camera, but this will cause the video call to be automatically hung up during the short absence of the user (the user does not want to hang up at this time), or turning the camera will also cause the video call to be automatically hung up, and the user needs to re-establish the video call, which has a high error rate and poor convenience, resulting in a poor user experience of the video call. Therefore, there is an urgent need for a better call processing solution to automatically hang up or maintain the video call. Summary of the invention

[0003] The embodiment of the present invention provides a call processing method, aiming to provide a more effective call processing solution to automatically hang up or maintain a video call. During a video call, the need to hang up or maintain the call can be determined based on the visual call status and voice call status of the two parties on the call. Since the visual call status and voice call status of the two parties on the call are determined by the image data and voice data of the two parties on the call, the intentions of the two parties on the call can be determined more accurately, the error rate of intention judgment can be reduced, and the hang-up or maintenance process can be performed more accurately, thereby improving the user's video call experience.

[0004] In a first aspect, an embodiment of the present invention provides a call processing method, the method comprising:

[0005] During a video call between two parties, obtaining image data and voice data of the two parties;

[0006] Determining the visual call status and the voice call status of the two parties of the call based on the image data and the voice data of the two parties of the call;

[0007] Based on the visual call status and voice call status of the two parties in the call, the video call is hung up or maintained.

[0008] Optionally, the step of determining the visual call status and the voice call status of the two parties of the call based on the image data and the voice data of the two parties of the call includes:

[0009] Based on the image data of the two parties of the call, determining the visual call status of the two parties of the call as a face state or a faceless state;

[0010] Based on the voice data of the two parties of the call, determine whether the voice call status of the two parties of the call is a voice state or a non-voice state.

[0011] Optionally, the step of hanging up or maintaining the video call based on the visual call status and the voice call status of the two parties of the call includes:

[0012] Determining a target call state combination of the two parties of the call based on the visual call state and the voice call state of the two parties of the call;

[0013] If the target call state combination belongs to a preset first type of state combination, first voice conversation data of the two call parties are obtained, the first type of state combination being a state combination when the visual call state of at least one call party is a faceless state and the voice call state is a voiceless state, and the first voice conversation data includes historical voice conversation data of the two call parties within a preset time;

[0014] Based on the first voice conversation data, hanging up or maintaining the video call;

[0015] If the target call state combination belongs to a preset second state combination, the second voice conversation data of the two call parties is obtained, and the second state combination is a state combination when the visual call states of the two call parties are both faceless and the voice call states of the two call parties are both voiced, then the second voice conversation data of the two call parties is obtained, and the second voice conversation data includes real-time voice conversation data of the two call parties;

[0016] Based on the second voice conversation data, hanging up or maintaining the video call;

[0017] If the target call state combination belongs to the preset third state combination, the video call is maintained. The second state combination is a state combination of the visual call state and the voice call state of the remaining two parties of the call except the first state combination and the second state combination.

[0018] Optionally, the first category state combination includes a first subcategory state combination and a second subcategory state combination, the first subcategory state combination is a state combination when the visual call states of both parties of the call are both in a faceless state, and the voice call state of only one party of the call is in a voice state, and the second subcategory state combination is other state combinations in the first category state combination except the first subcategory state combination. If the target call state combination belongs to the preset first category state combination, the step of obtaining the first voice conversation data of the two parties of the call includes:

[0019] If the target call state combination belongs to the first sub-category state combination, obtaining historical voice conversation data of the two call parties within a preset time and real-time voice conversation data of the two call parties as first voice conversation data;

[0020] If the target call state combination belongs to the second sub-category state combination, historical voice conversation data of the two call parties within a preset time is obtained as the first voice conversation data.

[0021] Optionally, based on the first voice conversation data, the step of hanging up or maintaining the video call includes:

[0022] Performing semantic understanding and keyword detection on the first voice dialogue data to determine a target waiting time or a target waiting event;

[0023] Based on the target waiting time or the target waiting event, determine the target maintenance time of the video call and start a maintenance mechanism;

[0024] Under the maintenance mechanism, maintaining the video call for the target maintenance duration;

[0025] If a change in the category to which the target call status combination belongs is detected within the target maintenance duration, the maintenance mechanism is cancelled;

[0026] If no change is detected in the category to which the target call status combination belongs within the target maintenance time, the video call is hung up when the maintenance time is exceeded.

[0027] Optionally, based on the target waiting time or the target waiting event, determining the target maintaining time of the video call and starting the maintaining mechanism includes:

[0028] Determine the moment when the target waiting time is identified as the starting moment;

[0029] Determining the target waiting time as the target maintaining time of the video call;

[0030] Alternatively, the time when the target call state combination is detected is determined as the starting time;

[0031] According to the correspondence between the waiting event and the maintenance duration, the target maintenance duration corresponding to the target waiting event is matched, and different waiting events correspond to different maintenance durations;

[0032] Based on the starting time and the target maintenance duration, a maintenance mechanism is started.

[0033] Optionally, the step of hanging up or maintaining the video call based on the second voice conversation data includes:

[0034] Extracting semantic features from the second voice conversation data to obtain the conversation semantics of the two parties of the conversation;

[0035] Based on the conversation semantics, determining the conversation logic and conversation relevance between the two parties of the call;

[0036] If the conversation logic and the conversation relevance do not meet the preset logic conditions and relevance conditions, the video call is hung up;

[0037] If the conversation logic and the conversation relevance meet the preset logic conditions and relevance conditions, the video call is maintained.

[0038] In a second aspect, an embodiment of the present invention further provides a call processing device, the call processing device comprising:

[0039] An acquisition module, used to acquire image data and voice data of both parties during a video call;

[0040] A first processing module, configured to determine a visual call state and a voice call state of the two parties of the call based on the image data and the voice data of the two parties of the call;

[0041] The second processing module is used to hang up or maintain the video call based on the visual call status and voice call status of the two parties of the call.

[0042] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the call processing method provided in the embodiment of the present invention when executing the computer program.

[0043] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the call processing method provided in the embodiment of the invention are implemented.

[0044] In an embodiment of the present invention, during a video call between two parties, image data and voice data of the two parties are obtained; based on the image data and voice data of the two parties, the visual call status and voice call status of the two parties are determined; based on the visual call status and voice call status of the two parties, the video call is hung up or maintained. During a video call, the present invention can jointly determine the need to hang up or maintain the process based on the visual call status and voice call status of the two parties. Since the visual call status and voice call status of the two parties are determined by the image data and voice data of the two parties, the intentions of the two parties can be determined more accurately, reducing the error rate of intention judgment, and then more accurately hang up or maintain the process, thereby improving the user's video call experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 is a flow chart of a call processing method provided by an embodiment of the present invention;

[0047] Figure 2 is a structural diagram of a call processing device provided by an embodiment of the present invention;

[0048] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] like Figure 1 As shown, Figure 1 1 is a flow chart of a call processing method provided by an embodiment of the present invention. The call processing method comprises the steps of:

[0051] 101. During a video call between two parties, image data and voice data of the two parties are obtained.

[0052] In an embodiment of the present invention, the two parties of the above-mentioned call conduct a video call through a smart terminal, and each party of the call corresponds to a smart terminal. The above-mentioned smart terminal can be a mobile phone, tablet, computer, robot, smart gateway, or other electronic device with built-in or external camera, microphone, and display.

[0053] When users use smart terminals for video call services, user A (corresponding to smart terminal A) sends a video call request to user B (corresponding to smart terminal B), and establishes a network channel for the video call through the transit server platform (server platform such as WeChat / QQ) and routers. The specific implementation steps of the video call mainly include the following aspects:

[0054] Audio and video collection: When making a video call, it is necessary to collect the user's audio and video data. When user B chooses to answer user A's video call request, user A and user B's respective smart terminals call hardware devices such as cameras and microphones to collect data through the underlying audio and video collection API.

[0055] Audio and video coding: The audio and video data collected by smart terminal A and smart terminal B need to be compressed by source and channel coding to reduce bandwidth occupancy and delay during data transmission. Common technologies include H.264 video coding and AAC audio coding.

[0056] Network transmission: The encoded audio and video data is transmitted through the network. It is transmitted to the other user's smart terminal through the transit server platform. The audio and video data of smart terminal A is transmitted to smart terminal B, and the audio and video data of smart terminal B is transmitted to smart terminal A. The commonly used protocols are UDP and IP / PORT.

[0057] Audio and video decoding: After receiving the encoded audio and video data, smart terminal A and smart terminal B need to decode and restore it to the original audio and video data. Commonly used technologies are H.264 video decoding and AAC audio decoding. Match the corresponding decoding technology according to the type of encoding technology.

[0058] Playback and rendering: Smart terminal A and smart terminal B play and render the decoded audio and video data respectively, so that user A and user B can see the image and hear the voice of each other on their respective smart terminals, call the speakers and screen displays of the smart terminals, and play the audio and video data of the other user at this time.

[0059] The above-mentioned call processing method is mainly used for hanging up or maintaining a video call. The above-mentioned call processing method can be deployed in a server or in a smart terminal. The activation of the above-mentioned call processing method requires the consent of the user. It can be understood that since the smart terminal can collect the image data and voice data of its own user, and can also receive the image data and voice data of the other party's user, the smart terminal can also execute the call processing method executed by the above-mentioned server. To avoid repeated description, the embodiment of the present invention takes the server as an example for explanation. The process of the smart terminal executing the call processing method can refer to the process of the server executing the call processing method. Therefore, the process of the smart terminal executing the call processing method will not be repeated.

[0060] After a video call is established, when both parties are in the video call, the camera and microphone of the terminal device can collect image data and voice data of the corresponding user in real time to obtain image data and voice data of both parties.

[0061] Specifically, during a video call between two parties, the smart terminals of the two parties use cameras to synchronously collect various information data within the FOV (angle of view). The microphone continuously collects audio data from both parties of the video call, and the collected audio signals are processed by the multimedia chip, audio encoder, and video encoder of the smart terminal.

[0062] 102. Determine the visual call status and voice call status of the two parties based on the image data and voice data of the two parties.

[0063] In an embodiment of the present invention, the above-mentioned visual call status is used to indicate whether the two parties of the call are in front of the smart terminal. Specifically, the above-mentioned visual call status is used to indicate whether the two parties of the call are within the FOV viewing angle of the camera of their respective smart terminals.

[0064] The above-mentioned voice call status is used to indicate whether there is voice communication between the two parties of the call.

[0065] After obtaining the image data of the two parties in the call, the image data of the two parties in the call can be processed separately by image processing technology, and then the visual call status of the two parties in the call can be determined separately according to the image processing results. For example, the image data of the two parties in the call can be classified separately by convolutional neural network CNN, support vector machine SVM, etc. The goal of image classification is the visual call status, that is, the image data is input into the convolutional neural network CNN or the support vector machine SVM, and the corresponding visual call status is output. In some possible embodiments, the above-mentioned image processing can only detect whether there is a user portrait. If there is a user portrait, it means that the user is still in front of the smart terminal. If there is no user portrait, it means that the user is not in front of the smart terminal.

[0066] After obtaining the voice data of both parties of the call, the voice data of both parties of the call can be processed by natural language processing technology, and then the voice call status of both parties of the call can be determined respectively according to the natural language processing results. For example, the voice data of both parties of the call can be processed by natural language processing by recurrent neural network RNN, long short-term memory network LSTM, BERT, etc., and the voice call status can be output. In some possible embodiments, the above natural language processing can only detect whether there is user voice. If there is user voice, it means that the user has voice communication, and if there is no user voice, it means that the user does not have voice communication.

[0067] 103. Based on the visual call status and voice call status of the two parties in the call, the video call is hung up or maintained.

[0068] In an embodiment of the present invention, after determining the visual call status and voice call status of the two parties of the call, the intentions of the two parties of the call can be jointly determined based on the visual call status and the voice call status. If one of the two parties of the call intends to hang up, the video call is hung up. If the intention of both parties of the call is not to hang up, the video call is maintained.

[0069] The visual call status and the voice call status can be classified into two categories through a binary classification model, and the binary classification result is two categories: hang up or maintain. If the binary classification result is hang up, the video call is hung up, and if the binary classification result is maintain, the video call is maintained. The visual call status and the voice call status can also be judged by preset rules, and whether to hang up the video call or maintain the video call is determined according to the judgment result.

[0070] The above-mentioned hang-up processing can be understood as disconnecting the communication connection between the smart terminals corresponding to the two parties of the call, and the above-mentioned maintenance processing can be understood as maintaining the communication connection between the smart terminals corresponding to the two parties of the call.

[0071] In an embodiment of the present invention, during a video call between two parties, image data and voice data of the two parties are obtained; based on the image data and voice data of the two parties, the visual call status and voice call status of the two parties are determined; based on the visual call status and voice call status of the two parties, the video call is hung up or maintained. During a video call, the present invention can jointly determine the need to hang up or maintain the process based on the visual call status and voice call status of the two parties. Since the visual call status and voice call status of the two parties are determined by the image data and voice data of the two parties, the intentions of the two parties can be determined more accurately, reducing the error rate of intention judgment, and then more accurately hang up or maintain the process, thereby improving the user's video call experience.

[0072] It should be noted that the call processing method provided in the embodiment of the present invention can be applied to devices such as smart phones, tablets, computers, robots, servers, etc. that can perform the call processing method.

[0073] It can be understood that in the specific implementation of this application, user data, image data, voice data, face data and other related data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data, as well as the training and use of large models need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0074] Optionally, in the step of determining the visual call status and voice call status of the two parties of the call based on the image data and voice data of the two parties of the call, the visual call status of the two parties of the call can be determined as a face state or a faceless state based on the image data of the two parties of the call; and the voice call status of the two parties of the call can be determined as a voice state or a non-voice state based on the voice data of the two parties of the call.

[0075] In an embodiment of the present invention, during a video call between two parties, the first image data and the first voice data of the first party can be obtained through the first smart terminal, and the second image data and the second voice data of the second party can be obtained through the second smart terminal. The first party and the second party are the two parties. The first smart terminal corresponds to the first party, and the second smart terminal corresponds to the second party.

[0076] When the number of the first call party and the second call party is only one person respectively, the first call party refers to user A, and the second call party refers to user B.

[0077] When the smart terminal camera collects video image information, face recognition can be used to determine the real-time number of people in the corresponding smart terminal within the FOV. The faces that appear in the camera's field of view are recorded and numbered, such as the faces in smart terminal A: A1, A2.........A n ; Faces in smart terminal B: B1, B2..........B n .

[0078] The first image data can be processed for face recognition by a face recognition model to obtain a face recognition result of the first image data. If there is no face data in the face recognition result, it can be determined that the visual call state of the first caller is a faceless state. If there is face data related to the first caller in the face recognition result, it can be determined that the visual call state of the first caller is a face-in state. The face data related to the first caller can be the face data of the first caller, or the face data of a person who has an associated relationship with the first caller. The above-mentioned associated relationship can be a family member, friend, colleague, etc. For example, in A1, A2.........A n In the example, the face of the first caller is A1, and the face of the first caller's daughter is A2. If A2 is recognized without A1, it can also be determined that the visual call state of the first caller is a face state.

[0079] Similarly, the face recognition model can be used to perform face recognition processing on the second image data to obtain the face recognition result of the second image data. If there is no face data in the face recognition result, it can be determined that the visual call state of the second call party is a faceless state. If there is face data related to the second call party in the face recognition result, it can be determined that the visual call state of the second call party is a face state. Among them, the face data related to the second call party can be the face data of the second call party, or it can be the face data of a person who has an associated relationship with the second call party. The above relationship association can be a relationship between family, friends, colleagues, etc. For example, in B1, B2..........B n In the example, the face of the first caller is B1, and the face of the first caller's daughter is B2. If B2 is recognized without B1, it can also be determined that the visual call state of the second caller is a face state.

[0080] The first voice data may be processed for voice recognition by a voice recognition model to obtain a voice recognition result of the first voice data. If the voice recognition result contains human voice data, the voice call state of the first caller may be determined to be a voice state. If the voice recognition result contains no human voice data, the voice call state of the first caller may be determined to be a voice-free state. For example, if the voice recognition result shows that someone is speaking, the voice call state of the first caller may be determined to be a voice state. If the voice recognition result shows that no one is speaking or only background sound is recognized, the voice call state of the first caller may be determined to be a voice-free state.

[0081] Similarly, the second voice data can be processed by voice recognition through the voice recognition model to obtain the voice recognition result of the second voice data. If the voice recognition result contains human voice data, the voice call state of the second caller can be determined to be a voice state. If the voice recognition result contains human voice data, the voice call state of the second caller can be determined to be a no-voice state. For example, if the voice recognition result shows that someone is speaking, the voice call state of the second caller can be determined to be a voice state. If the voice recognition result shows that no one is speaking, or only background sound is recognized, the voice call state of the second caller can be determined to be a no-voice state.

[0082] By determining the visual call status of the two parties through their image data and determining the voice call status of the two parties through their voice data, the intentions of the two parties can be determined more accurately.

[0083] Optionally, in the step of hanging up or maintaining the video call based on the visual call status and voice call status of the two parties to the call, a target call state combination of the two parties to the call can be determined based on the visual call status and voice call status of the two parties to the call; if the target call state combination belongs to a preset first type of state combination, first voice conversation data of the two parties to the call is obtained, the first type of state combination is a state combination when the visual call status of at least one party of the call is a faceless state, and the voice call status is a voiceless state, and the first voice conversation data includes historical voice conversation data of the two parties to the call within a preset time; based on the first voice conversation data, the video call is hung up or maintained; if the target call state If the combination belongs to the preset second-category state combination, the second voice conversation data of the two parties of the call is obtained. The second-category state combination is the state combination when the visual call state of both parties of the call is a faceless state, and the voice call state of both parties is a voice state. Then the second voice conversation data of the two parties of the call is obtained, and the second voice conversation data includes the real-time voice conversation data of the two parties of the call; based on the second voice conversation data, the video call is hung up or maintained; if the target call state combination belongs to the preset third-category state combination, the video call is maintained. The second-category state combination is the state combination of the visual call state and the voice call state of the other two parties of the call except the first-category state combination and the second-category state combination.

[0084] In the embodiment of the present invention, the visual call state includes a face state and a non-face state, and the voice call state includes a voice state and a non-voice state. For a call party, the visual call state and the voice call state can be combined into 4 call states, namely, a face state with no voice, a face state with voice, a face state with voice, a face state with voice, and a face state with no voice. Each call party has 4 call states, so there are 16 call states for the two call parties. Taking smart terminal A and smart terminal B as an example, smart terminal A includes 4 states: face state with no voice, face state with voice, face state with voice, face state with voice, and face state with no voice. Smart terminal B also includes 4 states: face state with no voice, face state with voice, face state with voice, face state with voice, and face state with no voice. The above 16 call states are specifically shown in the following Table 1:

[0085] Table 1

[0086]

[0087] As shown in Table 1 above, when there is a no-voice and no-face state on one side of smart terminal A or smart terminal B, the call status of the two parties is a first-category state combination, and the first-category state combination indicates that at least one party is not in front of its own terminal device, that is, the first-category state combination is a state combination when the visual call status of at least one party is a no-face state, and the voice call status is a no-voice state. When there is a voice and no-face state on both sides of smart terminal A and smart terminal B, the call status of the two parties is a second-category state combination, and the second-category state combination indicates that the two parties are still talking, that is, the second-category state combination is a state combination when the visual call status of the two parties is a no-face state, and the voice call status of the two parties is a voice state. The call status of the remaining two parties in Table 1 is a third-category state combination.

[0088] The detected visual call status and voice call status of the first caller are combined to obtain the call status of the first caller, the detected visual call status and voice call status of the second caller are combined to obtain the call status of the second caller, and then the call status of the first caller is combined with the call status of the second caller to obtain the target call status combination of the two callers.

[0089] The target call status combination is searched in Table 1, so that the type of the target call status combination can be determined.

[0090] If the target call state combination belongs to the first state combination, the historical voice conversation data of the two parties in the call within a preset time is obtained as the first voice conversation data. Semantic understanding and keyword detection are performed on the first voice conversation data, and according to the results of the semantic understanding and keyword detection, it is determined whether to hang up or maintain the video call.

[0091] If the target call state combination belongs to the second state combination, the real-time voice conversation data of the two parties of the call is obtained as the second voice conversation data. The second voice conversation data is subjected to a conversation semantic logic judgment, and it is determined whether to hang up or maintain the video call according to the conversation semantic logic judgment.

[0092] If the target call state combination belongs to the preset third category state combination, the video call is directly maintained.

[0093] By combining the visual call status and voice call status of the two parties of the call, the call status of the two parties is determined, and the call status is classified, the call intentions of the two parties can be more accurately determined, and the hang-up process or maintenance process can be performed more accurately.

[0094] Optionally, the first category of state combinations includes a first subcategory of state combinations and a second subcategory of state combinations, the first subcategory of state combinations being a state combination in which the visual call states of both parties to the call are both faceless states, and the voice call state of one and only one party to the call is in a voice state, and the second subcategory of state combinations being other state combinations in the first category of state combinations except the first subcategory of state combinations, and in the step of obtaining first voice conversation data of both parties to the call if the target call state combination belongs to the preset first category of state combination, if the target call state combination belongs to the first subcategory of state combination, then historical voice conversation data of both parties to the call within a preset time and real-time voice conversation data of both parties to the call can be obtained as first voice conversation data; if the target call state combination belongs to the second subcategory of state combination, then historical voice conversation data of both parties to the call within a preset time and real-time voice conversation data of both parties to the call can be obtained as first voice conversation data.

[0095] In an embodiment of the present invention, the above-mentioned first subcategory status combination indicates that both parties of the call are not in front of their respective smart terminals, but one of the callers has voice data. In order to accurately determine whether the voice data is background sound or the caller's voice, while obtaining the historical voice conversation data of the two parties of the call within a preset time, the real-time voice conversation data of the two parties of the call can be additionally obtained, so that the video call can be hung up or maintained more accurately.

[0096] The above-mentioned second subcategory status combination indicates that at least one of the two parties in the call has left his or her smart terminal. Therefore, the historical voice conversation data of the two parties in the call within a preset time can be obtained, and the historical voice conversation data can be used to determine whether the left party will come back. If so, the call will be maintained. If not or the waiting time has expired, the call will be hung up.

[0097] The preset time may be T time before the current time point, where T is a value greater than 0, and the value unit may be seconds or minutes.

[0098] Optionally, in the step of hanging up or maintaining the video call based on the first voice conversation data, semantic understanding and keyword detection can be performed on the first voice conversation data to determine a target waiting time or a target waiting event; based on the target waiting time or the target waiting event, the target maintenance time of the video call is determined, and a maintenance mechanism is started; under the maintenance mechanism, the video call is maintained for the target maintenance time; if a change is detected in the category of the target call status combination within the target maintenance time, the maintenance mechanism is canceled; if no change is detected in the category of the target call status combination within the target maintenance time, the video call is hung up when the maintenance time is exceeded.

[0099] In an embodiment of the present invention, when one of the two parties in a call has no voice and no face, the conversation data within T time before this time point is obtained as historical voice conversation data, semantic understanding is performed on the historical voice conversation data, and keywords therein are identified. The keywords may include specific event keywords and time-sensitive words, such as: I'm going to pour water; I'm going to get something and will be back in about 3 minutes, etc.

[0100] When one of the two parties in a call has voice but no face, and the other party has no voice and no face, the conversation data within T time before this time point is obtained as historical voice conversation data and real-time voice conversation data, and semantic understanding is also performed on the historical voice conversation data and real-time voice conversation data, and keywords are identified, including specific event keywords and time-sensitive words.

[0101] When a specific time interval sensitive word is identified in the first voice conversation data, the time interval corresponding to the identified specific time interval sensitive word can be directly determined as the target waiting time, and the target waiting time can be determined as the target maintenance time, and the maintenance mechanism is turned on. The above maintenance mechanism is mainly used to wait for the caller who has left to come back. When the target maintenance time is exceeded and the category to which the target call status combination belongs is not detected to change, it means that the caller who has left has not come back, or the waiting caller has also left. It can be understood that the face and voice information of the caller who has left are not detected. At this time, the server platform can automatically disconnect the video call link; if within the target maintenance time, the microphone or camera of the smart terminal without voice and face detects the voice or face image data of the corresponding person, the target maintenance time of this maintenance mechanism will be reset to zero, and the video call link will continue to be maintained.

[0102] Alternatively, when an event keyword corresponding to a waiting event is identified in the first voice conversation data, the waiting event corresponding to the event keyword is determined as a target waiting event, each waiting event is pre-set with a corresponding waiting time, and the target waiting time of the target waiting event is determined as the target maintenance time, and the maintenance mechanism is turned on. Within the target maintenance time, if the microphone or camera of the smart terminal without voice or face detects the voice or face image data of the corresponding caller, the target maintenance time minute time mechanism is reset to zero, and the video call link continues to be maintained. When the target maintenance time is exceeded and the face and voice information of the corresponding caller are not detected, the server platform automatically disconnects the video call link.

[0103] In a possible embodiment, time interval sensitive words may be identified first, and when no specific time interval sensitive words are identified, keywords for identifying specific event scenes may be detected.

[0104] By performing semantic understanding and keyword detection on the first voice conversation data, we can more accurately understand the call intentions of the two parties and determine whether there is a default agreement between the two parties. If so, the call will not be hung up. If not or the agreement has timed out, the call will be automatically hung up.

[0105] Optionally, in the step of determining the target maintenance duration of the video call based on the target waiting duration or the target waiting event, and starting the maintenance mechanism, the moment when the target waiting time is identified can be determined as the starting time; the target waiting time is determined as the target maintenance duration of the video call; or, if the target waiting time is not identified, the moment when the target call state combination is detected (or the moment when the face of one of the call parties cannot be detected) is determined as the starting time; according to the correspondence between the waiting event and the maintenance duration, the target maintenance duration corresponding to the target waiting event is matched, and different waiting events correspond to different maintenance durations; based on the starting time and the target maintenance duration, the maintenance mechanism is started.

[0106] In an embodiment of the present invention, when a specific time interval sensitive word is recognized in the first voice conversation data, such as: I will be back in 3 minutes, the above-mentioned "3 minutes" is the target waiting time, 3+t1 is the target maintenance time, and the server platform maintains the video call link for 3+t1, which can be a time mechanism of minutes, starting from the time t1 when the specific time sensitive word is recognized.

[0107] When no specific time interval sensitive words are identified, the keyword detection and identification of the specific event scene continues. When the keyword of the specific event scene is identified in the first voice conversation data, such as: I am going to get takeout, that is, the preset event scene keyword in the database appears, the waiting event corresponding to the event scene keyword is determined as the target waiting event. The preset event scene keywords in the database correspond to the waiting events. The server platform actively queries the preset table of the database. The preset table records the correspondence between each waiting event and the waiting time, and formulates the target maintenance time t2 according to the waiting time corresponding to the target waiting event. The moment when the camera detects the target call state combination is determined as the starting moment, or the moment when the camera cannot detect the face of the corresponding caller side is used as the starting point to maintain the video call link with the target maintenance time t2.

[0108] The above preset table is shown in Table 2 below:

[0109] Table 2

[0110]

[0111]

[0112] By turning on the maintenance mechanism through the target waiting time or the target waiting event, you can wait for the agreed situation, so that the automatic hang-up can be more accurate and will not hang up the call appointment that has not timed out.

[0113] Optionally, in the step of hanging up or maintaining the video call based on the second voice conversation data, semantic features can be extracted from the second voice conversation data to obtain the conversation semantics of the two parties to the call; based on the conversation semantics, the conversation logic and conversation relevance between the two parties to the call are determined; if the conversation logic and conversation relevance do not meet the preset logic conditions and relevance conditions, the video call is hung up; if the conversation logic and conversation relevance meet the preset logic conditions and relevance conditions, the video call is maintained.

[0114] In an embodiment of the present invention, for the specific situation where both parties in the call have voice but no face, when the video call is in a connected state, the microphone and the camera collect information and send it through the network, through the video call server, which can be equipped with an AI big model / translation engine, and the AI ​​big model / translation engine is used to perform voice recognition and semantic understanding on the second voice conversation data of both parties in the video call.

[0115] Speech recognition: Extract features from the input second voice conversation data, then perform sequence modeling through a deep neural network, and finally output the corresponding text. This is usually done using models trained using recurrent neural networks (RNNs) or convolutional neural networks (CNNs).

[0116] Semantic understanding: Encode the text from speech recognition, then perform semantic modeling through a deep neural network, and finally output the semantic representation corresponding to the second voice dialogue data. Usually, models such as attention mechanisms or graph convolutional networks can be used.

[0117] Through semantic understanding, it determines whether the conversation between the two parties in the video call is logical and contextually relevant. When it is identified that there is no obvious logical relevance, the server will directly disconnect the video call link; otherwise, the video call link will continue to be maintained.

[0118] In a possible embodiment, in the specific case that both parties of the call have voice but no face, the shooting direction of the smart terminal can be adjusted, such as the front and rear camera, to further confirm whether both parties of the call have no face. Of course, for the call party, if there is no face, the shooting direction of the smart terminal can also be adjusted to further determine whether there is really no face in front of the smart terminal, thereby improving the accuracy of the visual call status.

[0119] like Figure 2 As shown, an embodiment of the present invention provides a call processing device, the call processing device comprising:

[0120] An acquisition module 201 is used to acquire image data and voice data of two parties during a video call between the two parties;

[0121] A first processing module 202, configured to determine a visual call state and a voice call state of the two parties of the call based on the image data and voice data of the two parties of the call;

[0122] The second processing module 203 is used to hang up or maintain the video call based on the visual call status and voice call status of the two parties of the call.

[0123] Optionally, the first processing module 202 is also used to determine whether the visual call status of the two parties of the call is a face state or a non-face state based on the image data of the two parties of the call; and to determine whether the voice call status of the two parties of the call is a voice state or a non-voice state based on the voice data of the two parties of the call.

[0124] Optionally, the second processing module 203 is also used to determine the target call state combination of the two call parties based on the visual call state and voice call state of the two call parties; if the target call state combination belongs to the preset first category state combination, the first voice conversation data of the two call parties are obtained, the first category state combination is a state combination when the visual call state of at least one call party is a faceless state, and the voice call state is a voiceless state, and the first voice conversation data includes the historical voice conversation data of the two call parties within a preset time; based on the first voice conversation data, the video call is hung up or maintained; if the target call state combination belongs to the preset second category state combination, the first voice conversation data is obtained. The second voice conversation data of the two parties of the call, the second type of state combination is a state combination when the visual call states of the two parties of the call are both in a faceless state, and the voice call states of the two parties of the call are both in a voice state, then the second voice conversation data of the two parties of the call are obtained, and the second voice conversation data includes the real-time voice conversation data of the two parties of the call; based on the second voice conversation data, the video call is hung up or maintained; if the target call state combination belongs to a preset third type of state combination, the video call is maintained, and the second type of state combination is a state combination of the visual call state and the voice call state of the two parties of the call except the first type of state combination and the second type of state combination.

[0125] Optionally, the first category state combination includes a first subcategory state combination and a second subcategory state combination, the first subcategory state combination being a state combination when the visual call states of both parties to the call are both faceless states, and the voice call state of one and only one party to the call is in a voice state, and the second subcategory state combination is other state combinations in the first category state combination except the first subcategory state combination, and the second processing module 203 is also used to obtain historical voice conversation data of the two parties to the call within a preset time and real-time voice conversation data of the two parties to the call as first voice conversation data if the target call state combination belongs to the first subcategory state combination; if the target call state combination belongs to the second subcategory state combination, obtain historical voice conversation data of the two parties to the call within a preset time as first voice conversation data.

[0126] Optionally, the second processing module 203 is also used to perform semantic understanding and keyword detection on the first voice conversation data to determine a target waiting time or a target waiting event; based on the target waiting time or the target waiting event, determine the target maintenance time of the video call and start a maintenance mechanism; under the maintenance mechanism, maintain the video call for the target maintenance time; if a change is detected in the category of the target call status combination within the target maintenance time, cancel the maintenance mechanism; if no change is detected in the category of the target call status combination within the target maintenance time, hang up the video call when the maintenance time is exceeded.

[0127] Optionally, the second processing module 203 is also used to determine the moment when the target waiting time is identified as the starting moment; determine the target waiting time as the target maintenance time of the video call; or, if the target waiting time is not identified, determine the moment when the target call state combination is detected as the starting moment; according to the correspondence between the waiting event and the maintenance time, match the target maintenance time corresponding to the target waiting event, different waiting events correspond to different maintenance times; based on the starting moment and the target maintenance time, start the maintenance mechanism.

[0128] Optionally, the second processing module 203 is also used to extract semantic features of the second voice conversation data to obtain the conversation semantics of the two parties of the call; based on the conversation semantics, determine the conversation logic and conversation relevance between the two parties of the call; if the conversation logic and the conversation relevance do not meet the preset logic conditions and relevance conditions, hang up the video call; if the conversation logic and the conversation relevance meet the preset logic conditions and relevance conditions, maintain the video call.

[0129] It should be noted that the call processing device provided in the embodiment of the present invention can be applied to devices such as smart phones, tablets, computers, robots, servers, etc. that can perform the call processing method.

[0130] The call processing device provided in the embodiment of the present invention can implement each process implemented by the call processing method in the above method embodiment and can achieve the same beneficial effect. To avoid repetition, it will not be described here.

[0131] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, it includes: a memory 302, a processor 301, and a computer program of a call processing method stored in the memory 302 and executable on the processor 301, wherein:

[0132] The processor 301 is used to call the computer program stored in the memory 302 and execute the following steps:

[0133] During a video call between two parties, obtaining image data and voice data of the two parties;

[0134] Determining the visual call status and the voice call status of the two parties of the call based on the image data and the voice data of the two parties of the call;

[0135] Based on the visual call status and voice call status of the two parties in the call, the video call is hung up or maintained.

[0136] Optionally, the step of determining the visual call status and the voice call status of the two parties of the call based on the image data and the voice data of the two parties of the call, performed by the processor 301, includes:

[0137] Based on the image data of the two parties of the call, determining the visual call status of the two parties of the call as a face state or a faceless state;

[0138] Based on the voice data of the two parties of the call, determine whether the voice call status of the two parties of the call is a voice state or a non-voice state.

[0139] Optionally, the step of hanging up or maintaining the video call based on the visual call status and the voice call status of the two parties of the call, performed by the processor 301, includes:

[0140] Determining a target call state combination of the two parties of the call based on the visual call state and the voice call state of the two parties of the call;

[0141] If the target call state combination belongs to a preset first type of state combination, first voice conversation data of the two call parties are obtained, the first type of state combination being a state combination when the visual call state of at least one call party is a faceless state and the voice call state is a voiceless state, and the first voice conversation data includes historical voice conversation data of the two call parties within a preset time;

[0142] Based on the first voice conversation data, hanging up or maintaining the video call;

[0143] If the target call state combination belongs to a preset second state combination, the second voice conversation data of the two call parties is obtained, and the second state combination is a state combination when the visual call states of the two call parties are both faceless and the voice call states of the two call parties are both voiced, then the second voice conversation data of the two call parties is obtained, and the second voice conversation data includes real-time voice conversation data of the two call parties;

[0144] Based on the second voice conversation data, hanging up or maintaining the video call;

[0145] If the target call state combination belongs to the preset third state combination, the video call is maintained. The second state combination is a state combination of the visual call state and the voice call state of the remaining two parties of the call except the first state combination and the second state combination.

[0146] Optionally, the first category state combination includes a first subcategory state combination and a second subcategory state combination, the first subcategory state combination being a state combination when the visual call states of both parties of the call are both in a faceless state, and the voice call state of one and only one party of the call is in a voice state, and the second subcategory state combination is other state combinations in the first category state combination except the first subcategory state combination, and the step of obtaining the first voice conversation data of the two parties of the call executed by the processor 301 if the target call state combination belongs to the preset first category state combination includes:

[0147] If the target call state combination belongs to the first sub-category state combination, obtaining historical voice conversation data of the two call parties within a preset time and real-time voice conversation data of the two call parties as first voice conversation data;

[0148] If the target call state combination belongs to the second sub-category state combination, historical voice conversation data of the two call parties within a preset time is obtained as the first voice conversation data.

[0149] Optionally, the step of hanging up or maintaining the video call based on the first voice conversation data, performed by the processor 301, includes:

[0150] Performing semantic understanding and keyword detection on the first voice dialogue data to determine a target waiting time or a target waiting event;

[0151] Based on the target waiting time or the target waiting event, determine the target maintenance time of the video call and start a maintenance mechanism;

[0152] Under the maintenance mechanism, maintaining the video call for the target maintenance duration;

[0153] If a change in the category to which the target call status combination belongs is detected within the target maintenance duration, the maintenance mechanism is cancelled;

[0154] If no change is detected in the category to which the target call status combination belongs within the target maintenance time, the video call is hung up when the maintenance time is exceeded.

[0155] Optionally, the step of determining the target maintenance duration of the video call based on the target waiting duration or the target waiting event and starting a maintenance mechanism, performed by the processor 301, includes:

[0156] Determine the moment when the target waiting time is identified as the starting moment;

[0157] Determining the target waiting time as the target maintaining time of the video call;

[0158] Alternatively, if the target waiting time is not identified, the time when the target call status combination is detected is determined as the starting time;

[0159] According to the correspondence between the waiting event and the maintenance duration, the target maintenance duration corresponding to the target waiting event is matched, and different waiting events correspond to different maintenance durations;

[0160] Based on the starting time and the target maintenance duration, a maintenance mechanism is started.

[0161] Optionally, the step of hanging up or maintaining the video call based on the second voice conversation data, performed by the processor 301, includes:

[0162] Extracting semantic features from the second voice conversation data to obtain the conversation semantics of the two parties of the conversation;

[0163] Based on the conversation semantics, determining the conversation logic and conversation relevance between the two parties of the call;

[0164] If the conversation logic and the conversation relevance do not meet the preset logic conditions and relevance conditions, the video call is hung up;

[0165] If the conversation logic and the conversation relevance meet the preset logic conditions and relevance conditions, the video call is maintained.

[0166] It should be noted that the electronic device provided in the embodiment of the present invention can be applied to devices such as smart phones, tablets, computers, robots, servers, etc. that can perform the call processing method.

[0167] The electronic device provided by the embodiment of the present invention can implement each process implemented by the call processing method in the above method embodiment and can achieve the same beneficial effect. To avoid repetition, it will not be described here.

[0168] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the call processing method provided by the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0169] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The computer-readable storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0170] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A call processing method, characterized in that: The method comprises the following steps: During a video call between two parties, obtaining image data and voice data of the two parties; Determining the visual call status and the voice call status of the two parties of the call based on the image data and the voice data of the two parties of the call; Based on the visual call status and voice call status of the two parties in the call, the video call is hung up or maintained.

2. The call processing method according to claim 1, characterized in that: The step of determining the visual call status and the voice call status of the two parties of the call based on the image data and the voice data of the two parties of the call includes: Based on the image data of the two parties of the call, determining the visual call status of the two parties of the call as a face state or a faceless state; Based on the voice data of the two parties of the call, determine whether the voice call status of the two parties of the call is a voice state or a non-voice state.

3. The call processing method according to claim 2, characterized in that: The step of hanging up or maintaining the video call based on the visual call status and the voice call status of the two parties of the call includes: Determining a target call state combination of the two parties of the call based on the visual call state and the voice call state of the two parties of the call; If the target call state combination belongs to a preset first type of state combination, first voice conversation data of the two call parties are obtained, the first type of state combination being a state combination when the visual call state of at least one call party is a faceless state and the voice call state is a voiceless state, and the first voice conversation data includes historical voice conversation data of the two call parties within a preset time; Based on the first voice conversation data, hanging up or maintaining the video call; If the target call state combination belongs to a preset second state combination, the second voice conversation data of the two call parties is obtained, and the second state combination is a state combination when the visual call states of the two call parties are both faceless and the voice call states of the two call parties are both voiced, then the second voice conversation data of the two call parties is obtained, and the second voice conversation data includes real-time voice conversation data of the two call parties; Based on the second voice conversation data, hanging up or maintaining the video call; If the target call state combination belongs to the preset third state combination, the video call is maintained. The second state combination is a state combination of the visual call state and the voice call state of the remaining two parties of the call except the first state combination and the second state combination.

4. The call processing method according to claim 3, characterized in that: The first category state combination includes a first subcategory state combination and a second subcategory state combination, the first subcategory state combination is a state combination when the visual call states of both parties of the call are both in a faceless state, and the voice call state of only one party of the call is in a voice state, and the second subcategory state combination is other state combinations in the first category state combination except the first subcategory state combination, and if the target call state combination belongs to the preset first category state combination, the step of obtaining the first voice conversation data of the two parties of the call includes: If the target call state combination belongs to the first sub-category state combination, historical voice conversation data of the two call parties within a preset time and real-time voice conversation data of the two call parties are obtained as first voice conversation data; If the target call state combination belongs to the second sub-category state combination, historical voice conversation data of the two call parties within a preset time is obtained as the first voice conversation data.

5. The call processing method according to claim 4, characterized in that: The step of hanging up or maintaining the video call based on the first voice conversation data includes: Performing semantic understanding and keyword detection on the first voice dialogue data to determine a target waiting time or a target waiting event; Based on the target waiting time or the target waiting event, determine the target maintenance time of the video call and start a maintenance mechanism; Under the maintenance mechanism, maintaining the video call for the target maintenance duration; If a change in the category to which the target call status combination belongs is detected within the target maintenance duration, the maintenance mechanism is cancelled; If no change is detected in the category to which the target call status combination belongs within the target maintenance time, the video call is hung up when the maintenance time is exceeded.

6. The call processing method according to claim 5, characterized in that: The step of determining the target maintenance duration of the video call based on the target waiting duration or the target waiting event and starting the maintenance mechanism includes: Determine the moment when the target waiting time is identified as the starting moment; Determining the target waiting time as the target maintaining time of the video call; Alternatively, if the target waiting time is not identified, the time when the target call status combination is detected is determined as the starting time; According to the correspondence between the waiting event and the maintenance duration, the target maintenance duration corresponding to the target waiting event is matched, and different waiting events correspond to different maintenance durations; Based on the starting time and the target maintenance duration, a maintenance mechanism is started.

7. The call processing method according to any one of claims 3 to 6, characterized in that: The step of hanging up or maintaining the video call based on the second voice conversation data includes: Extracting semantic features from the second voice conversation data to obtain the conversation semantics of the two parties of the conversation; Based on the conversation semantics, determining the conversation logic and conversation relevance between the two parties of the call; If the conversation logic and the conversation relevance do not meet the preset logic conditions and relevance conditions, the video call is hung up; If the conversation logic and the conversation relevance meet the preset logic conditions and relevance conditions, the video call is maintained.

8. A call processing device, characterized in that: The call processing device comprises: An acquisition module, used to acquire image data and voice data of both parties during a video call; A first processing module, configured to determine a visual call state and a voice call state of the two parties of the call based on the image data and the voice data of the two parties of the call; The second processing module is used to hang up or maintain the video call based on the visual call status and voice call status of the two parties of the call.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the call processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the call processing method according to any one of claims 1 to 7 are implemented.

Images