Mobile phone video call processing method and system based on GPT

Abstract

The invention relates to the field of mobile phone video calls, in particular to a GPT-based mobile phone video call processing method and system. The position of the head of a tracked person is determined through a camera of a mobile phone, then audio signals received by a microphone array are decomposed, primary noise reduction is carried out on the decomposed component audio signals, then coordinates of a sound source of the component audio signals are calculated, and the sound source of the component audio signals is obtained. The method comprises the following steps: acquiring component audio signals, judging whether the component audio signals are sent by a tracked person or not by establishing a sound source ball, only outputting the component audio signals sent by the tracked person, and removing the component audio signals of which the sound source is not in the sound source ball, so as to filter the audio signals; the problem that in the prior art, in the video call process, it is difficult to guarantee the definition when the voice of a call person needing to conduct a video call is transmitted to a person at the other end is solved.

Description

Technical Field

[0001] The present invention relates to the field of mobile phone video calls, and in particular to a mobile phone video call processing method and system based on GPT. Background Art

[0002] As mobile phone functions become increasingly rich, mobile phone video call functions are becoming more and more widely used, greatly facilitating people's daily work, such as talking with relatives and friends through video calls or holding meetings with colleagues. In the process of making video calls outdoors, it is necessary to filter the external noise to ensure the clarity of the audio during the video call. The existing mainstream software algorithms include noise reduction filtering, spectral subtraction and adaptive filtering, which can filter the noise in the external environment to a certain extent.

[0003] However, when there are multiple voices with similar volumes at one end of a video call, the differences in sound information such as the frequency, sound intensity and volume of each voice are relatively small, making it difficult to filter out sounds other than those of the caller. Especially when other callers need to be added during the video call, it is even more difficult to separate the callers so that the person on the other end of the video call can hear them clearly. Therefore, the prior art is difficult to ensure the clarity of the voice of the caller who needs to make a video call when it is transmitted to the other end during the video call, thereby reducing the user experience. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a mobile phone video call processing method and system based on GPT, which solves the problem that the prior art is difficult to ensure the clarity of the voice of the person who needs to make a video call when it is transmitted to the person on the other end during a video call.

[0005] To achieve the above object, the present invention provides the following technical solutions: A method for processing a mobile phone video call based on GPT, the method comprising the following steps: S1. Set the person to be tracked and obtain the video data and audio signal containing the person to be tracked; S2, identifying the position of the head of the tracked person relative to the camera in real time according to the video data to obtain the head coordinates of the tracked person; S3, decomposing the audio signal to obtain component audio signals; S4, calculating the coordinates of the sound source position of the component audio signal according to the coordinates of the microphone and the signal time difference between any two microphones in the microphone array; S5. Establish a sound source sphere with the coordinates of the sound source position as the sphere center, and determine whether the coordinates of the head position of the tracked person are all located within the sound source sphere; If yes, the component audio signal is output and the corresponding text is generated based on GPT; If not, the component audio signal is deleted.

[0006] Preferably, in step S2, the following steps are specifically included: S21, obtaining the internal parameters of the camera, and identifying the head of the tracked person in real time according to the video data; S22. Calculate the coordinates of the center point of the head of the tracked person according to the pixel points of the head of the tracked person; the expression of the center point coordinates is: ; In the above formula, Indicates the center coordinates of the tracked person’s head. Indicates the number of pixels on the head of the tracked person. Represents the coordinates of the i-th pixel point on the head of the tracked person; S23. Calculate the head coordinates of the tracked person according to the camera internal parameters and the center point coordinates; the expression of the head coordinates is: ; In the above formula, and is the center point coordinate, that is, the pixel coordinate of the center point obtained through video data, , and is the spatial coordinate corresponding to the center point, is the scale factor, , and are the rotation parameters of the three coordinate axes, is the translation parameter, is the intrinsic parameter matrix of the camera.

[0007] Preferably, in step S3, the following steps are specifically included: S31, obtaining audio signals received by any two microphones in the microphone array; the expression of the audio signals received by any two microphones is: ; In the above formula, and Respectively represent the audio signals received by the i-th and j-th microphones, and Represent the audio coefficients of the i-th and j-th microphones, usually 1 / 2, is the audio signal of the sound source, and Respectively represent the time from the sound source to the two microphones, Represents the time difference between the audio signal from the sound source to the i-th and j-th microphones, and denote the Gaussian noise of the i-th and j-th microphones respectively; S32, fusing the audio signals received by the two microphones to obtain a first fusion signal; the expression of the first fusion signal is: ; In the above formula, represents the first fusion signal, and Respectively represent the audio signals received by the i-th and j-th microphones, and denote the first fusion weights of the i-th and j-th microphones respectively; S33, fusing the audio signals received by the two microphones through a blocking matrix to obtain a second fused signal; the expression of the second fused signal is: ; In the above formula, represents the second fusion signal, represents the blocking matrix, and Respectively represent the audio signals received by the i-th and j-th microphones; S34, constructing a fusion signal according to the first fusion signal and the second fusion signal, and obtaining a one-dimensional vector of the fusion signal; the expression of the one-dimensional vector of the fusion signal is: ; In the above formula, represents the one-dimensional vector of the fused signal, represents the first fusion signal, represents the second fusion signal; S35, performing de-averaging and whitening filtering on the one-dimensional vector in sequence to obtain a whitened signal; S36, constructing a filter function; the expression of the filter function is: ; In the above formula, represents the filter function, represents the covariance matrix of the speech signal in the whitened signal, Represents the covariance matrix of the noise signal; S37, filtering the whitened signal by a filter function to obtain a component audio signal after noise reduction; the expression of the component audio signal is: ; In the above formula, Represents component audio signal, is the filter function, is the unmixing matrix, is the speech signal in the whitened signal, is the noise signal in the whitened signal.

[0008] Preferably, in step S35, the following steps are specifically included: S351, de-average the one-dimensional vector to obtain a de-averaged vector; the expression of the de-averaged vector is: ; In the above formula, It means to find the mathematical expectation. represents the mean-removed vector, express dimensional mixing matrix, represents a one-dimensional vector; S352, performing whitening filtering on the de-meaned vector to obtain a whitened signal; the expression of the whitened signal is: ; In the above formula, represents the whitening signal, represents the whitening matrix, represents the mean-removed vector.

[0009] Preferably, in step S4, the following steps are specifically included: S41, obtaining a signal time difference of the audio signal obtained by any two microphones in the microphone array in a section of the audio signal to obtain a signal time difference sequence; the expression of the signal time difference sequence is: , represents the signal time difference between the i-th microphone and the j-th microphone at time T; S42, constructing a normal distribution model of the signal time difference corresponding to any two microphones according to the signal time difference sequence; the expression of the normal distribution model is: ; In the above formula, It means that at time T-1, the signal time difference between the i-th microphone and the j-th microphone is In the case of , the signal time difference between the i-th microphone and the j-th microphone at time T is The probability of represents the normal function, Indicates the time difference between the signals acquired by the i-th microphone and the j-th microphone. is the standard deviation of the signal time difference series; S43, calculating the probability of occurrence of any time difference at any time according to the normal distribution model; S44, calculating the self-correlation of the signal time difference between any two microphones, marking the two microphones as reference microphones, and calculating the mutual correlation between the signal time difference between any two microphones and the signal time difference between two reference microphones; S45, normalizing the mutual correlation to calculate the support of the signal time difference between any two microphones to the signal time difference between two reference microphones; the calculation formula of the support is: ; In the above formula, Indicates and The signal time difference between the i-th and j-th microphones is the support of the signal time difference between the i-th and j-th microphones as the reference microphones. Indicates and The self-correlation of the ith microphone and the cross-correlation between the ith and jth microphones as reference microphones, represents a microphone array; S46, updating the signal time difference between the i-th and j-th microphones according to the support of the signal time difference between any two other microphones to the signal time difference between the two reference microphones; the calculation formula of the updated signal time difference between the i-th and j-th microphones is: ; In the above formula, represents the updated signal time difference between the i-th and j-th microphones at time T, It means that at time T, and The signal time difference between the microphones is Indicates and The signal time difference between the i-th and j-th microphones is the support of the signal time difference between the i-th and j-th microphones as the reference microphones; S47, calculating the coordinates of the sound source position of the component audio signal according to the signal time difference between any two microphones and the coordinates of the microphones; the expression of the coordinates of the sound source position is: ; In the above formula, is the coordinate expression of the sound source position, , and are the coordinates of the 1st, 2nd and 3rd microphones in the microphone array, V is the speed of sound propagation in the air, and They respectively represent the signal time difference between the first microphone and the second microphone, and the signal time difference between the first microphone and the third microphone at time T.

[0010] Preferably, in step S44, the following steps are specifically included: S441, calculating the self-correlation of any two microphones according to the historical signal time difference, and marking the two microphones as reference microphones; the calculation formula of the self-correlation is: ; In the above formula, represents the self-correlation of the signal time difference between the i-th and j-th microphones as the reference microphone, represents the current signal time difference between the i-th and j-th microphones as reference microphones, represents the average value of the signal time difference between the i-th and j-th microphones as reference microphones; S442: Calculate the mutual correlation between the signal time difference between any two microphones and two reference microphones based on the self-correlation between the two microphones.

[0011] Preferably, in step S5, the following steps are specifically included: S51, obtaining the head sizes of several persons, and obtaining the maximum distance between any two positions of the heads; S52, setting the sound source radius according to the maximum distance; S53. A sound source sphere is established according to the coordinates of the sound source position and the sound source radius. The coordinate expression in the sound source sphere is: ; In the above formula, is the coordinate expression of the sound source position, is the coordinate of any position in the sound source sphere, is the distortion coefficient; S54, determining whether the head position coordinates of the tracked person are all located within the sound source sphere; If yes, the component audio signal is output and the corresponding text is generated based on GPT; If not, the component audio signal is deleted.

[0012] Preferably, in step S43, the calculation formula of the occurrence probability is: ; In the above formula, It means that when there is a signal time difference between the i-th microphone and the j-th microphone in the microphone array at time T, the signal time difference is The probability of It means that at time T-1, the signal time difference between the i-th microphone and the j-th microphone is The probability of It means that at time T-1, the signal time difference between the i-th microphone and the j-th microphone is In the case of , the signal time difference between the i-th microphone and the j-th microphone at time T is The probability of Represents a microphone array.

[0013] Preferably, in step S442, the calculation formula of the mutual correlation is: ; In the above formula, Indicates and The self-correlation of the ith microphone and the cross-correlation between the ith and jth microphones as reference microphones, represents the autocorrelation of the signal time difference between the i-th and j-th microphones, Indicates and The self-correlation of the signal time differences between the microphones, represents the current signal time difference between the i-th and j-th microphones, Indicates and The current signal time difference between the microphones.

[0014] The technical solution also provides a processing system for implementing the audio and video processing method for video calls, including a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the audio and video processing method for video calls is implemented.

[0015] Compared with the prior art, the present invention provides a mobile phone video call processing method and system based on GPT, which has the following beneficial effects: 1. The present invention first determines the position of the head of the tracked person through the camera of the mobile phone, then decomposes the audio signal received by the microphone array, and performs initial noise reduction on the decomposed component audio signals. Subsequently, the coordinates of the sound source of the component audio signal are calculated, and a sound source sphere is established to determine whether the component audio signal is emitted by the tracked person, and only the component audio signal emitted by the tracked person is output, thereby filtering the audio signal.

[0016] 2. The present invention realizes the recognition and positioning of the head of the tracked person by using the video data acquired in real time by the camera, thereby being able to obtain the determination of the head coordinates of the tracked person, so as to facilitate the subsequent confirmation of whether each audio component signal is emitted by the tracked person.

[0017] 3. In order to ensure the accuracy of the signal time difference between two selected reference microphones and avoid errors caused by external noise interference or internal circuit interference, the signal time difference between the reference microphones is corrected according to the acquired historical signal time difference data to ensure the accuracy of the signal time difference and the accuracy of the sound source position calculation.

[0018] 4. The present invention obtains a complete audio signal by fusing the audio signals acquired by two microphones, and performs de-averaging and whitening filtering to obtain a whitened signal to obtain a component audio signal after noise reduction, thereby obtaining the main component of the audio signal.

[0019] 5. The present invention can determine whether the head position coordinates of the tracked person are all located within the sound source sphere based on the maximum distance between any two positions of the head, the position coordinates of the head and the sound source position, and determine whether the corresponding component audio signals need to be output. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 The present invention is a flowchart of the audio and video processing method of the video call. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0022] Those skilled in the art can understand that all or part of the steps in the following embodiments can be completed by instructing the relevant hardware through a program, so the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0023] In order to solve the problem that it is difficult to ensure the clarity of the voice of the caller who needs to make a video call when it is transmitted to the other party during the video call in the prior art, so as to improve the user experience, the present invention provides a mobile phone video call processing method based on GPT, which locates the source of the collected audio information in real time, saves the audio information corresponding to the position of the caller, and deletes the remaining audio information, thereby filtering the audio information to remove environmental noise and other voices. The method includes the following steps: S1. Set the person to be tracked, and obtain the video data and audio signal containing the person to be tracked; the person to be tracked refers to the person participating in the video call, and the person to be called can manually select the corresponding person in the mobile phone video screen through the user.

[0024] S2. According to the video data, the position of the head of the tracked person relative to the camera is identified in real time to obtain the head coordinates of the tracked person. For the camera of the video call, in addition to realizing the real-time acquisition of the image, the position of the tracked person can also be realized through the camera. The following is a detailed introduction of a method for calculating the coordinates of the center position of the head of the tracked person through the camera to realize the position estimation of the source of the sound of the caller. In step S2, the following steps are specifically included; S21, obtaining parameters in the camera, and identifying the head of the tracked person in real time according to the video data; the head of the tracked person is obtained by a neural network model such as a YOLO model, etc.; S22. Calculate the coordinates of the center point of the head of the tracked person according to the pixel points of the head of the tracked person. Since the head of the tracked person is a relatively large area, in order to estimate the sound source, the center point of the head of the tracked person is used instead of the sound point of the tracked person. The expression of the center point coordinates is: ; In the above formula, Indicates the center coordinates of the tracked person’s head. Indicates the number of pixels on the head of the tracked person. Represents the coordinates of the i-th pixel point on the head of the tracked person; S23. Calculate the head coordinates of the tracked person according to the camera internal parameters and the center point coordinates; the expression of the head coordinates is: ; In the above formula, and is the center point coordinate, that is, the pixel coordinate of the center point obtained through video data, , and is the spatial coordinate corresponding to the center point, is the scale factor, , and are the rotation parameters of the three coordinate axes, is the translation parameter, is the intrinsic parameter matrix of the camera.

[0025] S3, decomposing the audio signal to obtain component audio signals; the audio signal acquired by the microphone array in real time contains the voice signal of the tracked person and may also contain a large amount of external noise signals, and the sound sources of various noises are different, some are far away and some are near, so it is necessary to separate the component audio signals containing various noises and the voice of the tracked person, and calculate the sound source coordinates of each audio signal, so as to filter the component audio signals that do not correspond to the head coordinates of the tracked person. In step S3, the following steps are specifically included: S31, obtaining audio signals received by any two microphones in the microphone array; the expression of the audio signals received by any two microphones is: ; In the above formula, and Respectively represent the audio signals received by the i-th and j-th microphones, and Represent the audio coefficients of the i-th and j-th microphones, usually 1 / 2, is the audio signal of the sound source, and Respectively represent the time from the sound source to the two microphones, Represents the time difference between the audio signal from the sound source to the i-th and j-th microphones, and denote the Gaussian noise of the i-th and j-th microphones respectively; S32, fusing the audio signals received by the two microphones to obtain a first fusion signal; the expression of the first fusion signal is: ; In the above formula, represents the first fusion signal, and Respectively represent the audio signals received by the i-th and j-th microphones, and denote the first fusion weights of the i-th and j-th microphones respectively; S33, fusing the audio signals received by the two microphones through a blocking matrix to obtain a second fused signal; the expression of the second fused signal is: ; In the above formula, represents the second fusion signal, represents the blocking matrix, and Respectively represent the audio signals received by the i-th and j-th microphones; S34, constructing a fusion signal according to the first fusion signal and the second fusion signal, and obtaining a one-dimensional vector of the fusion signal; the expression of the one-dimensional vector of the fusion signal is: ; In the above formula, represents the one-dimensional vector of the fused signal, represents the first fusion signal, represents the second fusion signal; S35, the one-dimensional vector is de-meaned and whitened in turn to obtain a whitened signal; the noise of the obtained one-dimensional vector needs to be filtered, and the real-time filtering of the noise signal is first implemented in the following manner, the amount of calculation is relatively small, and it is convenient to implement further filtering and screening in the subsequent step S4. Therefore, in step S35, the following steps are specifically included: S351, de-average the one-dimensional vector to obtain a de-averaged vector; the expression of the de-averaged vector is: ; In the above formula, It means to find the mathematical expectation. represents the mean-removed vector, express dimensional mixing matrix, represents a one-dimensional vector; S352, performing whitening filtering on the de-meaned vector to obtain a whitened signal; the expression of the whitened signal is: ; In the above formula, represents the whitening signal, represents the whitening matrix, represents the mean-removed vector; S36, constructing a filter function; the expression of the filter function is: ; In the above formula, represents the filter function, represents the covariance matrix of the speech signal in the whitened signal, Represents the covariance matrix of the noise signal; S37, filtering the whitened signal by a filter function to obtain a component audio signal after noise reduction; the expression of the component audio signal is: ; In the above formula, Represents component audio signal, is the filter function, is the unmixing matrix, is the speech signal in the whitened signal, is the noise signal in the whitened signal; the component audio signal is the sound signal after real-time preliminary noise reduction, and further judgment is needed to determine whether filtering is required.

[0026] S4, according to the coordinates of the microphone and the signal time difference between any two microphones in the microphone array, calculate the coordinates of the sound source position of the component audio signal; for the microphone receiving the audio signal, a large amount of external interference or abnormal structure of the microphone itself may lead to signal reception errors, resulting in the sound emitted by the same sound source, when being received by the same two microphones, the time difference of the signals received between the two may also be different, so it is necessary to correct the signal time difference between the two microphones to improve its credibility. In step S4, the following steps are specifically included: S41, obtaining a signal time difference of the audio signal obtained by any two microphones in the microphone array in a section of the audio signal to obtain a signal time difference sequence; the expression of the signal time difference sequence is: , represents the signal time difference between the i-th microphone and the j-th microphone at time T, that is, the time difference between the same component audio signal being received by the i-th microphone and the j-th microphone respectively; S42, constructing a normal distribution model of the signal time difference corresponding to any two microphones according to the signal time difference sequence; the expression of the normal distribution model is: ; In the above formula, It means that at time T-1, the signal time difference between the i-th microphone and the j-th microphone is In the case of , the signal time difference between the i-th microphone and the j-th microphone at time T is The probability of represents the normal function, Indicates the time difference between the signals acquired by the i-th microphone and the j-th microphone. is the standard deviation of the signal time difference series; S43. Calculate the probability of any time difference occurring at any time according to the normal distribution model; the calculation formula for the probability of occurrence is: ; In the above formula, It means that when there is a signal time difference between the i-th microphone and the j-th microphone in the microphone array at time T, the signal time difference is The probability of It means that at time T-1, the signal time difference between the i-th microphone and the j-th microphone is The probability of It means that at time T-1, the signal time difference between the i-th microphone and the j-th microphone is In the case of , the signal time difference between the i-th microphone and the j-th microphone at time T is The probability of represents a microphone array; S44, calculating the self-correlation of the signal time difference between any two microphones, marking the two microphones as reference microphones, and calculating the mutual correlation between the signal time difference between any two microphones and the signal time difference between two reference microphones; since the microphones themselves have different stabilities in receiving audio signals, it is necessary to select a relatively stable signal time difference from a large amount of signal time difference data, so as to assign weights to the corresponding two microphones, so as to more accurately calculate the signal time difference between the two microphones to be calculated. In step S44, the following steps are specifically included: S441. Calculate the self-correlation of any two microphones based on their historical signal time differences, and mark the two microphones as reference microphones; the self-correlation is used to reflect the stability of the two microphones when receiving the sound from the same sound source. If the value of the signal time difference fluctuates greatly over time, it means that the credibility of the signal time difference obtained from the two microphones is relatively low. Of course, the historical time based on the historical signal time difference needs to be relatively short to ensure the consistency of the relative position between the sound source and the two microphones. For example, for the signal time difference data within 1S, the calculation formula for the self-correlation is: ; In the above formula, represents the self-correlation of the signal time difference between the i-th and j-th microphones as the reference microphone, represents the current signal time difference between the i-th and j-th microphones as reference microphones, represents the average value of the signal time difference between the i-th and j-th microphones as reference microphones; S442, calculating the mutual correlation between the signal time difference between any two microphones and the two reference microphones according to the self-correlation between the two microphones; the calculation formula of the mutual correlation is: ; In the above formula, Indicates and The self-correlation of the ith microphone and the cross-correlation between the ith and jth microphones as reference microphones, represents the autocorrelation of the signal time difference between the i-th and j-th microphones, Indicates and The self-correlation of the signal time differences between the microphones, represents the current signal time difference between the i-th and j-th microphones, Indicates and The current signal time difference between the microphones; S45, normalizing the mutual correlation to calculate the support of the signal time difference between any two microphones to the signal time difference between two reference microphones; the calculation formula of the support is: ; In the above formula, Indicates and The signal time difference between the i-th and j-th microphones is the support of the signal time difference between the i-th and j-th microphones as the reference microphones. Indicates and The self-correlation of the ith microphone and the cross-correlation between the ith and jth microphones as reference microphones, represents a microphone array; S46, updating the signal time difference between the i-th and j-th microphones according to the support of the signal time difference between any two other microphones to the signal time difference between the two reference microphones; the calculation formula of the updated signal time difference between the i-th and j-th microphones is: ; In the above formula, represents the updated signal time difference between the i-th and j-th microphones at time T, It means that at time T, and The signal time difference between the microphones is Indicates and The signal time difference between the i-th and j-th microphones is the support of the signal time difference between the i-th and j-th microphones as the reference microphones; S47, calculating the coordinates of the sound source position of the component audio signal according to the signal time difference between any two microphones and the coordinates of the microphones; the expression of the coordinates of the sound source position is: ; In the above formula, is the coordinate expression of the sound source position, , and are the coordinates of the 1st, 2nd and 3rd microphones in the microphone array, V is the speed of sound propagation in the air, and They respectively represent the signal time difference between the first microphone and the second microphone, and the signal time difference between the first microphone and the third microphone at time T.

[0027] S5. Establish a sound source sphere with the coordinates of the sound source position as the sphere center, and determine whether the coordinates of the head position of the tracked person are all located within the sound source sphere; If yes, the component audio signal is output and the corresponding text is generated based on GPT; If not, the component audio signal is deleted.

[0028] Since the sound source of the external environment noise is often relatively far away from the head of the tracked person, a sound source ball is set here to set a larger range, so as to retain as many human voices as possible that are closer to the sound source position, so as to achieve the filtering function. In step S5, the following steps are specifically included: S51, obtaining head sizes of several persons, and obtaining the maximum distance between any two positions of the head; the head sizes can be obtained through big data or through images; S52, setting the sound source radius according to the maximum distance; S53. A sound source sphere is established according to the coordinates of the sound source position and the sound source radius. The coordinate expression in the sound source sphere is: ; In the above formula, is the coordinate expression of the sound source position, is the coordinate of any position in the sound source sphere, is the distortion coefficient, where It is usually greater than 1 and less than 3, and is set based on the width of the human head and shoulders. Here, only the x-axis and y-axis coordinates on the horizontal plane are calculated. The reason is that the height of a person's head may vary for various reasons, but its projection position on the ground remains unchanged, so the vertical position of the sound source can be determined based on this.

[0029] S54, determining whether the head position coordinates of the tracked person are all located within the sound source sphere; If yes, the component audio signal is output, and corresponding text is generated based on GPT. This indicates that the component audio signal is emitted by the person being tracked, so it is retained and output. Meanwhile, the corresponding text is generated based on GPT, which can eliminate the influence of some factors such as accents of people from different regions, and further provide a reference for the other end of the caller to understand the other party's speech. If not, the component audio signal is deleted.

[0030] When the present invention is implemented, the position of the head of the tracked person is first determined by the camera of the mobile phone, and then the audio signal received by the microphone array is decomposed, and the decomposed component audio signal is initially denoised. Subsequently, the coordinates of the sound source of the component audio signal are calculated, and a sound source sphere is established to determine whether the component audio signal is emitted by the tracked person, and only the component audio signal emitted by the tracked person is output, thereby filtering the audio signal.

[0031] The present invention also provides an audio and video processing system for video calls, the system comprising: A processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, an audio and video processing method for a video call is implemented.

[0032] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A mobile phone video call processing method based on GPT, characterized in that: The processing method comprises the following steps: S1. Set the person to be tracked and obtain the video data and audio signal containing the tracked person; S2, identifying the position of the head of the tracked person relative to the camera in real time according to the video data to obtain the head coordinates of the tracked person; S3, decomposing the audio signal to obtain component audio signals; S4, calculating the coordinates of the sound source position of the component audio signal according to the coordinates of the microphone and the signal time difference between any two microphones in the microphone array; S5. Establish a sound source sphere with the coordinates of the sound source position as the sphere center, and determine whether the coordinates of the head position of the tracked person are all located within the sound source sphere; If yes, the component audio signal is output and the corresponding text is generated based on GPT; If not, the component audio signal is deleted.

2. The processing method according to claim 1, characterized in that: In step S2, the following steps are specifically included: S21, obtaining the internal parameters of the camera, and identifying the head of the tracked person in real time according to the video data; S22. Calculate the coordinates of the center point of the head of the tracked person according to the pixel points of the head of the tracked person; the expression of the center point coordinates is: ； In the above formula, Indicates the center coordinates of the tracked person’s head. Indicates the number of pixels on the head of the tracked person. Represents the coordinates of the i-th pixel point on the head of the tracked person; S23. Calculate the head coordinates of the tracked person according to the camera internal parameters and the center point coordinates; the expression of the head coordinates is: ； In the above formula, and is the center point coordinate, that is, the pixel coordinate of the center point obtained through video data, , and is the spatial coordinate corresponding to the center point, is the scale factor, , and are the rotation parameters of the three coordinate axes, is the translation parameter, is the intrinsic parameter matrix of the camera.

3. The processing method according to claim 1, characterized in that: In step S3, the following steps are specifically included: S31, obtaining audio signals received by any two microphones in the microphone array; the expression of the audio signals received by any two microphones is: ； In the above formula, and Respectively represent the audio signals received by the i-th and j-th microphones, and denote the audio coefficients of the i-th and j-th microphones, respectively. is the audio signal of the sound source, and Respectively represent the time from the sound source to the two microphones, Represents the time difference between the audio signal from the sound source to the i-th and j-th microphones, and denote the Gaussian noise of the i-th and j-th microphones respectively; S32, fusing the audio signals received by the two microphones to obtain a first fusion signal; the expression of the first fusion signal is: ； In the above formula, represents the first fusion signal, and Respectively represent the audio signals received by the i-th and j-th microphones, and denote the first fusion weights of the i-th and j-th microphones respectively; S33, fusing the audio signals received by the two microphones through a blocking matrix to obtain a second fused signal; the expression of the second fused signal is: ； In the above formula, represents the second fusion signal, represents the blocking matrix, and Respectively represent the audio signals received by the i-th and j-th microphones; S34, constructing a fusion signal according to the first fusion signal and the second fusion signal, and obtaining a one-dimensional vector of the fusion signal; the expression of the one-dimensional vector of the fusion signal is: ； In the above formula, represents the one-dimensional vector of the fused signal, represents the first fusion signal, represents the second fusion signal; S35, performing de-averaging and whitening filtering on the one-dimensional vector in sequence to obtain a whitened signal; S36, constructing a filter function; the expression of the filter function is: ； In the above formula, represents the filter function, represents the covariance matrix of the speech signal in the whitened signal, Represents the covariance matrix of the noise signal; S37, filtering the whitened signal by a filter function to obtain a component audio signal after noise reduction; the expression of the component audio signal is: ； In the above formula, Represents component audio signal, is the filter function, is the unmixing matrix, is the speech signal in the whitened signal, is the noise signal in the whitened signal.

4. The processing method according to claim 3, characterized in that: In step S35, the following steps are specifically included: S351, de-average the one-dimensional vector to obtain a de-averaged vector; the expression of the de-averaged vector is: ； In the above formula, It means to find the mathematical expectation. represents the mean-removed vector, express dimensional mixing matrix, represents a one-dimensional vector; S352, performing whitening filtering on the de-meaned vector to obtain a whitened signal; the expression of the whitened signal is: ； In the above formula, represents the whitening signal, represents the whitening matrix, represents the mean-removed vector.

5. The processing method according to claim 1, characterized in that: In step S4, the following steps are specifically included: S41, obtaining a signal time difference of the audio signal obtained by any two microphones in the microphone array in a section of the audio signal to obtain a signal time difference sequence; the expression of the signal time difference sequence is: , represents the signal time difference between the i-th microphone and the j-th microphone at time T; S42, constructing a normal distribution model of the signal time difference corresponding to any two microphones according to the signal time difference sequence; the expression of the normal distribution model is: ； In the above formula, It means that at time T-1, the signal time difference between the i-th microphone and the j-th microphone is In the case of , the signal time difference between the i-th microphone and the j-th microphone at time T is The probability of represents the normal function, Indicates the time difference between the signals acquired by the i-th microphone and the j-th microphone. is the standard deviation of the signal time difference series; S43, calculating the probability of occurrence of any time difference at any time according to the normal distribution model; S44, calculating the self-correlation of the signal time difference between any two microphones, marking the two microphones as reference microphones, and calculating the mutual correlation between the signal time difference between any two microphones and the signal time difference between two reference microphones; S45, normalizing the mutual correlation to calculate the support of the signal time difference between any two microphones to the signal time difference between two reference microphones; the calculation formula of the support is: ； In the above formula, Indicates and The signal time difference between the i-th and j-th microphones is the support of the signal time difference between the i-th and j-th microphones as the reference microphones. Indicates and The self-correlation of the ith microphone and the cross-correlation between the ith and jth microphones as reference microphones, represents a microphone array; S46, updating the signal time difference between the i-th and j-th microphones according to the support of the signal time difference between any two other microphones to the signal time difference between the two reference microphones; the calculation formula of the updated signal time difference between the i-th and j-th microphones is: ； In the above formula, represents the updated signal time difference between the i-th and j-th microphones at time T, It means that at time T, and The signal time difference between the microphones is Indicates and The signal time difference between the i-th and j-th microphones is the support of the signal time difference between the i-th and j-th microphones as the reference microphones; S47, calculating the coordinates of the sound source position of the component audio signal according to the signal time difference between any two microphones and the coordinates of the microphones; the expression of the coordinates of the sound source position is: ； In the above formula, is the coordinate expression of the sound source position, , and are the coordinates of the 1st, 2nd and 3rd microphones in the microphone array, V is the speed of sound propagation in the air, and They respectively represent the signal time difference between the first microphone and the second microphone, and the signal time difference between the first microphone and the third microphone at time T.

6. The processing method according to claim 5, characterized in that: In step S44, the following steps are specifically included: S441, calculating the self-correlation of any two microphones according to the historical signal time difference, and marking the two microphones as reference microphones; the calculation formula of the self-correlation is: ； In the above formula, represents the self-correlation of the signal time difference between the i-th and j-th microphones as the reference microphone, represents the current signal time difference between the i-th and j-th microphones as reference microphones, represents the average value of the signal time difference between the i-th and j-th microphones as reference microphones; S442: Calculate the mutual correlation between the signal time difference between any two microphones and two reference microphones based on the self-correlation between the two microphones.

7. The processing method according to claim 1, characterized in that: In step S5, the following steps are specifically included: S51, obtaining the head sizes of several persons, and obtaining the maximum distance between any two positions of the heads; S52, setting the sound source radius according to the maximum distance; S53. A sound source sphere is established according to the coordinates of the sound source position and the sound source radius. The coordinate expression in the sound source sphere is: ； In the above formula, is the coordinate expression of the sound source position, is the coordinate of any position in the sound source sphere, is the distortion coefficient; S54, determining whether the head position coordinates of the tracked person are all located within the sound source sphere; If yes, the component audio signal is output and the corresponding text is generated based on GPT; If not, the component audio signal is deleted.

8. The processing method according to claim 5, characterized in that: In step S43, the calculation formula of the occurrence probability is: ； In the above formula, It means that when there is a signal time difference between the i-th microphone and the j-th microphone in the microphone array at time T, the signal time difference is The probability of It means that at time T-1, the signal time difference between the i-th microphone and the j-th microphone is The probability of It means that at time T-1, the signal time difference between the i-th microphone and the j-th microphone is In the case of , the signal time difference between the i-th microphone and the j-th microphone at time T is The probability of Represents a microphone array.

9. The processing method according to claim 6, characterized in that: In step S442, the calculation formula of the mutual correlation is: ； In the above formula, Indicates and The self-correlation of the ith microphone and the cross-correlation between the ith and jth microphones as reference microphones, represents the autocorrelation of the signal time difference between the i-th and j-th microphones, Indicates and The self-correlation of the signal time differences between the microphones, represents the current signal time difference between the i-th and j-th microphones, Indicates and The current signal time difference between the microphones.

10. A processing system for implementing the processing method according to any one of claims 1 to 9, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the processing method according to any one of claims 1 to 9 is implemented.

Images