A method and system for processing audio and video data

Abstract

The present invention relates to the field of video compression technology, and in particular to an audio and video data processing method and system, including: collecting live video of a person, obtaining key points of each frame of live image, calculating the matching degree of each two key points in two adjacent frames of live image, obtaining the matching key points of each key point in each frame of live image in the adjacent frame of live image according to the matching degree, calculating the difference between each key point of each frame of live image and the matching key points in the adjacent frame of live image, calculating the stability between two adjacent frames of live image, obtaining the key frames of the live video of the person according to the stability, and compressing the live video of the person according to the key frames of the live video of the person through the H.264 video compression algorithm. The present invention adaptively selects key frames, effectively retains the action details of the person, and improves the quality of the reconstructed image when the compression result of the live video of the person is decompressed according to the H.264 video compression algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of video compression, and in particular to an audio and video data processing method and system. Background Art

[0002] With the popularity of live broadcast technology, more and more people now choose to share their lives, knowledge and skills through live broadcast platforms. In this process, the transmission and processing efficiency of video data is one of the key factors in the live broadcast experience. As an efficient video encoding standard, the H.264 video compression algorithm is widely used in live broadcast, video conferencing, network streaming and other fields. It can compress video data into a smaller size to reduce the demand for storage space and transmission bandwidth. It has many advantages such as high compression efficiency, wide device support, and real-time transmission performance.

[0003] The existing H.264 video compression algorithm selects key frames based on the overall grayscale changes between two adjacent frames, and then compresses the video through the key frames. However, for videos in certain special scenarios, such as live streaming videos of people, since the character action details in the live streaming videos are relatively rich and there is rapid motion of the characters, selecting key frames based solely on the overall grayscale changes between two adjacent frames will result in inaccurate estimation of some action details and inability to effectively retain the character's action details, resulting in low quality of the reconstructed image when the compression result of the live streaming videos of people is decompressed according to the H.264 video compression algorithm. Summary of the invention

[0004] The present invention provides an audio and video data processing method and system to solve the existing problems.

[0005] The present invention provides an audio and video data processing method, the method comprising the following steps:

[0006] Collecting live video of people including several frames of live images;

[0007] Perform key point detection on each frame of live image to obtain the key points of each frame of live image;

[0008] Calculate the matching degree of each two key points in two adjacent frames of live images; obtain the matching key points of each key point in each frame of live images in the adjacent frame of live images according to the matching degree;

[0009] According to the grayscale change of the pixels in the neighborhood of each key point, the difference between each key point of each live image frame and the matching key points in the adjacent live images is calculated;

[0010] Obtain the attention heat map of each live broadcast image frame, and calculate the stability between two adjacent live broadcast images based on the attention weights of the pixels in the neighborhood of each key point;

[0011] According to the stability between two adjacent frames of live images, the key frame of the character live video is obtained; according to the key frame of the character live video, the character live video is compressed by using the H.264 video compression algorithm.

[0012] Furthermore, the calculation of the matching degree of every two key points in two adjacent frames of live broadcast images includes the following specific steps:

[0013] Obtain the serial number of each key point in the live image sorted according to the position in the live image; for two key points to be matched in the live images of two adjacent frames, use the vectors of the key points to be matched pointing to other key points in the corresponding live image as the vectors to be processed; the vectors to be processed corresponding to the two key points to be matched form a vector combination; obtain the matching degree according to the similarity between the vectors to be processed in the vector combination; the serial numbers of other key points corresponding to the end points of the two vectors to be processed in the vector combination are the same.

[0014] Furthermore, the step of obtaining the matching key points of each key point in each frame of live image in the adjacent frame of live image according to the matching degree includes the following specific steps:

[0015] Calculate the i-th key point x of the k-th frame live image k,i The degree of matching with all key points of the live image of the k+1th frame is to match all key points of the live image of the k+1th frame with the key point x k,i The key point with the largest matching degree is recorded as key point x k,i Match key points in live images in adjacent frames.

[0016] Furthermore, the specific calculation formula for calculating the difference between each key point of each frame of live image and the matching key point in the adjacent frame of live image is as follows:

[0017] According to the gray value difference between the pixel points of the key point and the corresponding matching key point within a preset neighborhood range, the neighborhood grayscale difference is obtained; according to the coordinates of the key point and the corresponding matching key point in the image, the position distance is obtained; according to the position distance and the neighborhood grayscale difference, the difference is obtained.

[0018] Furthermore, the calculation of the stability between two adjacent live image frames includes the following specific steps:

[0019] Each key point is fused with the average of the attention weights of all pixels of the corresponding matching key point in a preset neighborhood range to obtain the fused attention weight of each key point. According to the fused attention weight and the difference, the initial stability of each key point is obtained; the average initial stability of the key points in each live image is the stability of the corresponding live image.

[0020] Furthermore, the step of obtaining the key frame of the live video of the person includes the following specific steps:

[0021] The latter of the two adjacent α×M live image frames with the smallest stability is used as the key frame of the character live video, where α represents the preset key frame ratio, and M represents the number of all live image frames included in the character live video.

[0022] Furthermore, the method for obtaining the neighborhood range of each key point is specifically as follows:

[0023] An area centered on each key point of each frame of live image and having a size equal to w×w is used as the neighborhood range of each key point of each frame of live image, where w represents the preset side length.

[0024] Furthermore, the step of obtaining the attention heat map of each frame of the live broadcast image includes the following specific steps:

[0025] Using the Vision Transformer model, we can obtain the attention heat map of each live image frame, including the attention weight of each pixel in each live image frame.

[0026] Furthermore, the step of obtaining the key points of each frame of live broadcast image includes the following specific steps:

[0027] The openpose key point detection algorithm is used to perform key point detection on each frame of live image to obtain the key points of each frame of live image. The number of key points of each character in each frame of live image is N, where N represents the preset number of key points.

[0028] The present invention also proposes an audio and video data processing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of the audio and video data processing method when executing the computer program.

[0029] The beneficial effect of the technical solution of the present invention is: the present invention utilizes the key point matching results of each frame of live image, calculates the difference between each key point of each frame of live image and the matching key point in the adjacent frame of live image according to the grayscale change of pixel points in the neighborhood of each key point, and obtains the stability of two adjacent frames of live image according to the difference in attention weights of each key point in the attention heat map, which is used to measure the changes in the positions with higher attention in the two adjacent frames of live image, and then adaptively selects key frames, effectively retains the action details of the characters, and improves the quality of the reconstructed image when the compression result of the character live video is decompressed according to the H.264 video compression algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 The present invention is a flowchart of the steps of an audio and video data processing method. DETAILED DESCRIPTION

[0032] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the audio and video data processing method proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0033] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0034] A specific solution of an audio and video data processing method provided by the present invention is described in detail below with reference to the accompanying drawings.

[0035] Since the action details in live streaming videos of people are relatively rich and there are contents of rapid movement of people, selecting key frames only based on the overall grayscale changes of two adjacent frames of images will result in inaccurate estimation of some action details and inability to effectively retain the action details of the people, resulting in low quality of the reconstructed image when the compression results of the live streaming videos of people are decompressed according to the H.264 video compression algorithm.

[0036] Considering that when watching a live video of a person, more attention is paid to the person's movements and postures than the background, and in the live video of a person, the background area changes less, while the motion area, that is, the area corresponding to the person, changes more. Therefore, the key points of the key parts of the human body are used to analyze the changes in the person's movements, and it is determined whether the multiple frames corresponding to the changes in the details in two adjacent frames of live images need to be marked as key frames; however, due to the small difference in the change of the human body structure between two adjacent frames of live images, the key points of different adjacent frames of live images are matched according to the positional relationship between each key point in different frames of live images and the change of its details according to the grayscale change standard in its neighborhood, so as to achieve the purpose of highlighting the changes in some details; and the attention of different key points is different, and the parts with higher attention need to be better preserved or highlighted. The key points with higher attention weights should be given higher weights to measure the stability between two adjacent frames of live images, and then the key frames are adaptively selected to effectively retain the details of the person's movements and improve the quality of the reconstructed image when the compression result of the live video of the person is decompressed according to the H.264 video compression algorithm.

[0037] See also Figure 1 , which shows a flowchart of a method for processing audio and video data provided by an embodiment of the present invention, the method comprising the following steps:

[0038] S001, collecting live video of people including several frames of live images.

[0039] Specifically, a live video of a person is collected through a camera, and the live video of the person includes several frames of live images.

[0040] S002, perform key point detection on each frame of the live image, obtain the key points of each frame of the live image, calculate the matching degree of every two key points in two adjacent frames of the live image, obtain the matching key points of each key point in each frame of the live image in the adjacent frame of the live image according to the matching degree, calculate the difference between each key point of each frame of the live image and the matching key points in the adjacent frame of the live image according to the grayscale change of the pixel points in the neighborhood range of each key point, obtain the attention heat map of each frame of the live image, and calculate the stability between two adjacent frames of the live image according to the attention weight of the pixel points in the neighborhood range of each key point.

[0041] 1. Perform key point detection on each frame of live image to obtain the key points of each frame of live image.

[0042] It should be noted that when watching a live video, people mainly focus on the movements of the characters and do not pay attention to changes in the background. Therefore, this embodiment analyzes the movement changes of the characters by analyzing the key points of each character in each frame of the live image.

[0043] The preset number of key points N is used to obtain the key points of each character. The implementer can set the number of key points according to the actual implementation situation, for example, N=25.

[0044] Specifically, the openpose key point detection algorithm (human posture estimation algorithm) is used to perform key point detection on each frame of live image to obtain the key points of each frame of live image. The number of key points of each character in each frame of live image is N, and the number of all key points in each frame of live image is r×N, where r represents the number of characters in each frame of live image.

[0045] It should be noted that the openpose key point detection algorithm is a method of human posture estimation and key point detection based on deep learning technology, which is used to identify human movements, gestures, etc. The openpose key point detection algorithm is an existing technology and will not be described in detail here.

[0046] 2. Calculate the matching degree of every two key points in two adjacent frames of live images, and obtain the matching key points of each key point in each frame of live images in the adjacent frames of live images according to the matching degree.

[0047] It should be noted that, since the time interval between two adjacent live images is short, the position change of the same key point belonging to the same person in the two adjacent live images will not be too large. However, due to the complexity of the character's movements in the live video, the distortion or deformation of the body part may cause deviations in the positioning of the key points. For example, when explaining the product or changing the posture, the key points of the body will be blocked in some positions, resulting in the overlap of some key points. At this time, the positional relationship between different key points will be greatly affected, and the connection relationship between different human key points is not clear enough, which will affect the accuracy of the matching relationship between the key points in the two adjacent live images. However, between adjacent frames, there will be a certain correlation between the image data in the neighborhood of the corresponding key points. Therefore, it is necessary to calculate the matching degree of each two key points in the two adjacent live images according to the position of the key points in the two adjacent live images, and obtain the matching key points of each key point in each live image frame in the adjacent live image frame according to the matching degree.

[0048] Specifically, the degree of match between any two key points in two adjacent frames of live images is calculated, and the serial number of each key point in the live image is obtained according to the position sorting in the live image. It should be noted that the sorting method can be sorted according to preset rules, and it is only necessary to ensure that the sorting method in the live image is the same. In the embodiment of the present invention, the position relationship from top to bottom and from left to right is selected for sorting. For two key points to be matched in the live images of two adjacent frames, the vectors of the key points to be matched pointing to other key points in the corresponding live image are used as the vectors to be processed; the vectors to be processed corresponding to the two key points to be matched form a vector combination; the degree of match is obtained according to the similarity between the vectors to be processed in the vector combination; the serial numbers of other key points corresponding to the end points of the two vectors to be processed in the vector combination are the same.

[0049] Remember the i-th key point in the k-th frame and the tth key point closest to it The connecting vector between is the vector to be processed And for any t∈{1,2,…,N}, That is, as t increases, the distance between two key points also gradually increases, so that when analyzing the relationship between key points, the matching order between key points can be guaranteed to be fixed, so as to improve the accuracy of subsequent matching; when t traverses from 1 to N, the vector sequence is calculated respectively With vector sequence Its specific form is as follows:

[0050]

[0051] Considering that as the position of the characters in the live broadcast screen changes, the number of key frames in some frames will decrease or increase, so the above The length of the vector is n i , The length of the vector is n j , and n i ,n j ≤N; Direct use and To compare, even and It represents the same key point in two frames, and the matching degree between the two frames is also low. Therefore, the DTW matching method is used to match the above vector sequence, and the difference between the two vectors is the modulus length of the difference between the corresponding vectors, and the final matching result can be obtained.

[0052] In one embodiment of the present invention, the matching degree between the ith key point of the kth live image frame and the jth key point of the k+1th live image frame is calculated, and the specific calculation formula is:

[0053]

[0054] In the formula, M(x k,i ,x k+1,j ) represents the i-th key point x of the k-th live image k,i and the jth key point x of the live image of the k+1th frame k+1,j The degree of matching, Indicates the vector to be processed from the i-th key point of the k-th live image to the t-th key point of the k-th live image. It represents the vector to be processed from the jth key point of the k+1th frame of live image to the tth key point of the k+1th frame of live image, ‖‖ represents the modulus of the vector, r represents the number of characters in each frame of live image, N represents the number of preset key points, and r×N represents the number of all key points in each frame of live image.

[0055] The change of the corresponding vector sequence of the two key points corresponding to the same key point of the same person in two adjacent frames of live images will be small. In the two adjacent frames of live images, the difference in the continuous direction between each key point and other parts is small, and the length of the connection line does not change much. Therefore, the cosine of the corresponding vector angle is used as the weight coefficient to perform weighted averaging on the modulus of the vector difference to obtain the matching degree of any two key points in the two adjacent frames of live images, and the key points corresponding to the vector sequence with a small angle and similar modulus are given a higher matching degree. That is, in the embodiment of the present invention, cosine similarity and modulus difference are selected as the basis for calculating the similarity between vectors. In other embodiments of the present invention, other existing algorithms for measuring similarity such as Euclidean distance can also be used, which will not be limited or elaborated here.

[0056] Further, calculate the i-th key point x of the k-th live image k,i The degree of matching with all key points of the live image of the k+1th frame is to match all key points of the live image of the k+1th frame with the key point x k,i The key point with the largest matching degree is recorded as key point x k,i Match key points in live images in adjacent frames.

[0057] 3. According to the grayscale changes of the pixels in the neighborhood of each key point, the difference between each key point in each frame of the live image and the matching key points in the adjacent frames of the live image is calculated.

[0058] It should be noted that according to the matching key points of each key point in the adjacent frames of live broadcast images, the change information of the body structure of the person in the adjacent frames of live broadcast images can be obtained. However, when the overall structure of the human body changes little, there may be some changes in posture, such as gestures, head, elbow rotation, etc. during the live broadcast. At this time, the position change of the corresponding key node is not large, but there is more detailed change information. In order to retain this detailed information, the general outline of the moving area should be obtained in combination with the frame difference map, and then the neighborhood range of each key point should be obtained. By analyzing the grayscale changes of the pixels in the neighborhood of the two matching key points of the two adjacent frames of live broadcast images, the difference between the two matching key points of the two adjacent frames of live broadcast images can be obtained.

[0059] The preset side length w is used to obtain the neighborhood range of each key point. The implementer can set the side length according to the actual implementation situation, for example, w=9.

[0060] Specifically, the neighborhood grayscale difference is obtained according to the grayscale value difference between the key point and the corresponding matching key point in the preset neighborhood range; the position distance is obtained according to the coordinates of the key point and the corresponding matching key point in the image; the difference is obtained according to the position distance and the neighborhood grayscale difference. In one embodiment of the present invention, an area centered on each key point of each frame of live image and equal to w×w in size is used as the neighborhood range of each key point of each frame of live image; the difference between each key point of each frame of live image and the matching key point in the adjacent frame of live image is calculated, wherein the difference between the i-th key point of the k-th frame of live image and the matching key point in the adjacent frame of live image is calculated, and the specific calculation formula is:

[0061]

[0062] In the formula, y k+1,i Represents the i-th key point x of the k-th live image k,i Matching key points in the live images of adjacent frames, D(x k,i ,y k+1,i ) represents the i-th key point x of the k-th live image k,i and the matching key point y in the live image of the adjacent frame k+1,i The difference between A(x k,i ,y k+1,i ) represents the i-th key point x of the k-th live image k,i and the matching key point y in the live image of the adjacent frame k+1,i The neighborhood grayscale difference, h(f k,i,z ) represents the zth pixel point f in the neighborhood of the ith key point of the kth frame live image k,i,z The gray value, h(f k+1,i,z ) represents the i-th key point x of the k-th live imagek,i The zth pixel point f in the neighborhood of the matching key point in the live image of the adjacent frame k+1,i,z The gray value of || represents the absolute value, w represents the preset side length, and w×w represents the size of the neighborhood range of each key point in each frame of the live image, that is, the number of pixels in the neighborhood range of each key point in each frame of the live image. Indicates that the i-th key point of the k-th live image points to the i-th key point x of the k-th live image k,i The vector of the matching key points in the live images of adjacent frames, ‖‖ represents the modulus of the vector, and norm() represents the linear normalization function. That is, the formula uses The modulus length represents the position distance, and the neighborhood grayscale difference is obtained by the average value method.

[0063] In order to highlight the detail changes in the neighborhood of each key point, the difference in grayscale changes of pixels in the neighborhood of two matching key points of two adjacent frames of live images, that is, the difference in the mean grayscale values ​​of pixels in the neighborhood of the two matching key points, is used as the weight to weight the degree of position change of the two matching key points. For two matching key points with a smaller degree of position change, if the difference in grayscale changes of pixels in the neighborhood of the two matching key points is large, their position change should be amplified to highlight some detail change information. Otherwise, the position change of the corresponding key point is an approximate translation transformation, and the accuracy of its inference is higher.

[0064] 4. Obtain the attention heat map of each frame of the live image, and calculate the stability between two adjacent frames of the live image based on the attention weights of the pixels in the neighborhood of each key point.

[0065] It should be noted that the difference between the two matching key points of two adjacent frames of live images reflects the changes of each key point between the two adjacent frames of live images. For different motion situations, the location of the detail information that needs attention is different. For example, during the live broadcast, people pay more attention to the area where the host's facial expression or posture changes. Therefore, after obtaining the attention heat map of each frame of the live image, the average attention weight in the neighborhood of each key point is calculated, and then the difference between the two matching key points of the two adjacent frames of the live image is weighted and averaged, so that the details of some key points with higher attention levels are better preserved.

[0066] Specifically, the Vision Transformer model is used to obtain the attention heat map of each frame of the live image, including the attention weight of each pixel point of each frame of the live image.

[0067] It should be noted that there are two parameters that need to be set in the Vision Transformer model, namely the number of attention heads and the number of attention layers. The implementers can set the number of attention heads and the number of attention layers according to the actual implementation situation. For example, the number of attention heads is 8 and the number of attention layers is 12. The Vision Transformer model is an existing technology and will not be described in detail here.

[0068] Further, the stability between two adjacent frames of live images is calculated, and the average of the attention weights of all pixels of each key point and the corresponding matching key point in the preset neighborhood range is fused to obtain the fused attention weight of each key point. According to the fused attention weight and the difference, the initial stability of each key point is obtained; the average initial stability of the key point in each live image is the stability of the corresponding live image. In the embodiment of the present invention, the weight fusion method selects the average fusion, that is, the fused attention weight μ k,k+1 for The stability between the kth live image and the k+1th live image is calculated using the following formula:

[0069]

[0070] Where W k,k+1 represents the stability between the kth live image and the k+1th live image, N represents the number of preset key points, μ k,i Represents the i-th key point x of the k-th live image k,i The mean of the attention weights of all pixels in the neighborhood, μ k+1,i Represents the i-th key point x of the k-th live image k,i Matching key points y in the live images of adjacent frames k+1,i The mean of the attention weights of all pixels in the neighborhood of k,i ,y k+1,i ) represents the i-th key point x of the k-th live image k,i and the matching key point y in the live image of the adjacent frame k+1,i , norm() represents the linear normalization function, r represents the number of characters in each frame of live image, N represents the number of preset key points, and r×N represents the number of all key points in each frame of live image.

[0071] For two matching key points with high differences, the stability of the same key point corresponding to the same person is low. Therefore, the attention weight is used to weighted average the inverse of the difference between the two matching key points to obtain the stability of two adjacent frames of live broadcast images.

[0072] S003. Obtain key frames of the live video of the person based on the stability between two adjacent live image frames, and compress the live video of the person using an H.264 video compression algorithm based on the key frames of the live video of the person.

[0073] The key frame ratio α is preset to obtain all key frames. The implementer can set the key frame ratio according to the actual implementation situation, for example, α=15%.

[0074] Specifically, the stability between all two adjacent live image frames is calculated, and the latter live image frame among α×M adjacent live image frames with the smallest stability is used as the key frame of the character live video, where M represents the number of all live image frames included in the character live video.

[0075] Furthermore, the key frames of the live video of the characters are input into the H.264 video compression algorithm to obtain the compression result of the live video of the characters, and the live video data of the compression result of the live video of the characters is transmitted, which can effectively improve the transmission efficiency while ensuring the video quality.

[0076] The present invention also proposes an audio and video data processing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of the audio and video data processing method when executing the computer program.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for processing audio and video data, characterized in that: The method comprises the following steps: Collecting live video of people including several frames of live images; Perform key point detection on each frame of live image to obtain the key points of each frame of live image; Calculate the matching degree of each two key points in two adjacent frames of live images; obtain the matching key points of each key point in each frame of live images in the adjacent frame of live images according to the matching degree; According to the grayscale change of the pixels in the neighborhood of each key point, the difference between each key point of each live image frame and the matching key points in the adjacent live images is calculated; Obtain the attention heat map of each live broadcast image frame, and calculate the stability between two adjacent live broadcast images based on the attention weights of the pixels in the neighborhood of each key point; According to the stability between two adjacent frames of live images, a key frame of the live video of the person is obtained; and according to the key frame of the live video of the person, the live video of the person is compressed by using the H.264 video compression algorithm; The specific steps of calculating the stability between two adjacent live frames are as follows: Each key point is fused with the average of the attention weights of all pixels of the corresponding matching key point in a preset neighborhood range to obtain the fused attention weight of each key point. According to the fused attention weight and the difference, the initial stability of each key point is obtained; the average initial stability of the key points in each live image is the stability of the corresponding live image.

2. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of calculating the matching degree of each two key points in two adjacent frames of live broadcast images are as follows: Obtain the serial number of each key point in the live image sorted according to the position in the live image; for two key points to be matched in the live images of two adjacent frames, use the vectors of the key points to be matched pointing to other key points in the corresponding live image as the vectors to be processed; the vectors to be processed corresponding to the two key points to be matched form a vector combination; the matching degree is obtained according to the similarity between the vectors to be processed in the vector combination; the serial numbers of other key points corresponding to the end points of the two vectors to be processed in the vector combination are the same.

3. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of obtaining the matching key points of each key point in each frame of live image in the adjacent frame of live image according to the matching degree are as follows: Calculate the i-th key point x of the k-th frame live image k,i The degree of matching with all key points of the live image of the k+1th frame is to match all key points of the live image of the k+1th frame with the key point x k,i The key point with the largest matching degree is recorded as key point x k,i Match key points in live images in adjacent frames.

4. The method for processing audio and video data according to claim 1, characterized in that: The specific calculation formula for calculating the difference between each key point of each live image frame and the matching key point in the adjacent live image frame is as follows: According to the gray value difference between the pixel points of the key point and the corresponding matching key point within a preset neighborhood range, the neighborhood grayscale difference is obtained; according to the coordinates of the key point and the corresponding matching key point in the image, the position distance is obtained; according to the position distance and the neighborhood grayscale difference, the difference is obtained.

5. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of obtaining the key frame of the live video of the character are as follows: The latter of the two adjacent α×M live image frames with the smallest stability is used as the key frame of the character live video, where α represents the preset key frame ratio, and M represents the number of all live image frames included in the character live video.

6. The method for processing audio and video data according to claim 1, characterized in that: The method for obtaining the neighborhood range of each key point is specifically as follows: An area centered on each key point of each frame of live image and having a size equal to w×w is used as the neighborhood range of each key point of each frame of live image, where w represents the preset side length.

7. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of obtaining the attention heat map of each frame of the live broadcast image are as follows: Using the Vision Transformer model, we can obtain the attention heat map of each live image frame, including the attention weight of each pixel in each live image frame.

8. The method for processing audio and video data according to claim 1, characterized in that: The specific steps of obtaining the key points of each frame of live broadcast image are as follows: The openpose key point detection algorithm is used to perform key point detection on each frame of live image to obtain the key points of each frame of live image. The number of key points of each character in each frame of live image is N, where N represents the preset number of key points.

9. An audio and video data processing system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images