Video processing method, sending end, receiving end, storage medium and program product

Abstract

The application discloses a video processing method, a sending end, a receiving end, a storage medium and a program product, comprising: a video sending end acquiring a first video content type of a target video, determining first identification information and a first frame rate parameter according to the first video content type, taking the first identification information as target identification information and taking the first frame rate parameter as a target frame rate parameter; collecting the target video according to the target frame rate parameter, and encoding the target identification information and the target video to obtain a video code stream; and sending the video code stream to a video receiving end, so that the video receiving end decodes the video code stream to obtain a decoded video and the target identification information, and then displays the decoded video according to the target frame rate parameter corresponding to the target identification information. The application can determine the corresponding target frame rate parameter according to the first video content type of the target video to process the video, which not only can save bandwidth resources, but also can reduce playing lag and enhance subjective experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and particularly relates to a video processing method, a sending end, a receiving end, a storage medium and a program product. BACKGROUND

[0002] At present, in order to better integrate work efficiency and safety, remote work and home office gradually become a new normal. In order to enable enterprises to operate normally, many enterprises introduce video conference systems. The construction of the video conference system solves the communication problem between enterprises in different places to a certain extent, provides a convenient and fast communication way for branch offices distributed in different regions, reduces the travel cost of enterprises, so that people can realize face-to-face audio and video communication without going out. Among them, in the process of using the video conference system, in order to better communicate, it is often necessary to share some materials such as documents, videos, pictures and the like with other participants by using video conference auxiliary flow video.

[0003] However, the existing video conference often has problems such as waste of bandwidth resources or playing lag, for example: when sharing document content, it may take several seconds to turn a page once, the content changes slowly, and if a higher frame rate is used for playing at this time, it will cause waste of bandwidth resources; in addition, when sharing video content, if a lower frame rate is used for playing at this time, it will cause playing lag and poor subjective experience. SUMMARY

[0004] The embodiments of the present application provide a video processing method, a video sending end, a video receiving end, a storage medium and a program product, which can not only save bandwidth resources, but also reduce playing lag and enhance subjective experience.

[0005] In a first aspect, the embodiments of the present application provide a video processing method applied to a video sending end, the method comprising: acquiring a first video content type of a target video, determining corresponding first identification information and a first frame rate parameter according to the first video content type, and taking the first identification information as target identification information and taking the first frame rate parameter as a target frame rate parameter; collecting the target video according to the target frame rate parameter, and encoding the target identification information and the collected target video to obtain a video code stream; and sending the video code stream to a video receiving end, so that the video receiving end decodes the video code stream to obtain a decoded video and the target identification information, and then displays the decoded video according to the target frame rate parameter corresponding to the target identification information.

[0006] In a second aspect, the embodiments of the present application further provide a video processing method, applied to a video receiving end, the method comprising: receiving a video code stream from a video sending end; decoding the video code stream to obtain a decoded video and target identification information; determining a corresponding target frame rate parameter according to the target identification information, and displaying the decoded video according to the target frame rate parameter.

[0007] In a third aspect, the embodiments of the present application further provide a video sending end, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the video processing method of the first aspect.

[0008] In a fourth aspect, the embodiments of the present application further provide a video receiving end, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the video processing method of the second aspect.

[0009] In a fifth aspect, the embodiments of the present application further provide a computer readable storage medium, storing computer executable instructions, wherein the computer executable instructions are used to execute the video processing method as described above.

[0010] In a sixth aspect, the embodiments of the present application further provide a computer program product, comprising a computer program or computer instructions, wherein the computer program or the computer instructions are stored in a computer readable storage medium, a processor of a computer device reads the computer program or the computer instructions from the computer readable storage medium, and the processor executes the computer program or the computer instructions, so that the computer device executes the video processing method as described above.

[0011] In the embodiments of the present application, first, the video sending end obtains the first video content type of the target video, determines the corresponding first identification information and the first frame rate parameter according to the first video content type, and takes the first identification information as the target identification information and the first frame rate parameter as the target frame rate parameter. Then, the video sending end collects the target video according to the target frame rate parameter, encodes the target identification information and the collected target video, obtains a video code stream, and sends the video code stream to the video receiving end. Finally, the video receiving end decodes the video code stream to obtain a decoded video and the target identification information, and displays the decoded video according to the target frame rate parameter corresponding to the target identification information. According to the technical scheme of the embodiments of the present application, the corresponding target frame rate parameter can be determined according to the first video content type of the target video, different frame rate parameters are used for video processing for different video content types, not only can save bandwidth resources in the case of playing low frame rate required content such as sharing document content, but also can reduce the situation of playing lag in the case of playing high frame rate required content such as sharing video content, and enhance the subjective experience. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a structural schematic diagram of an implementation environment for executing a video processing method provided by an embodiment of the present application;

[0013] Figure 2 is a flowchart of a video processing method on the video sending end side provided by an embodiment of the present application;

[0014] Figure 3 is a flowchart of a video processing method on the video sending end side provided by another embodiment of the present application;

[0015] Figure 4 is a flowchart of a video processing method on the video sending end side provided by another embodiment of the present application;

[0016] Figure 5 is a flowchart of a video processing method on the video sending end side provided by another embodiment of the present application;

[0017] Figure 6 is a flowchart of a video processing method on the video sending end side provided by another embodiment of the present application;

[0018] Figure 7 is a flowchart of a video processing method on the video sending end side provided by another embodiment of the present application;

[0019] Figure 8 is a flowchart of a video processing method on the video sending end side provided by another embodiment of the present application;

[0020] Figure 9This is a flowchart of a video processing method on the video transmitting end side provided in another embodiment of this application;

[0021] Figure 10 This is a flowchart of a video processing method on the video transmitting end side provided in another embodiment of this application;

[0022] Figure 11 This is a flowchart of a video processing method on the video receiving end side according to an embodiment of this application;

[0023] Figure 12 This is a schematic diagram of the implementation environment for performing a video processing method according to an embodiment of this application;

[0024] Figure 13 This is an overall flowchart of a video processing method provided in one embodiment of this application;

[0025] Figure 14 This is an overall flowchart of a video processing method provided in another embodiment of this application;

[0026] Figure 15 This is a schematic diagram of the structure of a video transmitting end provided in one embodiment of this application;

[0027] Figure 16 This is a schematic diagram of the structure of a video receiver provided in one embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical methods, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0030] Currently, to better balance work efficiency and security, remote work and working from home are gradually becoming the new normal. To ensure normal business operations, many companies have introduced video conferencing systems. The construction of video conferencing systems has, to some extent, solved communication problems between companies in different locations, providing a convenient and fast communication method for branches located in different regions, reducing travel costs for businesses, and enabling people to conduct face-to-face audio and video communication without leaving their homes. In the process of using video conferencing systems, for better communication, it is often necessary to share materials such as documents, videos, and images with other participants using the video conferencing feed.

[0031] However, existing video conferencing systems often suffer from problems such as wasted bandwidth resources or playback stuttering. For example, when sharing document content, it may take several seconds to turn the page, and the content changes slowly. If a high frame rate is used to play the content at this time, it will lead to wasted bandwidth resources. In addition, when sharing video content, if a low frame rate is used to play the content at this time, it will cause playback stuttering and a poor subjective experience.

[0032] Based on this, embodiments of this application provide a video processing method, a video sending end, a video receiving end, a computer-readable storage medium, and a computer program product, which can not only save bandwidth resources but also reduce playback stuttering and enhance the subjective experience.

[0033] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0034] like Figure 1 As shown, Figure 1 This is a schematic diagram of the implementation environment for performing a video processing method according to an embodiment of this application.

[0035] exist Figure 1 In the example, the implementation environment includes, but is not limited to, a video transmitter 100 and a video receiver 200, wherein a communication connection exists between the video transmitter 100 and the video receiver 200.

[0036] In one embodiment, the relative position and number of video transmitter 100 and video receiver 200 can be set according to the specific application scenario. This application embodiment does not specifically limit the relative position and number of video transmitter 100 and video receiver 200.

[0037] Those skilled in the art will understand that the implementation environment for performing the video processing method can be applied to 3G communication network systems, LTE communication network systems, 5G communication network systems, 6G communication network systems, and subsequent evolved mobile / fixed communication network systems, etc., and the embodiments of this application do not specifically limit this.

[0038] It will be understood by those skilled in the art that Figure 1 The implementation environment shown does not constitute a limitation on the embodiments of this application, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0039] Based on the above implementation environment, various embodiments of the video processing method on the video sending end side of this application are proposed below.

[0040] like Figure 2 As shown, Figure 2 This is a flowchart of a video processing method on the video transmitting end side according to an embodiment of this application. This video processing method can be applied to... Figure 1 The video sending end in the process includes, but is not limited to, steps S100, S200 and S300.

[0041] Step S100: Obtain the first video content type of the target video, determine the corresponding first identification information and first frame rate parameter based on the first video content type, and use the first identification information as the target identification information and the first frame rate parameter as the target frame rate parameter.

[0042] Step S200: Acquire the target video according to the target frame rate parameter, and encode the target identification information and the acquired target video to obtain the video bitstream;

[0043] Step S300: Send the video stream to the video receiver so that the video receiver can decode the video stream to obtain the decoded video and target identification information, and then display the decoded video according to the target frame rate parameter corresponding to the target identification information.

[0044] Specifically, in the video processing, firstly, the video sending end obtains the first video content type of the target video, determines the corresponding first identification information and first frame rate parameter based on the first video content type, and uses the first identification information as the target identification information and the first frame rate parameter as the target frame rate parameter; then, the video sending end captures the target video according to the target frame rate parameter, encodes the target identification information and the captured target video to obtain a video stream, and sends the video stream to the video receiving end; finally, the video receiving end decodes the video stream to obtain the decoded video and target identification information, obtains the corresponding target frame rate parameter based on the target identification information, and then displays the decoded video according to the target frame rate parameter.

[0045] It should be noted that the target video mentioned above refers to the content to be displayed on the video sending end; in other words, it refers to the content that the video sending end needs to share with the video receiving end.

[0046] Additionally, it should be noted that the first video content type mentioned above can be a document type, a non-document type such as video footage, or other content types. This application embodiment does not specifically limit the type of the first video content type.

[0047] It is worth noting that the first video content type, the first identifier information, and the first frame rate parameter have a one-to-one correspondence. For example, when the first video content type is a document type, the first identifier information corresponds to the value 1, and the first frame rate parameter corresponds to f1; when the first video content type is a non-document type, the first identifier information corresponds to the value 0, and the first frame rate parameter corresponds to f2. Since the playback frame rate requirement for document types is lower than that for non-document types, f1 can be set to be less than f2.

[0048] It should be noted that the method for obtaining the first video content type of the target video in step S100 above can be as follows: the user can input the first video content type during initial sharing, and simultaneously input the first identifier information and the first frame rate parameter; alternatively, the first video content type can be determined through the motion vector parameters of the encoded frames during the video encoding process, the process of which is similar to the following. Figure 3 The method and steps are similar; alternatively, the first video content type can be determined by the image features of the video images in the target video, and the process is as follows. Figure 7 The method and steps are similar; other acquisition methods are also possible. This application embodiment does not specifically limit the acquisition method of the first video content type.

[0049] Additionally, it is worth noting that after obtaining the first video content type, the video sending end can determine the corresponding first identification information and first frame rate parameter by looking up a table locally; it can also send the first video content type to the cloud server and receive the first identification information and first frame rate parameter fed back by the cloud server; or it can use other acquisition methods. This application embodiment does not specifically limit the acquisition method of the first identification information and first frame rate parameter.

[0050] Additionally, it is worth noting that once the target frame rate parameter is determined as the first frame rate parameter, the embodiments of this application can adaptively adjust the video acquisition and encoding frame rate parameters based on the first frame rate parameter. Furthermore, the video sending end will encode the first identification information and the target video together and write them into a specific position in the bitstream, and then transmit them to the video receiving end for sharing via the network. The video receiving end decodes the received video bitstream and extracts the first identification information, determines the first frame rate parameter based on the first identification information, and adaptively adjusts the video display frame rate parameter based on the first frame rate parameter.

[0051] It is worth noting that the embodiments of this application can determine the corresponding target frame rate parameter according to the first video content type of the target video. Different frame rate parameters are used for video processing for different video content types. This not only saves bandwidth resources when playing content with low frame rate requirements, such as shared document content, but also reduces playback stuttering when playing content with high frame rate requirements, such as shared video content, thus enhancing the subjective experience.

[0052] In addition, such as Figure 3 As shown, Figure 3 This is a flowchart of a video processing method on the video sending end side provided in another embodiment of this application. After encoding the target identification information and the acquired target video in step S200 above, the video processing method of this application embodiment also includes, but is not limited to, steps S410, S420 and S430.

[0053] Step S410: Obtain the encoded frame during the encoding process and determine the motion vector parameters of the encoded frame;

[0054] Step S420: Determine the second video content type of the target video based on the motion vector parameters, and determine the corresponding second identification information and second frame rate parameters based on the second video content type;

[0055] Step S430: Update the target identification information to the second identification information and update the target frame rate parameter to the second frame rate parameter.

[0056] Specifically, in this application embodiment, the target identification information and target frame rate parameters are continuously updated. In one embodiment, during the encoding of the target identification information and target video, the video transmitting end determines the second video content type of the target video based on the motion vector parameters of the encoded frames in the encoding process. Then, it determines the corresponding second identification information and second frame rate parameters based on the second video content type, thereby updating the target identification information and target frame rate parameters respectively, so that the second identification information and second frame rate parameters are used in subsequent acquisition and encoding processes.

[0057] Additionally, it should be noted that the second video content type mentioned above, like the first video content type mentioned above, can be a document type, a non-document type such as video footage, or other content types. This application embodiment does not specifically limit the type of the second video content type.

[0058] It is worth noting that the second video content type, the second identification information, and the second frame rate parameter are in a one-to-one correspondence. For example, when the second video content type is a document type, the second identification information corresponds to the value 1, and the second frame rate parameter corresponds to f1; when the second video content type is a non-document type, the second identification information corresponds to the value 0, and the second frame rate parameter corresponds to f2. Since the playback frame rate requirement for document types is lower than that for non-document types, f1 can be set to be less than f2.

[0059] Additionally, it is worth noting that after obtaining the second video content type, the video sending end can determine the corresponding second identification information and second frame rate parameters by looking up a table locally; it can also send the second video content type to the cloud server and receive the second identification information and second frame rate parameters fed back by the cloud server; or it can use other acquisition methods. This application embodiment does not specifically limit the acquisition method of the second identification information and second frame rate parameters.

[0060] In addition, such as Figure 4 As shown, Figure 4 This is a flowchart of a video processing method on the video sending end side provided in another embodiment of this application. Regarding the determination of the second video content type of the target video based on motion vector parameters in step S420 above, it includes, but is not limited to, steps S510, S520 and S530.

[0061] Step S510: Divide the encoded frame into multiple block images and determine the total number of images in all block images;

[0062] Step S520: Calculate the difference in motion vector parameters between all two adjacent block images, and determine the number of motion blocks in the motion block image based on all the differences;

[0063] Step S530: Determine the second video content type of the target video based on the total number of images and the number of motion blocks.

[0064] Specifically, the process of determining the second video content type of the target video based on motion vector parameters is as follows: First, the video transmitter divides the encoded frame into multiple image blocks, and counts the total number of images in all image blocks. Next, for each pair of adjacent image blocks, the video transmitter calculates the difference in motion vector parameters between the two adjacent image blocks. Based on the magnitude of the difference, it determines whether the second image block in the pair of adjacent image blocks is a moving block image, and counts the number of moving blocks in the pair of adjacent image blocks in all groups. Finally, the second video content type of the target video is calculated based on the total number of images and the number of moving blocks.

[0065] In addition, such as Figure 5 As shown, Figure 5 This is a flowchart of a video processing method on the video transmitting end side provided in another embodiment of this application. Regarding the determination of the number of motion blocks in the motion block image in the segmented image based on all the differences in step S520 above, it includes, but is not limited to, steps S610 and S620.

[0066] Step S610: For every two adjacent block images, if the difference between the motion vector parameters of the previous block image and the motion vector parameters of the next block image is greater than a first preset threshold, the next block image is recorded as a motion block image.

[0067] Step S620: Count the number of motion blocks in all motion block images.

[0068] Specifically, for each pair of adjacent image blocks in a group, if the difference between the motion vector parameters of the preceding image block and the motion vector parameters of the following image block is greater than a first preset threshold, it indicates that the changes between the preceding and following images are significant, and the video transmitter will record the following image block as a moving image. Conversely, if the difference between the motion vector parameters of the preceding image block and the motion vector parameters of the following image block is less than or equal to the first preset threshold, it indicates that the changes between the preceding and following images are minor, and the video transmitter will not record the following image block as a moving image.

[0069] It is understood that the first preset threshold mentioned above can be pre-input by the user or determined according to preset rules based on the current video transmission situation. This application embodiment does not specifically limit the method of determining the first preset threshold.

[0070] In addition, such as Figure 6 As shown, Figure 6 This is a flowchart of a video processing method on the video sending end side provided in another embodiment of this application. Regarding the determination of the second video content type of the target video based on the total number of images and the number of motion blocks in the above step S530, it includes, but is not limited to, steps S710 and S720.

[0071] Step S710: Calculate the ratio of the number of moving blocks to the total number of images;

[0072] Step S720: Compare the ratio with the second preset threshold, and determine the second video content type of the target video based on the comparison result.

[0073] Specifically, after calculating the total number of images and the number of motion blocks, the video transmitter calculates the ratio between the number of motion blocks and the total number of images. If the ratio is greater than a second preset threshold, the second video content type of the target video can be determined to be a non-document type; if the ratio is less than or equal to the second preset threshold, the second video content type of the target video can be determined to be a document type.

[0074] It is understood that the aforementioned second preset threshold can be pre-input by the user or determined according to preset rules based on the current video transmission situation. This application embodiment does not specifically limit the method of determining the second preset threshold.

[0075] In addition, such as Figure 7 As shown, Figure 7 This is a flowchart of a video processing method on the video transmitting end side provided in another embodiment of this application. Regarding the determination of the second video content type of the target video based on the total number of images and the number of motion blocks in the above step S530, it includes, but is not limited to, steps S810 and S820.

[0076] Step S810: For each encoded frame, determine a video content type based on the total number of images and the number of motion blocks;

[0077] Step S820: Select the video content type with the largest number as the second video content type of the target video.

[0078] Specifically, in order to make the second video content type more accurate, the video sending end will determine a video content type for each encoded frame. Therefore, for multiple encoded frames, the video sending end will determine multiple video content types. Finally, the video sending end will select the video content type with the most occurrences as the second video content type of the target video.

[0079] In addition, such as Figure 8 As shown, Figure 8 This is a flowchart of a video processing method on the video sending end side provided in another embodiment of this application. After acquiring the target video according to the target frame rate parameter in step S200 above, the video processing method of this embodiment of the application also includes, but is not limited to, steps S910, S920 and S930.

[0080] Step S910: Extract features from the video images of the target video to obtain image features;

[0081] Step S920: Determine the second video content type of the target video based on the image features, and determine the corresponding second identification information and second frame rate parameters based on the second video content type;

[0082] Step S930: Update the target identification information to the second identification information and update the target frame rate parameter to the second frame rate parameter.

[0083] Specifically, in this application embodiment, the target identification information and target frame rate parameters are continuously updated. In one embodiment, after the target video is acquired according to the target frame rate parameters, the video sending end extracts features from the video image of the target video to obtain image features. Then, the second video content type is determined according to the image features, and the second identification information and second frame rate parameters corresponding to the second video content type are determined, thereby realizing the update of the target identification information and target frame rate parameters respectively, so that the second identification information and second frame rate parameters are switched to be used in the subsequent acquisition and encoding process.

[0084] Additionally, it should be noted that the second video content type mentioned above, like the first video content type mentioned above, can be a document type, a non-document type such as video footage, or other content types. This application embodiment does not specifically limit the type of the second video content type.

[0085] It is worth noting that the second video content type, the second identification information, and the second frame rate parameter are in a one-to-one correspondence. For example, when the second video content type is a document type, the second identification information corresponds to the value 1, and the second frame rate parameter corresponds to f1; when the second video content type is a non-document type, the second identification information corresponds to the value 0, and the second frame rate parameter corresponds to f2. Since the playback frame rate requirement for document types is lower than that for non-document types, f1 can be set to be less than f2.

[0086] Additionally, it is worth noting that after obtaining the second video content type, the video sending end can determine the corresponding second identification information and second frame rate parameters by looking up a table locally; it can also send the second video content type to the cloud server and receive the second identification information and second frame rate parameters fed back by the cloud server; or it can use other acquisition methods. This application embodiment does not specifically limit the acquisition method of the second identification information and second frame rate parameters.

[0087] In addition, such as Figure 9 As shown, Figure 9 This is a flowchart of a video processing method on the video sending end side provided in another embodiment of this application. Regarding the determination of the second video content type of the target video based on image features in step S920 above, the video processing method of this application embodiment also includes, but is not limited to, steps S1010 and S1020.

[0088] Step S1010: Classify and identify the image features to obtain the probability value of the video content type of the target video;

[0089] Step S1020: Compare the probability value with the third preset threshold, and determine the second video content type of the target video based on the comparison result.

[0090] Specifically, after extracting the image features, the video sending end classifies and identifies the image features to obtain the probability value of the video content type of the target video; then, the video sending end compares the probability value with a third preset threshold, and finally determines the second video content type of the target video based on the comparison result.

[0091] Specifically, when the above probability value is the probability value of belonging to a category other than document such as video, if the probability value is greater than the third preset threshold, then the second video content type of the target video is determined to be video type; if the probability value is less than or equal to the third preset threshold, then the second video content type of the target video is determined to be document type.

[0092] In addition, when the above probability value is the probability value of belonging to the document category, if the probability value is greater than the third preset threshold, the second video content type of the target video is determined to be a document type; if the probability value is less than or equal to the third preset threshold, the second video content type of the target video is determined to be a non-document type.

[0093] It is understood that the aforementioned third preset threshold can be pre-input by the user or determined according to preset rules based on the current video transmission situation. This application embodiment does not specifically limit the method of determining the third preset threshold.

[0094] In addition, such as Figure 10 As shown, Figure 10 This is a flowchart of a video processing method on the video sending end side provided in another embodiment of this application. Regarding the determination of the second video content type of the target video based on the comparison result in step S1020 above, the video processing method of this application embodiment also includes, but is not limited to, steps S1110 and S1120.

[0095] Step S1110: For each video image, determine a comparison result based on the probability value and the third preset threshold;

[0096] Step S1120: Determine the second video content type of the target video based on the comparison results with the largest number.

[0097] Specifically, in order to make the second video content type more accurate, the video sending end will determine a video content type for each video image. Therefore, for multiple video images, the video sending end will determine multiple video content types. Finally, the video sending end will select the video content type with the most occurrences as the second video content type of the target video.

[0098] Based on the above implementation environment and video processing method on the video sending end, various embodiments of the video processing method on the video receiving end of this application are proposed below.

[0099] like Figure 11 As shown, Figure 11 This is a flowchart of a video processing method on the video receiving end side according to an embodiment of this application. This video processing method can be applied to... Figure 1 The video receiving end in the process includes, but is not limited to, steps S1200, S1300 and S1400.

[0100] Step S1200: Receive the video stream from the video transmitter;

[0101] Step S1300: Decode the video stream to obtain the decoded video and target identification information;

[0102] Step S1400: Determine the corresponding target frame rate parameter based on the target identification information, and display the decoded video based on the target frame rate parameter.

[0103] Specifically, in the video processing, firstly, the video sending end obtains the first video content type of the target video, determines the corresponding first identification information and first frame rate parameter based on the first video content type, and uses the first identification information as the target identification information and the first frame rate parameter as the target frame rate parameter; then, the video sending end captures the target video according to the target frame rate parameter, encodes the target identification information and the captured target video to obtain a video stream, and sends the video stream to the video receiving end; finally, the video receiving end decodes the video stream to obtain the decoded video and target identification information, obtains the corresponding target frame rate parameter based on the target identification information, and then displays the decoded video according to the target frame rate parameter.

[0104] Additionally, it is worth noting that after obtaining the second video content type, the video receiving end can determine the corresponding second identification information and second frame rate parameters by looking up a table locally; it can also send the second video content type to the cloud server and receive the second identification information and second frame rate parameters fed back by the cloud server; or it can use other acquisition methods. This application embodiment does not specifically limit the acquisition method of the second identification information and second frame rate parameters.

[0105] The video receiver decodes the received video stream and extracts the target identification information. Based on the target identification information, it determines the target frame rate parameter and adaptively adjusts the video display frame rate parameter.

[0106] It is worth noting that, since the video processing method on the receiving end side of this application corresponds to the video processing method on the sending end side of the above-described embodiments, the specific implementation methods and technical effects of the video processing method on the receiving end side of this application can be referred to the specific implementation methods and technical effects of the video processing method on the sending end side of the above-described embodiments.

[0107] Based on the above implementation environment, the video processing method on the video sending end side, and the video processing method on the video receiving end side, the following is an overall embodiment of the video processing method of this application.

[0108] like Figure 12 As shown, Figure 12 This is a schematic diagram illustrating the implementation environment for performing a video processing method according to one embodiment of this application. Specifically: First, the video sending end sets relevant parameters, including a video content type flag and frame rate. Then, it encodes the video content type flag and the captured original video together to output a video stream. This video stream is transmitted to the receiving end via the network. The video receiving end decodes the received stream and extracts the video content type flag. Based on the video content type flag, it adjusts the frame rate parameters for video display. Simultaneously, it uses the original video and the encoded information to perform video content detection and analysis, updating the video content type flag and frame rate parameters based on the detection results for subsequent video processing.

[0109] Among them, regarding Figure 12 The underlying principles and steps are as follows:

[0110] Parameter settings: Set parameters such as video content type flag and video frame rate.

[0111] Video capture: Capture video according to the set frame rate.

[0112] Video encoding: Common video encoders such as H.26x, AV1, and AVS are used to encode the captured video and video content type identifiers.

[0113] Network transmission: transmitting video streams from the sending end to the receiving end.

[0114] Video display: The decoded video is then displayed and rendered.

[0115] Video decoding: Decoding the video bitstream.

[0116] Content detection: Analyzes and judges the video content type using the video and its related information, and outputs the video content type detection results.

[0117] In addition, such as Figure 13 As shown, Figure 13This is an overall flowchart of a video processing method provided in one embodiment of this application. Specifically, it includes the following steps:

[0118] Step S1-1: Set the relevant parameters of the auxiliary stream video in the video conferencing system. The initial auxiliary stream video content type is document, its flag bit is set to 1, and the initial frame rate parameter is set to f1.

[0119] Step S1-2: Acquire the original video at frame rate f1, and use an H.265 video encoder to encode the original video and the video content type flag bit to obtain the video bitstream. The video type flag information is written into the reserved position in the protocol, and the motion vector information of the encoded frame is saved for use by the subsequent content detection module.

[0120] Step S1-3: Transmit the encoded video stream from step S1-2 to the video receiving end via a TCP network;

[0121] Steps S1-4: The video receiver uses the corresponding H.265 video decoder to decode the received bitstream and obtain the decoded video and content type flag.

[0122] Steps S1-5: The video receiver adjusts the video frame rate parameter according to the video type flag to display and play the decoded video. If the video content type flag is 0, the frame rate is adjusted to f2 (f2>>f1); otherwise, the frame rate is adjusted to f1.

[0123] Steps S1-6: The content detection algorithm analyzes and processes the motion vectors of the encoded frames generated in step S1-2. The specific method is as follows: The image is divided into blocks. For example, if the input image has a width of W and a height of H, the image is divided into blocks using a preset patchsize to obtain the total number of blocks in the image. The motion vector in each block is compared with the motion vector value of the corresponding block in the previous frame. If the difference between the two is greater than a certain threshold, the current block is considered to be moving and recorded as motionblock. The proportion of moving blocks in the entire image is calculated as ratio = motionblock / num. If ratio > T, which can be obtained through experiments as needed, the content type of the current image is video; otherwise, it is document type, and the current frame type flag is saved. The frame type flags of N consecutively stored frames are voted on, and the type with the most votes is the final video content type result of the current frame.

[0124] Steps S1-7: Based on the detection results of the previous step, modify the video content type flag and the frame rate parameters for video acquisition and encoding at the video sending end, as follows: If the video content type detection result is document, then set the video content type flag to 1 and the video frame rate parameter to f1; if the video content type detection result is non-document, then set the video content type flag to 0 and the video frame rate parameter to f2 (f2>>f1).

[0125] like Figure 14 As shown, Figure 14 This is an overall flowchart of a video processing method provided in another embodiment of this application. Specifically, it includes the following steps:

[0126] Step S2-1: Set the relevant parameters of the auxiliary stream video in the video conferencing system. The initial auxiliary stream video content type is document, its flag bit is set to 1, and the initial frame rate parameter is set to f1.

[0127] Step S2-2: Acquire the original video at frame rate f1, and use an H.265 hardware video encoder to encode the original video and the video content type flag bit to obtain the video bitstream, wherein the video type flag information is written into the reserved position in the protocol;

[0128] Step S2-3: Transmit the encoded video stream from step S2-2 to the video receiving end via a UDP network;

[0129] Step S2-4: The video receiver uses the corresponding H.265 video decoder to decode the received bitstream and obtain the decoded video and the video content type flag.

[0130] Step S2-5: The video receiver adjusts the video frame rate parameter according to the video type flag to display and play the decoded video. If the video content type flag is 0, the frame rate is adjusted to f2 (f2>>f1); otherwise, the frame rate is adjusted to f1.

[0131] Step S2-6: The content detection module uses a convolutional neural network for video content detection. The network input is the original video collected in step S2-2. Image features are extracted using a deep learning network (such as MobileNet, ResNet, etc.), and a classifier (such as softmax, sigmoid, etc.) is used to determine the category attribute of the image features. The specific method is as follows: The ResNet50 neural network is trained using a labeled dataset to obtain network parameters for subsequent feature extraction; the trained ResNet50 neural network is used to extract features from the input image to obtain image features; the image features are input into the sigmoid classifier to obtain the probability value prob of the current feature belonging to the video category; if the video category probability value prob > 0.5, the current frame type is determined to be a document; otherwise, it is a non-document, and the current frame type flag is saved; the frame type flags of N consecutively stored frames are voted on, and the type with the most votes is the final video content type detection result of the current frame.

[0132] Step S2-7: Based on the detection results of the previous step, modify the video content type flag and the frame rate parameters for video acquisition and encoding at the video sending end, as follows: If the video content type detection result is document, then set the video content type flag to 1 and the video frame rate parameter to f1; if the video content type detection result is non-document, then set the video content type flag to 0 and the video frame rate parameter to f2 (f2>>f1).

[0133] According to the technical solution of this application embodiment, dynamically adjusting the video frame rate through video content detection can ensure a good subjective experience for different video content; in addition, adaptive frame rate can control video acquisition, video encoding and other links, which can effectively utilize and save bandwidth resources; furthermore, the video receiving end shares the detection results of the sending end, ensuring synchronization while reducing computational complexity.

[0134] Based on the above implementation environment, the video processing method on the video transmitting end side, and the video processing method on the video receiving end side, various embodiments of the video transmitting end, video receiving end, computer-readable storage medium, and computer program product of this application are presented below.

[0135] like Figure 15 As shown, Figure 15This is a schematic diagram of the structure of a video transmitter provided in one embodiment of this application; another embodiment of this application discloses a video transmitter 100, including: a first memory 120, a first processor 110 and a computer program stored in the first memory 120 and executable on the first processor 110. When the first processor 110 runs the computer program, it executes the video processing method on the video transmitter side as in any of the preceding embodiments.

[0136] The first memory 120 serves as a non-transitory computer-readable storage medium, used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, the first memory 120 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the first memory 120 may optionally include memory remotely located relative to the first processor 110, and these remote memories can be connected to the implementation environment via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0137] The video transmitter 100 in this embodiment can be corresponding to, for example: Figure 1 The video transmitting end in the implementation environment of the illustrated embodiment belongs to the same application concept, and therefore has the same implementation principle and beneficial effects, which will not be described in detail here.

[0138] The non-transient software program and instructions required to implement the video processing method on the video transmitting end side of the above embodiments are stored in the first memory 120. When executed by the first processor 110, the video processing method on the video transmitting end side of the above embodiments is executed, for example, the method described above is executed. Figure 2 to Figure 10 The methods and steps in the text.

[0139] It is worth noting that, since the video transmitting end 100 of this application embodiment can execute the video processing method on the video transmitting end side of the above embodiment, the specific implementation method and technical effect of the video transmitting end 100 of this application embodiment can be referred to the specific implementation method and technical effect of the video processing method on the video transmitting end side described above.

[0140] In addition, such as Figure 16 As shown, Figure 16 This is a schematic diagram of the structure of a video receiver provided in one embodiment of this application; another embodiment of this application discloses a video receiver 200, including: a second memory 220, a second processor 210, and a computer program stored in the second memory 220 and executable on the second processor 210. When the second processor 210 runs the computer program, it executes the video processing method on the video receiver side as described in any of the preceding embodiments.

[0141] The second memory 220, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, the second memory 220 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the second memory 220 may optionally include memory remotely located relative to the second processor 210, and these remote memories can be connected to the implementation environment via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0142] The video receiver 200 in this embodiment can be corresponding to, for example: Figure 1 The video receivers in the implementation environment of the illustrated embodiments belong to the same application concept, and therefore have the same implementation principle and beneficial effects, which will not be described in detail here.

[0143] The non-transient software program and instructions required to implement the video processing method on the video receiver side of the above embodiments are stored in the second memory 220. When executed by the second processor 210, the video processing method on the video receiver side of the above embodiments is executed, for example, the method described above is executed. Figure 11 The methods and steps in the text.

[0144] It is worth noting that, since the video receiver 200 of this application embodiment can execute the video processing method on the video receiver side of the above embodiment, the specific implementation method and technical effects of the video receiver 200 of this application embodiment can be referred to the specific implementation method and technical effects of the video processing method on the video receiver side described above.

[0145] In addition, one embodiment of this application discloses a computer-readable storage medium storing computer-executable instructions for performing video processing methods as described in any of the preceding embodiments.

[0146] It is worth noting that, since the computer-readable storage medium of this application embodiment is capable of executing the video processing methods of the video transmitting end or video receiving end of the above embodiments, the specific implementation methods and technical effects of the computer-readable storage medium of this application embodiment can be referred to the specific implementation methods and technical effects of the video processing methods of the video transmitting end or video receiving end of the above embodiments.

[0147] Furthermore, one embodiment of this application discloses a computer program product, including a computer program or computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, causing the computer device to perform the video processing method as described in any of the preceding embodiments.

[0148] It is worth noting that, since the computer program product of this application embodiment can execute the video processing method on the video sending end side or video receiving end side of the above embodiments, the specific implementation method and technical effect of the computer program product of this application embodiment can be referred to the specific implementation method and technical effect of the video processing method on the video sending end side or video receiving end side.

[0149] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. A video processing method applied at a video sending end, the method comprising: Obtain the first video content type of the target video, determine the corresponding first identification information and first frame rate parameter based on the first video content type, and use the first identification information as the target identification information and the first frame rate parameter as the target frame rate parameter. The target video is acquired according to the target frame rate parameter, and the target identification information and the acquired target video are encoded to obtain a video stream; The video stream is sent to the video receiving end, so that the video receiving end can decode the video stream to obtain the decoded video and the target identification information, and then display the decoded video according to the target frame rate parameter corresponding to the target identification information; The method further includes, after encoding the target identification information and the acquired target video, the latter being: Determine the second video content type of the target video, and determine the corresponding second identification information and second frame rate parameter based on the second video content type; update the target identification information to the second identification information, and update the target frame rate parameter to the second frame rate parameter.

2. The video processing method according to claim 1, characterized in that, After encoding the target identification information and the acquired target video, the method further includes: Acquire encoded frames during the encoding process and determine the motion vector parameters of the encoded frames; The second video content type of the target video is determined based on the motion vector parameters.

3. The video processing method according to claim 2, characterized in that, Determining the second video content type of the target video based on the motion vector parameters includes: The encoded frame is divided into multiple block images, and the total number of images in all the block images is determined. Calculate the difference in motion vector parameters between all two adjacent image blocks, and determine the number of motion blocks in the image blocks based on all the differences. The second video content type of the target video is determined based on the total number of images and the number of motion blocks.

4. The video processing method according to claim 3, characterized in that, Determining the number of motion blocks in the segmented image based on all the differences includes: For every two adjacent block images, if the difference between the motion vector parameters of the previous block image and the motion vector parameters of the next block image is greater than a first preset threshold, the next block image is recorded as a motion block image. Count the number of motion blocks in all the motion block images.

5. The video processing method according to claim 3, characterized in that, The step of determining the second video content type of the target video based on the total number of images and the number of motion blocks includes: Calculate the ratio of the number of moving blocks to the total number of images; The ratio is compared with the second preset threshold, and the second video content type of the target video is determined based on the comparison result.

6. The video processing method according to any one of claims 3 to 5, characterized in that, The step of determining the second video content type of the target video based on the total number of images and the number of motion blocks includes: For each encoded frame, a video content type is determined based on the total number of images and the number of motion blocks; Select the video content type with the largest number of videos as the second video content type of the target video.

7. The video processing method according to claim 1, characterized in that, After acquiring the target video according to the target frame rate parameter, the method further includes: Feature extraction is performed on the video images of the target video to obtain image features; The second video content type of the target video is determined based on the image features.

8. The video processing method according to claim 7, characterized in that, The step of determining the second video content type of the target video based on the image features includes: The image features are classified and identified to obtain the probability value of the video content type of the target video; The probability value is compared with a third preset threshold, and the second video content type of the target video is determined based on the comparison result.

9. The video processing method according to claim 8, characterized in that, Determining the second video content type of the target video based on the comparison result includes: For each video image, a comparison result is determined based on the probability value and a third preset threshold; The second video content type of the target video is determined based on the comparison result with the largest number.

10. A video processing method applied at a video receiving end, the method comprising: The system receives a video stream from a video transmitter, wherein the video stream is obtained through the following methods: the video transmitter acquires a first video content type of the target video, determines corresponding first identification information and a first frame rate parameter based on the first video content type, uses the first identification information as target identification information and the first frame rate parameter as target frame rate parameter, captures the target video according to the target frame rate parameter, and encodes the target identification information and the captured target video; wherein, after encoding the target identification information and the captured target video, the video transmitter determines a second video content type of the target video, and determines corresponding second identification information and a second frame rate parameter based on the second video content type; the target identification information is updated to the second identification information, and the target frame rate parameter is updated to the second frame rate parameter; The video stream is decoded to obtain the decoded video and target identification information; The corresponding target frame rate parameter is determined based on the target identification information, and the decoded video is displayed based on the target frame rate parameter.

11. A video transmitting end, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the video processing method according to any one of claims 1 to 9.

12. A video receiver, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the video processing method of claim 10 when running the computer program.

13. A computer-readable storage medium storing computer-executable instructions for performing the video processing method according to any one of claims 1 to 9 or the video processing method according to claim 10.

14. A computer program product, comprising a computer program or computer instructions, characterized in that, The computer program or the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer program or the computer instructions from the computer-readable storage medium and executes the computer program or the computer instructions, causing the computer device to perform the video processing method according to any one of claims 1 to 9 or the video processing method according to claim 10.

Images