A cloud server-based video conferencing system

Abstract

The application discloses a kind of video conference systems based on cloud server, it is related to video conference technical field, including real-time acquisition conference audio stream data;Real-time semantic analysis is carried out to transcription text sequence, and target keyword in preset keyword library is extracted;Synchronous acquisition of the video picture data of current time;Conference semantic value score and picture physical complexity are input into preset two-factor coding decision table;Real-time adjustment of the coding parameter of video encoder;Generate semantic value time axis metadata;Generate structured conference index file.The application retains more effective information under the same storage capacity and guarantees key fragment picture quality;Avoid the repeated analysis of full-amount audio data, greatly improve the index generation efficiency, realize the collaborative closed loop of "coding is labeled";Reduce search cost, overall realize the complete technical effect chain from recording coding optimization to index efficient generation to user convenient search.

Description

Technical Field

[0001] This invention relates to the field of video conferencing technology, and in particular to a video conferencing system based on a cloud server. Background Technology

[0002] Video conferencing systems, as a real-time communication tool that transcends geographical limitations, have been widely used in various industries such as government, enterprises, education, healthcare, and finance. A typical video conferencing system usually consists of two parts: a client and a server. The client is responsible for audio and video acquisition, preprocessing, encoding, and network transmission, while the server is mainly responsible for core functions such as signaling exchange, data forwarding, audio and video transcoding, and recording. With the maturity of cloud computing technology, cloud-based video conferencing systems have become the mainstream architecture, allowing users to obtain high-quality video conferencing services without deploying complex local servers. However, existing cloud-based video conferencing systems still have the following technical problems in the storage and utilization of recorded files:

[0003] First, the storage efficiency of recorded files is low. Existing video conferencing systems typically use uniform encoding parameters to encode and store the entire meeting during the recording process. Regardless of whether the meeting content is a static PPT presentation or document display, or a dynamic video playback or multi-person discussion scenario, the system uses the same bitrate, frame rate, and quantization parameters for encoding. This "one-size-fits-all" encoding strategy leads to two problems: for static content with slow-moving images, the excessively high bitrate results in a waste of storage space; while for dynamic content with drastic image changes, the uniform bitrate may not be sufficient to guarantee clear image quality. Although there are adaptive encoding schemes in existing technologies that adjust encoding parameters based on image complexity, these schemes only judge from the physical level of the video signal and fail to consider the semantic importance of the meeting content itself.

[0004] Secondly, the post-recording retrieval and utilization of recorded files is difficult. With the popularization of cloud recording functions, the number of meeting recording files accumulated by enterprises and individuals has grown exponentially. If a user wants to find a key decision point, a speaker's point of view, or the discussion content of a certain topic in a meeting recording file that lasts for several hours, they often need to watch the entire video from beginning to end, which is extremely inefficient. Although in recent years some video conferencing platforms have begun to introduce AI summary generation functions, which can automatically generate meeting minutes and chapter divisions after the meeting, these functions are completely separated from the recording and encoding process.

[0005] However, current common solutions have many drawbacks, including: In existing video conferencing recording technologies, encoding decisions rely solely on the physical features of the image and cannot perceive the semantic importance of the meeting content. This leads to the use of the same encoding strategy for high-value static content and low-value dynamic content, making it difficult to balance storage efficiency and critical image quality. At the same time, recording and encoding are completely separated from subsequent index generation, adopting a two-stage model of "record and store first, then analyze and index." The semantic information generated during the recording process is not recorded and reused, resulting in index generation requiring repeated calculations on all audio data and only being able to start after recording is completed. This leads to low processing efficiency, long response time, and the inability to feed back the index analysis results to the encoding stage to optimize recording quality. There is a lack of coordination mechanism between the two. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0007] In view of the problems existing in the current cloud server-based video conferencing system, the present invention is proposed.

[0008] Therefore, the purpose of this invention is to provide a cloud server-based video conferencing system that addresses the problems in existing video conferencing recording technologies. These technologies rely solely on the physical features of the image for encoding decisions, failing to perceive the semantic importance of the meeting content. This leads to the use of the same encoding strategy for high-value static content and low-value dynamic content, making it difficult to balance storage efficiency and critical image quality. Furthermore, the recording and encoding processes are completely separate from the subsequent index generation, employing a two-stage model of "record and store first, then analyze and index." Semantic information generated during recording is not recorded and reused, resulting in index generation requiring repeated calculations of all audio data and only being able to begin after recording. This leads to low processing efficiency, long response times, and the inability to feed back index analysis results to the encoding stage to optimize recording quality, highlighting the lack of a collaborative mechanism between the two processes.

[0009] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0010] In a first aspect, embodiments of the present invention provide a video conferencing method based on a cloud server, comprising: acquiring real-time audio stream data during a video conference; performing speech activity detection and real-time speech transcription on the audio stream data to generate a time-continuous transcribed text sequence; performing real-time semantic analysis on the transcribed text sequence to extract target keywords from a preset keyword library, and calculating a meeting semantic value score corresponding to the current moment based on the weight value and frequency of occurrence of the target keywords; simultaneously acquiring video frame data at the current moment and calculating the physical complexity of the video frame data; and inputting the meeting semantic value score and the physical complexity of the video frame data into a preset two-factor coding decision table. Based on the output of the two-factor coding decision table, the coding parameters of the video encoder are adjusted in real time. The coding parameters include at least quantization parameters and I-frame intervals. During the coding process, the semantic value score of the meeting is associated with the corresponding video frame timestamp to generate semantic value timeline metadata, and the semantic value timeline metadata is embedded in the container format of the recording file. After the recording is completed, in response to the user's index generation instruction, the semantic value timeline metadata in the recording file is directly called. Based on the metadata, the time intervals with semantic value scores higher than the preset value threshold are selected. Only the audio data within the selected time intervals is transcribed and extracted to generate a structured meeting index file.

[0011] As a preferred embodiment of the cloud server-based video conferencing method of the present invention, the physical complexity of the image includes at least the image motion vector magnitude, texture richness, and inter-frame variation.

[0012] As a preferred embodiment of the cloud server-based video conferencing method of the present invention, the two-factor coding decision table includes at least the following mapping rules: when the meeting semantic value score is higher than a first semantic threshold and the physical complexity of the image is lower than a first complexity threshold, the quantization parameter in the coding parameters is adjusted to a first quantization parameter value, and the I-frame interval in the coding parameters is adjusted to a first interval value, wherein the first quantization parameter value is lower than a normal quantization parameter value, and the first interval value is shorter than a normal I-frame interval; when the meeting semantic value score is lower than a second semantic threshold and the physical complexity of the image is higher than a second complexity threshold, the quantization parameter in the coding parameters is adjusted to a second quantization parameter value, and the I-frame interval in the coding parameters is adjusted to a second interval value, wherein the second quantization parameter value is higher than a normal quantization parameter value, and the second interval value is longer than a normal I-frame interval; wherein the first semantic threshold is higher than the second semantic threshold, and the first complexity threshold is lower than the second complexity threshold.

[0013] As a preferred embodiment of the cloud server-based video conferencing method of the present invention, the calculation method of the physical complexity of the image includes: extracting the pixel-level motion vector field of the current video frame relative to the previous video frame, and calculating the mean amplitude of the motion vector field as a first complexity component; extracting texture features of the current video frame, and calculating the contrast and entropy values ​​of the gray-level co-occurrence matrix as a second complexity component; calculating the inter-frame brightness change and chromaticity change between the current video frame and the previous video frame as a third complexity component; and performing a weighted summation of the first complexity component, the second complexity component, and the third complexity component to obtain the physical complexity of the image.

[0014] As a preferred embodiment of the cloud server-based video conferencing method of the present invention, the calculation method of the meeting semantic value score includes: pre-constructing a keyword library and configuring a corresponding basic weight value for each keyword in the keyword library; performing real-time word segmentation on the transcribed text sequence, matching the word segmentation results with the keyword library, and extracting the successfully matched target keywords and their occurrence positions; calculating the meeting semantic value score corresponding to the current moment based on the basic weight value of each target keyword, the frequency of occurrence within a preset time window, and the semantic correlation between the target keywords; wherein, when multiple semantically related keywords co-occur within the preset time window, the meeting semantic value score obtains an additional collaborative weighting value.

[0015] As a preferred embodiment of the cloud server-based video conferencing method of the present invention, the step of embedding the semantic value timeline metadata into the container format of the recording file specifically includes: defining a custom metadata field in the container format of the recording file; writing the semantic value timeline metadata into the custom metadata field in timestamp order, wherein each metadata record contains at least timestamp information and the corresponding meeting semantic value score; and when the encoding parameters are adjusted, synchronously recording the encoding parameter change event in the custom metadata field, wherein the encoding parameter change event includes at least the change timestamp, the encoding parameter value before the change, and the encoding parameter value after the change.

[0016] As a preferred embodiment of the cloud server-based video conferencing method of the present invention, the generation of the structured meeting index file specifically includes: filtering time intervals with meeting semantic value scores higher than a preset value threshold based on the semantic value time axis metadata, and generating a list of high-value time intervals; transcribing the audio data in each time interval of the high-value time intervals to generate corresponding transcribed text segments; extracting summaries from each transcribed text segment to generate content summaries corresponding to each time interval; associating and storing the list of high-value time intervals, the transcribed text segments corresponding to each time interval, and the content summaries to generate the structured meeting index file; the structured meeting index file also includes the start timestamp, end timestamp, and duration information of each high-value time interval.

[0017] Secondly, to further address the aforementioned technical problems, the present invention provides a cloud server-based video conferencing system, comprising: an audio acquisition module, used to acquire real-time audio stream data during a video conference, perform speech activity detection and real-time speech transcription on the audio stream data, and generate a time-continuous transcribed text sequence; a semantic analysis module, used to perform real-time semantic analysis on the transcribed text sequence, extract target keywords from a preset keyword library, and calculate the meeting semantic value score corresponding to the current moment based on the weight value and frequency of occurrence of the target keywords; a video acquisition module, used to synchronously acquire video frame data at the current moment and calculate the physical complexity of the video frame data; and an encoding decision module, used to integrate the meeting semantic value score with... The system employs several modules: a physical complexity input to a pre-defined two-factor coding decision table; a parameter adjustment module to adjust the video encoder's parameters in real time based on the output of the two-factor coding decision table; a metadata generation module to associate the meeting's semantic value score with the corresponding video frame timestamp during encoding, generating semantic value timeline metadata, and embedding this metadata into the container format of the recorded file; and an index generation module to respond to the user's index generation command after recording, directly calling the semantic value timeline metadata in the recorded file, filtering out time intervals with semantic value scores higher than a pre-defined value threshold, and performing speech transcription and summary extraction only on the audio data within the selected time intervals to generate a structured meeting index file.

[0018] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements any step of the video conferencing method based on a cloud server as described in the first aspect of the present invention.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of a video conferencing method based on a cloud server as described in the first aspect of the present invention.

[0020] The beneficial effects of this invention are as follows: This invention calculates the semantic value score of each time period of the meeting through real-time speech transcription and semantic analysis, and constructs a two-factor coding decision table in combination with the physical complexity of the image. Differentiated coding parameters are adjusted for meeting content with different semantic values ​​and different physical complexities, so that storage space is tilted towards high-value content, retaining more effective information and ensuring the image quality of key segments under the same storage capacity. During the encoding process, semantic value time axis metadata is generated synchronously and embedded into the recording file. When generating the index later, this metadata is directly called to filter high-value time periods, and only targeted speech transcription and summary extraction are performed on the filtered intervals, avoiding the repeated analysis of the full audio data, greatly improving the index generation efficiency, and realizing a collaborative closed loop of "coding as annotation". The final generated structured index file supports users to quickly locate and jump to the core content of the meeting, reducing the search cost. Moreover, the configurability of the keyword library allows the solution to be flexibly adapted to various industry scenarios such as enterprise business, telemedicine, distance education, government finance, etc. Overall, it realizes a complete technical effect chain from recording and encoding optimization to efficient index generation and convenient user retrieval. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0022] Figure 1 This is a flowchart illustrating the implementation of the present invention in Example 1.

[0023] Figure 2 This is a specific flowchart of the present invention in Example 1. Detailed Implementation

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0026] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0027] Example 1

[0028] Reference Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a video conferencing method based on a cloud server, including the following steps:

[0029] S1: During the video conference, acquire the audio stream data in real time, perform voice activity detection and real-time speech transcription on the audio stream data, and generate a time-continuous transcribed text sequence.

[0030] Preferably, the voice activity detection includes: performing frame processing on the audio stream data according to a preset time segment length (preferably 20ms to 40ms), calculating the short-time energy and zero-crossing rate of each frame of audio signal, and determining the current frame as a voice frame when the short-time energy exceeds a first energy threshold and the zero-crossing rate is within a preset range; merging the time intervals that are continuously determined to be voice frames to generate a voice activity time period.

[0031] Furthermore, real-time speech transcription employs a streaming speech recognition engine, which supports incremental recognition: whenever new speech frame data is detected, the speech frame data is input into the streaming speech recognition engine for real-time decoding, and partial recognition results are output; when the end of the speech activity is detected, all speech frame data within the speech activity time period are finally decoded, and complete transcribed text segments are output; the transcribed text segments corresponding to each speech activity time period are spliced ​​together in chronological order to generate a temporally continuous transcribed text sequence.

[0032] Furthermore, speech activity detection also includes: preprocessing the audio stream data with noise suppression and echo cancellation. Noise suppression uses spectral subtraction or Wiener filtering, and echo cancellation uses an adaptive filter. The preprocessed audio stream data is then input into speech activity detection and real-time speech transcription.

[0033] Specifically, the streaming speech recognition engine is built on deep learning acoustic and language models. The acoustic model adopts a connection-temporal classification (CTC) or attention mechanism architecture, and the language model adopts an N-gram model or a pre-trained language model based on Transformer. The streaming speech recognition engine supports speech recognition of Mandarin Chinese, mixed Chinese and English, and multiple dialects.

[0034] For example, a technology company is conducting a cloud-based video conference on "Project X Project Approval". During the conference, the system processes the audio stream data in 30ms segments, calculating the short-time energy and zero-crossing rate of each frame in real time. When the short-time energy of a frame exceeds a preset threshold and the zero-crossing rate is within the range of 0.1 to 0.6, the frame is determined to be a speech frame. The time intervals of consecutive speech frames are merged into a speech activity time period. The system uses a streaming speech recognition engine based on a connection-time classification (CTC) architecture to incrementally decode the detected speech frames. Whenever the project manager begins to speak and explain the project background, the system outputs partial recognition results in real time. When the project manager finishes a complete speech, the system performs final decoding on all speech frames within the speech activity time period, outputs a complete transcribed text segment, and concatenates the transcribed text of each time period in sequence into a continuous transcribed text sequence: "Hello everyone, today we are holding a Project X Project Approval Meeting. First, I will introduce the project background and market demand analysis..."

[0035] S2: Perform real-time semantic analysis on the transcribed text sequence, extract target keywords from the preset keyword library, and calculate the semantic value score of the meeting at the current moment based on the weight value and frequency of the target keywords.

[0036] Preferably, the calculation method for the semantic value score of the meeting includes:

[0037] Build a keyword library in advance and configure a corresponding basic weight value for each keyword in the keyword library;

[0038] The transcribed text sequence is processed by word segmentation in real time, and the segmentation results are matched with a keyword database to extract the target keywords that are successfully matched and their positions.

[0039] Based on the basic weight value of each target keyword, the frequency of occurrence within the preset time window, and the semantic relevance between target keywords, calculate the semantic value score of the meeting at the current moment.

[0040] When multiple semantically related keywords appear together within a preset time window, the semantic value score of the meeting receives an additional collaborative weighting value.

[0041] Specifically, the semantic value score of the meeting changes dynamically and continuously over time.

[0042] Furthermore, the preset time window is a time interval centered on the current moment, extending forward and backward by a first duration (preferably 30 to 60 seconds). The semantic value score of the meeting is continuously calculated using the preset time window as a sliding window.

[0043] Specifically, the formula for calculating the semantic value score of a meeting is as follows:

[0044] ;

[0045] in, express The semantic value score of the meeting at any given moment; The current moment; This indicates the number of target keywords that were successfully matched within a preset time window; For the first The index number of each target keyword; Indicates the first The base weight value of each target keyword; Indicates the first The frequency of occurrence of each target keyword within a preset time window; Indicates the first The time decay factor of a target keyword is higher when it is closer to the current time. This indicates the number of semantically related keyword groups that appear together within a preset time window; j is the index number of the j-th semantically related keyword group. This represents the collaborative weighted value of the j-th semantically related keyword group; This indicates the frequency of common occurrence of the j-th semantically related keyword group within a preset time window.

[0046] Furthermore, the semantic relevance is calculated based on a pre-defined semantic relevance graph. The semantic relevance graph uses keywords as nodes and semantic relationships between keywords as edges. Semantic relationships include synonymy, hyponymy, collocation, and co-occurrence. The semantic relevance between two keywords is determined by the path distance between corresponding nodes in the semantic relevance graph and the weight of the edge. The shorter the path distance and the higher the edge weight, the higher the semantic relevance.

[0047] Specifically, the keyword library is configured according to the application scenario:

[0048] For corporate business meeting scenarios, the keyword library includes keywords such as "contract," "signing," "decision-making," "approval," "budget," "investment," "risk," "goal," and "achievement."

[0049] For telemedicine scenarios, the keyword library includes keywords such as "diagnosis", "conclusion", "surgery", "pathology", "examination", "medication", "treatment plan", and "consultation".

[0050] For distance education scenarios, the keyword database includes keywords such as "exam points", "key points", "difficult points", "summary", "homework", "exam", "knowledge points", and "review".

[0051] For government and financial meeting scenarios, the keyword library includes keywords such as "policy", "regulation", "approval", "compliance", "regulation", "audit", "risk control", and "resolution".

[0052] Furthermore, the keyword database also supports user-defined configurations. Users can add, delete, or modify keywords and their corresponding basic weight values ​​through the system interface, and the system updates the keyword database in real time based on the user's configuration.

[0053] For example, the system uses a corporate business meeting keyword library (containing keywords such as "contract", "signing", "decision", "approval", "budget", "investment", "risk", "goal", and "achieve") to perform real-time word segmentation and keyword matching on the transcribed text sequence generated in step S1. When the finance manager mentions "the total project budget is 5 million yuan, and the investment return period is about 18 months", the system matches the two target keywords "budget" and "investment" within a 30-second time window. The basic weight value of "budget" is 0.8, and the basic weight value of "investment" is 0.7. The two have a pairing relationship in the semantic association graph, and the system assigns an additional collaborative weighting value of 0.3. Finally, the semantic value score of the meeting at the current moment is calculated as S(t) = 0.8 × 2 × 0.95 + 0.7 × 1 × 0.92 + 0.3 × 1 = 2.464 (out of 5). This score changes dynamically and continuously with the time axis, forming a fluctuating semantic value curve.

[0054] S3: Synchronously acquire the video frame data at the current moment and calculate the physical complexity of the video frame data.

[0055] Preferably, the physical complexity of the image includes at least the magnitude of the image motion vector, the richness of texture, and the amount of inter-frame variation.

[0056] Furthermore, the methods for calculating the physical complexity of a scene include:

[0057] Extract the pixel-level motion vector field of the current video frame relative to the previous video frame, and calculate the mean magnitude of the motion vector field as the first complexity component.

[0058] Extract texture features from the current video frame and calculate the contrast and entropy values ​​of the gray-level co-occurrence matrix as the second complexity components;

[0059] The inter-frame luminance and chrominance changes between the current video frame and the previous video frame are calculated as the third complexity component.

[0060] The physical complexity of the image is obtained by weighted summation of the first, second, and third complexity components.

[0061] Specifically, the extraction of pixel-level motion vector fields employs either block matching motion estimation or optical flow methods: the block matching motion estimation method divides the current video frame into several macroblocks, searches for the block that best matches each macroblock within a preset search range of the reference frame (the previous video frame), calculates the motion vector of each macroblock, and summarizes them to obtain the pixel-level motion vector field; the optical flow method calculates the gradient of image brightness in the spatiotemporal domain, solves the optical flow constraint equation, and obtains the motion vector of each pixel.

[0062] Furthermore, the formula for calculating the first complexity component is as follows:

[0063] ;

[0064] in, Indicates the first complexity component; This indicates the total number of macroblocks or pixels in the current video frame; For the first The index number of a macroblock or pixel; and They represent the first The motion vector components of a macroblock or pixel in the horizontal and vertical directions.

[0065] Specifically, the gray-level co-occurrence matrix is ​​constructed as follows: the current video frame is converted into a grayscale image, and the joint distribution of grayscale values ​​between pixel pairs that are separated by a preset distance (preferably 1 pixel) and located in preset directions (preferably 0°, 45°, 90°, and 135°) in the grayscale image is statistically analyzed to generate the gray-level co-occurrence matrix; the contrast ratio of the gray-level co-occurrence matrix is ​​calculated using the following formula:

[0066] ;

[0067] The formula for calculating the entropy of the gray-level co-occurrence matrix is:

[0068] ;

[0069] in, The contrast of the gray-level co-occurrence matrix; Indicates the total number of gray levels; This represents the row index of the gray-level co-occurrence matrix; For the column index of the gray-level co-occurrence matrix; It is the square of the difference between the grayscale values ​​of two pixels; Represents the th element in the gray-level co-occurrence matrix. Line 1 The element values ​​of the column, i.e., the grayscale value. and grayscale The probability of pixel pairs appearing; This represents the entropy value of the gray-level co-occurrence matrix.

[0070] Furthermore, the second complexity component is The weighted sum of the contrast and entropy values ​​of the gray-level co-occurrence matrix is ​​expressed as follows:

[0071] ;

[0072] Where is the second complexity component; The contrast of the gray-level co-occurrence matrix; This represents the entropy value of the gray-level co-occurrence matrix; The weighting coefficient of contrast in the second complexity component; is the weighting coefficient of the entropy value in the second complexity component.

[0073] Furthermore, the formula for calculating the third complexity component is as follows:

[0074] ;

[0075] in, Represents the third complexity component. , , These represent the luminance components of the current video frame and the previous video frame in the YUV color space, respectively. chromaticity components and chromaticity components The average change on , , These represent the corresponding weight coefficients, and .

[0076] Specifically, the formula for calculating the physical complexity of the image is:

[0077] ;

[0078] in, Indicates the physical complexity of the image; Indicates the first complexity component; Indicates the second complexity component; Represents the third complexity component; , , These represent the corresponding weight coefficients, and .

[0079] Furthermore, the weight coefficients of the first complexity component, the second complexity component, and the third complexity component are dynamically adjusted according to the video conferencing scenario: for screen sharing scenarios, the weight coefficient of the second complexity component (texture richness) is increased; for camera shooting scenarios, the weight coefficients of the first complexity component (motion vector amplitude) and the third complexity component (inter-frame variation) are increased.

[0080] For example, at the same moment the finance manager shares the screen to display the project budget sheet, the system simultaneously acquires the video frame data at that moment. Calculations show that the average pixel-level motion vector amplitude of the current video frame relative to the previous video frame is 0.8 (relatively small motion), the contrast ratio of the gray-level co-occurrence matrix is ​​120, and the entropy value is 7.2. Weighted calculations yield the second complexity component. =0.55×120+0.45×7.2=69.24 (rich texture, due to the large amount of data and table lines in the budget table), the inter-frame luminance change is 2.1, the chrominance change U is 0.8, and the chrominance change V is 0.6. The third complexity component is obtained by weighted calculation. =0.7×2.1+0.15×0.8+0.15×0.6=1.68, and finally the physical complexity of the screen at this moment is approximately 0.45×0.8 + 0.35×69.24 + 0.20×1.68 = 24.94, which belongs to a typical document display scene with low motion and high texture.

[0081] S4: Input the semantic value score of the meeting and the physical complexity of the screen into the preset two-factor coding decision table.

[0082] Preferably, the two-factor coding decision table includes at least the following mapping rules:

[0083] When the semantic value score of the meeting is higher than the first semantic threshold and the physical complexity of the picture is lower than the first complexity threshold, the quantization parameter in the encoding parameters is adjusted to the first quantization parameter value, and the I-frame interval in the encoding parameters is adjusted to the first interval value, wherein the first quantization parameter value is lower than the normal quantization parameter value, and the first interval value is shorter than the normal I-frame interval.

[0084] When the semantic value score of the meeting is lower than the second semantic threshold and the physical complexity of the picture is higher than the second complexity threshold, the quantization parameter in the encoding parameters is adjusted to the second quantization parameter value, and the I-frame interval in the encoding parameters is adjusted to the second interval value, wherein the second quantization parameter value is higher than the normal quantization parameter value, and the second interval value is longer than the normal I-frame interval.

[0085] Among them, the first semantic threshold is higher than the second semantic threshold, and the first complexity threshold is lower than the second complexity threshold.

[0086] Furthermore, the two-factor coding decision table is used to output corresponding coding parameter adjustment instructions based on the input meeting semantic value score and the physical complexity of the image.

[0087] Specifically, the two-factor coding decision table defines the mapping relationship between the semantic value score range and the physical complexity range and the coding parameters.

[0088] Preferably, the two-factor coding decision table also includes intermediate mapping rules between the two extreme cases mentioned above:

[0089] When the semantic value score of the meeting is between the second semantic threshold and the first semantic threshold, and the physical complexity of the picture is between the second complexity threshold and the first complexity threshold, the encoding parameters maintain the normal quantization parameter values ​​and the normal I-frame interval;

[0090] When the semantic value score of the meeting is higher than the first semantic threshold and the physical complexity of the picture is higher than the second complexity threshold, the quantization parameter is adjusted to the third quantization parameter value and the I-frame interval is adjusted to the third interval value. The third quantization parameter value is between the first quantization parameter value and the normal quantization parameter value, and the third interval value is between the first interval value and the normal I-frame interval.

[0091] When the semantic value score of the meeting is lower than the second semantic threshold and the physical complexity of the image is lower than the first complexity threshold, the quantization parameter is adjusted to the fourth quantization parameter value and the I-frame interval is adjusted to the fourth interval value. The fourth quantization parameter value is between the second quantization parameter value and the normal quantization parameter value, and the fourth interval value is between the second interval value and the normal I-frame interval.

[0092] S5: Based on the output of the two-factor coding decision table, adjust the coding parameters of the video encoder in real time. The coding parameters include at least the quantization parameters and the I-frame interval.

[0093] Specifically, the quantization parameter is adjusted by modifying the quantization parameter (QP) value of the video encoder. The smaller the quantization parameter value, the higher the quantization accuracy, the better the image quality, and the higher the bitrate; the larger the quantization parameter value, the lower the quantization accuracy, the worse the image quality, and the lower the bitrate. The I-frame interval is adjusted by modifying the group of pictures (GOP) size of the video encoder. The I-frame interval is measured in frames. An I-frame is an intra-coded frame that is encoded independently of other frames. The smaller the I-frame interval, the better the random accessibility of the video, the more stable the image quality, but the higher the bitrate.

[0094] Specifically, the video encoder adopts the H.264, H.265, H.264SVC or H.265SVC standard. The quantization parameters and I-frame interval are dynamically set by calling the video encoder's application programming interface (API), and take effect in real time during the encoding process without restarting the encoder or interrupting the encoding process.

[0095] Specifically, the regular quantization parameter value and the regular I-frame interval are the parameter values ​​of the video encoder under the default configuration. The regular quantization parameter value is preferably 28 to 32 (for the H.264 / H.265 standard), and the regular I-frame interval is preferably 2 to 5 times the frame rate (for example, when the frame rate is 30fps, the regular I-frame interval is 60 to 150 frames, corresponding to one I-frame every 2 to 5 seconds).

[0096] Specifically, the first quantization parameter value is lower than the conventional quantization parameter value, preferably 18 to 24; the second quantization parameter value is higher than the conventional quantization parameter value, preferably 36 to 42; the first interval value is shorter than the conventional I-frame interval, preferably 0.5 to 1 times the frame rate (for example, when the frame rate is 30fps, the first interval value is 15 to 30 frames); the second interval value is longer than the conventional I-frame interval, preferably 8 to 15 times the frame rate (for example, when the frame rate is 30fps, the second interval value is 240 to 450 frames).

[0097] For example, the system inputs the meeting semantic value calculated in step S2 and the image physical complexity calculated in step S3 into a two-factor coding decision table for lookup decision. The decision table determines that the current meeting semantic value score is higher than the first semantic threshold and the image physical complexity is lower than the first complexity threshold, satisfying the mapping condition of "high semantic value + low physical complexity". Therefore, the coding parameters are adjusted: the quantization parameter (QP) of the video encoder (using the H.265 standard) is adjusted from the current normal value of 30 to the first quantization parameter value of 22 (improving quantization accuracy and better image quality), and the I-frame interval is adjusted from the normal value of 90 frames (corresponding to 3 seconds, frame rate of 30fps) to the first interval value of 20 frames (corresponding to about 0.67 seconds, that is, inserting an I-frame every 0.67 seconds), ensuring the ultimate clarity during the budget table display, making the table numbers and text edges in the screen-shared content sharp and distinguishable.

[0098] S6: During the encoding process, the semantic value score of the meeting is associated with the corresponding video frame timestamp to generate semantic value timeline metadata, and the semantic value timeline metadata is embedded in the container format of the recorded file.

[0099] Specifically, embedding semantic value timeline metadata into the container format of the recorded file includes:

[0100] Define custom metadata fields in the container format of the recorded file;

[0101] Write semantic value timeline metadata into a custom metadata field in timestamp order. Each metadata record must contain at least timestamp information and the corresponding meeting semantic value score.

[0102] When the encoding parameters are adjusted, the encoding parameter change event is synchronously recorded in the custom metadata field. The encoding parameter change event includes at least the change timestamp, the encoding parameter value before the change, and the encoding parameter value after the change.

[0103] Preferably, the container format for the recorded file is MP4, MOV, MKV, or FLV, and the custom metadata field is embedded in the user data area of ​​the container format or in a custom atom (UUID atom); for MP4 format, the custom metadata field is embedded in a moov atom or an udta atom.

[0104] Specifically, when the video encoder is H.264 or H.265, the semantic value timeline metadata is embedded in the video bitstream through supplementary enhancement information (SEI) messages. The SEI message carries the semantic value score and the corresponding timestamp information, and is transmitted and stored synchronously with the video frames. The SEI message adopts the User Data Unregistered type, and its payload contains the binary encoding of the UUID and semantic value timeline metadata.

[0105] Furthermore, the semantic value timeline metadata is sampled and recorded at a preset time interval (preferably per second or per frame). The format of each metadata record is: (timestamp, semantic value score, encoding parameter status identifier). The encoding parameter status identifier is used to indicate whether the encoding parameter at the current moment is the adjusted parameter value.

[0106] Specifically, semantic value timeline metadata is generated in real time and written to custom metadata fields during the recording process. When the recording ends due to network interruption, system failure, or user-initiated stop, the generated metadata is saved together with the encoded audio and video data, and the ungenerated metadata is not generated.

[0107] For example, while adjusting the encoding parameters, the system associates the semantic value score S(t) = 2.464 at that moment with the timestamp of the current video frame (e.g., 18 minutes and 35 seconds after the start of the meeting) to generate a semantic value timeline metadata record. The format is (timestamp: 00:18:35, semantic value score: 2.464, encoding parameter status identifier: adjusted to high value mode). This metadata is embedded in the H.265 video stream through a Supplemental Enhancement Information (SEI) message of type User Data Unregistered and stored synchronously with the corresponding video frame. At the same time, the system defines a metadata field in the udta atom of the MP4 recording file and writes all semantic value score records into this field in the order of timestamps. When the encoding parameters are adjusted again (e.g., when the discussion moves to a non-critical stage), the system synchronously records the encoding parameter change event, including the change timestamp (e.g., 00:25:12), the quantization parameter 22 before the change, and the quantization parameter 30 after the change, forming a complete encoding parameter change log.

[0108] S7: After recording is completed, in response to the user's index generation command, the semantic value timeline metadata in the recording file is directly called. Based on the metadata, the time intervals with semantic value scores higher than the preset value threshold are selected. Only the audio data within the selected time intervals are transcribed and extracted to generate a structured meeting index file.

[0109] Specifically, generating a structured meeting index file includes:

[0110] Based on the semantic value timeline metadata, the time intervals in which the semantic value score of the meeting is higher than the preset value threshold are selected, and a list of high-value time periods is generated.

[0111] The audio data within each time interval of the high-value time period list is transcribed into speech to generate corresponding transcribed text fragments.

[0112] Extract summaries from each transcribed text fragment to generate content summaries for each time interval;

[0113] The list of high-value time periods, the transcribed text fragments corresponding to each time period, and the content summary are linked and stored to generate a structured meeting index file;

[0114] The structured meeting index file also includes start timestamps, end timestamps, and duration information for each high-value time period.

[0115] Preferably, the preset value threshold is set by the system default or by the user. When the preset value threshold is set by the user, the higher the threshold, the fewer high-value time periods are selected and the more concise the index file is; the lower the threshold, the more high-value time periods are selected and the more detailed the index file is.

[0116] Specifically, the summary extraction employs either an extractive summarization method or a generative summarization method: the extractive summarization method calculates the weights of sentences in each transcribed text segment (based on term frequency-inverse document frequency (TF-IDF), TextRank, or BERT sentence embedding similarity) and selects the sentences with the highest weights to form a summary; the generative summarization method uses a Transformer-based pre-trained language model (such as BART, T5, or GPT series models) to understand and rewrite the transcribed text segments, generating coherent summary text.

[0117] Furthermore, the structured meeting index file is in JSON, XML, or HTML format. The index file is stored in association with the recording file. Users can view the index file through the meeting management interface or player interface, and can jump to the corresponding playback position in the recording file by clicking on the high-value time period or transcribed text fragment in the index file.

[0118] For example, after the meeting, the project manager clicks the "Generate Meeting Index" command. The system directly calls the semantic value timeline metadata embedded in the MP4 recording file, sets a preset value threshold of 1.5, and quickly filters out all time intervals with a semantic value score higher than 1.5 throughout the meeting, generating a list of high-value time periods. These include intervals such as "00:18:30~00:22:15" (budget discussion stage) and "00:28:00~00:35:20" (technical solution review stage). The system only performs targeted speech-to-text transcription and summary extraction on the audio data within these selected high-value time intervals (approximately 45% of the total meeting time), using a BART-based generation method. The abstracting model generates summaries from the transcribed text of each interval. The final output structured meeting index file is in JSON format, containing the start and end timestamps, the complete transcribed text, and a content summary for each interval (e.g., "Budget Discussion Interval Summary: The finance manager reported that the total budget for Project X is 5 million yuan, with funding sources including 3 million yuan of self-owned funds and 2 million yuan of external financing. The investment return period is approximately 18 months, and the participants discussed cost control"). The project manager can view this index file through the meeting management interface and click on any time period in the summary to automatically jump to the corresponding playback position in the recording file, quickly obtaining all the core content without having to watch the entire 40-minute meeting recording one by one.

[0119] In summary, this invention calculates the semantic value score of each time period of a meeting through real-time speech transcription and semantic analysis, and constructs a two-factor coding decision table based on the physical complexity of the video. It then implements differentiated coding parameter adjustments for meeting content with different semantic values ​​and physical complexities, tilting storage space towards high-value content. This preserves more effective information and ensures the quality of key segments within the same storage capacity. During the encoding process, semantic value timeline metadata is generated synchronously and embedded into the recording file. Later, during index generation, this metadata is directly called to filter high-value time periods, and targeted speech transcription and summary extraction are performed only on the selected intervals. This avoids redundant analysis of the entire audio data, significantly improving index generation efficiency and achieving a collaborative closed loop of "coding as annotation." The final generated structured index file allows users to quickly locate and jump to the core content of the meeting, reducing retrieval costs. Furthermore, the configurability of the keyword library allows the solution to flexibly adapt to various industry scenarios such as enterprise business, telemedicine, distance education, and government finance. Overall, it achieves a complete technical effect chain from recording and encoding optimization to efficient index generation and convenient user retrieval.

[0120] Example 2, an embodiment of the present invention, provides a cloud server-based video conferencing system, comprising: an audio acquisition module, used to acquire real-time audio stream data during a video conference, perform speech activity detection and real-time speech transcription on the audio stream data, and generate a time-continuous transcribed text sequence; a semantic analysis module, used to perform real-time semantic analysis on the transcribed text sequence, extract target keywords from a preset keyword library, and calculate the meeting semantic value score corresponding to the current moment based on the weight value and frequency of occurrence of the target keywords; a video acquisition module, used to synchronously acquire video frame data at the current moment and calculate the physical complexity of the video frame data; and an encoding decision module, used to combine the meeting semantic value score and the physical complexity of the video frame data. The system includes a complexity input based on a pre-defined two-factor coding decision table; a parameter adjustment module to adjust the video encoder's encoding parameters in real time based on the output of the two-factor coding decision table; a metadata generation module to associate the meeting's semantic value score with the corresponding video frame timestamp during the encoding process, generating semantic value timeline metadata, and embedding the semantic value timeline metadata into the container format of the recorded file; and an index generation module that, after recording, responds to the user's index generation command by directly calling the semantic value timeline metadata in the recorded file, filtering out time intervals with semantic value scores higher than a preset value threshold based on the metadata, and performing speech transcription and summary extraction only on the audio data within the filtered time intervals to generate a structured meeting index file.

[0121] Example 3 is an embodiment of the present invention, which differs from the previous embodiment in that:

[0122] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0124] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0125] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0126] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A video conferencing method based on a cloud server, characterized in that: include: During the video conference, the audio stream data of the conference is acquired in real time, and the audio stream data is subjected to voice activity detection and real-time speech transcription to generate a time-continuous transcribed text sequence. Real-time semantic analysis is performed on the transcribed text sequence to extract target keywords from a preset keyword library, and the semantic value score of the meeting at the current moment is calculated based on the weight value and frequency of occurrence of the target keywords. Simultaneously acquire video frame data at the current moment and calculate the physical complexity of the video frame data; The semantic value score of the meeting and the physical complexity of the screen are input into a preset two-factor coding decision table; Based on the output of the two-factor coding decision table, the coding parameters of the video encoder are adjusted in real time, and the coding parameters include at least the quantization parameters and the I-frame interval; During the encoding process, the semantic value score of the meeting is associated with the corresponding video frame timestamp to generate semantic value timeline metadata, and the semantic value timeline metadata is embedded in the container format of the recorded file; After recording is completed, in response to the user's index generation command, the semantic value timeline metadata in the recording file is directly called. Based on the metadata, time intervals with semantic value scores higher than a preset value threshold are selected. Speech transcription and summary extraction are performed only on the audio data within the selected time intervals to generate a structured meeting index file.

2. The video conferencing method based on a cloud server as described in claim 1, characterized in that: The physical complexity of the image includes at least the magnitude of the image motion vector, the richness of texture, and the amount of variation between frames.

3. The video conferencing method based on a cloud server as described in claim 1, characterized in that: The two-factor coding decision table includes at least the following mapping rules: When the meeting semantic value score is higher than the first semantic threshold and the physical complexity of the picture is lower than the first complexity threshold, the quantization parameter in the encoding parameters is adjusted to the first quantization parameter value, and the I-frame interval in the encoding parameters is adjusted to the first interval value, wherein the first quantization parameter value is lower than the normal quantization parameter value, and the first interval value is shorter than the normal I-frame interval. When the meeting semantic value score is lower than the second semantic threshold and the physical complexity of the picture is higher than the second complexity threshold, the quantization parameter in the encoding parameters is adjusted to the second quantization parameter value, and the I-frame interval in the encoding parameters is adjusted to the second interval value, wherein the second quantization parameter value is higher than the normal quantization parameter value, and the second interval value is longer than the normal I-frame interval. Wherein, the first semantic threshold is higher than the second semantic threshold, and the first complexity threshold is lower than the second complexity threshold.

4. The video conferencing method based on a cloud server as described in claim 1, characterized in that: The methods for calculating the physical complexity of the image include: Extract the pixel-level motion vector field of the current video frame relative to the previous video frame, and calculate the mean amplitude of the motion vector field as the first complexity component. Extract texture features from the current video frame and calculate the contrast and entropy values ​​of the gray-level co-occurrence matrix as the second complexity components; The inter-frame luminance and chrominance changes between the current video frame and the previous video frame are calculated as the third complexity component. The first complexity component, the second complexity component, and the third complexity component are weighted and summed to obtain the physical complexity of the image.

5. The video conferencing method based on a cloud server as described in claim 1, characterized in that: The calculation method for the semantic value score of the meeting includes: A keyword library is pre-built, and a corresponding basic weight value is configured for each keyword in the keyword library; The transcribed text sequence is processed by real-time word segmentation, and the segmentation results are matched with the keyword database to extract the target keywords that are successfully matched and their occurrence positions. The semantic value score of the meeting at the current moment is calculated based on the basic weight value of each target keyword, the frequency of occurrence within a preset time window, and the semantic relevance between the target keywords. When multiple semantically related keywords appear together within the preset time window, the meeting semantic value score receives an additional collaborative weighted value.

6. The video conferencing method based on a cloud server as described in claim 1, characterized in that: The step of embedding the semantic value timeline metadata into the container format of the recording file specifically includes: Define custom metadata fields in the container format of the recorded file; The semantic value timeline metadata is written into the custom metadata field in timestamp order, and each metadata record contains at least timestamp information and the corresponding meeting semantic value score; When the encoding parameters are adjusted, the encoding parameter change event is synchronously recorded in the custom metadata field. The encoding parameter change event includes at least the change timestamp, the encoding parameter value before the change, and the encoding parameter value after the change.

7. The video conferencing method based on a cloud server as described in claim 1, characterized in that: The generation of the structured meeting index file specifically includes: Based on the semantic value timeline metadata, time intervals with meeting semantic value scores higher than a preset value threshold are selected, and a list of high-value time periods is generated. The audio data within each time interval of the high-value time period list is transcribed into speech to generate corresponding transcribed text fragments. Summarize each of the transcribed text fragments to generate content summaries for each time interval; The high-value time period list, the transcribed text fragments corresponding to each time period, and the content summary are associated and stored to generate the structured meeting index file; The structured meeting index file also includes start timestamps, end timestamps, and duration information for each high-value time period.

8. A cloud server-based video conferencing system, based on the cloud server-based video conferencing method according to any one of claims 1 to 7, characterized in that: include, The audio acquisition module is used to acquire the audio stream data of the video conference in real time during the video conference, perform voice activity detection and real-time speech transcription on the audio stream data, and generate a time-continuous transcribed text sequence. The semantic analysis module is used to perform real-time semantic analysis on the transcribed text sequence, extract target keywords from the preset keyword library, and calculate the meeting semantic value score corresponding to the current moment based on the weight value and frequency of the target keywords. The video acquisition module is used to synchronously acquire video frame data at the current moment and calculate the physical complexity of the video frame data. The coding decision module is used to input the meeting semantic value score and the physical complexity of the screen into a preset two-factor coding decision table; The parameter adjustment module is used to adjust the encoding parameters of the video encoder in real time based on the output of the two-factor encoding decision table. The metadata generation module is used to associate the semantic value score of the meeting with the corresponding video frame timestamp during the encoding process, generate semantic value timeline metadata, and embed the semantic value timeline metadata into the container format of the recorded file; The index generation module, after recording is completed, responds to the user's index generation command by directly calling the semantic value timeline metadata in the recording file, filters out the time intervals with semantic value scores higher than the preset value threshold based on the metadata, and performs speech transcription and summary extraction only on the audio data within the filtered time intervals to generate a structured conference index file.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the video conferencing method based on a cloud server as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the video conferencing method based on a cloud server as described in any one of claims 1 to 7.

Images