Intelligent audio and video mixed transmission strategy and system

Abstract

The invention discloses an intelligent audio and video mixed transmission strategy and system. In the monitoring period, the system captures user interaction and audio and video data in real time, and video and audio streams are separated. A video stream is subjected to frame extraction and parameter comparison to generate a first anomaly assessment. And positioning a picture by using the interactive data, performing feature and semantic analysis, and comparing database information to obtain second anomaly assessment. And in combination with the two evaluation results, positioning an abnormal condition of the audio data, and optimizing according to a preset strategy to generate an optimized audio. According to the invention, abnormal analysis and reasonable mixed transmission are effectively carried out on the audio and video data, the transmission optimization effect of the system platform on the multi-terminal audio and video data is effectively realized, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of audio and video data analysis, and more specifically, to an intelligent audio and video hybrid transmission strategy and system. Background Art

[0002] Multi-screen system platforms are becoming increasingly popular in the fields of business, education, and entertainment. However, due to hardware failures, software conflicts, or network delays, video and audio data anomalies occur frequently, affecting user experience and business operations. Existing audio and video anomaly detection methods often rely on manual monitoring or feedback from a single device, and have shortcomings such as slow response, high false alarm rate, and inability to adapt to dynamic changes. They are also unable to quickly, in real time, and accurately evaluate and adjust the images, resulting in a poor user experience. Therefore, the development of an efficient intelligent audio and video hybrid data analysis strategy and related methods is an important issue that needs to be solved urgently. Summary of the invention

[0003] The present invention overcomes the defects of the prior art and proposes an intelligent audio and video hybrid transmission strategy and system.

[0004] The first aspect of the present invention provides an intelligent audio and video hybrid transmission strategy, including: In one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data; Extracting video frames from real-time video data to obtain picture data, analyzing and comparing whether basic parameters of the picture data are consistent with preset picture parameters, and generating a first abnormality assessment result; Based on the user interaction data, the interaction time node and the screen interaction type information are obtained, and the corresponding screen is intercepted from the screen data based on the interaction time node to form the detection screen data, and the feature extraction, image recognition and text recognition are performed on the detection screen data to generate the semantic recognition result; The real-time image comparison semantic information is filtered out from the image semantic database through user interaction data, similarity is calculated between the semantic recognition result and the real-time comparison semantic information, image anomaly analysis is performed through threshold comparison, and a second anomaly assessment result based on semantic analysis is generated; Acquire abnormal time node and abnormal condition information based on the first abnormality assessment result and the second abnormality assessment result, extract corresponding audio segment data from real-time audio and video data based on the abnormal time node, optimize the audio segment data based on the abnormal condition information through a preset audio processing strategy, and generate optimized audio segment data; The first abnormality assessment result and the second abnormality assessment result are sent to the user terminal device, the abnormal state of the picture is adjusted in real time by the user terminal device, and optimized real-time video data is generated, and the real-time video data is mixed with the optimized audio segment data to obtain the second audio and video data.

[0005] In this solution, within one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data, specifically: Through the system platform, count the number of currently connected user terminals and analyze the current audio and video data transmission volume; Set monitoring period based on the number of user terminals and the volume of audio and video data transmission; In a monitoring cycle, user interaction data and real-time audio and video data are obtained in real time through user terminals; The real-time audio and video data is analyzed for data format and audio and video are separated to obtain real-time video data and real-time audio data.

[0006] In this solution, the real-time video data is extracted by video frame extraction to obtain picture data, and the basic parameters of the picture data are analyzed and compared to see whether they are consistent with the preset picture parameters, and a first abnormality assessment result is generated, specifically: Extract key frames from real-time video data to obtain an image set; Perform image enhancement, smoothing and standardization preprocessing on the image set to obtain picture data; Counting picture parameter information based on picture data, the parameter information includes resolution, picture ratio, frame rate, and obtaining basic parameters; The basic parameters are compared with the preset screen parameters. If there are differences, a first abnormality assessment result is generated based on the parameter differences. At the same time, the abnormal time node is recorded in the first abnormality assessment result.

[0007] In this solution, the interaction time node and screen interaction type information are obtained based on the user interaction data, the corresponding screen is captured from the screen data based on the interaction time node to form the detection screen data, and the detection screen data is subjected to feature extraction, image recognition and text recognition to generate the semantic recognition result, specifically: Obtain interaction instructions and interaction time nodes through user interaction data; Set the pre-interaction time node and the post-interaction time node based on the interaction time node; Intercepting corresponding picture information from the picture data through the pre-interaction time node and the post-interaction time node to obtain the first image data and the second image data; Constructing an image recognition model based on CNN, performing image segmentation and screen image recognition on the first image data and the second image data respectively, and obtaining a screen recognition result; Converting the picture recognition result into first semantic information; Based on the OCR technology, the first image data and the second image data are pre-processed by grayscale conversion and noise reduction, and the characters are recognized and extracted by the OCR system to obtain the second semantic information; The first semantic information and the second semantic information are integrated to form a semantic recognition result.

[0008] In this solution, the real-time image comparison semantic information is filtered out from the image semantic database through user interaction data, similarity is calculated between the semantic recognition result and the real-time comparison semantic information, and image anomaly analysis is performed through threshold comparison to generate a second anomaly assessment result based on semantic analysis, specifically: Obtaining interaction instructions based on user interaction data; Based on the interactive instructions, the corresponding real-time comparison semantic information of the screen is filtered and obtained from the system database; The first semantic information and the second semantic information in the semantic recognition result are used to perform similarity calculation with the real-time comparison semantic information. During the similarity calculation process, the semantic information similarity between the texts is calculated by using the bag-of-words model and the cosine similarity, and finally the semantic similarity is obtained; The semantic similarity is compared with the familiarity threshold, and anomaly assessment is performed through comparative analysis. At the same time, the abnormal time node is recorded and a second anomaly assessment result is generated.

[0009] In this solution, the abnormal time node and abnormal condition information are obtained based on the first abnormal evaluation result and the second abnormal evaluation result, the corresponding audio segment data is extracted from the real-time audio and video data based on the abnormal time node, and the audio segment data is optimized based on the abnormal condition information through a preset audio processing strategy to generate optimized audio segment data, specifically: Obtain abnormal time node and abnormal condition information through the first abnormal evaluation result and the second abnormal evaluation result; Based on the abnormal time nodes, data segmentation and extraction are performed from real-time audio and video data to obtain corresponding audio segment data; By presetting the audio processing strategy, the processing strategy is matched based on the abnormal condition information, and the audio segment data is optimized according to the matching results to generate optimized audio segment data.

[0010] In this solution, the first abnormality evaluation result and the second abnormality evaluation result are sent to the user terminal device, the abnormal state of the picture is adjusted in real time by the user terminal device, and optimized real-time video data is generated, and the real-time video data is mixed with the optimized audio segment data to obtain the second audio and video data, specifically: Sending the first abnormality assessment result and the second abnormality assessment result to a user terminal device; Analyze abnormal conditions through user terminal devices, and based on preset adjustment instructions, select corresponding adjustment instructions for abnormal conditions of different screens for real-time adjustment; The adjusted video data is recorded to obtain optimized video data, and the optimized video data is mixed with the optimized audio segment data to obtain second audio and video data.

[0011] The second aspect of the present invention further provides an intelligent audio and video hybrid transmission system, the system comprising: a memory, a processor, the memory comprising an intelligent audio and video hybrid transmission program, the intelligent audio and video hybrid transmission program when executed by the processor implements the following steps: In one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data; Extracting video frames from real-time video data to obtain picture data, analyzing and comparing whether basic parameters of the picture data are consistent with preset picture parameters, and generating a first abnormality assessment result; Based on the user interaction data, the interaction time node and the screen interaction type information are obtained, and the corresponding screen is intercepted from the screen data based on the interaction time node to form the detection screen data, and the feature extraction, image recognition and text recognition are performed on the detection screen data to generate the semantic recognition result; The real-time image comparison semantic information is filtered out from the image semantic database through user interaction data, similarity is calculated between the semantic recognition result and the real-time comparison semantic information, image anomaly analysis is performed through threshold comparison, and a second anomaly assessment result based on semantic analysis is generated; Acquire abnormal time node and abnormal condition information based on the first abnormality assessment result and the second abnormality assessment result, extract corresponding audio segment data from real-time audio and video data based on the abnormal time node, optimize the audio segment data based on the abnormal condition information through a preset audio processing strategy, and generate optimized audio segment data; The first abnormality assessment result and the second abnormality assessment result are sent to the user terminal device, the abnormal state of the picture is adjusted in real time by the user terminal device, and optimized real-time video data is generated, and the real-time video data is mixed with the optimized audio segment data to obtain the second audio and video data.

[0012] In this solution, within one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data, specifically: Through the system platform, count the number of currently connected user terminals and analyze the current audio and video data transmission volume; Set monitoring period based on the number of user terminals and the volume of audio and video data transmission; In a monitoring cycle, user interaction data and real-time audio and video data are obtained in real time through user terminals; The real-time audio and video data is analyzed for data format and audio and video are separated to obtain real-time video data and real-time audio data.

[0013] The third aspect of the present invention also provides a computer-readable storage medium, which includes an intelligent audio and video hybrid transmission program. When the intelligent audio and video hybrid transmission program is executed by a processor, the steps of the intelligent audio and video hybrid transmission strategy as described in any one of the above items are implemented.

[0014] The present invention discloses an intelligent audio and video hybrid transmission strategy and system. During the monitoring cycle, the system captures user interactions and audio and video data in real time, and separates the video and audio streams. The video stream undergoes frame extraction and parameter comparison to generate a first anomaly assessment. The interactive data is used to locate the screen, perform feature and semantic analysis, and compare database information to obtain a second anomaly assessment. Combining the two evaluation results, the audio data anomaly is located and optimized according to a preset strategy to generate optimized audio. Through the present invention, the audio and video data can be effectively analyzed for anomalies and reasonably mixed for transmission, effectively realizing the system platform's transmission optimization effect on multi-terminal audio and video data, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of an intelligent audio and video hybrid transmission strategy of the present invention is shown; Figure 2 A block diagram of an intelligent audio and video hybrid transmission system of the present invention is shown. DETAILED DESCRIPTION

[0016] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0017] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0018] Figure 1 A flow chart of an intelligent audio and video hybrid transmission strategy of the present invention is shown.

[0019] like Figure 1 As shown, the first aspect of the present invention provides an intelligent audio and video hybrid transmission strategy, including: S102, within a monitoring period, real-time acquisition of user interaction data and real-time audio and video data, and data separation of the real-time audio and video data to obtain real-time video data and real-time audio data; S104, extracting video frames from real-time video data to obtain picture data, analyzing and comparing whether basic parameters of the picture data are consistent with preset picture parameters, and generating a first abnormality assessment result; S106, acquiring interaction time nodes and screen interaction type information based on the user interaction data, capturing corresponding screens from the screen data based on the interaction time nodes to form detection screen data, performing feature extraction, image recognition and text recognition on the detection screen data to generate semantic recognition results; S108, filtering out real-time image comparison semantic information from an image semantic database through user interaction data, performing similarity calculation between the semantic recognition result and the real-time comparison semantic information, performing image anomaly analysis through threshold comparison, and generating a second anomaly assessment result based on semantic analysis; S110, obtaining an abnormal time node and abnormal condition information based on the first abnormality assessment result and the second abnormality assessment result, extracting corresponding audio segment data from the real-time audio and video data based on the abnormal time node, optimizing the audio segment data based on the abnormal condition information through a preset audio processing strategy, and generating optimized audio segment data; S112, sending the first abnormality assessment result and the second abnormality assessment result to the user terminal device, adjusting the abnormal state of the picture in real time through the user terminal device, and generating optimized real-time video data, and mixing the real-time video data with the optimized audio segment data to obtain the second audio and video data.

[0020] The present invention can solve the limitations of existing audio and video equipment in content transmission and playback, and can flexibly transmit and strategically play video content and audio content through abnormal analysis and response strategies.

[0021] The system includes an audio and video transmission module and an audio and video strategy playback module. The audio and video transmission module is responsible for transmitting the audio and video content of a device to multiple user devices (or terminal devices of each platform) in real time and stably. The module uses preset audio encoding technology and network transmission protocol to ensure high quality and low latency of audio transmission. At the same time, the module also has the function of automatic recognition and classification of audio signals to distinguish the device's own audio from external transmission audio. The audio and video strategy playback module is responsible for recording abnormal conditions, strategy application and historical processing strategy.

[0022] The mixed audio and video data can be recorded and replace the original distorted abnormal picture data and distorted audio data. When using the real-time audio and video terminal, such as in monitoring applications, when an abnormal picture appears, the audio and video optimization transmission strategy of the present invention can optimize the distorted abnormal picture and distorted audio data and store them in the monitoring database, replacing the original lossy audio and video data, thereby improving the adaptability of audio and video data transmission optimization.

[0023] In addition, real-time analysis of abnormal situations can restore the original distorted audio and video data in real time. Timely abnormal analysis and data optimization and recovery can improve the authenticity of audio and video data and effectively optimize later data maintenance work, especially in applications based on audio and video monitoring needs.

[0024] According to an embodiment of the present invention, in one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data, specifically: Through the system platform, count the number of currently connected user terminals and analyze the current audio and video data transmission volume; Set monitoring period based on the number of user terminals and the volume of audio and video data transmission; In a monitoring cycle, user interaction data and real-time audio and video data are obtained in real time through user terminals; The real-time audio and video data is analyzed for data format and audio and video are separated to obtain real-time video data and real-time audio data.

[0025] It should be noted that in the present invention, the system platform is the central platform for image monitoring, which is responsible for performing image capture and analysis on different user terminals. The monitoring period is dynamically set based on the amount of audio and video data transmission tasks of the current user terminal, specifically based on the analysis of the number of user terminals and the amount of audio and video data transmission. The larger the amount of tasks, the smaller the monitoring period can be set, thereby reducing the system computing power consumption, and can also be set by the user according to actual conditions.

[0026] The user terminal includes a terminal interaction device and a display device, which is used to display the picture and audio content to the user, such as a mobile terminal, a tablet, a computer, etc. The system platform is connected to the user terminal through the Internet. The system platform is responsible for real-time picture analysis, real-time audio analysis and abnormal monitoring and control of different user terminals.

[0027] According to an embodiment of the present invention, the video frame extraction is performed on the real-time video data to obtain the picture data, analyze and compare whether the basic parameters of the picture data are consistent with the preset picture parameters, and generate a first abnormality assessment result, specifically: Extract key frames from real-time video data to obtain an image set; Perform image enhancement, smoothing and standardization preprocessing on the image set to obtain picture data; Counting picture parameter information based on picture data, the parameter information includes resolution, picture ratio, frame rate, and obtaining basic parameters; The basic parameters are compared with the preset screen parameters. If there are differences, a first abnormality assessment result is generated based on the parameter differences. At the same time, the abnormal time node is recorded in the first abnormality assessment result.

[0028] It should be noted that the first abnormality evaluation result is evaluated based on the picture transmission parameters, which is used to evaluate the transmission loss and effectively reflect the impact of the picture abnormality during the transmission process. The first abnormality evaluation result includes abnormal condition information and the corresponding time node of the abnormality.

[0029] According to an embodiment of the present invention, the acquisition of interaction time nodes and screen interaction type information based on user interaction data, the corresponding screen capture from the screen data based on the interaction time node, the formation of detection screen data, the feature extraction, image recognition and text recognition of the detection screen data, and the generation of semantic recognition results are specifically as follows: Obtain interaction instructions and interaction time nodes through user interaction data; Set the pre-interaction time node and the post-interaction time node based on the interaction time node; Intercepting corresponding picture information from the picture data through the pre-interaction time node and the post-interaction time node to obtain the first image data and the second image data; Constructing an image recognition model based on CNN, performing image segmentation and screen image recognition on the first image data and the second image data respectively, and obtaining a screen recognition result; Converting the picture recognition result into first semantic information; Based on the OCR technology, the first image data and the second image data are pre-processed by grayscale conversion and noise reduction, and the characters are recognized and extracted by the OCR system to obtain the second semantic information; The first semantic information and the second semantic information are integrated to form a semantic recognition result.

[0030] It should be noted that the setting of the time node before and after the interaction is used to analyze whether the changes in the previous and subsequent screens conform to the corresponding interactive procedures. For example, when the user performs operations such as pausing, zooming in, entering a certain interface, and exiting a certain interface, whether the corresponding screen changes accordingly. The image recognition model improves the recognition rate by training historical screen information. The screen recognition results correspond to different results based on different scenarios. For example, in the application process of commercial tablets, commercial conference multi-displays, etc., it is generally to identify the specific operation interface of the software screen, such as the first-level interface and the second-level interface, to determine whether there is a screen abnormality during the interaction process. OCR technology is a text recognition technology that performs text recognition and extraction on image screens. The OCR system extracts and recognizes character features based on the CNN network.

[0031] According to an embodiment of the present invention, the real-time comparison semantic information of the screen is filtered out from the screen semantic database through user interaction data, similarity is calculated between the semantic recognition result and the real-time comparison semantic information, and screen abnormality analysis is performed through threshold comparison to generate a second abnormality assessment result based on semantic analysis, specifically: Obtaining interaction instructions based on user interaction data; Based on the interactive instructions, the corresponding real-time comparison semantic information of the screen is filtered and obtained from the system database; The first semantic information and the second semantic information in the semantic recognition result are used to perform similarity calculation with the real-time comparison semantic information. During the similarity calculation process, the semantic information similarity between the texts is calculated by using the bag-of-words model and the cosine similarity, and finally the semantic similarity is obtained; The semantic similarity is compared with the familiarity threshold, and anomaly assessment is performed through comparative analysis. At the same time, the abnormal time node is recorded and a second anomaly assessment result is generated.

[0032] It should be noted that the real-time picture comparison semantic information includes first and second comparison semantic information, which are compared with the first semantic information and the second semantic information respectively. In the present invention, the first abnormality assessment result performs picture abnormality analysis from the perspective of transmitted data, and the second abnormality assessment result performs picture abnormality analysis from the perspective of interactive abnormality, and the abnormal conditions reflected by the two are different. Abnormal conditions include black screen, white screen, flower screen, green screen, freeze, third-party application dialog box freeze, loading page freeze, etc. The second abnormality assessment result includes abnormal condition information and the corresponding time node when the abnormality occurs.

[0033] According to an embodiment of the present invention, the abnormal time node and abnormal condition information are obtained based on the first abnormal evaluation result and the second abnormal evaluation result, the corresponding audio segment data is extracted from the real-time audio and video data based on the abnormal time node, and the audio segment data is optimized based on the abnormal condition information through a preset audio processing strategy to generate optimized audio segment data, specifically: Obtain abnormal time node and abnormal condition information through the first abnormal evaluation result and the second abnormal evaluation result; Based on the abnormal time nodes, data segmentation and extraction are performed from real-time audio and video data to obtain corresponding audio segment data; By presetting the audio processing strategy, the processing strategy is matched based on the abnormal condition information, and the audio segment data is optimized according to the matching results to generate optimized audio segment data.

[0034] It is worth mentioning that, in the process of audio and video data transmission, generally speaking, it is difficult to evaluate and analyze the abnormal audio data in real time. Therefore, the present invention evaluates and analyzes the abnormal picture in real time. When the picture is abnormal, the corresponding audio data often has corresponding audio distortion, excessive noise, and repetitive playback of a certain audio data, which is specifically caused by the abnormal data. Therefore, the present invention records the abnormal video picture at the time node, intercepts and analyzes the corresponding audio data, and performs corresponding audio shielding, audio noise reduction and other processing to reduce the impact of the audio abnormality. The adjustment process is based on the preset audio processing strategy. Different abnormal conditions correspond to different optimization processing. For example, when the video picture is stuck or stuck, the audio can be shielded. When the picture is blurred and distorted, the audio data often has corresponding distortion, excessive noise, etc. At this time, the corresponding optimization strategy can be audio noise reduction for the audio data. The preset audio processing strategy can be set by the user. The optimized audio segment data is used to replace the original audio segment data.

[0035] According to an embodiment of the present invention, the first abnormality assessment result and the second abnormality assessment result are sent to the user terminal device, the abnormal state of the picture is adjusted in real time by the user terminal device, and optimized real-time video data is generated, and the real-time video data is mixed with the optimized audio segment data to obtain the second audio and video data, specifically: Sending the first abnormality assessment result and the second abnormality assessment result to a user terminal device; Analyze abnormal conditions through user terminal devices, and based on preset adjustment instructions, select corresponding adjustment instructions for abnormal conditions of different screens for real-time adjustment; The adjusted video data is recorded to obtain optimized video data, and the optimized video data is mixed with the optimized audio segment data to obtain second audio and video data.

[0036] It should be noted that the preset adjustment instructions are application adjustment strategies pre-set by the user. For example, if the screen abnormality is a decrease in screen resolution or a blurred screen, the screen can be adjusted by adjusting the transmission line, adding instructions such as adjusting network resource allocation, etc. If the screen abnormality is a freeze in the operating interface, etc., it can be adjusted by sending a refresh instruction.

[0037] Audio and video data mixing includes steps such as audio and video data encoding, time alignment, and encapsulation of synthesized data.

[0038] When performing audio and video, after adjustment, the optimized audio and video data is recorded, replacing the audio and video data before optimization, and stored in the system database.

[0039] In audio and video picture control, there are two methods: real-time control and delayed control. In delayed data control, after optimizing and analyzing to obtain the second audio and video data, the audio and video data of the original abnormal picture is replaced to achieve effective data maintenance of the audio and video data and reduce unnecessary data storage.

[0040] According to an embodiment of the present invention, it also includes: In N monitoring cycles, the user's N audio transmission data is obtained through the system platform; Taking one user as the analysis unit, based on the audio waveform comparison method, the waveform similarity comparison is performed on N audio transmission data. If the similarity is greater than the preset similarity threshold, the user is marked as the user to be analyzed; Analyze all users and record all users to be analyzed to form a user group to be analyzed; In the user group to be analyzed, all audio transmission data of each user is subjected to feature extraction based on frequency, amplitude, and audio waveform to obtain user audio features; Using user audio features as clustering sample data, the user groups to be analyzed are clustered and grouped based on the DBSCAN clustering model to form multiple user groups; During N monitoring cycles, the audio abnormality and corresponding audio abnormality handling strategies of each user group are monitored and recorded to obtain audio abnormality handling record information; Each user group corresponds to an audio exception processing record; Taking a user group as an analysis unit, extracting high-frequency abnormalities in the audio abnormality processing record information to obtain high-frequency abnormality information, and associating the corresponding high-frequency abnormality information with the corresponding processing strategy to obtain the audio abnormality general strategy information; Analyze all user groups, obtain multiple audio anomaly general policy information, and store the multiple audio anomaly general policy information in a system database; In the next monitoring cycle, determine whether the real-time user belongs to the user group to be analyzed. If so, based on the user group to which the real-time user belongs, obtain the general strategy information for audio anomalies from the system database to perform audio anomaly judgment and processing.

[0041] The general response strategy (i.e., the general strategy information for audio anomalies) generated by the present invention can be applied to large-scale audio transmission processing. When the system data processing pressure is greater than the warning threshold, the strategy can be adopted to reduce the system's data analysis pressure while ensuring the audio anomaly processing capability. Especially in large-scale user platforms, it can effectively reduce the system server pressure.

[0042] In the extraction of high-frequency abnormal situations, specifically, the number of occurrences of audio abnormal situations is counted, and the audio abnormal situations that are higher than the preset number value are recorded, and the corresponding processing strategies are recorded to obtain the general strategy information of audio abnormalities. In the all audio transmission data of each user, the all audio transmission data includes the user's N audio transmission data.

[0043] The waveform similarity comparison of the N audio transmission data is performed, that is, taking a certain audio transmission data and the next audio transmission data for audio waveform comparison, the waveform comparison can reflect the similarity of audio features, and the comparison process forms multiple similarity values. In the user group to be analyzed analyzed by the present invention, its N audio transmission data have certain similarities. Therefore, each user in the group has certain audio demand habits and regularity of audio transmission data. For example, in a video on demand or audio on demand platform, users often order similar audio and video data based on a certain interest when using the device terminal. When the corresponding high-frequency abnormal information is analyzed, the corresponding strategy is imported into the database as cache data. In the subsequent real-time audio transmission, the general strategy can be used to handle and judge the exception, reduce the pressure of system analysis data, and improve system stability.

[0044] Figure 2 A block diagram of an intelligent audio and video hybrid transmission system of the present invention is shown.

[0045] The second aspect of the present invention further provides an intelligent audio and video hybrid transmission system 2, which includes: a memory 21 and a processor 22, wherein the memory includes an intelligent audio and video hybrid transmission program, and when the intelligent audio and video hybrid transmission program is executed by the processor, the following steps are implemented: In one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data; Extracting video frames from real-time video data to obtain picture data, analyzing and comparing whether basic parameters of the picture data are consistent with preset picture parameters, and generating a first abnormality assessment result; Based on the user interaction data, the interaction time node and the screen interaction type information are obtained, and the corresponding screen is intercepted from the screen data based on the interaction time node to form the detection screen data, and the feature extraction, image recognition and text recognition are performed on the detection screen data to generate the semantic recognition result; The real-time image comparison semantic information is filtered out from the image semantic database through user interaction data, similarity is calculated between the semantic recognition result and the real-time comparison semantic information, image anomaly analysis is performed through threshold comparison, and a second anomaly assessment result based on semantic analysis is generated; Acquire abnormal time node and abnormal condition information based on the first abnormality assessment result and the second abnormality assessment result, extract corresponding audio segment data from real-time audio and video data based on the abnormal time node, optimize the audio segment data based on the abnormal condition information through a preset audio processing strategy, and generate optimized audio segment data; The first abnormality assessment result and the second abnormality assessment result are sent to the user terminal device, the abnormal state of the picture is adjusted in real time by the user terminal device, and optimized real-time video data is generated, and the real-time video data is mixed with the optimized audio segment data to obtain the second audio and video data.

[0046] The present invention can solve the limitations of existing audio and video equipment in content transmission and playback, and can flexibly transmit and strategically play video content and audio content through abnormal analysis and response strategies.

[0047] The system includes an audio and video transmission module and an audio and video strategy playback module. The audio and video transmission module is responsible for transmitting the audio and video content of a device to multiple user devices (or terminal devices of each platform) in real time and stably. The module uses preset audio encoding technology and network transmission protocol to ensure high quality and low latency of audio transmission. At the same time, the module also has the function of automatic recognition and classification of audio signals to distinguish the device's own audio from external transmission audio. The audio and video strategy playback module is responsible for recording abnormal conditions, strategy application and historical processing strategy.

[0048] The mixed audio and video data can be recorded and replace the original distorted abnormal picture data and distorted audio data. When using the real-time audio and video terminal, such as in monitoring applications, when an abnormal picture appears, the audio and video optimization transmission strategy of the present invention can optimize the distorted abnormal picture and distorted audio data and store them in the monitoring database, replacing the original lossy audio and video data, thereby improving the adaptability of audio and video data transmission optimization.

[0049] In addition, real-time analysis of abnormal situations can restore the original distorted audio and video data in real time. Timely abnormal analysis and data optimization and recovery can improve the authenticity of audio and video data and effectively optimize later data maintenance work, especially in applications based on audio and video monitoring needs.

[0050] According to an embodiment of the present invention, in one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data, specifically: Through the system platform, count the number of currently connected user terminals and analyze the current audio and video data transmission volume; Set monitoring period based on the number of user terminals and the volume of audio and video data transmission; In a monitoring cycle, user interaction data and real-time audio and video data are obtained in real time through user terminals; The real-time audio and video data is analyzed for data format and audio and video are separated to obtain real-time video data and real-time audio data.

[0051] It should be noted that in the present invention, the system platform is the central platform for image monitoring, which is responsible for performing image capture and analysis on different user terminals. The monitoring period is dynamically set based on the amount of audio and video data transmission tasks of the current user terminal, specifically based on the analysis of the number of user terminals and the amount of audio and video data transmission. The larger the amount of tasks, the smaller the monitoring period can be set, thereby reducing the system computing power consumption, and can also be set by the user according to actual conditions.

[0052] The user terminal includes a terminal interaction device and a display device, which is used to display the picture and audio content to the user, such as a mobile terminal, a tablet, a computer, etc. The system platform is connected to the user terminal through the Internet. The system platform is responsible for real-time picture analysis, real-time audio analysis and abnormal monitoring and control of different user terminals.

[0053] According to an embodiment of the present invention, the video frame extraction is performed on the real-time video data to obtain the picture data, analyze and compare whether the basic parameters of the picture data are consistent with the preset picture parameters, and generate a first abnormality assessment result, specifically: Extract key frames from real-time video data to obtain an image set; Perform image enhancement, smoothing and standardization preprocessing on the image set to obtain picture data; Counting picture parameter information based on picture data, the parameter information includes resolution, picture ratio, frame rate, and obtaining basic parameters; The basic parameters are compared with the preset screen parameters. If there are differences, a first abnormality assessment result is generated based on the parameter differences. At the same time, the abnormal time node is recorded in the first abnormality assessment result.

[0054] It should be noted that the first abnormality evaluation result is evaluated based on the picture transmission parameters, which is used to evaluate the transmission loss and effectively reflect the impact of the picture abnormality during the transmission process. The first abnormality evaluation result includes abnormal condition information and the corresponding time node of the abnormality.

[0055] According to an embodiment of the present invention, the acquisition of interaction time nodes and screen interaction type information based on user interaction data, the corresponding screen capture from the screen data based on the interaction time node, the formation of detection screen data, the feature extraction, image recognition and text recognition of the detection screen data, and the generation of semantic recognition results are specifically as follows: Obtain interaction instructions and interaction time nodes through user interaction data; Set the pre-interaction time node and the post-interaction time node based on the interaction time node; Intercepting corresponding picture information from the picture data through the pre-interaction time node and the post-interaction time node to obtain the first image data and the second image data; Constructing an image recognition model based on CNN, performing image segmentation and screen image recognition on the first image data and the second image data respectively, and obtaining a screen recognition result; Converting the picture recognition result into first semantic information; Based on the OCR technology, the first image data and the second image data are pre-processed by grayscale conversion and noise reduction, and the characters are recognized and extracted by the OCR system to obtain the second semantic information; The first semantic information and the second semantic information are integrated to form a semantic recognition result.

[0056] It should be noted that the setting of the time node before and after the interaction is used to analyze whether the changes in the previous and subsequent screens conform to the corresponding interactive procedures. For example, when the user performs operations such as pausing, zooming in, entering a certain interface, and exiting a certain interface, whether the corresponding screen changes accordingly. The image recognition model improves the recognition rate by training historical screen information. The screen recognition results correspond to different results based on different scenarios. For example, in the application process of commercial tablets, commercial conference multi-displays, etc., it is generally to identify the specific operation interface of the software screen, such as the first-level interface and the second-level interface, to determine whether there is a screen abnormality during the interaction process. OCR technology is a text recognition technology that performs text recognition and extraction on image screens. The OCR system extracts and recognizes character features based on the CNN network.

[0057] According to an embodiment of the present invention, the real-time comparison semantic information of the screen is filtered out from the screen semantic database through user interaction data, similarity is calculated between the semantic recognition result and the real-time comparison semantic information, and screen abnormality analysis is performed through threshold comparison to generate a second abnormality assessment result based on semantic analysis, specifically: Obtaining interaction instructions based on user interaction data; Based on the interactive instructions, the corresponding real-time comparison semantic information of the screen is filtered and obtained from the system database; The first semantic information and the second semantic information in the semantic recognition result are used to perform similarity calculation with the real-time comparison semantic information. During the similarity calculation process, the semantic information similarity between the texts is calculated by using the bag-of-words model and the cosine similarity, and finally the semantic similarity is obtained; The semantic similarity is compared with the familiarity threshold, and anomaly assessment is performed through comparative analysis. At the same time, the abnormal time node is recorded and a second anomaly assessment result is generated.

[0058] It should be noted that the real-time picture comparison semantic information includes first and second comparison semantic information, which are compared with the first semantic information and the second semantic information respectively. In the present invention, the first abnormality assessment result performs picture abnormality analysis from the perspective of transmitted data, and the second abnormality assessment result performs picture abnormality analysis from the perspective of interactive abnormality, and the abnormal conditions reflected by the two are different. Abnormal conditions include black screen, white screen, flower screen, green screen, freeze, third-party application dialog box freeze, loading page freeze, etc. The second abnormality assessment result includes abnormal condition information and the corresponding time node when the abnormality occurs.

[0059] According to an embodiment of the present invention, the abnormal time node and abnormal condition information are obtained based on the first abnormal evaluation result and the second abnormal evaluation result, the corresponding audio segment data is extracted from the real-time audio and video data based on the abnormal time node, and the audio segment data is optimized based on the abnormal condition information through a preset audio processing strategy to generate optimized audio segment data, specifically: Obtain abnormal time node and abnormal condition information through the first abnormal evaluation result and the second abnormal evaluation result; Based on the abnormal time nodes, data segmentation and extraction are performed from real-time audio and video data to obtain corresponding audio segment data; By presetting the audio processing strategy, the processing strategy is matched based on the abnormal condition information, and the audio segment data is optimized according to the matching results to generate optimized audio segment data.

[0060] It is worth mentioning that, in the process of audio and video data transmission, generally speaking, it is difficult to evaluate and analyze the abnormal audio data in real time. Therefore, the present invention evaluates and analyzes the abnormal picture in real time. When the picture is abnormal, the corresponding audio data often has corresponding audio distortion, excessive noise, and repetitive playback of a certain audio data, which is specifically caused by the abnormal data. Therefore, the present invention records the abnormal video picture at the time node, intercepts and analyzes the corresponding audio data, and performs corresponding audio shielding, audio noise reduction and other processing to reduce the impact of the audio abnormality. The adjustment process is based on the preset audio processing strategy. Different abnormal conditions correspond to different optimization processing. For example, when the video picture is stuck or stuck, the audio can be shielded. When the picture is blurred and distorted, the audio data often has corresponding distortion, excessive noise, etc. At this time, the corresponding optimization strategy can be audio noise reduction for the audio data. The preset audio processing strategy can be set by the user. The optimized audio segment data is used to replace the original audio segment data.

[0061] According to an embodiment of the present invention, the first abnormality assessment result and the second abnormality assessment result are sent to the user terminal device, the abnormal state of the picture is adjusted in real time by the user terminal device, and optimized real-time video data is generated, and the real-time video data is mixed with the optimized audio segment data to obtain the second audio and video data, specifically: Sending the first abnormality assessment result and the second abnormality assessment result to a user terminal device; Analyze abnormal conditions through user terminal devices, and based on preset adjustment instructions, select corresponding adjustment instructions for abnormal conditions of different screens for real-time adjustment; The adjusted video data is recorded to obtain optimized video data, and the optimized video data is mixed with the optimized audio segment data to obtain second audio and video data.

[0062] It should be noted that the preset adjustment instructions are application adjustment strategies pre-set by the user. For example, if the screen abnormality is a decrease in screen resolution or a blurred screen, the screen can be adjusted by adjusting the transmission line, adding instructions such as adjusting network resource allocation, etc. If the screen abnormality is a freeze in the operating interface, etc., it can be adjusted by sending a refresh instruction.

[0063] Audio and video data mixing includes steps such as audio and video data encoding, time alignment, and encapsulation of synthesized data.

[0064] When performing audio and video, after adjustment, the optimized audio and video data is recorded, replacing the audio and video data before optimization, and stored in the system database.

[0065] In audio and video picture control, there are two methods: real-time control and delayed control. In delayed data control, after optimizing and analyzing to obtain the second audio and video data, the audio and video data of the original abnormal picture is replaced to achieve effective data maintenance of the audio and video data and reduce unnecessary data storage.

[0066] The third aspect of the present invention also provides a computer-readable storage medium, which includes an intelligent audio and video hybrid transmission program. When the intelligent audio and video hybrid transmission program is executed by a processor, the steps of the intelligent audio and video hybrid transmission strategy as described in any one of the above items are implemented.

[0067] The present invention discloses an intelligent audio and video hybrid transmission strategy and system. During the monitoring cycle, the system captures user interactions and audio and video data in real time, and separates the video and audio streams. The video stream undergoes frame extraction and parameter comparison to generate a first anomaly assessment. The interactive data is used to locate the screen, perform feature and semantic analysis, and compare database information to obtain a second anomaly assessment. Combining the two evaluation results, the audio data anomaly is located and optimized according to a preset strategy to generate optimized audio. Through the present invention, the audio and video data can be effectively analyzed for anomalies and reasonably mixed for transmission, effectively realizing the system platform's transmission optimization effect on multi-terminal audio and video data, and improving the user experience.

[0068] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0069] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0070] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0071] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: a mobile storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0072] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0073] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent audio and video hybrid transmission strategy, characterized in that: include: In one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data; Extracting video frames from real-time video data to obtain picture data, analyzing and comparing whether basic parameters of the picture data are consistent with preset picture parameters, and generating a first abnormality assessment result; Based on the user interaction data, the interaction time node and the screen interaction type information are obtained, and the corresponding screen is intercepted from the screen data based on the interaction time node to form the detection screen data, and the feature extraction, image recognition and text recognition are performed on the detection screen data to generate the semantic recognition result; The real-time image comparison semantic information is filtered out from the image semantic database through user interaction data, similarity is calculated between the semantic recognition result and the real-time comparison semantic information, image anomaly analysis is performed through threshold comparison, and a second anomaly assessment result based on semantic analysis is generated; Based on the first abnormality assessment result and the second abnormality assessment result, an abnormal time node and abnormal condition information are obtained, corresponding audio segment data is extracted from the real-time audio and video data based on the abnormal time node, and the audio segment data is optimized based on the abnormal condition information through a preset audio processing strategy to generate optimized audio segment data; The first abnormality assessment result and the second abnormality assessment result are sent to the user terminal device, the abnormal state of the picture is adjusted in real time by the user terminal device, and optimized real-time video data is generated, and the real-time video data is mixed with the optimized audio segment data to obtain the second audio and video data.

2. According to claim 1, the intelligent audio and video hybrid transmission strategy is characterized in that: In one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data, specifically: Through the system platform, count the number of currently connected user terminals and analyze the current audio and video data transmission volume; Set monitoring period based on the number of user terminals and the volume of audio and video data transmission; In a monitoring cycle, user interaction data and real-time audio and video data are obtained in real time through user terminals; The real-time audio and video data is analyzed for data format and audio and video are separated to obtain real-time video data and real-time audio data.

3. According to claim 2, the intelligent audio and video hybrid transmission strategy is characterized in that: The video frame extraction is performed on the real-time video data to obtain the picture data, and the basic parameters of the picture data are analyzed and compared to see whether they are consistent with the preset picture parameters, and a first abnormality assessment result is generated, specifically: Extract key frames from real-time video data to obtain an image set; Perform image enhancement, smoothing and standardization preprocessing on the image set to obtain picture data; Counting picture parameter information based on picture data, the parameter information includes resolution, picture ratio, frame rate, and obtaining basic parameters; The basic parameters are compared with the preset screen parameters. If there are differences, a first abnormality assessment result is generated based on the parameter differences. At the same time, the abnormal time node is recorded in the first abnormality assessment result.

4. According to claim 3, the intelligent audio and video hybrid transmission strategy is characterized in that: The method of acquiring the interaction time node and the screen interaction type information based on the user interaction data, capturing the corresponding screen from the screen data based on the interaction time node to form the detection screen data, performing feature extraction, image recognition and text recognition on the detection screen data to generate the semantic recognition result is specifically as follows: Obtain interaction instructions and interaction time nodes through user interaction data; Set the pre-interaction time node and the post-interaction time node based on the interaction time node; Intercepting corresponding picture information from the picture data through the pre-interaction time node and the post-interaction time node to obtain the first image data and the second image data; Constructing an image recognition model based on CNN, performing image segmentation and screen image recognition on the first image data and the second image data respectively, and obtaining a screen recognition result; Converting the picture recognition result into first semantic information; Based on the OCR technology, the first image data and the second image data are pre-processed by grayscale conversion and noise reduction, and the characters are recognized and extracted by the OCR system to obtain the second semantic information; The first semantic information and the second semantic information are integrated to form a semantic recognition result.

5. The intelligent audio and video hybrid transmission strategy according to claim 4 is characterized in that: The real-time image comparison semantic information is filtered out from the image semantic database through user interaction data, similarity is calculated between the semantic recognition result and the real-time comparison semantic information, and image anomaly analysis is performed through threshold comparison to generate a second anomaly assessment result based on semantic analysis, specifically: Obtaining interaction instructions based on user interaction data; Based on the interactive instructions, the corresponding real-time comparison semantic information of the screen is filtered and obtained from the system database; The first semantic information and the second semantic information in the semantic recognition result are calculated similarly with the real-time comparison semantic information. During the similarity calculation process, the semantic information similarity between the texts is calculated by using the bag-of-words model and the cosine similarity, and finally the semantic similarity is obtained; The semantic similarity is compared with the familiarity threshold, and anomaly assessment is performed through comparative analysis. At the same time, the abnormal time node is recorded and a second anomaly assessment result is generated.

6. The intelligent audio and video hybrid transmission strategy according to claim 5, characterized in that: The method of obtaining the abnormal time node and abnormal condition information based on the first abnormal evaluation result and the second abnormal evaluation result, extracting the corresponding audio segment data from the real-time audio and video data based on the abnormal time node, optimizing the audio segment data based on the abnormal condition information through a preset audio processing strategy, and generating optimized audio segment data is specifically as follows: Obtain abnormal time node and abnormal condition information through the first abnormal evaluation result and the second abnormal evaluation result; Based on the abnormal time nodes, data segmentation and extraction are performed from real-time audio and video data to obtain corresponding audio segment data; By presetting the audio processing strategy, the processing strategy is matched based on the abnormal condition information, and the audio segment data is optimized according to the matching results to generate optimized audio segment data.

7. The intelligent audio and video hybrid transmission strategy according to claim 6, characterized in that: The first abnormality assessment result and the second abnormality assessment result are sent to the user terminal device, the abnormal state of the picture is adjusted in real time by the user terminal device, and optimized real-time video data is generated, and the real-time video data is mixed with the optimized audio segment data to obtain the second audio and video data, specifically: Sending the first abnormality assessment result and the second abnormality assessment result to a user terminal device; Analyze abnormal conditions through user terminal devices, and based on preset adjustment instructions, select corresponding adjustment instructions for abnormal conditions of different screens for real-time adjustment; The adjusted video data is recorded to obtain optimized video data, and the optimized video data is mixed with the optimized audio segment data to obtain second audio and video data.

8. An intelligent audio and video hybrid transmission system, characterized in that: The system includes: a memory and a processor, wherein the memory includes an intelligent audio and video hybrid transmission program, and when the intelligent audio and video hybrid transmission program is executed by the processor, the following steps are implemented: In one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data; Extracting video frames from real-time video data to obtain picture data, analyzing and comparing whether basic parameters of the picture data are consistent with preset picture parameters, and generating a first abnormality assessment result; Based on the user interaction data, the interaction time node and the screen interaction type information are obtained, and the corresponding screen is intercepted from the screen data based on the interaction time node to form the detection screen data, and the feature extraction, image recognition and text recognition are performed on the detection screen data to generate the semantic recognition result; The real-time image comparison semantic information is filtered out from the image semantic database through user interaction data, similarity is calculated between the semantic recognition result and the real-time comparison semantic information, image anomaly analysis is performed through threshold comparison, and a second anomaly assessment result based on semantic analysis is generated; Based on the first abnormality assessment result and the second abnormality assessment result, an abnormal time node and abnormal condition information are obtained, corresponding audio segment data is extracted from the real-time audio and video data based on the abnormal time node, and the audio segment data is optimized based on the abnormal condition information through a preset audio processing strategy to generate optimized audio segment data; The first abnormality assessment result and the second abnormality assessment result are sent to the user terminal device, the abnormal state of the picture is adjusted in real time by the user terminal device, and optimized real-time video data is generated, and the real-time video data is mixed with the optimized audio segment data to obtain the second audio and video data.

9. The intelligent audio and video hybrid transmission system according to claim 8, characterized in that: In one monitoring cycle, user interaction data and real-time audio and video data are obtained in real time, and the real-time audio and video data are separated to obtain real-time video data and real-time audio data, specifically: Through the system platform, count the number of currently connected user terminals and analyze the current audio and video data transmission volume; Set monitoring period based on the number of user terminals and the volume of audio and video data transmission; In a monitoring cycle, user interaction data and real-time audio and video data are obtained in real time through user terminals; The real-time audio and video data is analyzed for data format and audio and video are separated to obtain real-time video data and real-time audio data.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes an intelligent audio and video hybrid transmission program. When the intelligent audio and video hybrid transmission program is executed by a processor, the steps of the intelligent audio and video hybrid transmission strategy according to any one of claims 1 to 8 are implemented.

Images