Screen projection detection method and device, readable storage medium and computer program product

Abstract

The invention provides a screen projection detection method and device, a readable storage medium and a computer program product. The method comprises the following steps: in response to a screen projection operation of a user, carrying out synchronous recording processing on a screen projection process to obtain first audio and video data of screen projection equipment and second audio and video data of screen-projected equipment; preprocessing the first audio and video data and the second audio and video data to obtain first image data, first audio data, second image data and second audio data; performing structural similarity analysis on the first image data and the second image data to obtain image similarity of the first image data and the second image data; performing time-frequency correlation analysis on the first audio data and the second audio data to obtain audio similarity of the first audio data and the second audio data; and determining a screen projection detection score based on the image similarity and the time-frequency similarity, the screen projection detection score being used for detecting the screen projection quality. Therefore, the screen projection detection quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to video technology, and particularly relates to a screen projection detection method and device, a readable storage medium and a computer program product. BACKGROUND

[0002] Screen projection technology refers to a process of wirelessly transmitting audio and video content from one device to another display device for display, and is widely used in home entertainment, conference demonstration and remote office scenarios. In order to improve user experience, the stability and quality of screen projection become key technical indicators, and the screen projection effect needs to be detected and evaluated.

[0003] In the prior art, whether screen projection is successful is usually determined by manual observation or simple system prompts, and there is a lack of quantitative analysis means for image and audio synchronization, definition and transmission stability. Some schemes attempt to evaluate screen projection quality by comparing the consistency of original audio and video data and received data, but these methods often rely on specific protocol support and are difficult to fully reflect the quality difference perceived by users. SUMMARY

[0004] Embodiments of the present application provide a screen projection detection method, device, readable storage medium and computer program product, which can objectively evaluate screen projection quality by comprehensively analyzing image features and time-frequency information.

[0005] The technical solution of the embodiments of the present application is as follows: The embodiments of the present application provide a screen projection detection method, which comprises the following steps: In response to a screen projection operation of a user, synchronously recording and processing a screen projection process to obtain first audio and video data of a screen projection device and second audio and video data of a device being projected on; Preprocessing the first audio and video data and the second audio and video data to obtain first image data and first audio data, and second image data and second audio data; Performing structural similarity analysis on the first image data and the second image data to obtain image similarity of the first image data and the second image data; Performing time-frequency correlation analysis on the first audio data and the second audio data to obtain audio similarity of the first audio data and the second audio data; Determining a screen projection detection score based on the image similarity and the time-frequency similarity, wherein the screen projection detection score is used to detect screen projection quality.

[0006] The embodiments of the present application provide an electronic device, which comprises: A memory configured to store computer executable instructions or computer programs; A processor is configured to execute computer-executable instructions or computer programs stored in the memory to implement the screen projection detection method provided in the embodiments of the present application.

[0007] The embodiments of the present application provide a computer readable storage medium storing computer programs or computer-executable instructions, which are configured to be executed by a processor to implement the screen projection detection method provided in the embodiments of the present application.

[0008] The embodiments of the present application provide a computer program product including computer programs or computer-executable instructions, which are configured to be executed by a processor to implement the screen projection detection method provided in the embodiments of the present application.

[0009] In the technical solution of the embodiments of the present application, in response to a screen projection operation of a user, a screen projection process is synchronously recorded to obtain first audio and video data of a screen projection device and second audio and video data of a screen projection target device; the first audio and video data and the second audio and video data are preprocessed to obtain first image data and first audio data and second image data and second audio data; the first image data and the second image data are analyzed in terms of structural similarity to obtain image similarity of the first image data and the second image data; the first audio data and the second audio data are analyzed in terms of time-frequency correlation to obtain audio similarity of the first audio data and the second audio data; and the screen projection detection score is determined based on the image similarity and the time-frequency similarity, wherein the screen projection detection score is used to detect screen projection quality. In this way, on the one hand, the calculation complexity of image coding is reduced and the image comparison efficiency is improved through grayscale and convolution operators; on the other hand, more accurate quality evaluation is achieved at the image and audio levels through structural similarity algorithms and dynamic time warping algorithms, thereby improving the comprehensiveness and objectivity of screen projection quality detection, and the accuracy and quantifiability are higher than those of traditional methods which rely on subjective experience. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 FIG. 1 is a structural schematic diagram of a screen projection detection system architecture provided in the embodiments of the present application; Figure 2 FIG. 2 is a structural schematic diagram of an electronic device provided in the embodiments of the present application; Figure 3 FIG. 3 is a first flowchart of a screen projection detection method provided in the embodiments of the present application; Figure 4 FIG. 4 is a second flowchart of a screen projection detection method provided in the embodiments of the present application; Figure 5 FIG. 5 is a schematic diagram of an image structure provided in the embodiments of the present application; Figure 6 FIG. 6 is a third flowchart of a screen projection detection method provided in the embodiments of the present application; Figure 7 is a fourth flowchart of a screen projection detection method provided by an embodiment of the present application; Figure 8 is a fifth flowchart of a screen projection detection method provided by an embodiment of the present application; Figure 9 is a sixth flowchart of a screen projection detection method provided by an embodiment of the present application.

[0011] It should be noted that the above "first", "second" are only used to distinguish different schemes, and do not represent the degree of superiority or priority in the implementation process. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by a person of ordinary skill in the art without making creative labor fall within the scope of protection of the present application.

[0013] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0014] In the following description, the terms "first, second, third" are only used to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0015] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as commonly understood by a person skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0016] In related technologies, with the development of science and technology, screen projection technology has become a common demand in daily life and work. Users can wirelessly project content from mobile phones, tablets and other devices to smart TVs or projectors to enjoy visual experience on large screens. However, in actual application, due to network delay, device performance difference or software problem, the phenomenon of unsuccessful image and sound projection may occur. The existing screen projection detection method mainly relies on subjective experience, and often cannot provide detailed screen projection success rate analysis and optimization suggestions.

[0017] To solve the above problems, the embodiment of the present application provides a screen projection detection method, an electronic device, a computer readable storage medium and a computer program product, which can objectively and accurately describe the effect and quality between screen projection and original content.

[0018] Taking the application of the embodiment of the present application in the audio and video player and the audio discrimination scene as an example, referring to Figure 1 , Figure 1 is a structural schematic diagram of a screen projection detection system architecture provided by the embodiment of the present application, and the example Figure 1 involved in the server 100, the terminal 200 and the network 300. The terminal 200 connects the server 100 through the network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0019] In some embodiments, the screen projection detection system provided by the embodiment of the present application can be realized by the server and the terminal cooperatively. The terminal device includes a screen projection device and a screen projection receiving device.

[0020] The screen projection detection method provided by the embodiments of the present application can be executed by the intelligent terminal device, the screen projection receiving end device and the background processing server. For example, the terminal 200 responds to the screen projection operation of the user, the screen projection device initiates the screen projection operation and obtains the first audio and video data, the screen projection receiving device receives the screen projection device for playing the content and obtains the second audio and video data; the server 100 sends a screen projection detection request, and sends the first audio and video data and the second audio and video data, the server 100 receives the screen projection detection request, and obtains the screen projection detection score by the screen projection detection method provided by the embodiment of the present application, and sends the screen projection detection score to the terminal 200, and the terminal 200 receives the screen projection detection score sent by the server 100 to detect the screen projection quality.

[0021] In some embodiments, the screen projection device can be a terminal with a device screen projection function, and can transmit screen projection data to the screen projection receiving device. For example, the screen projection device can be an electronic device configured with a screen, such as a mobile phone, a notebook computer, a netbook, a tablet computer, a television, etc. The example embodiments of the screen projection device include but are not limited to electronic devices loaded with iOS, android, microsoft or other operating systems.

[0022] In some embodiments, the screen projection receiving device is a terminal that can be connected with the screen projection device and receive the screen projection data sent by the screen projection device. For example, the screen projection receiving device can be an electronic device configured with a screen, such as a mobile phone, a notebook computer, a netbook, a tablet computer, a television, etc. In some embodiments, the screen projection receiving device 200 can also be a projector with a projection imaging function. The screen projection receiving device is only a word used in the embodiment of the present application, and the functions performed thereby have been described in the embodiment, and the name thereof does not constitute any limitation on the embodiment.

[0023] In the embodiments of the present application, the screen projection device and the screen projection target device can establish a screen projection connection based on various wired communication or wireless communication modes. The communication mode in which the screen projection device and the screen projection target device establish the screen projection connection can be referred to as a screen projection connection mode. For example, the wireless communication mode can be near field communication (NFC) communication, wireless fidelity (WiFi) communication, ultra wide band (UWB) communication, Bluetooth communication, ZigBee communication, and the like. For example, the screen projection device and the screen projection target device can have one or more short-range communication modules such as an NFC communication module, a WiFi communication module, a UWB communication module, a Bluetooth communication module, a ZigBee communication module, and the like. The screen projection device can emit a signal through a short-range communication module (for example, a Bluetooth communication module) to detect and scan the screen projection target device (for example, a television) in the vicinity of the screen projection device, so that the screen projection device can discover the screen projection target device in the vicinity through a short-range wireless communication protocol, establish a wireless communication connection with the electronic device in the vicinity, and perform data transmission with the screen projection target device in the vicinity.

[0024] In some embodiments, the terminal device or the server can implement the screen projection detection method provided in the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program in the operating system or a software module. In summary, the above-mentioned computer-executable instructions can be any form of instructions, and the above-mentioned computer programs can be any form of application programs, modules, or plug-ins. The terminal device includes but is not limited to a mobile phone, a computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, and the like.

[0025] Referring to Figure 2 , Figure 2 FIG. 1 is a structural schematic diagram of an electronic device provided in the embodiments of the present application, Figure 2 The electronic device 400 shown in FIG. 1 can be the server 100 or the terminal device 200 described above. Figure 2 The electronic device 400 includes at least one processor 410, a memory 430, and at least one network interface 420. The various components in the electronic device 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between the components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 440 in Figure 2 FIG. 1.

[0026] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0027] The memory 430 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 430 may optionally include one or more storage devices physically located away from the processor 410.

[0028] The memory 430 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 430 described in this application embodiment is intended to include any suitable type of memory.

[0029] In some embodiments, memory 430 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0030] Operating system 431 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 432 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A projection detection device 433 stored in memory 430 is shown. This device can be software in the form of programs or plug-ins, and includes the following software modules: a data processing module 4331 and a data determination module 4332. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0031] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in a hardware manner. For example, the apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to perform the screen projection detection method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can be implemented by using one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), or other electronic elements.

[0032] Referring to Figure 3 , Figure 3 is a schematic diagram of the principle of the screen projection detection method provided by the embodiments of the present application. First, the first audio and video data of the screen projection device and the second audio and video data of the screen projection target device are obtained, and the image encoder and the audio encoder are called to obtain the first image feature, the second image feature, and the first audio feature. Next, the first image feature and the first text feature are fused to obtain the fusion feature. Finally, the image abnormality parameter of the image data is obtained based on the fusion feature. Figure 3 is a flowchart of the screen projection detection method provided by the embodiments of the present application. The following will be described in combination with the steps shown in Figure 3 . As shown in Figure 3 , the method includes the following steps 301 to 305.

[0033] Step 301, in response to the screen projection operation of the user, the screen projection process is synchronously recorded to obtain the first audio and video data of the screen projection device and the second audio and video data of the screen projection target device.

[0034] In some embodiments, when the user clicks the screen projection button on the screen projection device (such as a mobile phone), the system simultaneously starts the screen recording function on the screen projection device and the screen projection target device (such as a television), ensuring that the audio and video data of the screen projection device and the screen projection target device can be synchronized during the screen projection process. The audio and video data of the screen projection device and the screen projection target device not only contains image information, but also contains audio information, providing a basis for subsequent structural similarity analysis and time-frequency correlation analysis. By synchronously recording the screen projection device and the screen projection target device, the system can ensure that the audio and video data timestamps of the screen projection device and the screen projection target device are consistent, facilitating subsequent time alignment and comparative analysis. The user triggers the screen projection operation by clicking the screen projection button on the mobile phone.

[0035] For example, an application with high-quality screen recording function is installed on the screen projection device (such as a mobile phone), and the screen recording parameters are configured, with a resolution of 720p (720x1080 pixels) to ensure image clarity. The frame rate is set to 30 frames per second to ensure smoothness of the image stream. The audio sampling rate is set to 44.1 kHz to capture high-quality sound. The "internal audio recording" function is enabled in the screen recording application to ensure that the recording process includes audio content from the application. The screen recording application is started and recording begins, recording the entire wireless screen projection process. Ensure that the screen projection picture includes all active content on the mobile phone screen during recording. The screen projection target device (such as a television) starts the built-in screen recording function and configures the corresponding parameters (resolution and frame rate) to match the mobile phone screen recording settings, ensuring that the recording includes all screen projection content on the television screen and the corresponding audio.

[0036] Step 302, pre-processing the first audio and video data and the second audio and video data to obtain first image data and first audio data, and second image data and second audio data.

[0037] In the embodiments of the present application, by pre-processing the first audio and video data of the screen projection device (such as a mobile phone) and the second audio and video data of the screen projection target device (such as a television), the image data and audio data are decomposed to obtain the first image data and first audio data of the screen projection device, and the second image data and second audio data of the screen projection target device. By separately evaluating the image data and audio data of the screen projection device and the screen projection target device, and then combining the two screen projection quality evaluation results to determine whether the screen projection quality meets the requirements.

[0038] In some embodiments, pre-processing the first audio and video data and the second audio and video data includes: The first audio and video data is decomposed to obtain a first image sequence and a first audio file; the first image sequence and the first audio file are timestamped to obtain first image data and first audio data; the second audio and video data is decomposed to obtain a second image sequence and a second audio file; the second image sequence and the second audio file are timestamped to obtain second image data and second audio data.

[0039] Here, an image sequence refers to an ordered set of multiple independent image frames that are split from a continuous video stream according to its frame rate. For example, if the original video has a frame rate of 30 frames per second, then 30 images can be generated per second, forming an image sequence. Image sequences are used for subsequent image quality assessment and similarity calculation. For instance, FFmpeg can be used to decompose a recorded video file into an image sequence. Audio files are the pure audio information extracted from the original audio and video data, typically stored in formats such as WAV and MP3. For instance, the extraction of audio files can be completed using tools such as FFmpeg, facilitating subsequent spectral analysis and time synchronization detection.

[0040] Timestamp alignment refers to adjusting the playback order or sampling rate of image frames and audio segments by comparing their timestamp information, thereby ensuring that the image and audio are consistent in the temporal dimension. For example, in videos recorded on a mobile device, system latency may cause slight time discrepancies between audio and image. Timestamp alignment can eliminate these slight time discrepancies, ensuring synchronization between the image and audio. For instance, using FFmpeg's `-vsync` ensures that the timestamps of extracted image frames are consistent, thus aligning the timestamps in the image and audio files for accurate comparison.

[0041] Step 303: Perform structural similarity analysis on the first image data and the second image data to obtain the image similarity between the first image data and the second image data.

[0042] In this embodiment, the projection quality of the projection device and the projected device is analyzed by performing structural similarity analysis on the first and second image data. This involves comparing the similarity in brightness, contrast, and structural information between the first and second image data at the same timestamp, and then evaluating the quality based on the structural similarity. Here, structural similarity analysis is an image quality assessment method based on the visual perception characteristics of the human eye, which can effectively reduce computational load and improve recognition accuracy.

[0043] In some embodiments, see Figure 4 , Figure 3 Step 303 shown can be implemented through steps 401 to 403, as explained in detail below.

[0044] Step 401, performing gray processing on the first image data and the second image data to obtain a first gray image and a second gray image.

[0045] Here, the gray processing refers to the process of converting a color image into a gray image, that is, the color information of each pixel point i is represented by a brightness value (i.e., a gray value v i ). Gray processing can reduce the data volume of the image and improve the efficiency of subsequent processing while preserving the basic outline and shape information of the image. By performing gray processing on the image, the interference caused by color information can be removed, so that the image contrast focuses more on the consistency of shape and texture. After implementing gray processing on the image, the computational complexity can be reduced, the image processing speed can be accelerated, and thus the overall detection efficiency can be improved.

[0046] Step 402, performing convolution compression processing on the first gray image and the second gray image to obtain a first image feature and a second image feature.

[0047] Here, the convolution compression processing is to extract the key features of the image using convolution operation, and to convert the original image into an image feature with smaller scale but still reflecting the core information of the image through low-rank matrix compression. Convolution compression processing can significantly reduce the data volume without losing key information, facilitating subsequent similarity comparison. For example, convolution calculation is used for each picture to obtain a smaller data structure carrying the information of the whole picture, such as: convolution=720*k, where k is a convolution operator. If the computational load is reduced, a low-rank matrix is selected for k, so that the information content of the whole picture is compressed into the shadow area of the picture. See Figure 5 To compare the similarity of the image pictures of the screen projection device (mobile phone end) and the screen projection device (television end), it is only necessary to compare the similarity of the shadow area in the image. Convolution compression processing can effectively extract the key features such as edges and textures of the image, and compress the convolution compression processing to a smaller dimension, so that the comparison between images is more efficient and accurate. At the same time, the structural similarity analysis also reduces the requirement for hardware performance, and improves the applicability and scalability of the algorithm.

[0048] Step 403, performing structural similarity analysis on the first image feature and the second image feature to obtain an image similarity.

[0049] Here, structural similarity analysis is a method for measuring the similarity of two images in terms of brightness, contrast, and structural information. Structural similarity analysis not only considers the similarity at the pixel level, but also considers the overall structural features of the image, so it can more accurately reflect the similarity between images than traditional methods such as mean square error (MSE). By performing structural similarity analysis on the features of the compressed images, the consistency of the images before and after projection can be more accurately evaluated, and it can be determined whether the projection quality meets the expectations. Structural similarity analysis is particularly suitable for cases where there is slight distortion or deformation of the image, and has high robustness and adaptability.

[0050] Specifically, the structural similarity analysis of the first image feature and the second image feature includes: calculating the average brightness, variance, and covariance of the first image feature and the second image feature; and performing structural similarity analysis on the first image feature and the second image feature based on the average brightness, variance, and covariance to obtain an image similarity.

[0051] Here, the average brightness refers to the mean value of the overall pixel value of the image, which reflects the overall brightness of the image; the variance is used to measure the dispersion between the pixel value of the image and the average brightness, which represents the contrast of the image; and the covariance is used to describe the linear correlation between two image features, which can reflect the matching degree of the two images in terms of gray scale distribution. The structural similarity algorithm is used to obtain the similarity of the two shadow images, and the specific formula is as follows.

[0052] (1) wherein, wherein, μ x and μ y are the average brightness of images x and y; σ x 2 and σ y 2 are the variances of images x and y; σ xy is the covariance of images x and y, and C1 and C2 are constants. In this way, by calculating the above three statistical quantities of the image, the core feature information of the image can be quickly extracted, and the basic data for subsequent structural similarity analysis can be provided. This feature extraction method based on statistical quantity calculation helps to reduce the computational complexity in the image similarity analysis process and improves the overall efficiency of the image similarity analysis.

[0053] In some embodiments, the image similarity is used to determine an image projection score for projection quality detection, and the image projection score can be obtained by formula (2).

[0054] (2) wherein, Score pictureThe image projection score of the screen projection device and the screen projected device in the screen projection process is represented as N, which represents the number of images, and the value of N depends on the preset frame rate and the selected image time. For example, the first image data of the screen projection device and the second image data of the screen projected device are selected for 10s. Here, the 10s parameter can be adjusted according to the specific business requirements, so that 10*30 images are obtained. Then, the first image sequence and the second image sequence selected for 10s are compared, and the image projection score is obtained by formula (2): .

[0055] In this way, the comparison of each complete picture is compressed to compare the content of a smaller area carrying the information of the whole picture, and the calculation amount is reduced from 10*30*720*1080 to 10*30*720*rank(k). Here, when the convolution operator k selects a low-rank matrix, the calculation amount is reduced by rank(K) / 1080.

[0056] Step 304, audio correlation analysis is performed on the first audio data and the second audio data to obtain the audio similarity of the first audio data and the second audio data.

[0057] In the embodiments of the application, the screen projection quality of the screen projection device and the screen projected device can be analyzed by performing time-frequency similarity analysis on the first audio data and the second audio data, comparing the audio waveforms of the screen projection device and the screen projected device, checking whether there is a time delay in the audio waveforms recorded by the screen projection device and the screen projected device, and performing difference analysis on the frequency spectrum of the audio signals of the screen projection device and the screen projected device to detect the time synchronization of the audio.

[0058] In some embodiments, the time-frequency correlation analysis mainly includes dynamic time warping (DTW) algorithm and cosine similarity analysis. The dynamic time warping (DTW) algorithm is used to calculate the time alignment error between two audio signals, so as to determine whether there is a significant delay problem; and the cosine similarity is used to compare the similarity between the audio frequency spectrum vectors to evaluate the frequency consistency. Thus, by combining the time domain and frequency domain information, the quality of the audio projection is more comprehensively evaluated.

[0059] Specifically, referring to Figure 6 , Figure 3 The step 304 shown can be implemented by steps 601 to 603, which will be described in detail below.

[0060] Step 601, calculate the time alignment error of the first audio data and the second audio data, and obtain the time domain similarity based on the time alignment error.

[0061] Here, the time alignment error refers to the time offset between the two audio streams due to device delay or synchronization problems during audio signal transmission. The time alignment error reflects the inconsistency of the audio signals on the time axis between the mobile phone and the TV. By calculating the time alignment error, the consistency of the audio signal in the time domain can be evaluated. By quantifying the time alignment error, the system can effectively judge the synchronization status of the audio in the transmission process, thereby providing a basis for subsequent time domain similarity evaluation.

[0062] In an example, the dynamic time warping algorithm is applied to the audio data, and the time alignment error between the audio streams of the screen projection device and the screen projection target device is calculated, which can be obtained by formula (3).

[0063] (3) Wherein, S x represents the first audio data, S y represents the second audio data, and N represents the length of the audio data.

[0064] Step 602, calculate the spectral similarity of the first audio data and the second audio data, and obtain the frequency domain similarity based on the spectral similarity.

[0065] Here, the spectral similarity is used to measure the similarity of two audio signals in frequency distribution, which is usually achieved by comparing the shape and energy distribution of the spectrum. The higher the spectral similarity, the closer the two audio signals in frequency composition, indicating that the two audio signals match better in the frequency domain. Through the calculation of spectral similarity, the consistency of audio signal can be further verified in the frequency domain, and the comprehensiveness and accuracy of audio quality evaluation are improved.

[0066] In actual implementation, the time domain similarity and the frequency domain similarity are complementary to each other. The time domain similarity mainly focuses on the alignment of audio in the time axis, while the frequency domain similarity focuses more on the matching degree of audio in the frequency component. The combination of time domain similarity and frequency domain similarity helps to more comprehensively evaluate the overall similarity of audio signal.

[0067] In some embodiments, calculating the spectral similarity of the first audio data and the second audio data comprises: obtaining a first spectral feature based on the spectrum of the first audio data; obtaining a second spectral feature based on the spectrum of the second audio data; calculating the cosine similarity of the first spectral feature and the second spectral feature, and obtaining the spectral similarity based on the cosine similarity.

[0068] Here, the spectrum of the first audio data refers to the frequency component distribution obtained after converting the first audio data from the time domain to the frequency domain. The first audio data can be decomposed into a plurality of sinusoidal waves of different frequencies through Fourier transform (such as Fast Fourier Transform, FFT), and the energy size of each frequency component constitutes the spectrum of the first audio data. The first spectral feature represents the characteristic information of the first audio data in the frequency domain, and is usually stored in the form of a one-dimensional array, with each element representing the energy value of a certain frequency band. The second spectral feature is similar to the first spectral feature, and is a characteristic representation of the second audio data in the frequency domain. The second spectral feature reflects the frequency components and energy distribution of the second audio data. By comparing the first spectral feature and the second spectral feature, the similarity of the first audio data and the second audio data in the frequency structure can be determined.

[0069] In practical applications, in order to extract the spectral feature of the first audio data, the first audio data is usually preprocessed, such as windowing, framing, etc., and then converted to a frequency domain representation through an FFT algorithm. After converting the first audio data to a frequency domain representation through the FFT algorithm, the frequency distribution of the first audio data can be more intuitively displayed, facilitating subsequent comparison and analysis. The second audio data also needs to go through the same preprocessing process as the first audio data, including windowing, framing, and FFT conversion, to ensure that the first audio data and the second audio data are comparable.

[0070] The cosine similarity is an index for measuring the cosine value of the included angle between the first spectral feature and the second spectral feature, and is used to evaluate the directional similarity between the first spectral feature and the second spectral feature. For the first spectral feature and the second spectral feature, when the cosine similarity value increases, it indicates that the directions of the first spectral feature and the second spectral feature in the frequency domain space are closer, thereby indicating that the corresponding first audio data and second audio data have a higher degree of similarity in the frequency structure. The calculation formula of the cosine similarity is defined as the dot product between two vectors divided by the product of the lengths of the two features, which can be obtained by formula (4).

[0071] (4) Where X and Y are spectral vectors representing the first spectral feature and the second spectral feature, respectively. By calculating the cosine similarity, a value between -1 and 1 can be obtained, with a value closer to 1 indicating that the first spectral feature and the second spectral feature are more similar, and a value closer to -1 indicating that they are less similar. Finally, the cosine similarity is used as a quantitative result of the spectral similarity to evaluate the matching degree of the first audio data and the second audio data in the frequency structure.

[0072] Step 603, performing time-frequency correlation analysis on the first audio data and the second audio data based on the time domain similarity and the frequency domain similarity to obtain a time-frequency domain similarity.

[0073] Here, the time-frequency correlation analysis is a method that comprehensively considers time and frequency information. By combining the similarity results in the time domain and the frequency domain, the time-frequency correlation analysis method obtains a more comprehensive audio similarity score. The time-frequency correlation analysis method can more accurately reflect the overall matching of the audio signal and avoid the deviation that may be caused by single-dimensional evaluation. By introducing time-frequency correlation analysis, joint evaluation of the audio signal in multiple dimensions is realized, improving the objectivity and reliability of audio similarity determination, and thus more accurately detecting the screen projection quality.

[0074] In actual implementation, the calculation of the time alignment error in the entire step process provides a basis for subsequent time domain similarity evaluation, and the spectral similarity further confirms the consistency of the audio from the frequency dimension. Finally, through time-frequency correlation analysis, the evaluation results of the two are fused to form a more comprehensive audio similarity index. This multi-dimensional analysis method makes the audio quality evaluation more scientific and reasonable, and can be more effectively used for screen projection quality detection.

[0075] In some embodiments, the time-frequency domain similarity is used to determine an audio projection score for screen projection quality detection. The audio projection score can be obtained by formula (5).

[0076] (5) wherein Score sound is the comprehensive score of the time domain correlation analysis result DTW and the frequency domain correlation analysis result CosineSimilarity. By combining time synchronization and frequency consistency, the quality of the audio projection can be more comprehensively reflected. The system maps the audio projection score to a value between 0 and 1 to realize unified evaluation with the image score.

[0077] Step 305, determining a screen projection detection score based on the image similarity and the time-frequency similarity, wherein the screen projection detection score is used for detecting the screen projection quality.

[0078] Here, by mapping the image similarity and audio similarity into a two-dimensional space, a specified area is set according to the business requirements. If the comprehensive score value falls within the specified area, it is determined that the screen projection is successful, otherwise it is determined that the screen projection fails. In addition, according to the weight of the image and the audio in the user experience, a weighted calculation method can be used to generate the final screen projection detection score to more accurately reflect the overall situation of the screen projection quality. In this way, by synchronously recording, preprocessing, structural similarity analysis and time-frequency correlation analysis of the image and audio data in the screen projection process, and comprehensively obtaining the screen projection detection score, objective and quantitative evaluation of the screen projection quality is realized. The screen projection detection method not only improves the accuracy of screen projection detection, but also provides a quantifiable reference for subsequent performance optimization.

[0079] In some embodiments, based on the image similarity and the time-frequency domain similarity, the screen projection detection score is determined, including: determining an image screen projection score based on the image similarity; determining an audio screen projection score based on the time domain similarity and the frequency domain similarity; calculating the sum of the image screen projection score and the audio screen projection score, and determining the screen projection detection score based on the sum of the image screen projection score and the audio screen projection score.

[0080] Here, the image similarity is used to measure the similarity between the screen projection images of the mobile phone and the television. The images of the mobile phone and the television are respectively processed by grayscale, and a convolution operator is used to compress the images to generate a small data structure carrying the information of the whole picture, which is called a picture shadow structure. Then, the similarity of the two shadow images is compared based on the structural similarity algorithm (SSIM), so as to obtain the image similarity value. The higher the image similarity value, the closer the image quality, and the better the screen projection effect. The image screen projection score is a quantitative index calculated according to the image similarity, which is used to evaluate whether the image screen projection quality meets the user's demand. The image screen projection score can be converted into a score value between 0 and 1 by using linear mapping, which is conducive to the comprehensive calculation with other score dimensions in subsequent operations.

[0081] The time domain similarity is calculated by the dynamic time warping (DTW) algorithm to calculate the time alignment error between the audio streams of the mobile phone and the television, which is used to measure the synchronization of the audio signal on the time axis. The frequency domain similarity is calculated by extracting the frequency spectrum graph after Fourier transform of the audio signal, and comparing the correlation of the two groups of frequency spectrum vectors by using the cosine similarity algorithm, which reflects the consistency of the audio frequency. The time domain similarity and the frequency domain similarity together constitute the basis of the audio screen projection score. The audio screen projection score is a comprehensive score value calculated by weighting the time domain similarity and the frequency domain similarity. The combination of time synchronization and frequency consistency can more comprehensively reflect the quality of the audio screen projection. The system maps the audio screen projection score to a value between 0 and 1 to realize unified evaluation with the image score.

[0082] The image projection score and the audio projection score are equally important, and therefore the image projection score and the audio projection score can be added by simple summation to obtain a final projection detection score. The final projection detection score ranges from 0 to 2, and the higher the final projection detection score, the better the overall projection quality. For example, the projection detection score can be obtained by formula (6).

[0083] (6) Further, in order to improve the accuracy of the evaluation, the system can introduce a weight coefficient to weight and sum the image score and the audio score, and adjust the importance of each dimension according to the actual business needs. For example, in some scenarios, the image quality is more important than the sound, and the system can appropriately increase the weight of the image score; while in the scenario of video conference, the synchronization of the sound can be more critical, and the system can increase the weight of the audio score.

[0084] In the embodiments of the present application, the image projection score is determined based on image similarity, and the audio projection score is determined based on time domain similarity and frequency domain similarity, and then the image projection score and the audio projection score are comprehensively calculated to obtain the projection detection score. Through the above method, the projection quality can be objectively evaluated, so that optimization suggestions can be proposed to further improve user experience and enhance system reliability.

[0085] Embodiments The projection detection method provided by the embodiments of the present application provides a projection detection method for images and sounds, which evaluates the quality of the image and audio streams of the mobile terminal and the television terminal respectively, and then integrates the two quality evaluation results to confirm whether the projection quality meets the requirements. The effect and quality of the projection and the original content are objectively and accurately described. The following will be described in detail.

[0086] Referring to Figure 7 , Figure 7 FIG. 1 shows a flowchart of an image projection detection method provided by the embodiments of the present application. The images of the mobile terminal and the television terminal are subjected to grayscale processing to obtain grayscale images. The image sequence in a preset time period is taken, and a convolution calculation is performed on each picture in the obtained multiple pictures based on a convolution operator k to obtain a smaller data structure (picture shadow structure) carrying the information of the whole picture. The mobile terminal picture shadow structure and the television terminal picture shadow structure obtained by processing are compared in terms of similarity, and the projection image quality is evaluated based on the similarity size. Specifically: Step 701, obtaining mobile terminal and television terminal screen recording files, the files including audio and video and time stamps.

[0087] Here, install an application with high-quality screen recording function on the phone, configure the screen recording parameters, select 720p (720x1080 pixels) resolution to ensure image clarity. Set the frame rate to 30 frames per second to ensure image smoothness. Set the audio sampling rate to 44.1 kHz to capture high-quality sound. Enable the "internal audio recording" function in the screen recording application to ensure that the recording process includes audio content from the application. Start the screen recording application and begin recording, recording the entire wireless screen mirroring process. Ensure that the mirroring screen includes all active content on the phone screen during recording. Turn on the built-in screen recording function on the TV and configure the corresponding parameters (resolution and frame rate) to match the phone's screen recording settings, ensuring that the recording includes all mirroring content on the TV screen and its corresponding audio.

[0088] Step 702a, extract consecutive image frames from the phone-side video; Step 702b, extract consecutive image frames from the TV-side video.

[0089] Here, use the FFmpeg tool to decompose the recorded video file into an image sequence. Use FFmpeg's -vsync to ensure that the timestamps of the extracted image frames are consistent. Align the timestamps in the image and audio files for accurate comparison.

[0090] Step 703, compress the picture content by convolution.

[0091] Here, use the OpenCV library to process the image sequence. Extract the image frames from the phone and TV sides. To quickly detect the similarity of images, first perform grayscale processing on the phone and TV images, i.e., quantize each pixel point i to a grayscale value vi, to obtain a grayscale image. To improve detection accuracy, select a 10s image sequence, and the 10s parameter can be adjusted according to specific business requirements, resulting in 10*30 images. Before comparing image similarity, to reduce computational complexity, use convolution to obtain a smaller data structure that carries the information of the entire image. Refer to steps 401 to 403 above for specific steps.

[0092] Step 704, calculate SSIM based on the compressed picture and compare the similarity of each image frame.

[0093] Here, use the structural similarity algorithm to obtain the similarity of the two shadow images. Refer to formula (1) above for details.

[0094] Step 705, obtain the image mirroring score.

[0095] Here, refer to the above formula (2). The comparison of each complete picture is compressed into comparison for smaller area content carrying the information of the whole picture, so the calculation amount is reduced from 10*30*720*1080 to 10*30*720*rank(k), and if k is selected as a low-rank matrix, the calculation amount is reduced by rank(k) / 1080, and K is a convolution operator.

[0096] Referring to Figure 8 , Figure 8 An audio projection detection flowchart provided by an embodiment of the present application is shown in FIG. 8. Based on the dynamic time warping (DTW) algorithm, the time alignment error between the audio streams of the mobile terminal and the television terminal is calculated. The cosine similarity is used to compare the correlation of the frequency spectrum, and the frequency consistency is used to evaluate the quality of the projection audio stream. Specifically: Step 801, obtain the mobile terminal and the television terminal recording file, the file including audio and video and time stamp.

[0097] Here, the audio stream in the recorded video is extracted as a WAV format file. The audio waveform is compared using an audio signal processing tool to check whether there is a significant time delay in the audio waveform recorded by the mobile terminal and the television terminal.

[0098] Step 802a, use the dynamic time warping algorithm to obtain the time domain correlation between the audio.

[0099] Here, check whether there is a significant time delay in the audio waveform recorded by the mobile terminal and the television terminal. The dynamic time warping (DTW) algorithm is applied to the audio signal to calculate the time alignment error between the audio streams of the mobile terminal and the television terminal. Refer to the above formula (3) for details.

[0100] Step 802b, use the cosine similarity to compare the correlation of the frequency spectrum.

[0101] Here, refer to the above formula (4) for details.

[0102] Step 803, integrate the time domain and frequency domain results to obtain the audio projection score.

[0103] Here, refer to the above formula (5) for details.

[0104] Referring to Figure 9 , Figure 9An application schematic diagram for representing a screen projection scoring mapping mechanism provided by an embodiment of the present application. Since in the screen projection scenario, both the image and the sound directly affect the user's experience, in order to more highlight the importance of the image and the sound in the comprehensive screen projection score, the image score and the audio score are mapped to a vector space, i.e., refer to the above formula (6). The image screen projection score and the audio screen projection score are mapped to the vector space. According to the business content, a specified region can be obtained, i.e., the vector W falls within the specified region, it can be determined that the result of this screen projection is normal; if it falls outside the specified region, it is considered that the screen projection fails. The specified region can be dynamically adjusted according to the actual business requirements.

[0105] Through steps 301 to 305, the effect and quality between the screen projection and the original content are objectively and accurately described. By synchronously recording the audio and video data of the screen projection device and the device being projected, and extracting the image and audio features respectively, and then combining the structural similarity algorithm and the time-frequency correlation analysis, the screen projection quality can be objectively and quantitatively evaluated. Compared with the existing detection method relying on subjective experience, this method realizes automatic quality evaluation, which helps to improve the detection accuracy and provides data support for optimizing the screen projection performance.

[0106] Specifically, by decomposing and timestamp aligning the audio and video data, the consistency of the image and the audio in the time dimension is ensured, thereby improving the accuracy of subsequent image and audio similarity calculation and reducing the misjudgment caused by time misalignment; by grayscale and convolution compression processing, the calculation complexity of the image data can be effectively reduced while the key visual information is retained; the structural similarity analysis can better match the human eye perception characteristics and realize accurate judgment of the consistency of the image content; by using the average brightness, variance and covariance parameters in the structural similarity algorithm, the contrast, brightness and structural information of the image can be comprehensively reflected, thereby more accurately evaluating the similarity between the images; by combining the time domain and the frequency domain to evaluate the audio synchronicity, not only whether the sound content is consistent is considered, but also whether the playback time is aligned, thereby improving the comprehensiveness and accuracy of the audio quality evaluation; by calculating the cosine similarity of the spectrum vector, the consistency of the audio signal in the frequency dimension can be measured, thereby further improving the accuracy of the audio quality evaluation; by weighting and fusing the image and audio scores, the overall quality of the screen projection is comprehensively evaluated, which can more comprehensively reflect the actual experience of the user, and is more valuable than single-dimensional evaluation.

[0107] The following continues to describe an exemplary structure of the screen projection detection device 433 implemented as a software module provided by an embodiment of the present application. In some embodiments, as shown in FIG. 4B, the software modules in the screen projection detection device 433 stored in the memory 430 can include: Figure 2 ​The data processing module 4331 is configured to, in response to a screen projection operation of a user, perform synchronous recording processing on a screen projection process to obtain first audio and video data of a screen projection device and second audio and video data of a screen projection target device; perform preprocessing on the first audio and video data and the second audio and video data to obtain first image data and first audio data and second image data and second audio data; perform structure similarity analysis on the first image data and the second image data to obtain image similarity of the first image data and the second image data; and perform time-frequency correlation analysis on the first audio data and the second audio data to obtain audio similarity of the first audio data and the second audio data.

[0108] The data determination module 4332 is configured to determine a screen projection detection score based on the image similarity and the time-frequency similarity, where the screen projection detection score is used to detect screen projection quality.

[0109] In some embodiments, the data processing module 4331 is further configured to perform decomposition processing on the first audio and video data to obtain a first image sequence and a first audio file; perform timestamp alignment processing on the first image sequence and the first audio file to obtain the first image data and the first audio data; perform decomposition processing on the second audio and video data to obtain a second image sequence and a second audio file; and perform timestamp alignment processing on the second image sequence and the second audio file to obtain the second image data and the second audio data.

[0110] In some embodiments, the data processing module 4331 is further configured to perform grayscale processing on the first image data and the second image data to obtain a first grayscale image and a second grayscale image; perform convolution compression processing on the first grayscale image and the second grayscale image to obtain first image features and second image features; and perform structure similarity analysis on the first image features and the second image features to obtain the image similarity.

[0111] In some embodiments, the data processing module 4331 is further configured to calculate average brightness, variance, and covariance of the first image features and the second image features; and perform structure similarity analysis on the first image features and the second image features based on the average brightness, the variance, and the covariance to obtain the image similarity.

[0112] In some embodiments, the data processing module 4331 is further configured to perform time domain alignment analysis on the first audio data and the second audio data to obtain a time alignment error, the time alignment error being used for time domain correlation analysis; and perform frequency spectrum analysis on the first audio data and the second audio data to obtain a frequency spectrum similarity, the frequency spectrum similarity being used for frequency domain correlation analysis.

[0113] In some embodiments, the data processing module 4331 is further configured to obtain a first frequency spectrum feature based on a frequency spectrum of the first audio data; obtain a second frequency spectrum feature based on a frequency spectrum of the second audio data; calculate a cosine similarity of the first frequency spectrum feature and the second frequency spectrum feature, and obtain the frequency spectrum similarity based on the cosine similarity.

[0114] In some embodiments, the data determination module 4332 is further configured to determine an image projection scoring based on the image similarity; determine an audio projection scoring based on the time domain similarity and the frequency domain similarity; calculate a sum of the image projection scoring and the audio projection scoring, and determine the projection detection scoring based on the sum of the image projection scoring and the audio projection scoring.

[0115] Embodiments of the present application provide a computer program product, which includes a computer program or computer executable instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device performs the projection detection method provided in the embodiments of the present application.

[0116] Embodiments of the present application provide a computer readable storage medium storing computer executable instructions, wherein the computer executable instructions or computer programs are stored, and when the computer executable instructions or computer programs are executed by a processor, the processor will execute the projection detection method provided in the embodiments of the present application, for example, the projection detection method shown in the above. Figure 3

[0117] In some embodiments, the computer readable storage medium can be a RAM, a ROM, a flash memory, a magnetic surface memory, an optical disc, or a CD-ROM memory; or can be various devices including one or any combination of the above memories.

[0118] ​In some embodiments, the computer-executable instructions can be in the form of programs, procedures, modules, scripts, or code, and can be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0119] By way of example, a computer-executable instruction can be, but is not limited to, a file in a file system, can be stored in a part of a file that holds other programs or data, for example, one or more scripts stored in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, sub programs, or code portions.

[0120] By way of example, a computer-executable instruction can be deployed to execute on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected through a communication network.

[0121] To sum up, through the embodiments of the present application, the effect and quality between the screen projection and the original content are objectively and accurately described. By synchronously recording the audio and video data of the screen projection device and the screen projection device, and extracting the image and audio features respectively, and combining the structural similarity algorithm and the time-frequency correlation analysis, the screen projection quality can be objectively and quantitatively evaluated. Compared with the existing detection method which relies on subjective experience, the method realizes automatic quality evaluation, which helps to improve the detection accuracy and provides data support for optimizing the screen projection performance.

[0122] The above merely describes the embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting screen projection, characterized in that, The method comprises: In response to a user's screen projection operation, a screen projection process is synchronously recorded to obtain first audio and video data of a screen projection device and second audio and video data of a screen projection target device; The first audio and video data and the second audio and video data are preprocessed to obtain first image data and first audio data, and second image data and second audio data; The first image data and the second image data are analyzed for structural similarity to obtain image similarity of the first image data and the second image data; The first audio data and the second audio data are analyzed for time-frequency correlation to obtain audio similarity of the first audio data and the second audio data; The image similarity and the time-frequency similarity are used to determine a screen projection detection score, wherein the screen projection detection score is used to detect screen projection quality.

2. The method of claim 1, wherein, The preprocessing of the first audio and video data and the second audio and video data comprises: The first audio and video data are decomposed to obtain a first image sequence and a first audio file; The first image sequence and the first audio file are timestamp-aligned to obtain the first image data and the first audio data; The second audio and video data are decomposed to obtain a second image sequence and a second audio file; The second image sequence and the second audio file are timestamp-aligned to obtain the second image data and the second audio data.

3. The method of claim 1, wherein, The analysis of the first image data and the second image data for structural similarity comprises: The first image data and the second image data are processed for grayscale to obtain first and second grayscale images; The first and second grayscale images are compressed by convolution to obtain first and second image features; The first and second image features are analyzed for structural similarity to obtain the image similarity.

4. The method of claim 3, wherein, The analysis of the first and second image features for structural similarity comprises: The average brightness, variance and covariance of the first and second image features are calculated; The first and second image features are analyzed for structural similarity based on the average brightness, variance and covariance to obtain the image similarity.

5. The method of claim 1, wherein, The analysis of the first and second audio data for time-frequency correlation comprises: The first and second audio data are analyzed for time-domain alignment to obtain a time alignment error, which is used for time-domain correlation analysis; The first and second audio data are analyzed for frequency spectrum to obtain a frequency spectrum similarity, which is used for frequency-domain correlation analysis.

6. The method of claim 5, wherein, The analysis of the first and second audio data for frequency spectrum to obtain a frequency spectrum similarity comprises: First frequency spectrum features are obtained based on the frequency spectrum of the first audio data; Second frequency spectrum features are obtained based on the frequency spectrum of the second audio data; calculating a cosine similarity of the first spectral feature and the second spectral feature, obtaining the spectral similarity based on the cosine similarity.

7. The method according to any one of claims 1 to 6, characterized in that, The determining the screen projection detection score based on the image similarity and the time-frequency domain similarity comprises: determining an image screen projection score based on the image similarity; determining an audio screen projection score based on the time domain similarity and the frequency domain similarity; calculating a sum of the image screen projection score and the audio screen projection score, and determining the screen projection detection score based on the sum of the image screen projection score and the audio screen projection score.

8. An electronic device, comprising: comprise: a processor and a memory for storing a computer program capable of running on the processor, wherein the processor is configured to execute the computer program to perform the steps of the method of any one of claims 1 to 7.

9. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images