Audio and video content switching method based on multi-screen interaction

Abstract

The invention discloses an audio and video content switching method based on multi-screen interaction, and relates to the technical field of multimedia, and the method comprises the steps: collecting user behavior data, equipment state parameters and content feature data in real time through a multi-source sensor; constructing a dynamic attention index model based on the user behavior data, and quantifying the attention intensity of the user on the equipment in combination with a time decay function; calculating an optimal matching score of the content and the equipment through the attention index; according to the content adaptation degree score, dynamically adjusting a weight coefficient, and generating an equipment priority weight score; selecting a target device based on the weight priority score and the energy consumption cost, and executing seamless migration of the audio and video contents; according to audio and video switching result feedback, if switching fails, parameters of priority calculation weights are adjusted based on feedback data; according to the method, the device priority weight score is generated by dynamically adjusting the weight coefficient, the system can select the target device according to the priority score, and redundant switching of high-energy-consumption devices is avoided.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia technology, and in particular relates to a method for switching audio and video content based on multi-screen interaction. Background Art

[0002] With the popularization of smart devices and the rapid development of multimedia technology, users' audio and video consumption scenarios in daily life are becoming increasingly diverse. In environments such as homes, offices, and public places, users often have multiple display devices at the same time. These devices differ in screen size, resolution, network connection performance, etc., which causes users to face many inconveniences when switching audio and video content. Traditional audio and video switching methods usually rely on manual operations of users and cannot perform intelligent adaptation and switching based on users' real-time behavior, device status, and content characteristics, resulting in poor user experience. Although existing multi-screen interactive technologies have achieved interoperability between devices to a certain extent, there are still many problems in actual applications. The switching process between devices is not smooth enough, and the audio and video synchronization is poor, resulting in unsatisfactory switching effects. Therefore, we propose an audio and video content switching method based on multi-screen interaction. Summary of the Invention

[0003] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0004] The present invention provides a method for switching audio and video content based on multi-screen interaction, comprising the following steps:

[0005] Step S1: Collect user behavior data, device status parameters and content feature data in real time through multi-source sensors;

[0006] Step S2: Build a dynamic attention index model based on user behavior data, and combine it with a time decay function to quantify the user's attention intensity on the device;

[0007] Step S3: Analyze content features and generate an adaptation matrix using the attention index to calculate the optimal matching score between content and device;

[0008] Step S4: Dynamically adjust the weight coefficient based on the content suitability score to generate a device priority weight score;

[0009] Step S5: Selecting a target device based on the weighted priority score and energy consumption cost to perform seamless migration of audio and video content;

[0010] Step S6: According to the feedback of the audio and video switching result, if the switching fails, the parameters of the priority calculation weight are adjusted based on the feedback data.

[0011] Furthermore, the step S1 includes the following steps:

[0012] Step S11: Capture the user's head orientation angle through the millimeter-wave radar array, calculate the relative distance between the user and the device, and quantify the spatial relationship between the user and the device;

[0013] Step S12: Obtain network delay, battery percentage, and screen size through the device API;

[0014] Step S13: Record the frequency of the user's touch or voice operations in the past 5 minutes to measure the user's interactive activity on the device.

[0015] Furthermore, step S2 includes the following steps:

[0016] Step S21: Based on the user's recent behavior data, reduce the weight of historical data so that the model focuses more on recent behavior. Define a time decay function. The time decay formula is as follows:

[0017]

[0018] Where W(t) is the time decay weight, t is the current timestamp, and t0 is the timestamp of the user’s last operation or gaze;

[0019] Step S22: Calculate the dynamic attention index based on the influence of the user's gaze duration and angle deviation on the attention intensity. The formula is as follows:

[0020]

[0021] Where Ai is the user's dynamic attention index for device i, W(tk) is the time decay weight of the kth gaze, θk is the duration of the user's gaze on the device at the kth time, is the angular deviation between the user's head direction and the center of the device screen. Used to quantify the accuracy of the user's gaze. The smaller the angle deviation, the closer the value is to 1.

[0022] Furthermore, step S3 includes the following steps:

[0023] Step S31: Identify the scene type of the video content and extract key frame features using a convolutional neural network model;

[0024] Step S32: Define the adaptation rules between content and device, and generate the adaptation score matrix Mij. The score is based on the preset mapping relationship between content type and device type. The adaptation score matrix Mij is as follows:

[0025]

[0026] Where Mij is the compatibility score between content type i and device j, ranging from 0 to 1, i is the content type, and j is the device type.

[0027] Furthermore, step S4 includes the following steps:

[0028] Step S41: Reflect the impact of device power on priority, and define the weight coefficient formula of power impact as follows:

[0029]

[0030] In the formula, α is the power impact weight coefficient, B is the power percentage, and the smaller α is, the lower the device priority is;

[0031] Step S42: quantify the negative impact of network delay on transmission performance, and define the weight coefficient formula of network delay impact as follows:

[0032]

[0033] Where γ is the network delay impact weight coefficient, ND is the device network delay, and the larger γ is, the lower the device priority is;

[0034] Step S43: Calculate device priority based on user behavior, device status, and content compatibility. The formula is as follows:

[0035]

[0036] Where Priorityi is the priority score of device i. The larger the value, the higher the priority. λ is the time decay coefficient, Tg is the user's gaze duration, β is the operation frequency weight coefficient, Of is the user's operation frequency in the past 5 minutes, Omax is the maximum possible value of the operation frequency, which is used for normalization. Dr is the relative distance between the user and the device, δ is the distance weight coefficient, Cs is the content and device adaptation score from the adaptation matrix Mij, and Cmax is the maximum adaptation score, which is used for normalization.

[0037] Furthermore, in step S44, to avoid redundant switching of high energy consumption equipment, an energy cost factor EnergyCosti is introduced, and the candidate equipment selection formula is as follows:

[0038]

[0039] Where, Candidate is the candidate device finally selected, Priority i is the priority score of device i, and EnergyCost i is the energy consumption cost of device i. The smaller the value, the lower the energy consumption.

[0040] A dual-threshold mechanism is set: if the priority difference between the candidate device and the current device is greater than 15%, the switch is immediate; if the priority difference is between 5% and 15%, user confirmation is required.

[0041] Furthermore, step S5 includes the following steps:

[0042] Step S51: To ensure smooth switching, preload 3 seconds of content on the target device by preloading streaming media segments.

[0043] Step S52: Generate an audio-video synchronization algorithm by compensating for network delay and device processing time difference. The formula for the audio-video synchronization time difference is as follows:

[0044]

[0045] Where Δtsync is the time difference between audio and video synchronization, Dtarget is the distance between the target device and the user, Dsource is the distance between the source device and the user, c is the speed of sound, which is 343 meters per second, and Bufferadjust is the dynamic adjustment buffer, which ranges from 0 to 500 milliseconds.

[0046] Furthermore, step S6 includes the following steps:

[0047] Step S61: Based on the audio and video switching result feedback, the algorithm deviation is corrected, the user's manual switching operation is recorded, and the weight coefficient is dynamically updated. The formula is as follows:

[0048] αnew=αold×(1-0.1×sign(ΔP));

[0049] Where αnew is the updated power impact coefficient, αold is the power impact coefficient before the update, ΔP is the difference between the algorithm priority and the user's actual selection, and sign(ΔP) is the sign of the difference, which is +1 or -1.

[0050] Step S62: Regularly recalibrate the distance mapping table between the device and the user by improving the distance measurement accuracy.

[0051] The present invention has the following beneficial effects:

[0052] 1. The present invention generates a device priority weight score by dynamically adjusting the weight coefficient and comprehensively considering factors such as device power, network latency, and user operation frequency. The system can select target devices based on the priority score and energy consumption cost to avoid redundant switching of high-energy-consuming devices. This optimization strategy not only improves the utilization efficiency of device resources, but also reduces energy consumption and extends the service life of the device.

[0053] 2. The present invention constructs a dynamic attention index model by collecting user behavior data, device status parameters and content feature data in real time, which can accurately quantify the user's attention intensity on the device. Based on this, the system can intelligently analyze content features and generate an adaptation matrix, calculate the optimal matching score between content and device, and thus achieve seamless migration of audio and video content. This method can effectively reduce the user's manual device switching operations and improve the user experience in multi-screen interaction.

[0054] 3. The present invention uses an audio and video switching result feedback mechanism to dynamically adjust the priority calculation weight parameters according to the user's actual operation and correct the algorithm deviation. The system also improves the distance measurement accuracy and regularly recalibrates the distance mapping table between the device and the user to ensure the accuracy and adaptability of the system. This feedback mechanism enables the system to be continuously optimized to adapt to the usage habits of different users and environmental changes.

[0055] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 The present invention is a flowchart of a method for switching audio and video content based on multi-screen interaction. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] See also Figure 1 As shown, the present invention is a method for switching audio and video content based on multi-screen interaction, comprising the following steps:

[0060] Step S1: Collect user behavior data, device status parameters and content feature data in real time through multi-source sensors;

[0061] Step S2: Build a dynamic attention index model based on user behavior data, and combine it with a time decay function to quantify the user's attention intensity on the device;

[0062] Step S3: Analyze content features and generate an adaptation matrix using the attention index to calculate the optimal matching score between content and device;

[0063] Step S4: Dynamically adjust the weight coefficient based on the content suitability score to generate a device priority weight score;

[0064] Step S5: Selecting a target device based on the weighted priority score and energy consumption cost to perform seamless migration of audio and video content;

[0065] Step S6: According to the feedback of the audio and video switching result, if the switching fails, the parameters of the priority calculation weight are adjusted based on the feedback data.

[0066] Step S1 includes the following steps:

[0067] Step S11: Capture the user's head orientation angle through the millimeter-wave radar array, calculate the relative distance between the user and the device, and quantify the spatial relationship between the user and the device;

[0068] Step S12: Obtain network delay, battery percentage, and screen size through the device API;

[0069] Step S13: Record the frequency of the user's touch or voice operations in the past 5 minutes to measure the user's interactive activity on the device.

[0070] Step S2 includes the following steps:

[0071] Step S21: Based on the user's recent behavior data, reduce the weight of historical data so that the model focuses more on recent behavior. Define a time decay function. The time decay formula is as follows:

[0072]

[0073] Where W(t) is the time decay weight, t is the current timestamp, and t0 is the timestamp of the user’s last operation or gaze;

[0074] Step S22: Calculate the dynamic attention index based on the influence of the user's gaze duration and angle deviation on the attention intensity. The formula is as follows:

[0075]

[0076] Where Ai is the user's dynamic attention index for device i, W(tk) is the time decay weight of the kth gaze, θk is the duration of the user's gaze on the device at the kth time, is the angular deviation between the user's head direction and the center of the device screen. Used to quantify the accuracy of the user's gaze. The smaller the angle deviation, the closer the value is to 1.

[0077] Step S3 includes the following steps:

[0078] Step S31: Identify the scene type of the video content and extract key frame features using a convolutional neural network model;

[0079] Step S32: Define the adaptation rules between content and device, and generate the adaptation score matrix Mij. The score is based on the preset mapping relationship between content type and device type. The adaptation score matrix Mij is as follows:

[0080]

[0081] Where Mij is the compatibility score between content type i and device j, ranging from 0 to 1, i is the content type, and j is the device type.

[0082] Step S4 includes the following steps:

[0083] Step S41: Reflect the impact of device power on priority, and define the weight coefficient formula of power impact as follows:

[0084]

[0085] In the formula, α is the power impact weight coefficient, B is the power percentage, and the smaller α is, the lower the device priority is;

[0086] Step S42: quantify the negative impact of network delay on transmission performance, and define the weight coefficient formula of network delay impact as follows:

[0087]

[0088] Where γ is the network delay impact weight coefficient, ND is the device network delay, and the larger γ is, the lower the device priority is;

[0089] Step S43: Calculate device priority based on user behavior, device status, and content compatibility. The formula is as follows:

[0090]

[0091] Where Priorityi is the priority score of device i. The larger the value, the higher the priority. λ is the time decay coefficient, Tg is the user's gaze duration, β is the operation frequency weight coefficient, Of is the user's operation frequency in the past 5 minutes, Omax is the maximum possible value of the operation frequency, which is used for normalization. Dr is the relative distance between the user and the device, δ is the distance weight coefficient, Cs is the content and device adaptation score from the adaptation matrix Mij, and Cmax is the maximum adaptation score, which is used for normalization.

[0092] Step S44: To avoid redundant switching of high-energy-consuming devices, an energy cost factor EnergyCost i is introduced. The candidate device selection formula is as follows:

[0093]

[0094] Where, Candidate is the candidate device finally selected, Priority i is the priority score of device i, and EnergyCost i is the energy consumption cost of device i. The smaller the value, the lower the energy consumption.

[0095] A dual-threshold mechanism is set: if the priority difference between the candidate device and the current device is greater than 15%, the switch is immediate; if the priority difference is between 5% and 15%, user confirmation is required.

[0096] Step S5 includes the following steps:

[0097] Step S51: To ensure smooth switching, preload 3 seconds of content on the target device by preloading streaming media segments.

[0098] Step S52: Generate an audio-video synchronization algorithm by compensating for network delay and device processing time difference. The formula for the audio-video synchronization time difference is as follows:

[0099]

[0100] Where Δtsync is the time difference between audio and video synchronization, Dtarget is the distance between the target device and the user, Dsource is the distance between the source device and the user, c is the speed of sound, which is 343 meters per second, and Bufferadjust is the dynamic adjustment buffer, which ranges from 0 to 500 milliseconds.

[0101] Step S6 includes the following steps:

[0102] Step S61: Based on the audio and video switching result feedback, the algorithm deviation is corrected, the user's manual switching operation is recorded, and the weight coefficient is dynamically updated. The formula is as follows:

[0103] αnew=αold×(1-0.1×sign(ΔP));

[0104] Where αnew is the updated power impact coefficient, αold is the power impact coefficient before the update, ΔP is the difference between the algorithm priority and the user's actual selection, and sign(ΔP) is the sign of the difference, which is +1 or -1.

[0105] Step S62: Regularly recalibrate the distance mapping table between the device and the user by improving the distance measurement accuracy.

[0106] A specific application of this embodiment is:

[0107] Scene construction:

[0108] The user has three devices at home: a mobile phone with a 6.5-inch screen and 80% battery; a tablet with a 10.5-inch screen and 45% battery; and a TV with a 55-inch screen and 95% battery. The user is watching a sports video and gradually switches to conference mode content.

[0109] Step S1:

[0110] A millimeter-wave radar array (operating at 60 GHz and with an angular resolution of 0.1°) deployed in the living room captures the user's head orientation at a 15° angle in real time and calculates the user's relative distance from the TV to be 3.2 meters, from the tablet to be 1.5 meters, and from the phone to be 0.8 meters.

[0111] Use the device API to obtain the network latency (TV: 50ms, tablet: 120ms, mobile phone: 80ms), battery percentage (TV: 95%, tablet: 45%, mobile phone: 80%), and screen size of each device;

[0112] Record the user's touch operation frequency in the past 5 minutes (TV: 2 times, tablet: 5 times, mobile phone: 8 times), and the voice operation frequency is 0 times. Calculate the interaction activity score (mobile phone > tablet > TV);

[0113] Step S2:

[0114] Define the time decay function Where t0 is the timestamp of the user's last viewing of the TV (10 seconds ago), and the current timestamp t is t0+10. The calculated value is W(t) = e-0.02×10≈0.818;

[0115] The user's viewing time of the TV is θk=8, and the head orientation angle deviation is Calculate the TV's attention index:

[0116] ATV=0.818×(8×0.966)≈6.32

[0117] The attention index for tablets and mobile phones is 3.15 and 2.87 respectively;

[0118] Step S3:

[0119] A pre-trained convolutional neural network (ResNet-50) is used to identify the current video content as "motion video" and extract key frame features;

[0120] According to the preset adaptation rules, the adaptability score of sports video and TV is Msports video→TV=0.7, and the adaptability score of conference mode and tablet is Mconference mode→tablet=0.9;

[0121] Step S4:

[0122] Calculate the weight coefficient α based on the power consumption: α for TV = 1.2-0.4×0.95=0.82, α for tablet = 1.2-0.4×0.45=1.02;

[0123] Calculate γ based on network delay,

[0124] The priority score is calculated by comprehensive parameters. Taking TV as an example: λ = 0.1, β = 0.5, δ = 0.3, Cs = 0.7, then:

[0125]

[0126] Similarly, the priority scores for tablets and mobile phones are 0.18 and 0.15 respectively;

[0127] Introducing energy consumption costs (TV: 5.0, tablet: 2.0, mobile phone: 1.5), calculating candidate devices:

[0128]

[0129] The priority difference (tablet 0.18: TV 0.12) is 50%, exceeding the 15% threshold, triggering automatic switching;

[0130] Step S5:

[0131] Preload 3-second streaming media segments on the tablet to ensure smooth switching;

[0132] Calculate the audio and video synchronization time difference based on the distance between the target device (tablet) and the user (1.5 meters) and the source device (TV) (3.2 meters):

[0133] Achieve audio and video synchronization through dynamic buffer adjustment;

[0134] Step S6:

[0135] If the user manually switches back to TV, the difference ΔP is recorded as 0.12-0.18=-0.06, and the power weight coefficient is updated:

[0136] αnew=0.82×(1-0.1×(-1))=0.82×1.1=0.902;

[0137] Recalibrate the distance measurement error of the millimeter-wave radar and update the distance mapping table.

[0138] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0139] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for switching audio and video content based on multi-screen interaction, characterized by: The following steps are involved: Step S1: Collect user behavior data, device status parameters and content feature data in real time through multi-source sensors; Step S2: Build a dynamic attention index model based on user behavior data, and combine it with a time decay function to quantify the user's attention intensity on the device; Step S3: Analyze content features and generate an adaptation matrix using the attention index to calculate the optimal matching score between content and device; Step S4: Dynamically adjust the weight coefficient based on the content suitability score to generate a device priority weight score; Step S5: Selecting a target device based on the weighted priority score and energy consumption cost to perform seamless migration of audio and video content; Step S6: According to the feedback of the audio and video switching result, if the switching fails, the parameters of the priority calculation weight are adjusted based on the feedback data.

2. The method for switching audio and video content based on multi-screen interaction according to claim 1, characterized in that: The step S1 includes the following steps: Step S11: Capture the user's head orientation angle through the millimeter-wave radar array, calculate the relative distance between the user and the device, and quantify the spatial relationship between the user and the device; Step S12: Obtain network delay, battery percentage, and screen size through the device API; Step S13: Record the frequency of the user's touch or voice operations in the past 5 minutes to measure the user's interactive activity on the device.

3. The method for switching audio and video content based on multi-screen interaction according to claim 1, characterized in that: The step S2 includes the following steps: Step S21: Based on the user's recent behavior data, reduce the weight of historical data so that the model focuses more on recent behavior. Define a time decay function. The time decay formula is as follows: Where W(t) is the time decay weight, t is the current timestamp, and t0 is the timestamp of the user’s last operation or gaze; Step S22: Calculate the dynamic attention index based on the influence of the user's gaze duration and angle deviation on the attention intensity. The formula is as follows: Where Ai is the user's dynamic attention index for device i, W(tk) is the time decay weight of the kth gaze, θk is the duration of the user's gaze on the device at the kth time, is the angular deviation between the user's head direction and the center of the device screen. Used to quantify the accuracy of the user's gaze. The smaller the angle deviation, the closer the value is to 1.

4. The method for switching audio and video content based on multi-screen interaction according to claim 1, characterized in that: The step S3 includes the following steps: Step S31: Identify the scene type of the video content and extract key frame features using a convolutional neural network model; Step S32: Define the adaptation rules between content and device, and generate the adaptation score matrix Mij. The score is based on the preset mapping relationship between content type and device type. The adaptation score matrix Mij is as follows: Where Mij is the compatibility score between content type i and device j, ranging from 0 to 1, i is the content type, and j is the device type.

5. The method for switching audio and video content based on multi-screen interaction according to claim 1, characterized in that: The step S4 comprises the following steps: Step S41: Reflect the impact of device power on priority, and define the weight coefficient formula of power impact as follows: In the formula, α is the power impact weight coefficient, B is the power percentage, and the smaller α is, the lower the device priority is; Step S42: quantify the negative impact of network delay on transmission performance, and define the weight coefficient formula of network delay impact as follows: Where γ is the network delay impact weight coefficient, ND is the device network delay, and the larger γ is, the lower the device priority is; Step S43: Calculate device priority based on user behavior, device status, and content compatibility. The formula is as follows: Where Priorityi is the priority score of device i. The larger the value, the higher the priority. λ is the time decay coefficient, Tg is the user's gaze duration, β is the operation frequency weight coefficient, Of is the user's operation frequency in the past 5 minutes, Omax is the maximum possible value of the operation frequency, which is used for normalization. Dr is the relative distance between the user and the device, δ is the distance weight coefficient, Cs is the content and device adaptation score from the adaptation matrix Mij, and Cmax is the maximum adaptation score, which is used for normalization.

6. The method for switching audio and video content based on multi-screen interaction according to claim 5, characterized in that: Step S44: To avoid redundant switching of high-energy-consuming devices, an energy cost factor EnergyCost i is introduced. The candidate device selection formula is as follows: Where, Candidate is the candidate device finally selected, Priority i is the priority score of device i, and EnergyCost i is the energy consumption cost of device i. The smaller the value, the lower the energy consumption. A dual-threshold mechanism is set: if the priority difference between the candidate device and the current device is greater than 15%, the switch is immediate; if the priority difference is between 5% and 15%, user confirmation is required.

7. The method for switching audio and video content based on multi-screen interaction according to claim 1, characterized in that: The step S5 includes the following steps: Step S51: Preload 3 seconds of content on the target device by preloading streaming media segments; Step S52: Generate an audio-video synchronization algorithm by compensating for network delay and device processing time difference. The formula for the audio-video synchronization time difference is as follows: Where Δtsync is the time difference between audio and video synchronization, Dtarget is the distance between the target device and the user, Dsource is the distance between the source device and the user, c is the speed of sound, which is 343 meters per second, and Bufferadjust is the dynamic adjustment buffer, which ranges from 0 to 500 milliseconds.

8. The method for switching audio and video content based on multi-screen interaction according to claim 1, characterized in that: The step S6 includes the following steps: Step S61: Based on the audio and video switching result feedback, the algorithm deviation is corrected, the user's manual switching operation is recorded, and the weight coefficient is dynamically updated. The formula is as follows: αnew=αold×(1-0.1×sign(ΔP)); Where αnew is the updated power impact coefficient, αold is the power impact coefficient before the update, ΔP is the difference between the algorithm priority and the user's actual selection, and sign(ΔP) is the sign of the difference, which is +1 or -1. Step S62: Periodically recalibrate the distance mapping table between the device and the user.

Images