Application scenario recognition method and device, equipment storage medium and program product

Abstract

The application relates to an application scenario identification method and device, electronic equipment, a computer storage medium and a computer program product. The method comprises the following steps: acquiring at least one of current load information, current device temperature information, current network state information and current multimedia data information when an electronic device is in a target multimedia state; determining a target load scenario according to the current load information, determining a target temperature scenario according to the current device temperature information, determining a target network state scenario according to the current network state information, and determining a target multimedia type scenario according to the current multimedia data information; and obtaining a scenario identifier in the target multimedia state according to at least one of the target load scenario, the target temperature scenario, the target network state scenario or the target multimedia type scenario. The application can accurately identify the application scenario type in the multimedia state.

Description

Technical Field

[0001] This application relates to the field of computer application technology, and in particular to an application scenario identification method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the widespread use of electronic devices and the development of internet and multimedia technologies, more and more users are using electronic devices for online communication, work, and study. Users also use electronic devices for multimedia services, such as live streaming. Live streaming scenarios include various complex situations, and traditional technologies cannot accurately identify these diverse scenarios, such as those present in a live stream. Summary of the Invention

[0003] This application provides an application scene recognition method, apparatus, electronic device, and computer-readable storage medium, which can improve the accuracy of scene recognition.

[0004] Firstly, this application provides an application scenario identification method, the method comprising:

[0005] When the electronic device is in the target multimedia state, acquire at least one of the following: current load information, current device temperature information, current network status information, and current multimedia data information;

[0006] The target load scenario is determined based on the current load information, the target temperature scenario is determined based on the current device temperature information, the target network status scenario is determined based on the current network status information, and the target multimedia type scenario is determined based on the current multimedia data information.

[0007] Based on at least one of the target load scenario, target temperature scenario, target network status scenario, or target multimedia type scenario, obtain the scenario identifier under the target multimedia status.

[0008] Secondly, this application also provides an application scenario recognition device, the device comprising:

[0009] The information acquisition module is used to acquire at least one of the following when the electronic device is in the target multimedia state: current load information, current device temperature information, current network status information, and current multimedia data information;

[0010] The scenario determination module is used to determine the target load scenario based on the current load information, the target temperature scenario based on the current device temperature information, the target network status scenario based on the current network status information, and the target multimedia type scenario based on the current multimedia data information.

[0011] The scene marking module is used to obtain the scene identifier under the target multimedia state based on at least one of the target load scene, target temperature scene, target network state scene, or target multimedia type scene.

[0012] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps in the first aspect.

[0013] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method steps of the first aspect.

[0014] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the method of the first aspect.

[0015] The aforementioned application scenario identification method, device, electronic device, computer storage medium, and computer program product, when the electronic device is in the target multimedia state, acquire one or more of the following: current load information, current device temperature information, current network status information, and current multimedia data information. Based on each piece of information, the target load scenario, target temperature scenario, target network status scenario, and target multimedia type scenario are determined, thereby achieving accurate identification of the application scenario. Furthermore, based on one or more of the target load scenario, target temperature scenario, target network status scenario, or target multimedia type scenario, the scenario identifier in the target multimedia state is obtained, thus accurately obtaining the scenario identifier in the target multimedia state. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of a scene recognition framework in one embodiment;

[0018] Figure 2 This is a flowchart illustrating an application scenario identification method in one embodiment;

[0019] Figure 3 This is a flowchart illustrating a scenario in which the target network state is determined based on the current network state information, as shown in one embodiment.

[0020] Figure 4This is a flowchart illustrating a scenario in which a target multimedia type is determined based on the current multimedia data information, as shown in one embodiment.

[0021] Figure 5 This is a schematic diagram illustrating the training method of a multimodal recognition model in one embodiment;

[0022] Figure 6 This is a schematic diagram illustrating the process of identifying whether the multimedia state of the current application is in a live broadcast state in one embodiment;

[0023] Figure 7 This is a schematic diagram illustrating the process of multi-module linkage in one embodiment;

[0024] Figure 8 This is a schematic diagram of the recognition process of a multimodal deep learning model in one embodiment;

[0025] Figure 9 This is a schematic diagram of the training process of a multimodal recognition model in one embodiment;

[0026] Figure 10 This is a schematic diagram of the application scenario recognition device in one embodiment;

[0027] Figure 11 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0029] In related technologies, inaccurate scene identification for multimedia applications leads to mismatches between the performance configuration of electronic devices and the scenarios, potentially causing issues such as choppy playback or high power consumption. This application provides an application scene identification framework. Through the coordinated operation of modules within this framework, application scenes can be identified more accurately, providing a basis for accurately adjusting various performance parameters of electronic devices. For example... Figure 1As shown, the recognition framework for this application scenario includes a recognition module 102, an application activity management module 104, a processor (Central Processing Unit, CPU) 106, a temperature control module 108, a network module 110, a codec module 112, a multimodal recognition model 114, and a multimedia application 116. The application activity management module 104, processor 106, temperature control module 108, network module 110, codec module 112, multimodal recognition model 114, and multimedia application 116 all communicate with the recognition module 102. The recognition module 102 is located on an electronic device and can be a control module. The control module can be a central processing unit, a graphics processing unit, or an embedded processor, etc., and is not limited to these. The multimodal recognition model can be a deep learning model. The application activity management module is used to obtain top-level activity information and transmit this information to the recognition module 102. The network module is used to obtain current network status information and transmit it to the recognition module 102. The processor is used to obtain current load information and transmit it to the recognition module 102. The temperature control module detects the current temperature of the electronic device and transmits this information to the recognition module 102. The encoding / decoding module acquires image and audio data output by multimedia applications, extracts image features from the image data, extracts audio features from the audio data, and transmits these features to the multimodal recognition model. The multimodal recognition model identifies the content scene within the multimedia scenario based on the image and audio features and transmits this content scene to the recognition module 102. The encoding / decoding module can also transmit its own status information to the recognition module 102. The recognition module 102 performs a comprehensive judgment based on the received information, configuring or optimizing scene performance to adapt the electronic device to the scene.

[0030] In some exemplary embodiments, such as Figure 2 As shown, an application scenario identification method is provided. Taking the application of this method to electronic devices as an example, the electronic devices can be, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. This application scenario identification method includes the following steps 202 to 206. Wherein:

[0031] Step 202: When the electronic device is in the target multimedia state, acquire at least one of the following: current load information, current device temperature information, current network status information, and current multimedia data information.

[0032] Multimedia can be a combination of various media, including one or more forms such as text, graphics, images, video, audio, and animation. Multimedia applications can include video applications, live streaming applications, and the target multimedia state can be set as needed, such as a live streaming state, a video conferencing state, etc.

[0033] Current load information is used to indicate the current load of the processor, and is used to measure the ratio between the amount of tasks the processor is processing and the maximum amount of tasks it can process. For example, the maximum processing capacity of a single-core processor is considered to be 100%. If the tasks it is currently processing occupy 50% of the processing capacity, then its load is 50%.

[0034] Current device temperature information indicates the current temperature of the electronic device. This information can be obtained in real-time or periodically by a temperature control module. The temperature control module can be a hardware device, such as a temperature sensor, or a software module. A software module can estimate the overall temperature of the electronic device by collecting data from multiple hardware sensors (such as battery current, voltage, CPU usage, etc.).

[0035] Current network status information indicates the current network condition of electronic devices. This information can be obtained in real-time or periodically by network modules. It may include network quality parameters, such as bandwidth utilization, latency, jitter, or packet loss rate. Bandwidth utilization refers to the ratio of the bandwidth actually used by a network link or device to the total available bandwidth within a given time period; it measures the effective utilization of network bandwidth resources. Latency refers to the time interval between data transmission from the sender to the receiver. Jitter refers to the degree of variation in network latency, specifically the difference in latency between a series of data packets transmitted along the same path. Packet loss rate is the ratio of the number of lost data packets to the total number of data packets transmitted during network transmission. Packet loss can be caused by various factors, including network congestion, transmission errors, and network equipment failure.

[0036] The current multimedia data information may include video data currently output by the multimedia application, obtained by the encoding / decoding module. This video data may include image and audio data. The current multimedia data information may also include multimedia data stream number information obtained by the encoding / decoding module. This multimedia data stream number information may include the number of video data streams and the number of audio data streams.

[0037] For example, the processor obtains the current load information, the temperature control module obtains the current device temperature information, the network module obtains the current network status information, and the encoding / decoding module obtains the current multimedia data information.

[0038] Step 204: Determine the target load scenario based on the current load information, determine the target temperature scenario based on the current device temperature information, determine the target network status scenario based on the current network status information, and determine the target multimedia type scenario based on the current multimedia data information.

[0039] This can be achieved by pre-establishing a correspondence between load threshold ranges and load scenarios, comparing the current load information with the load threshold ranges to determine the corresponding load threshold ranges, and then obtaining the target load scenario based on the load scenario corresponding to the load threshold range.

[0040] A pre-established correspondence between temperature threshold ranges and temperature scenarios can be made. The current device temperature information can be compared with the temperature threshold range to determine the corresponding temperature threshold range. Then, the target temperature scenario can be obtained based on the temperature scenario corresponding to the temperature threshold range.

[0041] The relationship between network state score and network state scenario can be established in advance. Then, the current network state score can be calculated based on the current network state information. Based on the current network state score, the corresponding network state scenario can be obtained to obtain the target network state scenario.

[0042] The current multimedia data information can include video data output by multimedia applications and multimedia data stream number information. Based on the multimedia data stream number information, the data stream scenario can be determined. A multimodal recognition model can be pre-trained and used to identify the video data in the current multimedia data information to obtain the target multimedia type scenario.

[0043] Step 206: Obtain the scene identifier under the target multimedia state based on at least one of the target load scenario, target temperature scenario, target network state scenario, or target multimedia type scenario.

[0044] For example, one or more of the following can be used as the scene identifier under the target multimedia state: target load scenario, target temperature scenario, target network state scenario, or target multimedia type scenario.

[0045] For example, a preset number of scenarios can be selected as scenario identifiers for the target multimedia state, based on their priority from high to low. The scenarios from high to low priority can be configured as needed, such as network status scenarios, multimedia type scenarios, temperature scenarios, and load scenarios, etc., and are not limited to these.

[0046] The aforementioned application scenario identification method, when the electronic device is in the target multimedia state, acquires one or more of the following: current load information, current device temperature information, current network status information, and current multimedia data information. Based on each piece of information, it determines the target load scenario, target temperature scenario, target network status scenario, and target multimedia type scenario, thus achieving accurate identification of the application scenario. Furthermore, it obtains the scenario identifier in the target multimedia state based on one or more of the target load scenario, target temperature scenario, target network status scenario, or target multimedia type scenario, enabling accurate identification of the scenario in the multimedia state. This provides an accurate basis for the electronic device to allocate resources and adjust performance according to the scenario in the multimedia state, allowing the electronic device to adapt to the corresponding application scenario.

[0047] In some exemplary embodiments, determining the target load scenario based on the current load information includes: comparing the current load information with a load threshold range; determining the target load threshold range in which the current load information is located; and taking the load scenario corresponding to the target load threshold range as the target load scenario.

[0048] The load threshold range can be set as needed, with 2, 3, 4, or other different load threshold ranges. The number of load threshold ranges is the same as the number of load scenarios, with one load threshold range corresponding to one load scenario. For example, the load threshold ranges may include a first load threshold range, a second load threshold range, a third load threshold range, and a fourth load threshold range. The maximum value of the first load threshold range is less than the minimum value of the second load threshold range, the maximum value of the second load threshold range is less than the minimum value of the third load threshold range, and the maximum value of the third load threshold range is less than the minimum value of the fourth load threshold range. Different load threshold ranges correspond to different load scenarios, establishing a correspondence between load threshold ranges and load scenarios. The first load threshold range corresponds to the first load scenario, the second load threshold range corresponds to the second load scenario, the third load threshold range corresponds to the third load scenario, and the fourth load threshold range corresponds to the fourth load scenario. The load corresponding to the first load scenario is less than the load corresponding to the second load scenario, the load corresponding to the second load scenario is less than the load corresponding to the third load scenario, and the load corresponding to the third load scenario is less than the load corresponding to the fourth load scenario. For example, the first load threshold range is less than 30%, the second load threshold range is greater than or equal to 30% and less than 80%, the third load threshold range is greater than or equal to 80% and less than 90%, and the fourth load threshold range is greater than 90%. The first load scenario can be a light load scenario, the second load scenario can be a medium load scenario, the third load scenario can be a high load scenario, and the fourth load scenario can be an ultra-high load scenario, as shown in Table 1.

[0049] Table 1 CPU Load Scenarios

[0050] CPU load Scene <30％ Light load scenarios 30％～80％ Medium load scenarios 80％～90％ High load scenarios >90％ Ultra-high load scenarios

[0051] For example, by comparing the current load information with various load threshold ranges, the target load threshold range in which the current load information falls can be determined. Then, the load scenario corresponding to the target load threshold range is taken as the target load scenario. For instance, if the current load information includes a CPU load of 40%, and 40% falls between 30% and 80%, which is the second load threshold range, then the corresponding second load scenario is a medium load scenario. By associating different load threshold ranges with different load scenarios, and comparing the current load information with the load threshold ranges, the load scenario can be quickly and accurately determined.

[0052] In some exemplary embodiments, determining the target temperature scenario based on the current device temperature information includes: comparing the current device temperature information with a temperature threshold range; determining the target temperature threshold range in which the current device temperature information is located; and using the temperature scenario corresponding to the target temperature threshold range as the target temperature scenario.

[0053] Different temperature threshold ranges can be set to correspond to different temperature scenarios. Multiple temperature threshold ranges can be set as needed, such as 2, 3, 4, etc., and are not limited to this. The number of temperature threshold ranges is the same as the number of temperature scenarios; one temperature threshold range corresponds to one temperature scenario.

[0054] For example, the temperature threshold range may include a first temperature threshold range and a second temperature threshold range. The maximum value of the first temperature threshold range is less than the minimum value of the second temperature threshold range. The temperature scenario corresponding to the first temperature threshold range is the first temperature scenario, and the temperature scenario corresponding to the second temperature threshold range is the second temperature scenario. The temperature corresponding to the first temperature scenario is lower than the temperature corresponding to the second temperature scenario. For example, the first temperature threshold range is less than 45 degrees Celsius, and the second temperature threshold range is greater than or equal to 45 degrees Celsius. The first temperature scenario can be a normal temperature scenario, and the second temperature scenario can be a high temperature scenario.

[0055] In another example, the temperature threshold range may include a first temperature threshold range, a second temperature threshold range, and a third temperature threshold range. The maximum value of the first temperature threshold range is less than the minimum value of the second temperature threshold range, and the maximum value of the second temperature threshold range is less than the minimum value of the third temperature threshold range. The temperature scenario corresponding to the first temperature threshold range is the first temperature scenario, the temperature scenario corresponding to the second temperature threshold range is the second temperature scenario, and the temperature scenario corresponding to the third temperature threshold range is the third temperature scenario. The temperature corresponding to the first temperature scenario is lower than the temperature corresponding to the second temperature scenario, and the temperature corresponding to the second temperature scenario is lower than the temperature corresponding to the third temperature scenario. For example, the first temperature threshold range is less than 43 degrees Celsius, the second temperature threshold range is greater than or equal to 43 degrees Celsius and less than 60 degrees Celsius, and the third temperature threshold range is greater than or equal to 60 degrees Celsius. The first temperature scenario can be a normal temperature scenario, the second temperature scenario can be a high temperature scenario, and the third temperature scenario can be an ultra-high temperature scenario.

[0056] For example, by comparing the current device temperature information with the temperature threshold range, the target temperature range threshold where the current device temperature information is located can be determined. The temperature scenario corresponding to the target temperature threshold range is taken as the target temperature scenario. For example, if the current device temperature information is 45 degrees Celsius, it can be determined that 45 degrees Celsius is in the second temperature threshold range, and the temperature scenario is the second temperature scenario, such as a high temperature scenario.

[0057] Different temperature threshold ranges correspond to different temperature scenarios. The temperature scenario can be determined based on the current device temperature information and the temperature threshold range, which improves the accuracy and convenience of temperature scenario identification.

[0058] In some exemplary embodiments, the method further includes: if the target temperature scenario is a second temperature scenario or a third temperature scenario, freezing or closing background applications. The second temperature scenario is a high-temperature scenario, and the third temperature scenario is an ultra-high-temperature scenario. Freezing background applications can mean preventing background applications from using hardware resources. By freezing or closing background applications, the use of hardware resources such as processors can be reduced, power consumption can be reduced, and thus the temperature can be lowered.

[0059] In some exemplary embodiments, the current network status information includes one or more of the following: bandwidth utilization, latency, packet loss rate, and jitter duration. "Multiple" refers to two or more of these.

[0060] The current bitrate can be obtained through the codec module. Dividing the current bitrate by the available bandwidth yields the bandwidth utilization rate. Bitrate refers to the number of bits of data transmitted per unit time, and can be expressed in Kbps (kilobits per second) or Mbps (megabits per second). Bitrate determines the amount of data in a video. A higher bitrate means the video can contain more information, such as richer colors, higher resolution, and more complex details.

[0061] The correspondence between bandwidth utilization threshold ranges and network state scenarios can be pre-configured. The number of bandwidth utilization threshold ranges is the same as the number of network state scenarios, and one bandwidth utilization threshold range corresponds to one network state scenario. The division of bandwidth utilization threshold ranges can be customized as needed. For example, the bandwidth utilization threshold ranges include a first bandwidth utilization threshold range, a second bandwidth utilization threshold range, a third bandwidth utilization threshold range, and a fourth bandwidth utilization threshold range. The maximum value of the first bandwidth utilization threshold range is less than the minimum value of the second bandwidth utilization threshold range, the maximum value of the second bandwidth utilization threshold range is less than the minimum value of the third bandwidth utilization threshold range, and the maximum value of the third bandwidth utilization threshold range is less than the minimum value of the fourth bandwidth utilization threshold range. Network state scenarios include a first congestion scenario, a second congestion scenario, a third congestion scenario, and a fourth congestion scenario. The congestion level corresponding to the first congestion scenario is less than the congestion level corresponding to the second congestion scenario, the congestion level corresponding to the second congestion scenario is less than the congestion level corresponding to the third congestion scenario, and the congestion level corresponding to the third congestion scenario is less than the congestion level corresponding to the fourth congestion scenario. The first bandwidth utilization threshold range corresponds to the first congestion scenario, the second bandwidth utilization threshold range corresponds to the second congestion scenario, the third bandwidth utilization threshold range corresponds to the third congestion scenario, and the fourth bandwidth utilization threshold range corresponds to the fourth congestion scenario. For example, the first bandwidth utilization threshold range can be less than 70%, the second bandwidth utilization threshold range can be greater than or equal to 70% and less than 85%, the third bandwidth utilization threshold range can be greater than or equal to 85% and less than 95%, and the fourth bandwidth utilization threshold range can be greater than 95%, and so on. The first congestion scenario can be a normal network scenario, the second congestion scenario can be a mild network congestion scenario, the third congestion scenario can be a moderate network congestion scenario, and the fourth congestion scenario can be a severe network congestion scenario.

[0062] The correspondence between latency threshold ranges and network state scenarios can be pre-configured. The number of latency threshold ranges is the same as the number of network state scenarios, and one latency threshold range corresponds to one network state scenario. The division of latency threshold ranges can be customized as needed. For example, the latency threshold ranges include a first latency threshold range, a second latency threshold range, a third latency threshold range, and a fourth latency threshold range. The maximum value of the first latency threshold range is less than the minimum value of the second latency threshold range, the maximum value of the second latency threshold range is less than the minimum value of the third latency threshold range, and the maximum value of the third latency threshold range is less than the minimum value of the fourth latency threshold range. The first latency threshold range corresponds to the first congestion scenario, the second latency threshold range corresponds to the second congestion scenario, the third latency threshold range corresponds to the third congestion scenario, and so on. For example, the first latency threshold range can be less than 50 milliseconds (ms), the second latency threshold range can be greater than or equal to 50 ms and less than 100 ms, the third latency threshold range can be greater than or equal to 100 ms and less than 150 ms, and the fourth latency threshold range can be greater than 150 ms.

[0063] The correspondence between jitter duration threshold ranges and network state scenarios can be pre-configured. The number of jitter duration threshold ranges is the same as the number of network state scenarios, and one jitter duration threshold range corresponds to one network state scenario. The division of jitter duration threshold ranges can be customized as needed. For example, the jitter duration threshold ranges include a first jitter threshold range, a second jitter threshold range, a third jitter threshold range, and a fourth jitter threshold range. The maximum value of the first jitter threshold range is less than the minimum value of the second jitter threshold range, the maximum value of the second jitter threshold range is less than the minimum value of the third jitter threshold range, and the maximum value of the third jitter threshold range is less than the minimum value of the fourth jitter threshold range. The first jitter threshold range corresponds to the first congestion scenario, the second jitter threshold range corresponds to the second congestion scenario, the third jitter threshold range corresponds to the third congestion scenario, and the fourth jitter threshold range corresponds to the third congestion scenario. For example, the first jitter threshold range can be less than 30 milliseconds (ms), the second jitter threshold range can be greater than or equal to 30 ms and less than 50 ms, the third jitter threshold range can be greater than or equal to 50 ms and less than 70 ms, and the fourth jitter threshold range can be greater than 70 ms.

[0064] The mapping between packet loss rate threshold ranges and network state scenarios can be pre-configured. The number of packet loss rate threshold ranges is the same as the number of network state scenarios, and one packet loss rate threshold range corresponds to one network state scenario. The packet loss rate threshold ranges can be divided as needed. For example, the packet loss rate threshold ranges include a first packet loss rate threshold range, a second packet loss rate threshold range, a third packet loss rate threshold range, and a fourth packet loss rate threshold range. The maximum value of the first packet loss rate threshold range is less than the minimum value of the second packet loss rate threshold range, the maximum value of the second packet loss rate threshold range is less than the minimum value of the third packet loss rate threshold range, and the maximum value of the third packet loss rate threshold range is less than the minimum value of the fourth packet loss rate threshold range. The first packet loss rate threshold range corresponds to the first congestion scenario, the second packet loss rate threshold range corresponds to the second congestion scenario, the third packet loss rate threshold range corresponds to the third congestion scenario, and the fourth packet loss rate threshold range corresponds to the third congestion scenario. For example, the first packet loss rate threshold range can be less than 1%, the second packet loss rate threshold range can be greater than or equal to 1% and less than 2%, the third packet loss rate threshold range can be greater than or equal to 2% and less than 5%, and the fourth packet loss rate threshold range can be greater than 5%.

[0065] The correspondence between bandwidth utilization, latency, jitter, and packet loss rate in network status information and network status scenarios is shown in Table 2. It is understood that these values ​​can be adjusted as needed and are not limited to specific parameters. Figure 2 The values ​​listed in the document.

[0066] Table 2 Definitions of Network Status Indicators

[0067] Network Status Scenario Bandwidth utilization Delay duration Shaking duration Packet loss rate Normal network scenarios <70％ <50ms <50ms <1％ Mild network congestion scenarios 70％～85％ 50ms~100ms 30ms~70ms 1％～2％ Moderate network congestion scenarios 85％～95％ 100ms~150ms 50ms~70ms 2％～5％ Severe network congestion scenarios >95％ >150ms >70ms >5％

[0068] In some exemplary embodiments, determining the target network state scenario based on the current network state information may include: comparing the broadband utilization rate with a broadband utilization rate threshold range to determine the target broadband utilization threshold range in which the broadband utilization rate falls, and using the network state scenario corresponding to the target broadband utilization rate threshold range as the target network state scenario. For example, if the broadband utilization rate is 80%, the target broadband utilization rate threshold range can be determined as a second broadband utilization rate threshold range (e.g., 70% to 85%), and the second broadband utilization rate threshold range corresponds to a second congestion scenario, that is, the second congestion scenario is used as the target network state scenario. If the second congestion scenario is a mild network congestion scenario, then the target network state scenario is a mild network congestion scenario.

[0069] In some exemplary embodiments, determining the target network state scenario based on the current network state information may include: comparing the delay duration with a delay duration threshold range to determine the target delay threshold range in which the delay duration falls, and using the network state scenario corresponding to the target delay threshold range as the target network state scenario. For example, if the delay duration is 60ms, the target delay threshold range can be determined as a second delay threshold range (e.g., 50ms to 100ms), and the second delay threshold range corresponds to a second congestion scenario. That is, the second congestion scenario is used as the target network state scenario. If the second congestion scenario is a mild network congestion scenario, then the target network state scenario is a mild network congestion scenario.

[0070] In some exemplary embodiments, determining the target network state scenario based on the current network state information may include: comparing the jitter duration with a jitter duration threshold range to determine the target jitter threshold range in which the jitter duration falls, and using the network state scenario corresponding to the target jitter threshold range as the target network state scenario. For example, if the jitter duration is 40ms, the target jitter threshold range can be determined as a second jitter threshold range (e.g., 30ms to 50ms), and the second jitter threshold range corresponds to a second congestion scenario. That is, the second congestion scenario is used as the target network state scenario. If the second congestion scenario is a mild network congestion scenario, then the target network state scenario is a mild network congestion scenario.

[0071] In some exemplary embodiments, determining the target network state scenario based on the current network state information may include: comparing the packet loss rate with a packet loss rate threshold range to determine the target packet loss threshold range where the packet loss rate threshold falls, and using the network state scenario corresponding to the target packet loss rate threshold range as the target network state scenario. For example, if the packet loss rate is 1.5%, the target packet loss rate threshold range can be determined as a second packet loss rate threshold range (e.g., 1% to 2%), and the second packet loss rate threshold range corresponds to a second congestion scenario, that is, the second congestion scenario is used as the target network state scenario. If the second congestion scenario is a mild network congestion scenario, then the target network state scenario is a mild network congestion scenario.

[0072] By comparing one of the following metrics—bandwidth utilization, latency, jitter duration, or packet loss rate—with the corresponding threshold, the target network state scenario can be quickly determined.

[0073] In some exemplary embodiments, a comprehensive score can be calculated based on bandwidth utilization, latency, packet loss rate, and jitter duration from the current network status information, and the target network status scenario can be determined based on the comprehensive score. For example... Figure 3 As shown, determining the target network state scenario based on the current network state information includes steps 302 to 312. Wherein:

[0074] Step 302: Determine the broadband utilization score based on the broadband utilization rate and the broadband utilization rate threshold range.

[0075] The mapping between broadband utilization threshold ranges and broadband utilization score calculation methods can be configured, and this mapping can be represented by a function. Different broadband utilization threshold ranges correspond to different broadband utilization score calculation methods. For example, the first broadband utilization threshold corresponds to a broadband utilization score calculation method of 1, and the second broadband utilization threshold corresponds to a broadband utilization score calculation method of score = 1 - a(x - 70%) / (85% - 70%), where score is the score, a is a positive number, and x is the broadband utilization rate. Other thresholds are similar.

[0076] The broadband utilization rate is compared with the broadband utilization rate threshold range to determine the target broadband utilization rate threshold range. The broadband utilization rate score is then calculated using the broadband utilization rate score calculation method corresponding to the target broadband utilization rate threshold range.

[0077] Step 304: Determine the delay score based on the delay duration and the delay duration threshold range.

[0078] The mapping between latency threshold ranges and latency score calculation methods can be configured, and this mapping can be represented by a function. Different latency threshold ranges correspond to different latency score calculation methods. For example, the latency score calculation method corresponding to the first latency threshold can be 1, and the latency score calculation method corresponding to the second latency threshold can be score = 1 - b(x - 50) / (100 - 50), where score is the score, b is a positive number, and x is the latency. Other thresholds are similar.

[0079] The delay duration is compared with the delay duration threshold range to determine the target delay threshold range. The delay score is then calculated using the delay score calculation method corresponding to the target delay threshold range.

[0080] Step 306: Determine the packet loss rate score based on the packet loss rate and the packet loss rate threshold range.

[0081] The mapping between packet loss rate threshold ranges and packet loss rate score calculation methods can be configured, and this mapping can be represented by a function. Different packet loss rate threshold ranges correspond to different packet loss rate score calculation methods. For example, the packet loss rate score calculation method corresponding to the first packet loss rate threshold can be 1, and the packet loss rate score calculation method corresponding to the second packet loss rate threshold can be score = 1 - c(x - 1%) / (2% - 1%), where score is the score, c is a positive number, and x is the packet loss rate. Other thresholds are similar.

[0082] The packet loss rate is compared with the packet loss rate threshold range to determine the target packet loss rate threshold range. The packet loss rate score is then calculated using the packet loss rate score calculation method corresponding to the target packet loss rate threshold range.

[0083] Step 308: Determine the jitter score based on the jitter duration and the jitter duration threshold range.

[0084] The mapping between jitter duration threshold ranges and jitter score calculation methods can be configured, and this mapping can be represented by a function. Different jitter duration threshold ranges correspond to different jitter score calculation methods. For example, the first jitter duration threshold corresponds to a jitter score calculation method of 1, and the second jitter duration threshold corresponds to a jitter score calculation method of score = 1 - d(x - 50) / (100 - 50), where score is the score, d is a positive number, and x is the jitter duration. Other thresholds are similar.

[0085] The jitter duration is compared with the jitter duration threshold range to determine the target jitter threshold range. The jitter score is then calculated using the jitter score calculation method corresponding to the target jitter threshold range.

[0086] Step 310: Obtain a comprehensive score based on at least one of the broadband utilization score, the latency score, the packet loss rate score, or the jitter score.

[0087] The weights of each indicator in the network status information can be configured, that is, the corresponding weights can be configured for bandwidth utilization, latency, packet loss rate and jitter duration. The first weight corresponding to bandwidth utilization is greater than the second weight corresponding to latency, the second weight corresponding to latency is greater than the third weight corresponding to packet loss rate, and the second weight corresponding to latency is greater than the fourth weight corresponding to jitter duration.

[0088] For example, a comprehensive score is obtained by weighting at least one of the broadband utilization score, the latency score, the packet loss rate score, or the jitter score.

[0089] For example, a comprehensive score is obtained by weighting the broadband utilization score and its corresponding first weight, the latency score and its corresponding second weight, the packet loss rate score and its corresponding third weight, and the jitter score and its corresponding fourth weight.

[0090] Step 312: Based on the comprehensive score, determine the target network state scenario from the correspondence between the comprehensive score threshold range and the network state scenario.

[0091] The comprehensive score threshold range can be divided as needed. The number of comprehensive score threshold ranges is the same as the number of network state scenarios, with one comprehensive score threshold range corresponding to one network state scenario. For example, the comprehensive score threshold ranges include a first comprehensive score threshold range, a second comprehensive score threshold range, a third comprehensive score threshold range, and a fourth comprehensive score threshold range. The minimum value of the first comprehensive score threshold range is greater than the maximum value of the second comprehensive score threshold range, the minimum value of the second comprehensive score threshold range is greater than the maximum value of the third comprehensive score threshold range, and the minimum value of the third comprehensive score threshold range is greater than the maximum value of the fourth comprehensive score threshold range. Network state scenarios include a first congestion scenario, a second congestion scenario, a third congestion scenario, and a fourth congestion scenario. The congestion level corresponding to the first congestion scenario is less than that corresponding to the second congestion scenario, the second congestion scenario is less than that corresponding to the third congestion scenario, and the third congestion scenario is less than that corresponding to the fourth congestion scenario. The first comprehensive score threshold range corresponds to the first congestion scenario, the second comprehensive score threshold range corresponds to the second congestion scenario, the third comprehensive score threshold range corresponds to the third congestion scenario, and the fourth comprehensive score threshold range corresponds to the fourth congestion scenario. The first congestion scenario can be a normal network scenario, the second congestion scenario can be a mild network congestion scenario, the third congestion scenario can be a moderate network congestion scenario, and the fourth congestion scenario can be a severe network congestion scenario.

[0092] The overall score is compared with the overall score threshold range to determine the target overall score threshold range. The network state scenario corresponding to the target overall score threshold range is taken as the target network state scenario. For example, an overall score of 1 point and an overall score threshold range of 1 indicate the first congestion scenario, i.e., a normal network scenario. An overall score of 0.8 and an overall score threshold range greater than or equal to 0.7 and less than 1 indicate the second congestion scenario, i.e., a mild network congestion scenario.

[0093] By comprehensively calculating a score using multiple network parameters such as bandwidth utilization, latency, jitter duration, and packet loss rate, the target network state scenario can be determined more accurately based on the comprehensive score.

[0094] The network status scenario can be used to determine whether data streaming and distribution can be satisfied. If the network status scenario is normal, it means that data streaming and distribution can be satisfied. If the network status scenario is moderately congested or heavily congested, it means that data streaming and distribution cannot be satisfied, and the data stream needs to be adjusted.

[0095] In some exemplary embodiments, the multimedia type scenario includes a data stream path scenario; determining the target multimedia type scenario based on the current multimedia data information includes: obtaining the video stream path and audio stream path of the current multimedia application from the current multimedia data information; if the video stream path is one path and the audio stream is one path, determining the data stream path scenario as a first multimedia scenario; if the video stream path is multiple paths or the audio stream is multiple paths, determining the data stream path scenario as a second multimedia scenario; the second multimedia scenario represents a multi-user interactive scenario.

[0096] For example, the current multimedia data information obtained through the encoding / decoding module may include the number of video and audio streams of the multimedia application. The first multimedia scenario can be a typical multimedia scenario, such as a typical live streaming scenario. The second multimedia scenario can represent a multi-person interactive scenario, such as a multi-person PK scenario (i.e., a battle) or a multi-person live streaming scenario. The second multimedia scenario requires more hardware resources than the first multimedia scenario. The encoding / decoding module can quickly and accurately obtain the number of multimedia data streams, thereby determining which multimedia scenario it belongs to.

[0097] In some exemplary embodiments, multimedia type scenarios include content scenarios; a content scenario can refer to a scenario categorized based on the specific content of the multimedia. For example, if the multimedia is live streaming, the live streaming content scenario could be games, outdoor activities, shopping, talent shows, education, food broadcasts, etc. If the multimedia is web conferencing, the web conferencing content scenario could be different industry conferences, such as medical industry seminars, new energy seminars, etc., or other conferences. The current multimedia data information includes image data and audio data acquired by the encoding / decoding module. Figure 4 As shown, the scenario for determining the target multimedia type based on the current multimedia data information includes:

[0098] Step 402: Extract features from the image data to obtain image features.

[0099] The encoding / decoding module can collect image and audio data from video data played by multimedia applications. The video data played by the multimedia application can be live or on-demand. The recognition module 102 can use a convolutional neural network to extract features from the image data to obtain image features. For example, the recognition module 102 can also preprocess the image data first, and then extract features from the preprocessed image data to obtain image features. Preprocessing may include image denoising, etc. Image features are represented using image feature vectors.

[0100] Step 404: Extract features from the audio data to obtain audio features.

[0101] The recognition module 102 can use Mel-spectrum coefficients to extract features from the audio data to obtain audio features. Audio feature extraction can include extracting speech features. Audio features include speech features. For example, the recognition module 102 can also preprocess the audio data first, and then extract features from the preprocessed audio data to obtain audio features. Preprocessing can include audio noise reduction, etc. Audio features can be represented using audio feature vectors.

[0102] Step 406: Fuse the image features and the audio features to obtain fused features.

[0103] The image features and audio features are fused, that is, the image feature vector and the audio feature vector are fused to obtain the fused feature vector.

[0104] Step 408: Identify the fused features to obtain the content scene.

[0105] A trained neural network model is used to identify the fused feature vectors to obtain the content scene.

[0106] By collecting image and audio data from multimedia application video playback, extracting features separately, and then fusing them to obtain fused features, the content scene can be obtained by recognizing the fused features, thus accurately identifying the content scene.

[0107] In some exemplary embodiments, the fused features are identified using a multimodal recognition model to obtain the content scene; such as Figure 5 As shown, the training method for this multimodal recognition model includes:

[0108] Step 502: Obtain the multimedia sample dataset.

[0109] A multimedia sample dataset can be a collection of multimedia playback data for searching different content scenarios, such as live streaming data of different live content or video conferencing data of different conference content. Each multimedia sample in the multimedia sample dataset carries content scenario annotation information.

[0110] Step 504: Extract the video data and sample audio data of each multimedia sample data in the multimedia sample dataset.

[0111] The Fast Forward MPEG (FFmpeg) tool can be used to extract the sample image data and sample audio data of each multimedia sample data in the multimedia sample data.

[0112] Step 506: Extract image features from the sample image data to obtain sample image features.

[0113] Convolutional neural networks can be used to extract image features from sample image data. These features can be represented as feature vectors, or the sample image data can be preprocessed before feature extraction. Preprocessing can include image denoising, etc.

[0114] Step 508: Extract audio features from the sample audio data to obtain the sample audio features.

[0115] Mel-Frequency Cepstral Coefficients (MFCCs) can be used to extract features from sample audio data, resulting in sample audio features. Feature extraction from sample audio data can yield speech features. For example, the sample audio data can be preprocessed first, and then features can be extracted from the preprocessed data to obtain sample audio features. Preprocessing can include audio noise reduction, etc. Sample audio features can be represented as sample audio feature vectors.

[0116] Step 510: Fuse the sample image features and the sample audio features to obtain the sample fused features.

[0117] The sample image features and sample audio features are fused by a multimodal recognition model to obtain sample fusion features.

[0118] Step 512: Train the multimodal recognition model based on the sample fusion features and the content scene annotation information corresponding to the multimedia sample data. Once the preset termination condition is met, the trained multimodal recognition model is obtained.

[0119] A multimodal recognition model identifies fused features from samples to obtain a predicted content scene. Based on the predicted content scene and its annotation information, a loss is calculated to obtain a loss value. The parameters of the multimodal recognition model are then adjusted based on this loss value until a preset termination condition is met, resulting in a trained multimodal recognition model. The multimodal recognition model can include a multimodal deep neural network (MM-DNN) and a long short-term memory (LSTM) network. The MM-DNN is used to fuse image and audio feature vectors to generate a unified feature representation, i.e., a fused feature vector. The LSTM network is used to identify the fused feature vector to obtain a predicted content scene. Again, based on the predicted content scene and its annotation information, a loss is calculated to obtain a loss value. The parameters of the multimodal recognition model are then adjusted based on this loss value until a preset termination condition is met, resulting in a trained multimodal recognition model.

[0120] In some exemplary embodiments, the method further includes: obtaining the currently running application of the electronic device; if it is determined that the currently running application is a whitelisted application, obtaining the multimedia status fed back by the currently running application; if the multimedia status is a target multimedia status, obtaining the multimedia type scenario fed back by the currently running application, and then performing the step of obtaining at least one of the following when the electronic device is in the target multimedia status: current load information, current device temperature information, current network status information, and current multimedia data information.

[0121] The currently running application on an electronic device generally refers to the foreground application. Whitelisted applications can refer to cooperating third-party applications. Applications on the whitelist can obtain relevant information about the application. Multimedia status can include live streaming status, video-on-demand status, etc. The target multimedia status can be live streaming status. The multimedia type scenario reported by the current application can be the multimedia type scenario obtained by the current application itself. For example, multimedia type scenarios can include content scenarios, which can include game live streaming scenarios, outdoor live streaming scenarios, educational live streaming scenarios, etc.

[0122] If the current application is a whitelisted application, the multimedia type scenario fed back by the current application can be directly obtained. Then, by obtaining the current load information, current device temperature information, current network status information, and current multimedia data information, at least one of the target load scenario, target temperature scenario, target network status scenario, and target multimedia type scenario can be obtained to make corresponding adjustments to the electronic device. Furthermore, the target multimedia type scenario determined by the obtained multimedia data information can be used to optimize the multimedia type scenario fed back by the current application, making the determined multimedia type scenario more accurate and more granular.

[0123] In some exemplary embodiments, the above method further includes: obtaining the currently running application of the electronic device; if it is determined that the currently running application is not a whitelisted application, obtaining top-level activity information from the application activity management module; if the top-level activity information is identified as a target multimedia state, then performing the step of obtaining at least one of the following when the electronic device is in the target multimedia state: current load information, current device temperature information, current network status information, and current multimedia data information.

[0124] If the current application is not in the whitelist, the identification module 102 can obtain top-level activity information from the application activity management module. If the activity information indicates a target multimedia state, it will then obtain current load information, current device temperature information, current network status information, and current multimedia data information to determine the corresponding scenario. By working in conjunction with the application activity management module, it can quickly determine whether the current application is in a target multimedia state, such as a live streaming state.

[0125] In some exemplary embodiments, the target multimedia state is a live broadcast state; the method further includes: if the top-level activity information is not identified as a live broadcast state, then obtaining the state information of the encoding and decoding module; if the state information of the encoding and decoding module is a decoding state, then identifying it as a video-on-demand scenario.

[0126] Video-on-demand (VOD) is a media playback method where users can actively select and request specific audio or video content based on their own preferences and schedules. The status information from the codec module can quickly determine if the current playback scenario is VOD.

[0127] In some exemplary embodiments, the method further includes: adjusting the electronic device accordingly based on the scene identifier of the target multimedia state.

[0128] For example, scene identifiers may include one or more of the following: target load scene, target temperature scene, target network status scene, target multimedia type scene, etc. Electronic devices can establish mapping relationships between load scenes, temperature scenes, network status scenes, multimedia type scenes, and adjustment strategies individually, or they can establish mapping relationships between combinations of multiple scenes and adjustment strategies. When a single scene exists, the electronic device is adjusted accordingly based on the mapping relationship between the single scene and the adjustment strategy; when multiple scenes exist, the electronic device is adjusted accordingly based on the mapping relationships between multiple scenes and adjustment strategies. Adjustments to the electronic device may include pausing or closing background applications, adjusting video parameter values, or adjusting encoding / decoding parameters, etc. Video parameter values ​​may include one or more of the following: display resolution, frame rate, or bitrate.

[0129] In some exemplary embodiments, the method further includes: obtaining at least one of the current display resolution, current frame rate, and current bit rate from the codec module;

[0130] The video parameters extracted by the encoding / decoding module. Video parameters may include one or more of the following: display resolution, frame rate, or bitrate. Display resolution can be the resolution of the screen image, or the number of pixels a monitor can display. Frame rate can be the number of frames per second in the video. Bitrate can refer to the number of bits of data transmitted per unit of time.

[0131] Accordingly, the scene identifier includes the target network state scene. Based on the scene identifier of the target multimedia state, the electronic device is adjusted accordingly, including adjusting the current video parameters based on the target network state scene.

[0132] Furthermore, the current video parameters include at least one of the current display resolution, current frame rate, and current bitrate. Based on the target network state scenario, the current video parameters are adjusted, including: if the target network state scenario is a first-level congestion scenario, increasing the current display resolution, increasing the current frame rate, or increasing the current bitrate by at least one of the following adjustments.

[0133] For example, the current video parameters are adjusted according to the target network state scenario, including: if the target network state scenario is a second congestion scenario, a third congestion scenario, or a fourth congestion scenario, at least one of the following adjustments is made: reducing the current display resolution, reducing the current frame rate, or reducing the current bit rate.

[0134] The adjustment ranges for the second, third, and fourth congestion scenarios can be different. The adjustment range for video parameters in the second congestion scenario is smaller than that in the third congestion scenario, and the adjustment range for video parameters in the third congestion scenario is smaller than that in the fourth congestion scenario.

[0135] For example, the current video parameters are adjusted according to the target network state scenario, including: if the target network state scenario is the second congestion scenario, the current display resolution, current frame rate and current bit rate remain unchanged.

[0136] The following section uses the target multimedia state as an example to illustrate live streaming scene recognition, multi-module linkage, and electronic device adjustments. For example... Figure 6 As shown, the process of identifying whether the current application's multimedia status is live includes:

[0137] Step 601: Open a video application.

[0138] Users open video applications on their electronic devices.

[0139] Step 602: Determine whether the video stream application, i.e. the current application, belongs to the third-party collaborative customized application category (whitelist application). If yes, proceed to step 603; otherwise, proceed to step 605.

[0140] Step 603: The recognition module is linked with the application's status. When the recognition module listens to the feedback result of the current application, it marks the corresponding result, which includes live streaming, on-demand, and others.

[0141] Step 604: When the recognition result is a live broadcast status, the recognition module further links with the application to monitor the live broadcast type and whether it is a normal live broadcast scenario or a multi-person PK or multi-person voice chat scenario, marks the corresponding result, and notifies the corresponding module that needs to use the marked result later, and then proceeds to step 611.

[0142] Step 605: Obtain top-level Activity information through the application activity management module (Activity Manager module).

[0143] Step 606: Determine whether it is a live streaming Activity based on the top-level Activity information. If yes, proceed to step 607; otherwise, proceed to step 608.

[0144] Step 607: Mark as live and notify the module that will identify the live stream, then proceed to step 611.

[0145] Step 608: The recognition module further links with the codec module to obtain the current codec status information and determine whether the codec status information is in the decoding state. If yes, proceed to step 609; otherwise, proceed to step 610.

[0146] Step 609: Mark as on-demand scenario.

[0147] Step 610: Mark as other scenarios.

[0148] Step 611: Start the live stream.

[0149] Step 612: Multi-module collaboration to perform scene recognition within the live stream and optimize scene performance.

[0150] By linking with third-party applications and customizing relevant scenario interfaces, it is possible to accurately identify live streaming and video-on-demand scenarios, as well as various types of live streaming scenarios, making it easy to customize configurations for specific scenarios. For non-linked third-party applications, the linked application activity management module can accurately identify live streaming and video-on-demand scenarios.

[0151] like Figure 7 As shown, when an electronic device is in live streaming mode, the process of multi-module linkage includes:

[0152] (1) The Activity Manager module 104 transmits Activity information to the recognition module 102.

[0153] (2) When entering the live broadcast state, the identification module 102 and the CPU module 106 are linked in state, a series of load threshold ranges are set, the current CPU load information is monitored in real time, the load scenario is determined by comparing the current CPU load information with the corresponding load threshold range, and the target load scenario is marked. The target load scenario includes light load scenario, medium load scenario, high load scenario or ultra-high load scenario.

[0154] (3) Upon entering live streaming mode, the recognition module 102 and the temperature control module 108 are linked. The temperature control module 108 monitors the temperature of the electronic device in real time, obtains the current device temperature information, and sends the current device temperature information to the recognition module 102. The recognition module 102 determines the target temperature scenario based on the relationship between the current device temperature information and the temperature threshold range. If the temperature threshold range is a temperature threshold, the recognition module 102 determines that the current device temperature information exceeds the temperature threshold and identifies it as a high-temperature scenario; otherwise, it identifies it as a normal temperature scenario.

[0155] (4) Upon entering live streaming mode, the recognition module 102, encoding / decoding module 112, and network module 110 are linked in status. The recognition module 102 obtains video parameters such as the current display resolution, current frame rate, and current bitrate from the encoding / decoding module 112. The recognition module 102 also obtains current bandwidth and speed from the network module 110. The recognition module 102 also obtains network status information, including bandwidth utilization, latency, jitter duration, and packet loss rate. The recognition module 102 calculates a comprehensive score based on the network status information and determines the target network status scenario based on the comprehensive score. The target network status scenario is a normal network scenario, a slightly congested network scenario, a moderately congested network scenario, or a heavily congested network scenario. The recognition module 102 is also used to determine whether the target network status scenario meets the requirements for pushing and distributing the current encoding / decoding data.

[0156] (5) Upon entering live streaming mode, the recognition module 102 also works in conjunction with the encoding / decoding module 112. The recognition module 102 collects image and audio data from the encoding / decoding module 112 during the live streaming process. It performs preprocessing such as image denoising on the collected image data, extracts features from the preprocessed image data to obtain image feature vectors, performs preprocessing such as audio denoising on the audio data, extracts features from the preprocessed audio data to obtain audio feature vectors, and uses a multimodal deep learning model (i.e., multimodal recognition model 114) to perform unified feature representation of the extracted image and audio feature vectors. The multimodal deep learning model is then used to identify the live streaming scene and determine the current live streaming content scene category, such as game live streaming, talent show, outdoor live streaming, etc. Figure 8 As shown.

[0157] By linking with system modules such as the CPU and temperature control module, it identifies high-load and high-power scenarios, facilitating adjustments to electronic devices from a performance and power consumption perspective. For example, closing background applications can reduce load and power consumption, improving user experience. Linking with system modules such as the encoding / decoding module and network module, it identifies network status scenarios, allowing for adjustments to encoding / decoding parameters to improve display quality and live stream smoothness. Utilizing a multimodal deep learning model, it links with the encoding / decoding module to identify different live stream types, improving the accuracy of scenario identification. It identifies various complex scenarios in live streaming, facilitating customized solutions for different scenarios. This improves user experience in multiple aspects, including performance, power consumption, display effect, stability, and convenience. For example, identifying high-load scenarios allows for targeted adjustments to live stream video parameters to improve performance, and identifying high-temperature scenarios allows for the development of relevant power-saving measures.

[0158] Figure 9 This is a schematic diagram of the training process for a multimodal recognition model. Figure 9 As shown, the training process of a multimodal recognition model, also known as a multimodal deep learning model, includes:

[0159] Step 902: Obtain the dataset, which includes screen recording data of major live streaming types such as shopping live streaming, game live streaming, talent show live streaming, outdoor live streaming, education live streaming, and food live streaming.

[0160] Step 904, Data Acquisition. Use FFmpeg to extract the video and audio data from each screen recording.

[0161] Step 906: Perform image denoising and other preprocessing on the acquired video data, and perform audio denoising and other preprocessing on the acquired audio data.

[0162] Step 908: Use a convolutional neural network to extract image features from the preprocessed image data to obtain an image feature vector, and use Mel-spectrum coefficients to extract speech features to obtain an audio feature vector.

[0163] Step 910: Use a multimodal deep neural network (MM-DNN) to fuse the image feature vector and the audio feature vector together to generate a unified feature representation.

[0164] Step 912: The fused feature vector is fed into a Long Short-Term Memory (LSTM) network for model training and recognition. The multimodal deep learning model is continuously corrected and optimized based on the recognition results to obtain a trained multimodal recognition model.

[0165] By searching screen recording data of different live streaming types, and performing data collection, preprocessing, feature extraction and feature fusion on the screen recording data to obtain fused feature vectors, the multimodal recognition model was trained and optimized, which improved the accuracy of the model's scene recognition.

[0166] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0167] Based on the same inventive concept, this application also provides an apparatus for implementing the above-described method. The solution provided by this apparatus is similar to the solution described in the above-described method; therefore, the specific limitations in the embodiments provided below can be found in the limitations of the above-described method, and will not be repeated here.

[0168] In some exemplary embodiments, such as Figure 10 As shown, an application scene recognition device includes an information acquisition module 1010, a scene determination module 1020, and a scene marking module 1030.

[0169] The information acquisition module 1010 is used to acquire at least one of the following when the electronic device is in the target multimedia state: current load information, current device temperature information, current network status information, and current multimedia data information.

[0170] The scenario determination module 1020 is used to determine the target load scenario based on the current load information, the target temperature scenario based on the current device temperature information, the target network status scenario based on the current network status information, and the target multimedia type scenario based on the current multimedia data information.

[0171] The scene marking module 1030 is used to obtain the scene identifier of the target multimedia state based on at least one of the target load scene, target temperature scene, target network state scene, or target multimedia type scene.

[0172] In one embodiment, the scenario determination module 1020 is further configured to compare the current load information with the load threshold range; determine the target load threshold range in which the current load information is located; and take the load scenario corresponding to the target load threshold range as the target load scenario.

[0173] In one embodiment, the scene determination module 1020 is further configured to compare the current device temperature information with a temperature threshold range; determine the target temperature threshold range in which the current device temperature information is located; and use the temperature scene corresponding to the target temperature threshold range as the target temperature scene.

[0174] In one embodiment, the current network state information includes one or more of bandwidth utilization, latency, packet loss rate, and jitter duration; the scenario determination module 1020 is further configured to determine a bandwidth utilization score based on the bandwidth utilization and a bandwidth utilization threshold range; determine a latency score based on the latency and a latency threshold range; determine a packet loss score based on the packet loss rate and a packet loss threshold range; determine a jitter score based on the jitter duration and a jitter threshold range; obtain a comprehensive score based on at least one of the bandwidth utilization score, the latency score, the packet loss rate score, or the jitter score; and determine the target network state scenario based on the comprehensive score and the correspondence between the comprehensive score threshold range and the network state scenario.

[0175] In one embodiment, the multimedia type scenario includes a data stream path scenario; the scenario determination module 1020 is further configured to obtain the video stream path and audio stream path of the current multimedia application from the current multimedia data information; when the video stream path is one path and the audio stream is one path, the data stream path scenario is determined to be a first multimedia scenario; when the video stream path is multiple paths or the audio stream is multiple paths, the data stream path scenario is determined to be a second multimedia scenario; the second multimedia scenario represents a multi-person interactive scenario.

[0176] In one embodiment, the multimedia type scene includes a content scene; the current multimedia data information includes image data and audio data acquired by the encoding and decoding module; the scene determination module 1020 is further used to extract features from the image data to obtain image features; extract features from the audio data to obtain audio features; fuse the image features and the audio features to obtain fused features; and identify the fused features to obtain the content scene.

[0177] In one embodiment, the fused features are identified using a multimodal recognition model to obtain the content scene. The device further includes a model training module, which is used to acquire a multimedia sample dataset; extract sample image data and sample audio data for each multimedia sample data in the multimedia sample dataset; extract image features from the sample image data to obtain sample image features; extract audio features from the sample audio data to obtain sample audio features; fuse the sample image features and the sample audio features to obtain sample fused features; and train the multimodal recognition model based on the sample fused features and the content scene annotation information corresponding to the multimedia sample data, and obtain the trained multimodal recognition model when the preset termination condition is met.

[0178] In one embodiment, the apparatus further includes an application acquisition module and a multimedia information acquisition module. The application acquisition module is used to acquire the currently running applications of the electronic device.

[0179] The multimedia information acquisition module is used to acquire the multimedia status returned by the current application if it is determined that the current application is a whitelisted application; it is also used to acquire the multimedia type scenario returned by the current application if the multimedia status is a target multimedia status, and then the information acquisition module 1010 executes it.

[0180] In one embodiment, the above-described apparatus further includes an application acquisition module and an activity information acquisition module.

[0181] The application acquisition module is used to acquire the currently running applications of the electronic device.

[0182] The activity information acquisition module is used to obtain top-level activity information from the application activity management module if it is determined that the current application is not a whitelisted application.

[0183] The information acquisition module 1010 is also used to acquire at least one of the following when the electronic device is in the target multimedia state: current load information, current device temperature information, current network status information, and current multimedia data information, if the top-level activity information is identified as the target multimedia state.

[0184] In one embodiment, the target multimedia state is a live broadcast state; the device further includes a state information acquisition module and an identification module.

[0185] The status information acquisition module is used to acquire the status information of the encoding / decoding module if the top-level activity information is not identified as being in a live broadcast state.

[0186] The identification module is used to identify a video-on-demand scenario if the status information of the encoding / decoding module is in the decoding state.

[0187] In one embodiment, the device further includes an adjustment module. The adjustment module is used to make corresponding adjustments to the electronic device based on the scene identifier of the target multimedia state.

[0188] In one embodiment, the information acquisition module 1010 is further configured to acquire at least one of the current display resolution, current frame rate, and current bit rate from the encoding / decoding module.

[0189] The scene identifier includes the target network state scene. The adjustment module is also used to adjust at least one of the current display resolution, current frame rate, and current bitrate based on the target network state scene.

[0190] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0191] In one exemplary embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the electronic device includes an image sensor, an image signal processor, a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. The computer program is executed by the processor to implement the above methods. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0192] Those skilled in the art will understand that Figure 11The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0193] In one exemplary embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the method described above.

[0194] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the method described above.

[0195] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the method described above.

[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0197] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0198] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0199] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An application scenario recognition method, characterized in that, The method includes: When the electronic device is in the target multimedia state, acquire at least one of the following: current load information, current device temperature information, current network status information, and current multimedia data information; The target load scenario is determined based on the current load information, the target temperature scenario is determined based on the current device temperature information, the target network status scenario is determined based on the current network status information, and the target multimedia type scenario is determined based on the current multimedia data information. Based on at least one of the target load scenario, target temperature scenario, target network status scenario, or target multimedia type scenario, obtain the scenario identifier under the target multimedia status.

2. The method according to claim 1, characterized in that, Determining the target load scenario based on the current load information includes: Compare the current load information with the load threshold range; Determine the target load threshold range in which the current load information is located, and take the load scenario corresponding to the target load threshold range as the target load scenario.

3. The method according to claim 1, characterized in that, The step of determining the target temperature scenario based on the current device temperature information includes: Compare the current device temperature information with the temperature threshold range; Determine the target temperature threshold range where the current device temperature information is located, and take the temperature scene corresponding to the target temperature threshold range as the target temperature scene.

4. The method according to claim 1, characterized in that, The current network status information includes one or more of bandwidth utilization, latency, packet loss rate, and jitter duration; determining the target network status scenario based on the current network status information includes: Based on the broadband utilization rate and the broadband utilization threshold range, a broadband utilization score is determined; The delay score is determined based on the delay duration and the delay duration threshold range; Based on the packet loss rate and the packet loss rate threshold range, a packet loss rate score is determined; The jitter score is determined based on the jitter duration and the jitter duration threshold range. A comprehensive score is obtained based on at least one of the bandwidth utilization score, the latency score, the packet loss rate score, or the jitter score; Based on the comprehensive score, the target network state scenario is determined from the correspondence between the comprehensive score threshold range and the network state scenario.

5. The method according to claim 1, characterized in that, Multimedia type scenarios include data stream path scenarios; determining the target multimedia type scenario based on the current multimedia data information includes: Obtain the number of video streams and audio streams of the current multimedia application from the current multimedia data information; When the number of video streams is one and the number of audio streams is one, the scenario of the number of data streams is determined to be the first multimedia scenario; When the number of video streams is multiple or the number of audio streams is multiple, the scenario of the number of data streams is determined to be a second multimedia scenario; the second multimedia scenario represents a multi-person interactive scenario.

6. The method according to claim 1, characterized in that, The multimedia type scenario includes a content scenario; the current multimedia data information includes image data and audio data acquired by the encoding / decoding module; determining the target multimedia type scenario based on the current multimedia data information includes: Feature extraction is performed on image data to obtain image features; Feature extraction is performed on the audio data to obtain audio features; The image features and the audio features are fused to obtain fused features; The fused features are identified to obtain the content scene.

7. The method according to claim 6, characterized in that, The fused features are identified using a multimodal recognition model to obtain the content scene; The training methods for the multimodal recognition model include: Obtain multimedia sample datasets; Extract the sample image data and sample audio data of each multimedia sample data in the multimedia sample dataset; Image feature extraction is performed on the sample image data to obtain sample image features; Audio features are extracted from the sample audio data to obtain sample audio features; The sample image features and the sample audio features are fused to obtain sample fused features; The multimodal recognition model is trained based on the sample fusion features and the content scene annotation information corresponding to the multimedia sample data. Once the preset termination condition is met, the trained multimodal recognition model is obtained.

8. The method according to claim 1, characterized in that, The method further includes: Get the currently running application on the electronic device; If it is determined that the current application is a whitelisted application, then obtain the multimedia status returned by the current application; If the multimedia state is the target multimedia state, then obtain the multimedia type scenario fed back by the current application, and then execute the step of obtaining at least one of the following when the electronic device is in the target multimedia state: current load information, current device temperature information, current network status information, and current multimedia data information.

9. The method according to claim 1, characterized in that, The method further includes: Get the currently running application on the electronic device; If it is determined that the current application is not a whitelisted application, then obtain the top-level activity information from the application activity management module; If the top-level activity information is identified as a target multimedia state, then the step of obtaining at least one of the following when the electronic device is in the target multimedia state is executed: current load information, current device temperature information, current network status information, and current multimedia data information.

10. The method according to claim 9, characterized in that, The target multimedia state is a live broadcast state; the method further includes: If the top-level activity information is not identified as being in a live broadcast state, then the status information of the encoding / decoding module is obtained; If the status information of the encoding / decoding module is in decoding state, it is identified as a video-on-demand scenario.

11. The method according to claim 1, characterized in that, The method further includes: The electronic device is adjusted accordingly based on the scene identifier in the target multimedia state.

12. The method according to claim 11, characterized in that, The method further includes: Obtain at least one of the following from the encoding / decoding module: current display resolution, current frame rate, and current bit rate; The scene identifier includes a target network state scene. The step of adjusting the electronic device accordingly based on the scene identifier in the target multimedia state includes: Based on the target network state scenario, at least one of the current display resolution, current frame rate, and current bit rate is adjusted.

13. An application scene recognition device, characterized in that, The device includes: The information acquisition module is used to acquire at least one of the following when the electronic device is in the target multimedia state: current load information, current device temperature information, current network status information, and current multimedia data information; The scenario determination module is used to determine a target load scenario based on the current load information, a target temperature scenario based on the current device temperature information, a target network status scenario based on the current network status information, and a target multimedia type scenario based on the current multimedia data information; the scenario marking module is used to obtain a scenario identifier in the target multimedia status based on at least one of the target load scenario, target temperature scenario, target network status scenario, or target multimedia type scenario.

14. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 12.

Images