Method, Medium, and System for Dynamically Selecting a Codec

Abstract

This application relates to methods, media, and systems for dynamically selecting codecs. Exemplary embodiments relate to techniques for dynamically selecting codecs when transmitting video in real time. In some embodiments, a first codec initially encodes video data, and a second codec is evaluated to replace the first codec. The system switches to the second codec only if the increase in power consumption resulting from using the second codec is balanced by a sufficient increase in the quality of the video encoded by the second codec. In some embodiments, a codec is excluded from consideration if it is determined that the local device does not have sufficient processing resources to operate the codec, or if a mismatch is detected between the codec operating on the sending device and the codec operating on the receiving device.

Description

Background

[0001] Computing devices can run a variety of different applications, some of which can capture video and transmit the video over a network. For example, a video conferencing application can send and receive video in real time over the Internet.

[0002] Video can be encoded at a sending device by a codec and decoded at a receiving device. A codec can compress or otherwise manipulate video data so that the video data transmission is suitable for the bitrate that the network can support. There are many different codecs, and different codecs may incur different costs and benefits (e.g., in terms of battery consumption, encoded video quality, throughput, etc.).

[0003] There are two different types of codecs: hardware codecs, which are typically hard-coded into processing circuitry; and software codecs, which are software applications that can run on a dedicated or general-purpose processor. Hardware codecs may perform well in high-bandwidth situations, using less power and processing resources than software codecs when producing high-quality media encoding. However, hardware codecs may perform poorly in low-bandwidth situations because they produce low-quality media encoding and do not adjust well to stay within the bandwidth limit. Software media encoders can perform well in low-bandwidth situations, producing relatively high-quality encoding given the bandwidth limit and adhering to the bandwidth limit. However, their performance tends to be poor in high-bandwidth situations, in which case they will consume a large amount of power and processing resources.

[0004] The codec used to encode video at a sending device needs to be compatible with the codec used to decode video at a receiving device. Traditionally, the sending device and the receiving device (possibly with the help of an intermediate server) negotiate to determine a list of codecs that are supported by both the sending device and the receiving device. Then, the sending device typically selects from the list of compatible codecs by applying a predefined rule that selects a given codec to apply under specific network conditions.

[0005] Brief Description of the Several Views of the Drawings

[0006] To easily identify the discussion of any particular element or action, one or more of the most significant digits in the reference numerals refer to the figure number in which the element was first introduced.

[0007] Figure 1 Depicts an exemplary data exchange between devices that select and apply a codec to encode a video data stream.

[0008] Figure 2 Depicts an exemplary environment 200 suitable for embodiments.

[0009] Figures 3A - 3E Depicts an exemplary data structure suitable for use in the embodiments.

[0010] Figure 4 Illustrates an exemplary CPU test logic 400 according to an embodiment.

[0011] Figure 5 Illustrates an exemplary call initiation logic 500 according to an embodiment.

[0012] Figure 6 Illustrates an exemplary codec selection logic 600 according to an embodiment.

[0013] Figures 7A - 7B Illustrates an exemplary error analysis logic 700 according to an embodiment.

[0014] Figure 8 Illustrates an exemplary sender error analysis logic 800 according to an embodiment.

[0015] Figure 9 Depicts an illustrative computer system architecture that can be used to practice the exemplary embodiments described herein.

[0016] Figure 10 Illustrates an exemplary messaging service 1000 according to one embodiment. Detailed description

[0017] When selecting a codec according to traditional methods, several problems may occur. Especially on mobile devices where power management needs to be concerned, it is usually best to use hardware codecs as much as possible. However, different manufacturers may implement the same hardware codec in different ways. These differences may lead to incompatibilities, thus causing errors when decoding videos.

[0018] Hardware codecs are also prone to "overshoot", in which case the data encoded by the codec exceeds the data that the network can transmit. This can be particularly problematic when transmitting videos in real time, because the transmitting device only has limited ability to buffer data for smooth transmission.

[0019] In some cases, new codecs may be available or rolled out on older or traditional hardware. Although the hardware can technically be able to apply the codec, the codec may require more processing resources than the hardware can reasonably provide. Therefore, if the device applies the codec, the video processing performance may be affected. At the same time, if the hardware can support the new codec, it is desirable to use the new codec when it becomes available, because such a codec can provide higher quality at the same bit rate (and vice versa).

[0020] Further, the predefined rules applied by traditional codec selection algorithms tend to include fixed values that have historically been shown to produce good results on average. For example, one rule can direct the device to use a particular software codec when below a relatively low bitrate threshold, and then switch to a particular hardware codec when network conditions improve and the bitrate increases above the threshold. While these rules may make sense on average, the best codec to apply in practice (in terms of balancing quality, battery consumption, and processing resources) depends on the complexity of the scene being recorded. For example, if the video being recorded only shows a relatively simple scene (such as a user's face), the video quality may not improve significantly by switching to a more complex codec (although the more complex codec will still use increased processing and power resources). Nevertheless, traditional techniques may switch to a more complex codec based only on network conditions. By maintaining a simpler codec, which operates on fewer bits compared to a more complex codec, and since the simpler codec works with less data, it may be possible to increase the video resolution faster compared to if a more complex codec were used.

[0021] Exemplary embodiments provide techniques for dynamically adjusting a video codec while a video call is in progress. Multiple different aspects of codec selection are described, and it is contemplated that each function can be used individually or together.

[0022] According to a first embodiment, a sending device can consult a list on a server and / or its own device that indicates codecs that will not work with the sending device (possibly in combination with a specified receiving device). The list can be generated based on on-device CPU tests that compare available codecs to a CPU threshold; if the device's CPU does not meet the threshold performance requirements of a codec, that codec may not be considered for a given network transaction.

[0023] Codecs can also or alternatively be added to the list in other ways. For example, a hardware codec can be added to the list when it is known that the hardware codec of the sending device is incompatible with the corresponding hardware codec on the receiving device due to differences in the way the device manufacturer implemented the codec. Information in this regard can be learned during the processes described in connection with the second and third embodiments.

[0024] The list can include pre-determined codec information from historical transactions observed by the device and / or server. Alternatively or additionally, the list can be determined dynamically while transmitting video.

[0025] In some embodiments, if a codec is on the list, the codec is not used during the tests described in the second and third embodiments. This saves time and resources because codecs known not to work well will not be tested to determine if they can produce a good balance of battery life and quality. This embodiment also improves the overall video transmission quality because codecs known not to work well will not be considered as options during transmission; thus, video can only be transmitted through codecs known to work well. Even in the case of the first video transmission of a given mobile device, without testing the compatible codecs themselves, it can rely on the historical codec tests compiled by the intermediate server to avoid testing or using codecs that have been proven not to work well on other similar devices or in similar contexts.

[0026] According to the second embodiment, a codec is selected or changed based on the characteristics of the video being encoded. In these embodiments, the system can analyze the video being captured, identify the characteristics that define how much quality improvement is expected from switching different codecs, calculate the increased battery consumption from the new codec, and only switch to the new codec if it is necessary (if warranted) to improve quality by increasing battery consumption.

[0027] Generally, a software codec can make the quality measurement results available (e.g., through API calls). However, a hardware codec may not provide similar information. To determine or predict the video quality achieved by a hardware codec, a proxy of video characteristics (e.g., brightness from the camera, amount of motion calculated in the video, etc.) can be used to apply machine learning.

[0028] In some cases, the video frames encoded by an encoder on a sending device can be provided to a decoder on the same sending device. In these cases, the output from the decoder can be compared with the original video frames provided to the encoder to determine quality. However, this technique will consume more processing resources than retrieving the quality measurement results with an API, so it may only be used in limited cases (when information in this regard cannot be obtained without consuming processor cycles).

[0029] The second embodiment allows the device to dynamically adjust the applied codec instead of relying on static rules. Because the dynamic selection process takes into account the quality achieved by the codec in the context of the current scene in the video, it can avoid switching to a more complex (and more battery and processor intensive) codec when an improvement in video quality does not need to be compromised by reducing battery life. This result can be achieved even when the network conditions change to a more complex codec as traditionally required.

[0030] According to the third embodiment, the list of supported hardware codecs can be dynamically pruned based on statistical information such as video error rate, overshoot, and the number or rate of packet loss. This technique can be used to identify hardware codec mismatches (even when the device claims to support a given codec). If the measured statistical information exceeds a predetermined threshold, the hardware codec can be removed from the list of codecs supported by the device. Depending on the results, all hardware codecs may be prohibited (e.g., if there is too much overshoot, which may be a characteristic of the hardware codec), or specific hardware codecs may be prohibited. Alternatively, the hardware codec may be applied, but specific features may be disabled. In the latter case, the system can recalculate whether the improvement in quality (which will be reduced due to the reduced feature set) is still necessary to switch to the new codec.

[0031] The third embodiment allows the device to identify when there may be a mismatch in the way a particular hardware codec is implemented, allowing the device to remove such a codec from consideration (dynamically during the current video transmission and in future video transmissions involving similar device types). As a result, the video transmission quality is improved, and the error rate can be reduced. In addition, overshoot can be avoided, which is particularly beneficial for video call applications deployed in the global market, where many user devices may operate on slower or less reliable networks. Since the use of the hardware codec can be dynamically adjusted as the call progresses, the same technique can be used for newer mobile devices operating on reliable high-bandwidth networks and for older mobile devices operating on less reliable low-bandwidth networks. This simplifies the design of video transmission applications that support the described codec selection logic, allowing applications to be designed with shorter, more efficient code that can run on more devices. At the same time, it allows advanced hardware and networks to use their full capabilities while still supporting older devices and networks.

[0032] As an example of use, consider a video call where a first user operating a first type of mobile device calls a second user operating a second different type of mobile device. When the call is initiated, the device of the first user broadcasts the codecs it supports. However, the list is filtered to remove codecs that the device of the first user technically supports but for which the device of the first user does not have sufficient processing power to operate (and / or codecs that have historically been shown to be incompatible between the two types of mobile devices). The device of the second user responds with any corresponding codecs that the device of the first user supports and that the device of the second user also supports (which may also be filtered to remove those codecs for which the device of the second user does not have sufficient processing power). Thus, when negotiating which codec to use, the user devices do not even consider those codecs that the devices will not be able to operate effectively.

[0033] After the video call starts, the first user device begins to encode the video using the selected codec. At regular intervals, or when the network conditions change, the first user device evaluates alternative codecs. To do this, the first user device calculates the expected improvement in video quality and the expected change in battery consumption. The first user device weighs the expected improvement against the reduced battery life and switches to a new codec only if a compromise is necessary.

[0034] As the call progresses, the first device switches to a hardware codec that is also supported by the second user device, but the second user device notices an increase in error rate and packet loss. The first device interprets this increase as a codec mismatch (e.g., two different devices implement the same hardware codec in different ways), and the list of supported codecs is trimmed to remove the mismatched codec. Alternatively, the first user device can consider disabling features of the hardware codec to try to eliminate those features that cause the mismatch, but first tries to determine the video quality that will result when the features are disabled (to determine if it still makes sense to continue using the codec).

[0035] This brief overview is intended as a non - restrictive introduction to the concepts discussed in more detail below. However, before discussing further exemplary embodiments, a brief note on data privacy is provided first. The more detailed description of privacy settings and authentication will be addressed in conjunction with the following figures.

[0036] Some embodiments described herein utilize training data or metrics, which may include information voluntarily provided by one or more users. In such embodiments, data privacy can be protected in a variety of ways.

[0037] For example, before collecting or using user data, it may be necessary for the user to opt - in to any data collection. The user can also be provided with the opportunity to opt - out of any data collection. Before opting - in to data collection, the user can be provided with a description of how the data will be used, how long the data will be retained, and the safeguards in place to protect the data from disclosure.

[0038] Any information that identifies the user from whom the data is collected can be purged or de - associated from the data. In cases where any identifying information needs to be retained (e.g., to meet regulatory requirements), the user can be notified of the collection of the identifying information, the use of the identifying information, and the amount of time the identifying information will be retained. Information that specifically identifies the user can be removed and replaced with, for example, a generic identification number or other non - specific form of identification.

[0039] Once collected, the data can be stored in a secure data storage location that includes protection measures to prevent unauthorized access to the data. The data can be stored in an encrypted format. Identifying information and / or non-identifying information can be purged from the data storage after a predetermined period of time.

[0040] Although specific privacy protection techniques are described herein for illustrative purposes, those of ordinary skill in the art will recognize that privacy can also be protected in other ways. Additional details regarding data privacy are discussed in the section below that describes embodiments of the network.

[0041] Assuming that the privacy conditions of the user are met, the exemplary embodiments can be deployed in a variety of messaging systems, including messaging in social networks or on mobile devices (e.g., via a messaging client application or via the Short Message Service), and so on. An overview of the exemplary logic and processes involved in participating in a synchronized video conversation in a messaging system is provided next.

[0042] To aid understanding, a series of examples will be given first before the detailed description of the underlying implementation. Note that these examples are for illustrative purposes only, and the present invention is not limited to the embodiments shown.

[0043] Reference is now made to the accompanying drawings, in which like reference numerals are always used to refer to like elements. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding thereof. However, the novel embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate their description. The intention is to cover all modifications, equivalents, and alternatives consistent with the claimed subject matter.

[0044] In the drawings and the accompanying description, the designations "a" and "b" and "c" (and similar designators) are intended to be variables representing any positive integer. Thus, for example, if an implementation sets the value of a to a = 5, the complete set of components 122 shown as components 122-1 through 122-a can include components 122-1, 122-2, 122-3, 122-4, and 122-5. The embodiments are not limited to this context.

[0045] Figure 1 An exemplary data exchange between devices that select and apply codecs to encode a video data stream is depicted. In this example, a sending device 102 (such as a computer, mobile device, tablet, dedicated video conferencing device, etc.) is attempting to initiate a video call to a receiving device 106. An intermediate server 104 facilitates the establishment and data exchange of a video conference between the devices.

[0046] To exchange video data, the sending device 102 and the receiving device 106 need to agree on one or more codecs that are mutually compatible between the devices. If there are no mutually compatible codecs, the devices may not be able to exchange video data. To determine which codecs are supported by both devices, the sending device 102 can transmit a list of the codecs supported by the sending device 102 as part of a call initiation 108. The intermediate server 104 can receive the call initiation 108 and relay it to the receiving device 106.

[0047] The call initiation 108 can include a list of the codecs supported by the sending device 102. As described in more detail herein, the list of codecs can be filtered based on tests performed by the sending device 102 and / or the intermediate server 104. For example, the sending device 102 may technically support a first hardware codec, but may know from previous experience or CPU testing that it does not have the processing power to successfully or efficiently apply the first hardware codec. Thus, the list of available codecs advertised by the sending device 102 in the call initiation 108 can include all of the codecs available to the sending device 102, or can include only a subset of the codecs available to the sending device 102.

[0048] In response to receiving the call initiation 108, the receiving device 106 can reply with a call acceptance 110. Additionally, the call acceptance 110 can include a list of the codecs supported by the receiving device 106. In some cases, the receiving device can respond with a list of all of the codecs it supports (which may be filtered in a manner similar to that described above for the sending device 102), or can include only those codecs supported by the receiving device 106 that are also supported by the sending device 102 (as determined from the call initiation 108). The advantage of the former case is that if multiple devices are participating in a call and one device drops off, the remaining devices have a more comprehensive understanding of the codecs supported in the environment. Thus, they may be able to apply codecs that were not initially supported by the dropped-off device.

[0049] The sending device 102 and the receiving device 106 now both understand which codecs are supported by both devices. The sending device 102, which is encoding video data for transmission, can select one of the codecs to perform the encoding. The receiving device 106 can also encode video data for transmission and can do so using the same or a different codec than the codec used by the sending device 102. The sending device 102 and the receiving device 106 can thus transmit first encoded video data 112.

[0050] At some point, the sending device 102 may register a context change 114. The context change 114 may be a change in the underlying conditions of the network (e.g., an increase or decrease in the bitrate implemented by the network), but it may also be a change in the context of the video data. For example, when the user switches the mobile device from using its front camera to using its rear camera, or when there is a change in the location, position, or orientation of the camera identified by the user's mobile device (indicating a change in the scene being recorded), a change in the context of the video data may occur. Another example of a context change may be when the image processing software on the device recognizes a change in the complexity of the scene (e.g., when the user initially records a video of a face but then points the camera elsewhere and records a video of a landscape).

[0051] In response to the context change, different codecs may be more suitable for encoding the video data. Therefore, the sending device may select a new codec and encode the video data in a different way, resulting in second-encoded video data 116. This process may be repeated until the video transmission ends.

[0052] Although Figure 1 one device is referred to as the "sender" and another device is referred to as the "receiver", in most cases, the sender will also receive video data, and the receiver will also send video data. Additionally, although Figure 1 a video conference between a single sending device 102 and a single receiving device 106 is depicted, the exchange of video data may involve more devices. For example, in a video conference among three or more devices, each device may send video data to and receive video data from each device participating in the conference. In such a case, during call initiation 108, all devices in the conference may negotiate a list of codecs that are compatible with all devices. If a device is added to or removed from an ongoing video conference, the participating devices may renegotiate the available codecs.

[0053] The exemplary embodiments described herein may replace, add, or eliminate some of the processes described in conjunction with Figure 1 Before describing these processes, an overview of the various devices and data structures used is provided.

[0054] Figure 2 An exemplary environment 200 is depicted, which includes a sending device 102, an intermediate server 104, and a receiving device 106 suitable for use in the embodiments.

[0055] The sending device 102 can be any suitable device, such as a desktop computer or a laptop computer, a mobile device such as a phone or a tablet, a dedicated video conferencing device, or another type of device. The sending device 102 can include one or more input devices 202, such as a camera, a microphone, a keyboard, etc. For example, the camera and the microphone on the sending device 102 can act together to record the original video data to be encoded and transmitted over the network.

[0056] The sending device 102 can also include a processor 204 and a non-transitory computer-readable medium 206. The processor 204 can include a hardware processor circuit. The non-transitory computer-readable medium 206 is, for example, RAM, ROM, a hard disk drive, or other types of memory or storage. The non-transitory computer-readable medium 206 can store instructions configured to cause the processor 204 to execute one or more processes. For example, the instructions can include instructions implementing the following logic.

[0057] The non-transitory computer-readable medium 206 can also store one or more application programs (such as a video conferencing application program that communicates with a corresponding video conferencing application program on the receiving device 106). In addition, the video conferencing logic can support the call initiation logic 500 described in more detail Figure 5 below.

[0058] The sending device 102 can include a network interface 208 for communicating over the network. The network interface 208 can be a wired or wireless interface (such as a network interface card (NIC), an Ethernet adapter, or other suitable interface).

[0059] The intermediate server 104 can also include a processor 226, a non-transitory computer-readable medium 228, and a network interface 230.

[0060] The sending device 102 can support one or more codecs for encoding and decoding video data. Some codecs can be hardware codecs, while others can be software codecs. In some cases, the codec can be a hardware / software hybrid codec. The sending device 102 can maintain a list of available codecs 210, which can include each codec supported by the sending device 102. The receiving device 106 can maintain a corresponding list of available codecs 210 supported by the receiving device 106.

[0061] In Figure 3AAn example of a list of available codecs 210 is shown. The list of available codecs 210 includes a codec name 302 that identifies the codecs supported by the device. For each codec, the list may also include a codec type 304 (e.g., hardware or software) and a performance map 306. The performance map 306 maps conditions or contexts to the expected video quality produced by the codec. For example, the performance map may accept values such as the mobility of objects in a given video frame, average brightness and count, network bitrate and other current network conditions, type of receiving device, etc. as inputs. The performance map 306 may map these inputs to a value indicating the quality that the codec can achieve under the specified conditions (e.g., the resolution of the encoded video frame). The list of available codecs 210 may also include a value representing the power consumption 308 of the codec. As described in more detail below, the device may trade off the power consumption of the codec against the quality achievable by the codec to determine whether it is worth switching from an existing codec to a different codec. In some embodiments, if the codec has specific features that may be disabled, the performance map 306 may include multiple different maps (e.g., one for the codec as a whole, one for the codec with a specific combination of features disabled, one for the codec with other combinations of features disabled, etc.).

[0062] Returning to Figure 2 , although the sending device 102 and the receiving device 106 may technically support a given codec, in some cases, the codec may not work well. Some examples of these cases have been described above, but they include: when the device does not have a processor 204 powerful enough to operate the codec effectively; when the receiving device 106 generally will have enough processing power to decode the video stream but multiple input video streams (e.g., during a video conference) consume too much processor resources; when the hardware codec on the sending device 102 has been implemented in a different manner than the corresponding hardware codec on the receiving device 106; when network conditions cause the codec to overshoot; and other cases. Thus, each device may learn the situations in which a codec should not be applied and maintain a client filter list 214 that indicates those codecs in the list of available codecs 210 that should not be announced during the initial codec negotiation phase of a video exchange (or those codecs that should not be considered when applying a new codec in response to changing conditions during video transmission).

[0063] Figure 3BAn example of the client filter list 214 is depicted. The client filter list 214 includes each codec that has been filtered out of consideration and is indexed based on the codec name 310. For each codec, a rationale 312 for why the codec is included in the client filter list 214 can be optionally provided. The rationale 312 can indicate the conditions under which the codec should not be used (e.g., if it is determined that a particular type of receiving device implements the hardware codec differently from the sending device, the rationale 312 in the client filter list 214 of the sending device can indicate that the codec should not be used to send video to those types of receiving devices, but the codec can still be used to encode video data for other types of compatible devices). In some cases, the rationale can indicate that the codec should not be used under any conditions (e.g., because the sending device does not have sufficient processing power to run the codec). Additionally, the rationale 312 can indicate that the codec can be used under specific conditions, but some features should be disabled under those conditions. If no rationale 312 is provided, the device can default to the codec not being under consideration in all cases.

[0064] Each client can provide a copy of its client filter list 214 to the intermediate server 104. Based on the information in the client filter list 214, the server can build its own server filter list 232, as Figure 3C shown. The server filter list 232 can identify each codec by the codec name 314 and can also include information describing the context in which the codec should not be used. For example, the server filter list 232 can identify the sending client type 316 and the receiving client type 318. If a sending device 102 of the same type as the sending client type 316 is attempting to send video data to a receiving device 106 of the same type as the receiving client type 318, the codec corresponding to the codec name 314 can be added to the client filter list 214. Thus, the sending device 102 will not advertise the codec as available for use. In some embodiments, the sending device 102 can remove those codecs on the client filter list 214 from the list of available codecs 210 (effectively forgetting that the sending device 102 ever knew about the removed codecs), while in other embodiments, the list of available codecs 210 can be maintained, and the client filter list 214 can be used to prune on a per-transmission basis.

[0065] The server filter list 232 can include additional information, such as the rationale 312 from the client list, a list of features that can be disabled for a given codec, etc.

[0066] Returning to Figure 2, based on the list of available codecs 210 and the client filter list 214, the sending device 102 can select a codec to be used for encoding the video data. The selected codec 216 can include an encoder 218 that is configured to obtain the raw video data and compress, resize, or otherwise manipulate the raw video data before transmission over the network. Each codec also includes a corresponding decoder 220 that can be used to decode the video data encoded by the encoder 218 of the codec. The currently applied codec can be selected by the codec selection logic 600 which will be described in more detail in conjunction with Figure 6 The codec selection logic 600 will be described in more detail.

[0067] After encoding the raw video data with the encoder 218, the encoded video 222 is generated. The encoded video can be transmitted over the network to the receiving device 106. In some cases, the encoded video 222 can be received at one or more intermediate servers 104 before being received at the receiving device 106. Among other activities, the intermediate server 104 can perform network routing tasks. In some embodiments, the intermediate server 104 can inspect the video data being transmitted (approved by the participants in the video transmission) in order to optimize the transmission. The intermediate server 104 can also inspect error messages and network statistics to determine whether the encoded video 222 causes problems (e.g., indicating a mismatch of hardware codecs, overshoot in the network, etc.), and can flag this information for the sending device 102 and / or the receiving device 106 so that these devices can decide whether to update their client filter list 214. In other embodiments, the intermediate server 104 can be an end-to-end encryption (E2EE) server that cannot inspect the encoded video 222 because it is in an encrypted form that cannot be decrypted by the intermediate server 104.

[0068] The encoded video 222 can include a header indicating the selected codec 216 used for encoding the video data. The receiving device 106 can look up the header metadata, identify the codec used for encoding the data, and apply the corresponding codec to decode the data. Since the sending device 102 and the receiving device 106 negotiate compatible codecs during the initial phase of establishing the video transmission, the sending device 102 should only use codecs that the receiving device 106 can decode.

[0069] As described above, if a given codec uses more processing resources than can be reasonably provided by the transmitting device 102 or the receiving device 106, that codec may be excluded from use. One way to identify such codecs is to use them to encode or decode video data and then measure the load on the processor 204. If the processor load is consistently too high, the use of that codec is excluded. However, this requires the device applying the codec to test each codec individually, which is both time-consuming and resource-consuming. Additionally, if a given device does not fully support a codec, the quality of the video transmission may be affected until the device determines that it does not have sufficient processing resources and switches to another codec.

[0070] Accordingly, each device may maintain a list of CPU thresholds 224 associated with different codecs. The list of CPU thresholds 224 may indicate, for each codec, the amount of processing power required to effectively run the codec. For example, as Figure 3D shown, the CPU threshold 224 may identify a codec with respect to a codec name 320. For each identified codec, the CPU threshold 322 may represent the minimum score with respect to a processor stress test that the processor must be able to achieve in order to have a chance of effectively running that codec.

[0071] Returning to Figure 2 , each client device may develop its own list of CPU thresholds 224 based on its own internal testing. However, the CPU thresholds may also be developed at or transmitted to the intermediate server 104. Accordingly, the intermediate server 104 may maintain a list of CPU thresholds 224 based on the historical testing of many devices and may use these tests to develop its own list of values of CPU thresholds averaged over many different types of hardware. The intermediate server 104 may then transmit its list of CPU thresholds 224 to various client devices, which may use the server-derived CPU thresholds 224 to update their own local lists of CPU thresholds 224. Accordingly, each client device may benefit from the historical information collected by the intermediate server 104 without having to develop its own processing power tests.

[0072] Regardless of whether the list of CPU thresholds 224 is developed at the client, at the intermediate server 104, or at both the client and the intermediate server 104, the local client device may (e.g., using a combination Figure 4The more detailed CPU test logic 400 runs a processing power test to generate a processor score. The local client device can then compare this score to the value 322 of a CPU threshold in the list of CPU thresholds 224 to determine which codecs can be effectively run by the device and which codecs cannot be effectively run by the device. If the processor score is below the threshold required for a given codec, the codec can be added to the client filter list 214.

[0073] As the encoded video 222 is transmitted by the sending device 102 and decoded by the receiving device 106, many possible error conditions can occur. For example, if the codec on the sending device 102 encodes too much data for the current network conditions, the receiving device 106 may lose input data packets (resulting in lost data). This results in a situation called overshoot, which is common in some hardware codecs. In another example, a mismatch in the implementation between the encoder 218 on the sending device 102 and the decoder 220 on the receiving device 106 may cause decoder errors.

[0074] Information about error conditions can be calculated by the receiving device 106, the intermediate server 104, and / or the sending device 102. The resulting error statistics 234 can be sent to the sending device 102 and / or the intermediate server 104 so that each device can determine if the codec being used is problematic and add that codec to its respective filter list. An example of error statistics 234 is depicted in Figure 3E ; in this example, the error statistics 234 includes a list of decoder errors 324 reported by the receiving device 106 and a packet loss rate 326 (which can include the number, rate, or frequency of packets lost in the network). The error statistics 234 can be calculated or analyzed by error analysis logic 700 and / or sender error analysis logic 800 as described in more detail below with respect to Figures 7A - 7B and Figure 8 more detailed description.

[0075] Figures 3A - 3E Depicts an exemplary data structure suitable for use in an embodiment. Each of the depicted data structures has been discussed above in connection with Figure 2 discussion.

[0076] Figure 4 Shows an exemplary CPU test logic 400 according to an embodiment. When the local device does not have sufficient processing power to effectively run a codec (e.g., running the codec would use too large a proportion of the device's processor cycles to run the minimum set of capabilities on the device, resulting in the encoding of video data being too slow to support real-time streaming or transmission of the data), the CPU test logic 400 can be used to remove the codec from consideration.

[0077] The CPU test logic 400 can be implemented as instructions stored on a non-transitory computer-readable medium and can be configured to cause a processor to perform actions corresponding to the logic blocks described below. The CPU test logic 400 can be executed by the sending device 102, the receiving device 106, the intermediate server 104, or a combination of these devices.

[0078] When the local device receives an instruction to perform a CPU test, the process can begin at start block 402. At block 404, the local device retrieves a list of available codecs 210 from its memory or storage, where the list of available codecs 210 represents those codecs supported by the local device.

[0079] At block 406, the local device runs one or more processor tests. The processor tests can include stress tests that run the local device's processor under different conditions and at different loads and can generate one or more scores representing the processing power of the local device. For example, the processor tests can instruct the processor to perform complex mathematical operations, compress data blocks, search for prime numbers, encode video, encrypt data blocks, sort data, run simulations across multiple threads, use multiple CPU cores, etc. The output of the processor tests can be a processor score, the frame rate achieved during a video test, the speed at which an operation is performed, the number of operations performed, the rate of operations performed, etc. The output of the processor tests can represent the processing power of the local device's processor.

[0080] At block 408, the system retrieves a list of CPU thresholds 224. This list can be stored locally on the local device and / or can be stored on a remote server. If both the local device and the remote server maintain separate lists, the list of CPU thresholds 224 for the local device can be updated with information from the remote server (which may be more detailed). If the value 322 of the CPU threshold in the server's list of CPU thresholds 224 does not match the value 322 of the CPU threshold in the local device's list of CPU thresholds 224, the action to be taken may vary depending on the application. For example, in some cases, a more conservative value can be used to reduce the likelihood that the system will be unable to run a given codec. In some embodiments, the server's value can be used because the server's value can be generated based on historical data from many devices, which may be more accurate than the limited tests run by the local device. In other embodiments, the local device's value can be used under the assumption that unique characteristics of the local device's background may allow the local device to run codecs that other similarly powerful devices cannot run. In additional embodiments, these values can be combined (e.g., by averaging them or using a weighted combination).

[0081] At block 410, the system may select a first (next) codec to evaluate from the list of available codecs 210 retrieved at block 404. As previously described, the list of available codecs 210 may include a value 322 indicating a CPU threshold of processing power required to efficiently run the codec.

[0082] At block 412, the local device may compare the CPU test result from block 406 to a codec threshold value corresponding to the codec selected in block 410 in the list of CPU threshold values ​​224. If the CPU test result is not above the CPU threshold value for the selected codec, then at block 414, the codec may be added to a filter list (e.g., client filter list 214) of the local device. If the CPU test result is above the CPU threshold value, then processing may proceed to block 416.

[0083] At block 412, the codec may not be listed in the list of CPU thresholds 224, for example because the codec has not been tested before. In this case, the codec may be allowed to remain in the list of available codecs 210 (e.g., it is not added to the client filter list).

[0084] 214), but codecs can be marked for testing at runtime.

[0085] If a problem occurs with the processor (e.g., the CPU continues to exhibit a high load above a predetermined threshold amount, the encoder causes an error or program crashes, etc.), the codec can be added to the filter list and a note can be made in the list of CPU thresholds 224 indicating that the processor value determined by the local device in block 404 is insufficient to support the codec (e.g., the value 322 of the CPU threshold for the codec in the list of CPU thresholds 224 can be set to "less than" the processor value). The value 322 of the CPU threshold can be refined over time, but it will be known that any device executing at or below the processor power level of the local device will be insufficient to support the codec. This information can be provided to the intermediate server 104 so that the list of CPU thresholds 224 on the server can be updated and provided to other devices.

[0086] At block 416, the system may determine whether there are more codecs to be evaluated. If so, processing returns to block 410 and the next codec is selected for evaluation. If not, the filter list has been updated with codecs that are known to perform poorly given the processing power of the local device. Processing may then proceed to block 418 and terminate.

[0087] Figure 5An exemplary call initiation logic 500 according to an embodiment is shown. The call initiation logic 500 can be executed when initiating a video call and includes a codec negotiation process. Those of ordinary skill in the art will recognize how the call initiation logic 500 can be modified to support other types of video transmissions according to the context.

[0088] The call initiation logic 500 can be implemented as instructions stored on a non-transitory computer-readable medium and can be configured to cause a processor to perform actions corresponding to the logic blocks described below. The call initiation logic 500 can be executed by the sending device 102, the receiving device 106, the intermediate server 104, or a combination of these devices.

[0089] Processing can begin at block 502. At block 504, a call initiation instruction can be received (e.g., when a video conferencing application running on a local client device makes an outgoing call or receives a request to join a call). Initiating a call may involve negotiating codecs that can be used to encode and decode video during the call (e.g., those codecs that are compatible with or supported by the sending device 102 and any receiving device 106).

[0090] To this end, the system can filter its list of available codecs 210 to remove codecs known to perform poorly. Thus, at block 506, the system can retrieve a copy of its filtered list of codecs to be filtered out from the list of available codecs 210 supported by the device. For example, if the call initiation logic 500 is running on the sending device 102, the device can retrieve its client filter list 214.

[0091] The local client filter list 214 can be supplemented by a similar server filter list 232 stored on a remote server. The server filter list 232 can incorporate filtering information from multiple clients and multiple different contexts and can be collected over a longer period of time compared to the client filter list 214. This may help address differences between devices, such as device age and other processes that may use the device's processor when the codec is running. Thus, at block 508, the device can retrieve the server filter list 232. Because the list is incorporated into the client's own client filter list 214, the server can help on-ramp new devices, avoiding the need for each device to independently develop its own client filter list 214 through trial and error. At block 512, the local system can add the server filter list to its own client filter list 214.

[0092] At block 510, the device may retrieve the device type associated with the receiving device 106 (or with another device involved in the call initiation request received at block 504, which may be the case if the local device receives a call initiation from the sending device 102). For example, a video call may be facilitated by an intermediate server capable of identifying the device type of the other device. Alternatively, if the call initiation request at block 504 is the result of another device calling the local device, the call initiation message may specify the device type of the other device. In yet another embodiment, the local device may transmit a call request message to the other device before beginning the codec negotiation process and receive the type of the receiving device in response. Further still, the local device may continue to negotiate the codec with the other device, perhaps even start video transmission, and then learn the device type of the other device at that time.

[0093] According to rationale 312 for including codecs in the filter list, it is possible that some codecs are only incompatible with certain device combinations (e.g., a hardware codec running on a device from a first manufacturer may be able to interoperate with different device types from the same manufacturer, but may have been implemented differently on devices from different manufacturers). Thus, at block 514, the device checks the filter list to determine if there are any known incompatibilities between the device type of the sending device and the device type of the receiving device as determined at block 510. If so, at block 516, the incompatible codecs may be retained in the client filter list 214 such that the corresponding codecs may be removed from the list of available codecs. On the other hand, if a codec appears in the filter list (because it is incompatible for a particular receiving device combination not covered by the combination of the local device's device type and the device type of the other device determined at block 510), then that codec may be retained in the list of available codecs 210.

[0094] After completing and analyzing the filter list, at block 518, the device may filter its list of available codecs 210 to remove those codecs that are present on the filter list (except for the compatible codecs mentioned above). At block 520, as part of the call initiation process, the device may publish the filtered list of available codecs (see Figure 1 ). The sending device and the receiving device may negotiate the list of compatible codecs, and the sending device 102 may start recording and transmitting video data using one of the codecs on the list. The initial codec may be selected based on network conditions, the characteristics of the video being recorded, based on a default codec, or based on other factors. Then, the process may proceed to Figure 6 block 602 in

[0095] Figure 6 Shows an exemplary codec selection logic 600 according to an embodiment. The codec selection logic 600 can be used in conjunction with an ongoing video transmission (e.g., a video conference) to dynamically select a codec based on various factors, including the characteristics of the video being encoded.

[0096] The codec selection logic 600 can be implemented as instructions stored on a non-transitory computer-readable medium and can be configured to cause a processor to perform actions corresponding to the logic blocks described below. The codec selection logic 600 can be executed by the sending device 102, the receiving device 106, the intermediate server 104, or a combination of these devices.

[0097] When a video is encoded using an encoder of a first codec and transmitted to a receiving device, the process can begin at block 602. At block 604, the system can determine whether the background has changed since the previous iteration of the codec selection process. A background change may involve a change in network conditions (e.g., reduced bandwidth), a change in the set of receiving devices for the received video, or a change in the video being recorded. For example, the system can signal that the user has switched the input device 202 (e.g., from a front camera to a rear camera), indicating a change in the scene being captured. The system can signal (e.g., based on location and / or accelerometer data) that the device has been moved to a new location or has been placed in a different orientation or position. The system can otherwise register a change in the scene being captured (e.g., by calculating the amount of movement, average brightness, or number of objects in the scene). Given any of these conditions, the system can determine that it is time to evaluate whether a change to a different codec is necessary. Then, the process can proceed to block 608.

[0098] If the background has not changed since the last iteration, the system can still re-evaluate the codec being applied at a predetermined time interval (e.g., every minute, every two minutes, etc.). At block 606, the system can determine whether such an interval has passed. If not, the process can return to block 602, and the system can wait for a predetermined period of time before determining again whether the background has changed.

[0099] If the background has changed since the last iteration or a predetermined time interval has passed, starting from block 608, the system can evaluate whether it is necessary to change to a different codec. In an exemplary embodiment, the codec is not changed solely based on network conditions and not solely based on whether the new codec will improve the quality of the encoded video. For example, a more complex codec may theoretically improve the quality of the encoded video; however, in practice, if the scene is simple (e.g., recording a user's face), the improvement may be minimal. Rather, the question is whether changing to a more complex codec will sufficiently improve the quality of the video considering other factors that may also be important (e.g., reduced battery life due to using a more complex codec).

[0100] To this end, at block 608, the system determines the encoding quality and battery consumption of the current codec. For example, the encoding quality can be a quantization parameter or a peak signal-to-noise ratio (PSNR) value.

[0101] The encoding quality can be determined in several ways. For example, many software codecs expose a value that can represent or can be used as a proxy for encoding quality (commonly used to perform motion compensation).

[0102] Hardware codecs typically do not make this information visible, and thus it may be necessary to derive this information in other ways. In some embodiments, machine learning can be employed based on features extracted from frames of the video data. This information can be supplemented by proxies for brightness and motion received from the camera. Machine learning algorithms can be applied to the scene being recorded and can determine the expected quality of encoding using the current encoder.

[0103] In another example, the encoded video at the sending device can be provided to a corresponding decoder (also at the sending device). The decoded video can be compared to the original (pre-encoded) data, and the difference between the two can be used as the encoding quality value.

[0104] The battery consumption of the current codec can be determined based on measurements indicating the power draw level of the codec at the local device (e.g., based on the CPU load when the codec is applied), or based on the power consumption 308 from a list of available codecs 210.

[0105] At block 610, the system can determine background features indicating scene complexity (e.g., the amount of movement of objects in the scene, average brightness or number and / or complexity). This information can be used in conjunction with the performance mapping 306 of other available codecs to determine whether other available codecs will improve the quality of the encoded video.

[0106] To this end, at block 612, the system may select the first (next) available codec from the filtered list of available codecs 210. At block 614, the system may use the performance map 306 and the power consumption 308 to calculate the expected video quality (and the expected amount of power consumed to achieve that quality) that will be achieved by the selected codec under the conditions determined at block 610. At block 616, the system may use this information to calculate one or more improvement scores that represent the amount of improvement achieved by the selected codec compared to the currently applied codec. The score may be weighted based on the increase (or decrease) in battery consumption of the selected codec such that if the selected codec is expected to increase the power consumption of the device, the score is reduced. The weighting may be linear. Alternatively, in some cases, the weighting may be non-linear (e.g., exponential) such that the greater the increase in power consumption, the greater the penalty.

[0107] At block 618, the system may determine whether the improvement score determined at block 616 exceeds a minimum improvement threshold. That is, the system determines whether the improvement in quality (weighted based on the increase in battery consumption) is sufficient to warrant a change from the current codec. If the improvement is not significant and the battery consumption increases or remains the same, the system may continue to apply the existing codec.

[0108] If the improvement does exceed the threshold at block 618, the process proceeds to block 620 and the selected codec is added to the candidate list. If not, at block 622, the system determines whether there are any more codecs to be evaluated. If so, it returns to block 612. If not, it proceeds to block 624.

[0109] At block 624, the candidate list consists of zero or more codecs whose improvement scores exceed the minimum improvement threshold. If no such candidate is found, the process may return to block 602. If one such codec is found, at block 626, the candidate codec may be applied to replace the existing codec. If more than one such codec is found, at block 624, the codec with the best balance of quality improvement and battery consumption (e.g., the codec with the highest improvement score) is selected and that codec may be applied at block 626. Then, the process may return to block 602.

[0110] Figures 7A - 7B An exemplary error analysis logic 700 according to an embodiment is shown. The error analysis logic 700 includes logic for analyzing decoder errors ( Figure 7A ) and logic for analyzing packet loss rates ( Figure 7B) logic. This represents two ways of analyzing codec errors, which are used to determine whether codecs implemented on two different devices may be incompatible, or whether other problems in the network are causing poor codec performance; however, those of ordinary skill in the art will recognize that there are other ways to identify poor codec performance, and these other ways can be used to signal a dynamic codec change.

[0111] The error analysis logic 700 can be implemented as instructions stored on a non-transitory computer-readable medium and can be configured to cause a processor to perform actions corresponding to the logic blocks described below. The error analysis logic 700 can possibly be executed by the receiving device 106 and / or the intermediate server 104 with the help of the sending device 102.

[0112] As Figure 7A shown, when video is transmitted to the receiving device, the process can start at block 702. At block 704, the receiving device can receive the encoded video. At block 706, the receiving device can identify the codec used to encode the video and can provide the encoded video to the corresponding decoder.

[0113] At block 708, the system can measure any errors during the decoding process. For example, the decoder can report that one or more errors have occurred; alternatively, the video can be analyzed to determine whether any encoding / decoding artifacts have been added, whether data has been lost, whether there are any pixel misalignments or obvious incorrectness, etc.

[0114] If the decoder error rises to a level exceeding a predetermined threshold ("Yes" at block 710), then the system can attempt to reset the background of the video at block 712 by requesting key frames associated with the video data. The key frames can be sourced from buffers or memories on the sending device 102, the intermediate server 104, or the receiving device 106. Restoring to the key frame can mitigate some or all of the decoder errors. If the decoder error is measured again (block 714) and still exceeds the threshold ("Yes" at block 716), then the system can determine that there is an inconsistency or incompatibility between the encoder on the sending device 102 and the decoder on the receiving device 106. For example, the hardware encoder may have different implementations on the sending device 102 and the receiving device 106, resulting in decoder errors. Thus, at block 718, the receiving device 106 can signal to the intermediate server 104 and / or the sending device 102 that the codec is incompatible between the sending device type and the receiving device type. This information can trigger a change in the codec applied at the sending device 102 and can cause the sending / receiving device combination to be added to the client filter list 214 and / or the server filter list 232 regarding the current codec.

[0115] If the decoder error is initially insufficient to indicate incompatibility ("No" at block 710), or the use of key frames resolves the error ("No" at block 716), then the process can return to block 704 and additional video can be received and decoded. Otherwise, the process can proceed to block 720 and end.

[0116] Another problem may be caused by overshoot, which can be dynamically resolved by the Figure 7B process shown.

[0117] Processing begins at block 722, where receiving device 106 receives the input encoded video. At block 724, receiving device 106, intermediate server 104, or transmitting device 102 can identify the packet loss rate 326 associated with the video transmission. The packet loss rate can indicate how many data packets are transmitted over a period of time but not successfully delivered to the receiving device (which may include data packets received at the network interface 208 of receiving device 106 but discarded due to lack of buffer space or memory). If the packet loss rate is too high, this indicates that transmitting device 102 is encoding the video at too high a bit rate; or the network cannot transmit the video fast enough, or the decoder on receiving device 106 cannot decode the video fast enough. Thus, some information is being lost, which can be particularly problematic when the video is being transmitted in real time as part of an ongoing video conversation. Such video cannot be queued or cached indefinitely for future use, and thus the video can become incoherent, lag, or exhibit other problems.

[0118] In some cases, receiving device 106 may support a particular codec and may even have sufficient processing resources to run that codec under normal circumstances. However, during a video call where multiple transmitting devices are sending video to the receiving device, the processing capabilities of the receiving device may become overwhelmed. This can also be indicated by an excessive number or rate of lost data packets, suggesting that the transmitting device should switch to a simpler codec.

[0119] Thus, if the packet loss rate exceeds a predetermined threshold value ("Yes" at block 726), then at block 730, the device can signal to transmitting device 102 that there is a problem with the codec. Then, the process can proceed to block 732 and end.

[0120] If the packet loss rate does not exceed the predetermined threshold ("No" at block 726), then at block 728, the system can wait for a predetermined amount of time. Subsequently, the process can return to block 724, and the system can evaluate the packet loss rate for the next time interval.

[0121] Figure 8An exemplary sender error analysis logic 800 according to an embodiment is shown. The sender error analysis logic 800 can operate in conjunction with the error analysis logic 700 operating on the receiving device 106 and / or the intermediate server 104 to evaluate when a codec may need to be dynamically trimmed from a list of available codecs 210 such that the codec is changed and not used (at least) for subsequent video encoding during the current transmission.

[0122] The sender error analysis logic 800 can be implemented as instructions stored on a non-transitory computer-readable medium and can be configured to cause a processor to perform actions corresponding to the logic blocks described below. The sender error analysis logic 800 can potentially be executed by the sending device 102 with the help of the intermediate server 104 and / or the receiving device 106.

[0123] When the sending device 102 encodes and transmits video, the process begins at block 802. At some point (block 804), the sending device 102 can receive a report of codec incompatibility from the intermediate server 104 and / or the receiving device 106. The incompatibility report may indicate a mismatch between the sending device 102 and the receiving device 106 in the way their codecs are implemented (indicated by decoder errors), or that the codec used by the sending device 102 exceeds the available bandwidth or processing power of the receiving device 106.

[0124] At block 806, the system determines whether the current codec can continue to be used by eliminating features from the codec. For example, some codecs may support complex B-frame features, but this support may be turned off to use a simpler B-frame. If there are no removable features (e.g., the codec must be used as is or not used at all), then at block 812, the codec can be changed. A process for selecting the initial codec or a process described in conjunction with Figure 6 blocks 612 - 626 can be used to select a new codec.

[0125] If the codec includes features that can be disabled, it may be worthwhile to continue using the existing codec with a reduced feature set; however, this is not given, as there may be other available codecs that are superior to the diminished existing codec. Therefore, at block 808, a process similar to the process described in conjunction with Figure 6 blocks 612 - 626 above can be used to test the existing codec (with features disabled) against other available codecs.

[0126] If the existing codec continues to be superior to other available codecs ("Yes" at block 810), then at block 814, these features can be disabled and the existing codec can continue to be used. Otherwise, the process can proceed to block 812 and the codec can be changed.

[0127] In any case, codecs indicated as incompatible at block 604 can be added to the client filter list 214 and / or the server filter list 232 at block 816. Justifications 312 (such as "excessive overshoot" or "hardware implementation incompatibility between <sender device type> and <receiver device type>") can be indicated on the filter list so that the codec can be removed from the list of available codecs 210 under similar conditions in the future. Then, the process can return to block 802 and the system can wait for a new incompatibility indication.

[0128] Figure 9 An example of a system architecture and a data processing device that can be used to implement one or more illustrative aspects described herein in a stand-alone and / or networked environment is shown. Various network nodes (such as data server 910, web server 906, computer 904, and mobile device 902) can be interconnected via a wide area network 908 (WAN) (such as the Internet). Other networks (including private intranets, corporate networks, LANs, metropolitan area networks (MANs), wireless networks, personal area networks (PANs), etc.) can also be or alternatively be used. The illustrated network 908 and devices are for illustrative purposes and can be replaced with fewer or additional computer networks or devices. A local area network (LAN) can have one or more of any known LAN topologies and can use one or more of various different protocols (such as Ethernet). The data server 910, web server 906, computer 904, mobile device 902, and other devices (not shown) can be connected to one or more networks via twisted pair, coaxial cable, fiber optic, radio waves, or other communication media.

[0129] Computer software, hardware, and networks can be used in a variety of different system environments (including stand-alone, networked, remote access (also known as remote desktop), virtualized, and / or cloud-based environments, etc.).

[0130] As used herein and depicted in the figures, the term "network" refers not only to a system in which remote storage devices are coupled together via one or more communication paths, but also to independent devices that can be coupled to such a system with storage capabilities from time to time. Thus, the term "network" includes not only "physical networks" but also "content networks", where a "content network" consists of data belonging to a single entity that resides on all physical networks.

[0131] The data server 910 can provide overall access, control, and management of the database and control software to perform one or more illustrative aspects described herein. The data server 910 can be connected to the web server 906, through which users can interact with and obtain data according to requests. Alternatively, the data server 910 can act as the web server itself and be directly connected to the Internet. The data server 910 can be connected to the web server 906 via the network 908 (e.g., the Internet) through a direct connection, an indirect connection, or via some other network connection. Users can use the remote computer 904 or the mobile device 902 to interact with the data server 910, e.g., using a web browser to connect to the data server 910 via one or more externally public websites hosted by the web server 906.

[0132] The client computer 904 or the mobile device 902 can be used in conjunction with the data server 910 to access the data stored therein, or can be used for other purposes. For example, from the client computer 904, a user can use an Internet browser known in the art or access the web server 906 by executing a software application that communicates with the web server 906 and / or the data server 910 via a computer network such as the Internet.

[0133] The server and the application can be combined on the same physical machine and retain separate virtual or logical addresses, or can reside on separate physical machines. Figure 9 Only one example of the network architecture that can be used is shown, and those skilled in the art will recognize that, as further described herein, the specific network architecture and data processing devices used can vary and are secondary to the functions they provide. For example, the services provided by the web server 906 and the data server 910 can be combined on a single server.

[0134] Each device shown can be any type of known computer, server, or data processing device. The devices can each include a hardware processor 912 that controls the overall operation of the device. The device can also include a RAM 916, a ROM 918, a network interface 914, an input / output interface 920 (e.g., keyboard, mouse, display, printer, etc.), and a memory 922.

[0135] The input / output interface 920 can include various interface units and drivers for reading, writing, displaying, and / or printing data or files.

[0136] The RAM 916, ROM 918, and memory 922 can be non-transitory computer-readable media that store instructions configured to cause the respective devices to perform the techniques described herein, and can also store operating system software 924 for controlling the overall operation of the data server 910, control logic 926 for instructing the data server 910 to perform the aspects described herein, and other application software 928 that provides auxiliary, support, and / or other functions that may or may not be used in combination with the aspects described herein. The functions of the device can refer to operations or decisions made automatically based on rules encoded into the control logic, made manually by a user providing input into the system, and / or a combination of automated processing based on user input (e.g., queries, data updates, etc.).

[0137] The memory 922 can also store data used in the execution of one or more aspects described herein, including a first database 932 and a second database 930. In some embodiments, the first database can include the second database (e.g., as separate tables, reports, etc.). That is, depending on the system design, information can be stored in a single database or divided into different logical, virtual, or physical databases. The illustrated devices can each have an architecture similar to or different from those described. Those skilled in the art will understand that the functions described herein can be distributed across multiple data processing devices, e.g., to distribute the processing load across multiple computers, to isolate transactions based on geographical location, user access level, quality of service (QoS), etc.

[0138] One or more aspects may be implemented in computer-usable or readable data and / or computer-executable instructions executed by one or more computers or other devices as described herein (such as one or more aspects may be implemented in one or more program modules). Generally, program modules include routines, programs, objects, components, data structures, etc., which perform particular tasks or implement particular abstract data types when executed by a processor in a computer or other device. The modules may be written in source code programming languages that are subsequently compiled for execution, or may be written in scripting languages such as (but not limited to) HTML or XML. The computer-executable instructions may be stored on a computer-readable medium such as a non-volatile storage device. Any suitable computer-readable storage medium may be used (including hard disks, CD-ROMs, optical storage devices, magnetic storage devices, and / or any combination thereof). In addition, various transmission (non-storage) media representing data or events as described herein may be transmitted between a source and a destination in the form of electromagnetic waves traveling through a signal-conducting medium such as metal wires, optical fibers, and / or wireless transmission media (e.g., air and / or space). The various aspects described herein may be implemented as a method, a data processing system, or a computer program product. Thus, the various functions may be implemented in whole or in part in software, firmware, and / or hardware or hardware equivalents (such as integrated circuits, field-programmable gate arrays (FPGAs), etc.). Particular data structures may be used to more effectively implement one or more aspects described herein, and such data structures are contemplated within the scope of the computer-executable instructions and computer-usable data described herein.

[0139] The above embodiments may be executed by a messaging architecture, and examples thereof will be described next with reference to Figure 10 its examples.

[0140] Figure 10 An embodiment showing multiple servers implementing various functions of a messaging service 1000 suitable for exemplary embodiments is illustrated. It should be understood that different work and function distributions may be used in various embodiments of the messaging service 1000.

[0141] The messaging service 1000 may include a domain name front end 1002. The domain name front end 1002 may be assigned one or more domain names associated with the messaging service 1000 in the Domain Name System (DNS). The domain name front end 1002 may receive incoming connections and assign these connections to servers providing various messaging services.

[0142] The messaging service 1000 may include one or more chat servers 1004. The chat server 1004 may include a front-end server for receiving and transmitting user-to-user messaging updates such as chat messages. Based on workload balancing, the domain name front-end 1002 may allocate input connections to the chat server 1004.

[0143] The messaging service 1000 may include a back-end server 1008. The back-end server 1008 may perform specialized tasks in support of the chat operations of the front-end chat server 1004. A variety of different types of back-end servers 1008 may be used. It should be understood that in different embodiments, the allocation of task types to different back-end servers 1008 may vary. In some embodiments, some of the back-end services provided by dedicated servers may be combined into a single server or a group of servers, and in the embodiments described herein, each server performs multiple tasks divided among different servers. Similarly, in some embodiments, the tasks of some of the dedicated back-end servers 1008 described herein may be divided among different servers in different server groups.

[0144] The messaging service 1000 may include one or more offline storage servers 1010. The one or more offline storage servers 1010 may store messaging content for messaging clients that are currently offline for use when the messaging clients reconnect.

[0145] The messaging service 1000 may include one or more session servers 1012. The one or more session servers 1012 may maintain the session state of connected messaging clients.

[0146] The messaging service 1000 may include one or more presence servers 1014. The one or more presence servers 1014 may maintain the presence information of the messaging service 1000. The presence information may correspond to user-specific information that indicates whether a given user has an online messaging client and is available for chatting, has an online messaging client but is currently away from the client, does not have an online messaging client, and any other presence state.

[0147] The messaging service 1000 may include one or more push storage servers 1016. The one or more push storage servers 1016 may cache push requests and transmit the push requests to messaging clients. The push requests may be used to wake up messaging clients to notify the messaging clients that messaging updates are available and to otherwise perform server-side driven interactions with the messaging clients.

[0148] The messaging service 1000 may include one or more group servers 1018. The one or more group servers 1018 may maintain group lists, add users to groups, remove users from groups, and perform reception, caching, and forwarding of group chat messages.

[0149] The messaging service 1000 may include one or more block list servers 1020. The one or more block list servers 1020 may maintain user-specific block lists. The user-specific input block list indicates, for each user, one or more other users who are prohibited from transmitting messages to that user. Alternatively or additionally, the one or more block list servers 1020 may maintain user-specific output block lists, which indicate, for each user, one or more other users to whom that user is prohibited from transmitting messages. It should be understood that the input block list and the output block list may be stored in combination, for example, in a database, where the input block list and the output block list represent different views of the same repository of block information.

[0150] The messaging service 1000 may include one or more last-seen information servers 1022. The one or more last-seen information servers 1022 may receive, store, and maintain information indicating the last access location, status, messaging client, and other elements of the user's last access connection to the messaging service 1000.

[0151] The messaging service 1000 may include one or more key servers 1024. The one or more key servers may host public keys for public / private key encrypted communication.

[0152] The messaging service 1000 may include one or more profile photo servers 1026. The one or more profile photo servers 1026 may store and make available for retrieval profile photos of multiple users of the messaging service 1000.

[0153] The messaging service 1000 may include one or more spam logging servers 1028. The one or more spam logging servers 1028 may log known and suspected spam (e.g., unwanted messages, especially those of a promotional nature). The one or more spam logging servers 1028 may be operable to analyze messages to determine if they are spam and, in some embodiments, perform penalty measures against suspected spammers (users who send spam messages).

[0154] The messaging service 1000 may include one or more statistics servers 1030. One or more statistics servers may edit and store statistics related to the operation of the messaging service 1000 and the behavior of the users of the messaging service 1000.

[0155] The messaging service 1000 may include one or more web servers 1032. One or more web servers 1032 may make Hypertext Transfer Protocol (HTTP) and Hypertext Transfer Protocol Secure (HTTPS) connections with web browsers.

[0156] The messaging service 1000 may include one or more chat activity monitoring servers 1034. One or more chat activity monitoring servers 1034 may monitor user chats to identify unauthorized or discouraged behavior of the users of the messaging service 1000. One or more chat activity monitoring servers 1034 may work in concert with the spam logging server 1028 and the block list server 1020, where one or more chat activity monitoring servers 1034 identify spam or other discouraged behavior and provide spam information to the spam logging server 1028 and, where appropriate, block information to the block list server 1020.

[0157] The messaging service 1000 may include one or more synchronization servers 1036. One or more synchronization servers 1036 may synchronize the messaging service 1000 with contact information from the messaging client (such as the address book on a mobile phone) to identify the contacts of the users in the messaging service 1000.

[0158] The messaging service 1000 may include one or more multimedia servers 1038. One or more multimedia servers may store multimedia (such as images, videos, audio) transferred between messaging clients, cache multimedia for offline endpoints, and may perform transcoding of multimedia.

[0159] The messaging service 1000 may include one or more payment servers 1040. One or more payment servers 1040 may process payments from users. One or more payment servers 1040 may connect to an external third-party server to perform payments.

[0160] The messaging service 1000 may include one or more registration servers 1042. One or more registration servers 1042 may register new users of the messaging service 1000.

[0161] The messaging service 1000 may include one or more voice relay servers 1044. The one or more voice relay servers 1044 may relay Internet Protocol voice (VoIP) voice communications between messaging clients to perform VoIP calls.

[0162] In some embodiments, the messaging service 1000 may be an end-to-end encrypted (E2EE) messaging service, where the sending device encrypts the information and the receiving device decrypts the information. Intermediate servers of the messaging service 1000 may assist in establishing an E2EE session and may facilitate the communication transfer between devices, but may not be able to decrypt (and thus access) the content of the communication. In an E2EE environment, some adjustments may be required to the processes that the server would perform in a non-E2EE environment (eliminating these processes, adjusting them, or moving them to one or more client devices).

[0163] Some embodiments may be described using the phrases "one embodiment" or "an embodiment" and their derivatives. These terms mean that the particular features, structures, or characteristics described in connection with the embodiment are included in at least one embodiment. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment. Additionally, unless otherwise stated, the above features are considered to be used in any combination together. Thus, any features discussed separately can be used in combination with each other, unless it is noted that these features are incompatible with each other.

[0164] Generally referring to the symbols and terms used herein, the detailed description herein may be presented in terms of program procedures executed on a computer or computer network. These process descriptions and representations are used by those skilled in the art to most effectively convey the substance of their work to other skilled persons in the art.

[0165] A process is herein and generally considered to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transported, combined, compared, and otherwise manipulated. For reasons of common usage mainly, it has proven convenient at times to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, and so on. However, it should be noted that all of these and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to these quantities.

[0166] In addition, the manipulations performed are often expressed in terms (such as addition or comparison) that are commonly associated with intellectual operations performed by a human operator. In any of the operations forming part of one or more embodiments described herein, this ability of a human operator is not necessary or not desirable in most cases. Instead, these operations are machine operations. Useful machines for performing the operations of the various embodiments include general-purpose digital computers or similar devices.

[0167] Some embodiments may use the terms "coupled" and "connected" and their derivatives to describe. These terms are not necessarily meant to be synonyms of each other. For example, some embodiments may use the terms "connected" and / or "coupled" to describe to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0168] The various embodiments also relate to apparatus or systems for performing these operations. The apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. The processes presented herein are not inherently related to a particular computer or other device. Various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may prove convenient to construct more specialized devices to perform the required method steps. The structures required for the various such machines will emerge from the given description.

[0169] It is emphasized that the abstract of the disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that the abstract will not be used to interpret or limit the scope or meaning of the claims. In addition, as can be seen in the above detailed description, for the purpose of streamlining the disclosure, various features are grouped together in a single embodiment. The method of the disclosure is not to be construed as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Instead, as reflected by the appended claims, the subject matter of the invention lies in less than all of the features of a single disclosed embodiment. Accordingly, the appended claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment. In the appended claims, the terms "including" and "in which" are used as the plain English equivalents of the respective terms "comprising" and "wherein". In addition, the terms "first", "second", "third", etc. are used only as labels and are not intended to impose numerical requirements on their objects.

[0170] The foregoing description includes examples of the disclosed architecture. Of course, it is not possible to describe every conceivable combination of components and / or methods, but one of ordinary skill in the art will recognize that many further combinations and permutations are possible. Accordingly, the novel architecture is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

Claims

1. A method for dynamically selecting a codec, comprising: encoding video data using a first codec; receiving an instruction to re-evaluate the first codec; determining an expected quality improvement that a second codec is expected to achieve relative to the first codec; determining a change in the expected power consumption of the second codec; if a combination of the expected quality improvement and the change in the expected power consumption meets a predetermined requirement, switching to the second codec; and encoding video data using the second codec; wherein switching to the second codec includes calculating an improvement score based on the expected quality improvement and the change in the expected power consumption, and switching to the second codec only if the improvement score exceeds a value of a predetermined threshold.

2. The method according to claim 1, wherein in response to detecting a change in the background of the video data, receiving an instruction to re-evaluate the first codec.

3. The method according to claim 2, wherein detecting the change in the background of the video data includes one or more of the following: detecting a change in a recording device that captures the video data, detecting a change in a scene recorded in the video data, or detecting a change in a position or orientation of a device that captures the video data.

4. The method according to claim 1, further comprising: determining that a third codec can be used to encode the video data; determining that the third codec is on a filtered list of codecs excluded from use; and in response to receiving an instruction to re-evaluate the first codec, suppressing the evaluation of the third codec.

5. The method according to claim 4, further comprising constructing the filtered list by: selecting candidate codecs that can be used by a local device; retrieving a list of processor thresholds representing amounts of processing power used to operate corresponding codecs; running a processor test on the local device to calculate a processor score; determining that the processor score is below a value of a threshold associated with the candidate codec on the processor threshold list; and when the processor score is below the value of the threshold, adding the candidate codec to the filtered list.

6. The method according to claim 4, further comprising constructing the filtered list by: determining that the third codec is on a list of codecs compatible with a sending device and a receiving device; operating the third codec to encode the video data at the sending device; receiving measurement results of one or more of the following: decoder errors associated with operating a decoder corresponding to the codec, a number of data packets lost when transmitting the video data, or an overshoot associated with transmitting the video data; determining that the measurement results exceed a predetermined threshold; and in response to the measurement results exceeding the predetermined threshold, adding the third codec to the filtered list.

7. A non - transitory computer - readable storage medium, the computer - readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the following operations: Encode video data using a first codec; Receive an instruction to re - evaluate the first codec; Determine an expected quality improvement that the second codec is expected to achieve relative to the first codec; Determine a change in the expected power consumption of the second codec; If the combination of the expected quality improvement and the change in the expected power consumption meets a predetermined requirement, switch to the second codec; and Encode video data using the second codec; wherein switching to the second codec includes calculating an improvement score based on the expected quality improvement and the change in the expected power consumption, and switching to the second codec only when the improvement score exceeds a value of a predetermined threshold.

8. The computer - readable storage medium according to claim 7, wherein, in response to detecting a change in the background of the video data, receive an instruction to re - evaluate the first codec.

9. The computer - readable storage medium according to claim 8, wherein, detecting a change in the background of the video data includes one or more of the following: detecting a change in the recording device that captures the video data, detecting a change in the scene recorded in the video data, or detecting a change in the position or orientation of the device that captures the video data.

10. The computer - readable storage medium according to claim 7, wherein, the instructions further configure the computer to: Determine that a third codec can be used to encode the video data; Determine that the third codec is on a filtered list of codecs excluded from use; and In response to receiving an instruction to re - evaluate the first codec, suppress the evaluation of the third codec.

11. The computer - readable storage medium according to claim 10, wherein, the instructions further configure the computer to build the filtered list by: Selecting candidate codecs that the local device can use; Retrieving a list of processor thresholds representing the amount of processing power used to operate the respective codecs; Running a processor test on the local device to calculate a processor score; Determine that the processor score is lower than a value of a threshold associated with the candidate codec on the processor threshold list; and When the processor score is lower than the value of the threshold, add the candidate codec to the filtered list.

12. The computer - readable storage medium according to claim 10, wherein, the instructions further configure the computer to build the filtered list by: Determine that the third codec is on a list of codecs compatible with the sending device and the receiving device; Operate the third codec to encode the video data at the sending device; Receive measurement results of one or more of the following: decoder errors associated with operating a decoder corresponding to the codec, the number of data packets lost when transmitting the video data, or the overshoot associated with transmitting the video data; Determine that the measurement results exceed a predetermined threshold; and In response to the measurement results exceeding the predetermined threshold, add the third codec to the filter list.

13. A computing device, the computing device comprises: a processor; and a memory storing instructions that, when executed by the processor, configure the device to: Encode video data using a first codec; Receive an instruction to re-evaluate the first codec; Determine an expected quality improvement that the second codec is expected to achieve relative to the first codec; Determine a change in the expected power consumption of the second codec; If the combination of the expected quality improvement and the change in the expected power consumption meets a predetermined requirement, switch to the second codec; and Encode video data using the second codec; wherein switching to the second codec includes calculating an improvement score based on the expected quality improvement and the change in the expected power consumption, and switching to the second codec only when the improvement score exceeds a value of a predetermined threshold.

14. The computing device according to claim 13, wherein, In response to detecting a change in the background of the video data, receive an instruction to re-evaluate the first codec, the change in the background including one of the following: a change in the recording device that captures the video data, a change in the scene recorded in the video data, or a change in the position or orientation of the device that captures the video data.

15. The computing device according to claim 13, wherein, The instructions further configure the device to: Determine that a third codec can be used to encode the video data; Determine that the third codec is on a filter list of codecs excluded from use; and In response to receiving an instruction to re-evaluate the first codec, suppress the evaluation of the third codec.

16. The computing device according to claim 15, wherein, The instructions further configure the device to build the filter list by: Select candidate codecs that the local device can use; Retrieve a list of processor thresholds representing the amount of processing power used to operate the corresponding codecs; Run a processor test on the local device to calculate a processor score; Determine that the processor score is lower than a value of the threshold associated with the candidate codec on the processor threshold list; and When the processor score is lower than the value of the threshold, add the candidate codec to the filter list.

17. The computing device according to claim 15, wherein, The instructions further configure the device to build the filter list by: Determine that the third codec is on a list of codecs compatible with the sending device and the receiving device; Operate the third codec to encode the video data at the transmitting device; Receive measurements of one or more of the following: decoder errors associated with operating a decoder corresponding to the codec, the number of data packets lost when transmitting the video data, or the amount of overshoot associated with transmitting the video data; Determine that the measurements exceed a predetermined threshold; and In response to the measurements exceeding the predetermined threshold, add the third codec to the filtering list.

Images