Systems and methods for managing display electronics during audio / video conferences

Abstract

The systems first delineate a person or user from the background in a video image using a bounding box. The systems apply a dimming factor to pixels associated with the background in a video image. The dimming factor can be a percentage of the pixel's original intensity. The dimming factor is a variable; in first order, the dimming factor varies as a function of distance from the bounding box. In second order, the dimming factor has a rate of change that varies as it is applied across the background. The dimming factor is smaller adjacent to the person and larger at the periphery of the video image.

Description

BACKGROUND

[0001] Display electronics and screens are among the most power-consuming components in a client device. Many popular audio / video conferencing and video chat applications consume significant power through the operation of display electronics and screens. There is a continuing desire to reduce power consumption to improve battery life. Accordingly, continuous reductions in power consumption by display electronics and screens, especially during audio / video conferencing, are desirable. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a schematic representation of an exemplary environment in which teachings of the disclosure may be implemented. Fig.2 is an architectural block diagram of one or more application modules that may operate in a system for managing display electronics during audio / video conferencing, according to various embodiments. Fig. 3A and Fig. 3B are visual aids for discussing the algorithms and tasks that the embodiments implement. Fig. 4 illustrates a use case according to various embodiments. Fig. 5 shows an example flowchart of an example method for managing display electronics during audio / video conferences according to various embodiments. Fig. Figure 6 shows an exemplary method for managing display electronics during audio / video conferences based on software. Fig. 7 illustrates a timing controller as may be implemented in various embodiments. Fig. 8 is a block diagram of an example compute node that may include any of the embodiments disclosed herein. Fig. 9 illustrates a multiprocessor environment in which embodiments may be implemented. Fig. 10 is a block diagram of an example processor unit 1100 for executing computer-executable instructions as part of implementing technologies described herein. DETAILED DESCRIPTION

[0002] The display electronics and the display screen driven by the display electronics are among the most power-consuming components in a client device, which negatively impacts battery life. A new display technology, emissive display screens, is expected to become widely adopted for the display electronics in client devices. Some examples of emissive display screens include display screens using technologies such as OLED (organic light-emitting diodes) and micro-LEDs. Transmissive displays, which use mini-LEDs as backlights, are also popular due to the high contrast range they provide. Mini-LED displays are a technology that uses LEDs to backlight an LCD screen.

[0003] Because the display electronics and the display screen driven by the display electronics are in heavy use during a variety of popular audio / video conferencing and video chat applications, these popular audio / video conferencing and video chat applications consume significant power. Some non-limiting factors that control this power consumption include the need to meet requirements for image resolution, contrast, brightness, refresh rates, and the like without sacrificing transmission speed. Any reduction in power consumption by the display electronics and the display screen driven by the display electronics can greatly improve the battery life of client devices.Therefore, a technical challenge and opportunity is presented to reduce the power consumption of emitting display screens beyond existing display power saving techniques, especially during the use of audio / video conferencing applications.

[0004] Some approaches using OLED technologies dim the edge of the display in one or more layers by a certain percentage. The disadvantage of this approach is that only the edges can be dimmed, thus limiting its power savings. Furthermore, this solution does not proactively detect or exploit opportunities that arise during audio / video conferencing to dim even more pixels.

[0005] Embodiments provide a solution to this technical challenge and related problems in the form of technologically enhanced systems and methods for managing display electronics during audio / video conferencing. As described in more detail below, embodiments leverage opportunities presented during audio / video conferencing, such as the popular use of features like background blurring, background replacement filters, and the like, to dim pixels and reduce power consumption. In doing so, embodiments not only improve the energy efficiency of these display screens but also extend the battery life of client devices.

[0006] Embodiments have the potential to save up to 50% of display power in a client device using emissive displays such as OLED, microLED, and transmissive displays with mini-LEDs as backlights, an LCD panel technology using LED diodes to backlight an LCD display, for example, in audio / video conferencing, without any degradation of visual quality or user experience. Embodiments achieve this by dimming pixels that are not actively used to display the user or any "meaningful" content, where meaningful is described in more detail below.

[0007] Some embodiments may exhibit a reduction in power consumption of up to 50% during applications such as audio / video conferencing. This may be observed, for example, by (1) observing display power consumption in an active video conference call; or (2) recording the video conference call and playing back the recorded video conference call to observe operating power consumption. Additionally, optical sensors may be used to detect brightness and intensity while comparing an embodiment to a display screen that does not implement the embodiment. For example, if an image includes a person wearing a white dress and the background also includes some white content, the optical sensor may reveal that the white in the background will have less brightness than the dress.

[0008] As used herein, the terms "processor unit", "processing circuitry", or "processing unit" or "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory. A processor unit may be a SOC (system on a chip) and / or one or more DSPs (digital signal processor units), ASICs (application-specific integrated circuits), CPUs (central processing units), GPUs (graphics processing units), GPGPUs (general-purpose graphics processing units), APUs (accelerated processing units), FPGAs (field-programmable gate arrays), NPUs (neural network processing units), DPUs (data processing units), accelerators (e.g.,Graphics accelerators, compression accelerators, artificial intelligence accelerators), control cryptoprocessors (specialized processing units that execute cryptographic algorithms in hardware), server processing units, controllers, or any other suitable type of processing unit. Therefore, the processing unit may be referred to as an XPU (or xPU).

[0009] As used herein, the term "module" may refer to any hardware, software, firmware, electronic control component, processing logic, and / or processor device, individually or in any combination. In various embodiments, a module is one or more of the following: an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), an electronic circuit, a computer system including a processor (shared, dedicated, or group) and memory executing one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the functionality associated with the module.

[0010] For the sake of brevity, conventional techniques related to signal processing, data transmission, signaling, control, artificial intelligence (AI) models, machine learning models, image analysis, and other functional aspects of the systems (and the individual operating components of the systems) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures included herein are intended to represent exemplary functional relationships and / or physical or logical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may be present in an embodiment of the present disclosure.

[0011] Embodiments of the present disclosure may be described herein in terms of functional and / or logical block components that perform various steps or tasks. It should be understood that such block components may be implemented by any number of hardware, software, and / or firmware components configured to perform the specified functions. For example, an embodiment of the present disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, or the like, that can perform a variety of functions under the control of one or more microprocessors or other control devices.

[0012] As mentioned, audio and video conferencing and video chat applications can operate with lower power consumption using embodiments described herein. The technologically enhanced systems and methods for managing display electronics during audio / video conferencing are described in more detail below in conjunction with the figures.

[0013] Fig. Figure 1 is a schematic representation of an example environment 100 in which embodiments may be implemented. Multiple user devices, represented as user device 1 through user device N (N>1), may be communicatively coupled to one another via a network 106. A variety of different transmission protocols and architectures may be used in the network 106 and to support bidirectional communication between the user devices, such as, but not limited to, wireless, WIFI, 2G, 3G, 4G, 5G, etc.

[0014] The user devices are capable of sending and receiving media data including at least audio, video, and images; and the user devices are configured, such as with an installed device application, to execute at least one media application that consumes and processes audio, video, and images. Some example media applications include audio and video conferencing applications and video chat applications. Accordingly, the user devices generally include at least one camera (108-1, 108-2), a speaker 110-1, 110-2, a microphone 112-1, 112-2, a display 114-1, 114-2, a user input device 116-1, 116-2 (e.g., a keyboard or touchscreen), and a communication system 119-1, 118-2 that supports communication over the network 106. In various aspects of the disclosure, the network 106 includes a cloud server (not shown).

[0015] Although the user devices are drawn the same, in practice they may be any combination of available computing devices that meet the above criteria. For example, the user devices may include any combination of laptop computers, desktop computers, kiosks, and mobile phones.

[0016] Various aspects of this disclosure relate to a receiving-side device. In an exemplary embodiment, user device 1 is the receiving device, and a system for managing display electronics during audio / video conferences, shown generally as system 105, is shown on the right side in Fig.1 is enlarged. System 105 includes a control circuit 122. In one embodiment, system 105 may further include an unpacketization and demultiplexing (demux) system 124, a video encoding system 126, an encoding system 128, and a communication system 130. Other components, not shown to avoid clutter, may also be included in the receiving-side device.

[0017] In operation, system 105 may receive mixed-media data signals Rx, process video data signals Rx as described herein, generate respective controls for display electronics, and cause images and video to be displayed on the user's device. It will be appreciated that system 105 may perform these operations simultaneously for multiple users, such as while an audio / video conferencing application or a video chat application is in operation.

[0018] In various embodiments, as in Fig.1, the control circuit 122 is implemented as an enhanced computer system that includes a computer-readable storage device or media, memory 152 for storing instructions, algorithms, and / or programs, such as the program 156 and a plurality of preprogrammed thresholds and parameters, the processor 150 for executing the program 156, and the input / output (I / O) interface 158. The computer-readable storage device or media, memory 152, may include, for example, volatile and non-volatile storage ROM (Read-Only Memory), RAM (Random Access Memory), and KAM (Keep-Alive Memory). KAM is persistent or non-volatile memory that can be used to store various operating variables while the processor 150 is powered down.Memory 152 may be implemented using any of several known storage devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combination storage devices capable of storing data, some of which may represent executable instructions used by processor 150 in other aspects of the operation of server 120. In various embodiments, processor 150 is intended to implement system 105. Memory 152 may also be used by processor 150 to cache data, temporarily store results of comparisons and analyses, and the like.Information in memory 152 may be organized in a process during an initialization or installation operation and / or imported from an external source; it may also be programmed via a user I / O interface.

[0019] The input / output (I / O) interface 158 may be operatively coupled to the processor 150 via a bus and enables communication within the circuitry 122 as well as communication outside the circuitry 122. The input / output (I / O) interface 158 may include one or more wired and / or wireless network interfaces and may be implemented using any suitable method and means. In various embodiments, the input / output (I / O) interface 158 includes the hardware and software to support one or more communication protocols for wireless communication between the processor 150 and external sources, such as satellites, processing systems in the cloud, communication towers, and ground stations.In various embodiments, the input / output (I / O) interface 122 supports communication with technicians and / or one or more storage interfaces for direct connection to storage devices.

[0020] During operation of system 105, processor 150 loads and executes one or more algorithms, instructions, and rules embodied as a program 156, and therefore controls the overall operation of system 105. During operation of system 105, processor 150 may receive data from external sources via communication system 130. In various embodiments of system 105, control circuitry 122 may perform operations according to an algorithm associated with system 105; perform operations according to state machine logic; and perform operations according to logic in a programmable logic array.

[0021] Although the exemplary embodiment of system 105 is described in the context of control circuitry 122 implemented as a fully functional enhanced computer system, those skilled in the art will recognize that the mechanisms of the present disclosure are capable of being distributed as computer-executable instructions or a computer program product (e.g., program 156) and predefined parameters. Such a program product may include an arrangement of instructions organized as multiple interdependent program code modules, each configured to achieve a separate process and / or perform a separate algorithmic operation configured to manage the flow of data through system 105. The program code modules may each include an ordered collection of executable instructions or rules for implementing logical functions for the processes performed by system 105.The instructions in the program code modules, when executed by a processor (e.g., processor 150), cause the processor to receive and process signals and perform logic, calculations, methods, and / or algorithms as described herein. Such a program product may take many forms, and the present disclosure applies equally regardless of the type of computer-readable signal-bearing media used to carry out the distribution.

[0022] As mentioned, the first user device 102 is designated as a reference device or receiving-side device to distinguish it from multiple devices of other users participating in a mixed-media application or conference call. It will be appreciated that, in operation, the techniques and methods described for the receiving-side user may be employed for any user participating in the conference call or mixed-media application.

[0023] With further reference to Fig. 1 will now Fig. 2-6. Exemplary application process modules of the system 105 are discussed in conjunction with Fig. 2 described. Fig. 3A and Fig. 3B are visual aids for discussing the algorithms that the system 105 implements, and Fig. 4 illustrates a use case. Fig.5 provides an exemplary method 500 for operating the system 105, which in conjunction with Fig. 1 and Fig. 2 is described, and Fig. 6 provides an exemplary method for managing display electronics during audio / video conferences that is based on software.

[0024] Fig.Figure 2 provides a non-limiting example of an organization of application process modules in system 105. In an application, each application module may be implemented as one or more sub-modules, and the modules and sub-modules may be distributed on and among various server and / or device systems and components. In the example, N user devices are in operative communication with the receiver application module 202, providing NRx data input 203. The N user devices are also referred to as external devices. The NRx are mixed-media data signals. In various applications, the individual Rx are audio / video conferencing data or video chat data.

[0025] A depacketization, decoding, and demultiplexing module 204 (shortened here to demux) receives the NRx data input 203, which is a combined media stream (i.e., combined mixed-media data signals) comprising video signals, audio signals, and images or profile pictures for N users (the N users do not include duplicate users) via their respective user devices. The demux module 204 sorts the NRx signal into its N data stream components Rxn. A composition module 206 synchronizes the data and generates a video Rxn of the final composition for a single user (external user).

[0026] A display management module 208 in the receiver application module operates on the video data for the individual external user. The display management module may operate according to method 500. For illustrative purposes, the following description of method 500 may refer to the above in connection with Fig. 1, Fig. 2, Fig. 3A, Fig. 3B and Fig. 4. In various embodiments, portions of the method 500 may be performed by various components of the described system 105. Note that the method 500 may include any number of additional or alternative operations and tasks, wherein the Fig. 5 need not be performed in the order shown, and method 500 may be incorporated into a more comprehensive procedure or method, such as a video conference call application with additional functionality not described in detail here. Furthermore, one or more of the tasks shown in Fig. 5 may be omitted in one embodiment of the method 500 if the intended overall functionality remains intact.

[0027] At 502, the received multimedia data NRx is decoded, unpacketized, and demultiplexed to Rxn, as described above.

[0028] At 504, person segmentation or person detection is performed, for example, by module 210. In various aspects of the disclosure, person segmentation or person detection is performed as known in the industry, for example, using polygons or boxes. As an example, Fig.3A illustrates a simplified video image 300 showing the first user 302 in a video conferencing application and a small video inset 304 depicting another (external) user in the video conferencing application. As used herein, the video image 300 may be a full video image or may include incremental video data sufficient to perform a bounding box operation (at 506). The simplified background 306 represents vertical panels and a gray tabletop. The display management module converts the video image 300 using bounding boxes (at 506) by labeling the person-bounded region(s) as Region 1 (R1) and the non-person regions as Region 2 (R2).

[0029] The bounding box separates the person, or R1, from the background, or R2, and this bounding box can have any geometric shape, such as an n-sided polygon, a curved shape, or using depth information for foreground / background separation. System 105 displays a converted video image in which R1 has pixels displayed at their full original intensity; in other words, system 105 controls the display electronics to remain unchanged for R1. R2 is what is also referred to as the background, with the pixels associated with the background subject to further scrutiny.

[0030] A means for classification or a classification module 214 operates on R2 (at 510) to determine whether the background has meaning (meaning determined = M) or no meaning (no meaning determined = NM). Backgrounds with meaning are displayed by system 105 with pixels at full intensity, as are the pixels of region 1, i.e., their intensities remain unchanged from their original pixel intensity. Pixels associated with backgrounds with meaning may also be displayed by system 105 at full refresh rate, as may the pixels of region 1, which may have a higher refresh rate than pixels associated with backgrounds without meaning.

[0031] In various aspects of the disclosure, the classification module 214 may utilize text recognition algorithms and / or object recognition algorithms. In one embodiment, the classification module 214 or the classification means may further include a set of rules encoded in the program 156. In another embodiment, the classification module 214 or the classification means may further include a lookup table. See, for example, Table 1 below. Table 1, example lookup table Detected With meaning Without meaning Background blur filter is turned on X Structureless wall X Structureless ceiling X Text on an object X Books X Dynamic background X

[0032] In some embodiments, machine learning or an artificial intelligence (AI) module may be used in the classification module 214 or means for classification at 510. For example, a test data set may be generated to train an AI module to classify the content in R2. Using an AI module in this way may improve the speed and performance of the system 105.

[0033] After identifying a region 2 or background of no importance (NM), the embodiments then apply a dimming algorithm at 512 (indicated by the dimming module 216). Referring to Fig.Figure 3B shows the simplified video image 330 in which the first user has been segmented and sorted as Region 1 (R1) and the background has been sorted as Region 2 (R2). The video inset 304 is another Region 1 in the converted source video image / display, which is intended to remain unchanged (i.e., it will be displayed at its full original pixel intensity). It can be seen that the complete video image 330 is indicated by a rectangle.

[0034] The operations of the dimming module 216 can also be viewed as a means for managing display electronics. In one embodiment, the dimming module 216 or means for managing display electronics begins with the bounding box 332 placed around a person's periphery, as described herein, and a dimming factor or dimming amount for an individual pixel is determined based on a distance the pixel is from the bounding box 332. In a simplified manner, it can be viewed as concentric rings radiating outward from the bounding box 332, but applied to an irregular shape, as illustrated. In this aspect, the dimming factor is a function of the distance from the bounding box.

[0035] The pixels are dimmed by a dimming factor, and the dimming factor ranges from a minimum amount to a maximum amount. As a non-limiting example, for each pixel in a motion, line by line and pixel by pixel through the data in the video image 330, a pixel distance from the periphery of Region 1 is determined and a respective dimming factor is determined based thereon. If the pixel is adjacent to Region 1, the dimming factor is a first, minimum amount (e.g., 0.001%). As the pixel distance from Region 1 increases, the dimming factor increases until the dimming factor at the edge 336 of the video image is a second, maximum amount (e.g., 20% or 50%, etc.). In one embodiment, the pixels are dimmed non-linearly between the bounding box of Region 1 and the edge 336 or periphery of the video image.As a non-limiting example, the minimum amount is in a range of 0.001% to 0.01% and the maximum amount is in a range of 20% to 50%.

[0036] In another non-limiting example, the dimming factor further has a rate of change adjacent to the bounding box (i.e., near the person) that is at least 20% higher than the rate of change at the edge or periphery of the video image. For example, the system may implement the dimming factor by changing it more frequently closer to the bounding lines closer to the bounding box than to bounding lines near the periphery of the video image. The dimming factor is applied by system 105 to the original intensity pixel, resulting in a dimmed pixel, and system 105 controls the display electronics (at 514) to display a converted video image in the audio / video application that includes the collective dimmed pixel data combined with the original intensity pixel data from R1.An advantage of this methodology is that it produces a smooth visual transition adjacent to the person or R1, and the farther from R1 the pixels are (i.e., the more in the area of ​​no interest R2), the more quickly the system 105 transitions from a high initial intensity pixel display to a low (dimmed) pixel intensity, promoting a lowest power configuration.

[0037] Dimmed data of no significance (DNM) may be combined with undimmed pixel data of original intensity (i.e., R1 and M-R2 data), e.g., by a combining module 218, and the output of the combining module may be used to drive graphics processing (e.g., a graphics processing module 220) and / or may be applied directly (at 516) to the display electronics in a display module 222 to display a resulting converted video image. In various aspects of the disclosure, if a previously provided and set blur flag (BF) is reset, the system 105 may return to classifying the background with the classification module 214 or the means for classifying.Similarly, if previously provided metadata including the identification of Region 1 (the person), the bounding box, and the identification of Region 2 (the background) is subsequently withdrawn, the system 105 may responsively return to performing person segmentation and person detection (module 210) and thereby generate the bounding box for the video image as described herein.

[0038] System 105 may periodically perform a video image check to determine if an R2 previously determined to be NM (without significance) has now become significant (M). In some non-limiting examples, this periodicity may be every 10 frames, every 1-2 seconds, or any frequency suitable to meet power consumption requirements. When it is determined that the background has gained significance (or upon resetting a blur flag, as described above), system 105 may stop applying the dimming factors and drive the display electronics at the respective full original pixel intensities in the video image. In various embodiments, system 105 also determines whether the user is operating the audio / video conferencing application in a foreground mode of the user's device, and if not, system 105 may stop driving the display electronics.

[0039] As mentioned above, in some embodiments, the dimming module changes the rate of change of the dimming factor as it is applied to individual pixels in the R2 region. For example, the rate of change of the dimming factor may be the lowest closest to the bounding box 332, and successively distant boundary lines from region 1 may have a dimming factor that has an increasingly faster rate of change.This functionality can be referred to as a frequency of determining the dimming factor and is indicated in the figure as follows: Note that a first distance between a boundary line 338 and the bounding box 332 is smaller than a second distance between the boundary line 340 and the boundary line 338, and the second distance is in turn smaller than a third distance between the boundary line 340 and the boundary line 342; this corresponds to a small rate of change at the first distance, a larger rate of change at the second distance, and an even larger rate of change at the third distance.An advantage of this methodology is that it creates a smooth visual transition adjacent to the person or R1, and the farther from R1 the pixels are, the faster they descend from a high initial intensity pixel display to a low (dimmed) pixel intensity in the remaining parts of R2, promoting lowest power configuration and operation of the system 105.

[0040] In an optional configuration, the modules in the dashed box can be Fig.2 may be performed prior to data arrival at the receiving application, such as a cloud server or application server. In some embodiments, the receiving application module 202 may receive optional metadata inputs. The metadata inputs may include a blur flag (BF) notifying the receiving application module 202 that the background has been blurred by the user, filtered by the user, or the like. The metadata may also include the bounding boxes with R1 and R2 already defined. In an embodiment where the metadata provides the bounding box, thereby distinguishing the person from the background, the system 105 may proceed to the classification module 214. In an embodiment that provides the blur flag (BF), the system may proceed to the dimming algorithm represented by the dimming module 216.In such cases, this information may be immediately processed by the display management module 208, and operations previously associated with module 210 and module 212 may be omitted from operations performed by the system 105.

[0041] It should be noted that even if the external user or broadcaster has enabled background blur or background filtering, embodiments of the system 105 can still reduce power consumption with the dimming operations of the dimming module 216. Fig. 4 illustrates this. In video image 400, it can be seen that the background has been blurred. Video image 430 shows the output of dimming module 216; in particular, it can be seen that region 403 in video image 400 has more white and clearer shadows than the corresponding region 433 in video image 430.

[0042] Much of this discussion has been directed to a combined hardware or hardware / software application, such as may be implemented on a user device in a timing controller (see Fig. 7, TCON 700, described in more detail below), but at least some of the features described here may also be implemented in a software program. With reference to Fig. 6, another method 600 for managing display electronics during audio / video conferences is shown.

[0043] At 602, background service software may capture a displayed frame Rxn using a preset frame rate. At 604, person detection or person segmentation may be performed as described above. At 606, the software may generate a mask representing pixels of the person to distinguish regions R1 and R2. This mask may act as an invisible top-level window, sending (at 608) R1 for full-intensity display and, at 610, sending R2 background image data to a dimming algorithm. At 612, an alpha blending routine may apply a dimming factor to the background in the videoconferencing application window. The dashed line indicates a hardware or display electronics receiver for the software-generated dimmed R2.

[0044] Fig.7 shows a block diagram of an exemplary timing controller 700 including a classifier and dimming module 730. The timing controller 700 includes a video data receiver 710, a frame buffer 720, the classifier and dimming module 730, and a display driver 740. The timing controller 700 receives video data from a display module 750 located in a base of a mobile computing device and drives a display screen 760. The timing controller 700 and the display screen 760 may be located in a lid of a mobile client computing device (e.g., user device 102). The display driver 740 drives display screen control circuitry, such as row drivers 780 and column drivers 790.

[0045] The classification and dimming module 730 enables reduction of the power consumed by the display screen 760 by globally dimming the images to be displayed, as described above. In some embodiments, one or more image processing modules 770 are located before and / or after the classification and dimming module 730 in the stack 795. The image processing modules 770 can perform various image processing operations available on the client device.

[0046] Thus, systems and methods for managing display electronics during audio / video conferencing have been provided. Embodiments advantageously reduce power consumption during conference calls and other mixed-media applications. The following additional figures and description are intended to illustrate various contexts for using and applying the present disclosure.

[0047] Disclosed embodiments may be implemented in a compute node. In the Fig.8, a compute node 800 includes a compute engine (also referred to herein as "computational circuitry") 802, an input / output (I / O) subsystem 808, a data storage device 810, a communication circuitry subsystem 812, and optionally one or more peripheral devices 814. With respect to the present example, the compute node 800 or the compute circuitry 802 may perform the operations and tasks associated with the system 105. In other examples, respective compute nodes 800 may include other or additional components, such as those commonly found in a computer (e.g., a display, peripheral devices, etc.). Additionally, in some examples, one or more of the example components may be integrated with or otherwise form a part of another component.In some examples, compute node 800 may be implemented as a single device, such as an integrated circuit, an embedded system, a field programmable gate array (FPGA), a system on a chip (SOC), or other integrated system or device. In the illustrative example, compute node 800 includes or is implemented as a processor 804 and a memory 806. Processor 804 may be implemented as any type of processor capable of performing the functions described herein (e.g., executing compile functions and executing an application). For example, processor 804 may be implemented as one or more multi-core processors, a microcontroller, a processing unit, a specialized or special-purpose processing unit, or other processor or other processing / control circuitry.

[0048] In some examples, processor 804 may be implemented as, include, or be coupled to an FPGA, ASIC (Application Specific Integrated Circuit), reconfigurable hardware or hardware circuitry, or other specialized hardware to enable performance of the functions described herein. In some examples, processor 804 may also be implemented as a specialized x-processing unit (xPU), also known as a data processing unit (DPU), infrastructure processing unit (IPU), or network processing unit (NPU). Such an xPU may be embodied as a standalone circuit or circuit package, integrated within a SOC, or integrated with networking circuitry (e.g., in a SmartNIC or enhanced SmartNIC), acceleration circuitry, storage devices, or AI hardware (e.g., GPUs or programmed FPGAs).Such an xPU may be configured to receive programming to process one or more data streams external to the CPU or general-purpose processing hardware and to perform specific tasks and actions on the data streams (such as hosting microservices, performing service management or orchestration, organizing or managing server or data center hardware, managing service meshes, or collecting and distributing telemetry). However, it should be understood that an xPU, a SOC, a CPU, and other variations of the processor 804 may work in coordination with one another to perform many types of operations and instructions within and on behalf of the compute node 800. The memory 806 may be implemented as any type of volatile memory (e.g., dynamic random access memory (DRAM), etc.) or non-volatile memory or data storage capable of performing the functions described herein.Volatile memory can be a storage medium that requires power to maintain the state of the data stored on the medium. Examples of volatile memory include various types of RAM (random access memory), such as DRAM or SRAM (static random access memory). One specific type of DRAM that can be used in a memory module is synchronous dynamic random access memory (SDRAM).

[0049] In one example, the memory device is a block-addressable memory device, such as those based on NAND or NOR technologies. A memory device may also include a three-dimensional crosspoint memory device (e.g., Intel® 3D XPoint™ memory) or other byte-addressable, write-in-place non-volatile memory device. The memory device may refer to the die itself and / or to an encapsulated memory product. In some examples, the 3D crosspoint memory (e.g., Intel® 3D XPoint™ memory) may include a transistorless stackable crosspoint architecture where memory cells are located at the intersection of wordlines and bitlines and are individually addressable, and where bit storage is based on a change in volume resistance. In some examples, memory 806 may be fully or partially integrated with processor 804.The memory 806 may store various software and data used during operation, such as one or more applications, data manipulated by the application(s), libraries, and drivers.

[0050] Computing circuitry 802 is communicatively coupled to other components of compute node 800 via I / O subsystem 808, which may be implemented as circuitry and / or components to enable input / output operations with computing circuitry 802 (e.g., with processor 804 and / or main memory 806) and other components of computing circuitry 802. I / O subsystem 808 may, for example, be implemented as or otherwise include memory control hubs, input / output control hubs, integrated sensor hubs, firmware devices, communication links (e.g., point-to-point links, bus connections, wires, cables, light guides, traces on printed circuit boards, etc.), and / or other components and subsystems to enable the input / output operations.In some examples, the I / O subsystem 808 may form part of a system-on-a-chip (SoC) and may be integrated into the computing circuitry 802 along with the processor 804 and / or the memory 806 and / or other components of the computing circuitry 802.

[0051] The one or more illustrative data storage devices 810 may be implemented as any type of device configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. Individual data storage devices 810 may include a system partition that stores data and firmware code for the data storage device 810. Individual data storage devices 810 may also include one or more operating system partitions that store data files and executable files for operating systems, depending on, for example, the type of compute node 800.

[0052] The communication circuit 812 may be embodied as any communication circuit, device, or collection thereof capable of enabling communication over a network between the computing circuitry 802 and another computing device (e.g., an edge gateway of an implementing edge computing system).

[0053] The communication chip 812 may implement any of a number of wireless standards or protocols, including, but not limited to, Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.11 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long Term Evolution (LTE) Project along with all modifications, updates, and / or revisions (e.g., Advanced LTE Project, Ultra Mobile Broadband (UMB) Project (also referred to as "3GPP2"), etc.). IEEE 802.16 compliant Broadband Wireless Access (BWA) networks are commonly referred to as WiMAX networks, an acronym that stands for Worldwide Interoperability for Microwave Access, which is a certification mark for products that pass conformance and interoperability testing for the IEEE 802.16 standards.The communication device 812 may operate according to a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication device 812 may operate according to Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication chip 812 may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols designated as 3G, 4G, 5G, and beyond. The communication subsystem 812 may operate according to other wireless protocols in other embodiments.The communication subsystem 812 may include an antenna 1822 to facilitate wireless communications and / or to receive other wireless transmissions (such as AM or FM radio transmissions).

[0054] In some embodiments, the communication subsystem 812 may manage wired communication, such as electrical, optical, or any other suitable communication protocols (e.g., IEEE 802.3 Ethernet standards). As mentioned above, the communication subsystem 812 may include multiple communication components. For example, a first communication subsystem 812 may be dedicated to shorter-range wireless communications, such as Wi-Fi or Bluetooth, and a second communication system 812 may be dedicated to longer-range wireless communications, such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, a first communication system 812 may be dedicated to wireless communication, and a second communication system 812 may be dedicated to wired communication.

[0055] The example communications subsystem 812 includes a network interface controller (NIC) 820, which may also be referred to as a host fabric interface (HFI). The NIC 820 may be embodied as one or more daughter boards, subordinate cards, network interface cards, control chips, chipsets, or other devices that may be used by the compute node 800 to interface with another computing device (e.g., an edge gateway node). In some examples, the NIC 820 may be embodied as part of a system on a chip (SoC) that includes one or more processors or may be included in a multi-chip package that also includes one or more processors. In some examples, the NIC 820 may include a local processor (not shown) and / or local memory (not shown), both of which are local to the NIC 820.In such examples, the local processor of the NIC 820 may be capable of performing one or more of the functions of the computing circuitry 802 described herein. Additionally or alternatively, in such examples, the local memory of the NIC 820 may be integrated into one or more components of the client compute node at the board level, socket level, chip level, and / or other levels.

[0056] Additionally, in some examples, a respective compute node 800 may include one or more peripheral devices 814. Such peripheral devices 814 may include any type of peripheral device found in a computing device or server, such as audio input devices, a display, other input / output devices, interface devices, and / or other peripheral devices, depending on the particular type of compute node 800. In further examples, compute node 800 may be embodied by a respective edge compute node (whether a client, gateway, or aggregation node) in an edge computing system or similar forms of devices, computers, subsystems, circuitry, or other components.

[0057] In other examples, compute node 800 may be embodied as any type of device or collection of devices capable of performing various computing functions. Each compute node 800 may be implemented as a type of device, appliance, computer, or other "thing" capable of communicating with other edge, network, or endpoint components. For example, a compute node may be embodied as a personal computer, server, smartphone, mobile computing device, smart device, in-vehicle computing system (e.g., a navigation system), a standalone device with an outer casing, shell, etc., or any other device or system capable of performing the described functions.

[0058] Fig.9 illustrates a multiprocessor environment in which embodiments may be implemented. Processor units 902 and 904 further include caches 912 and 914, respectively. Caches 912 and 914 may store data (e.g., instructions) used by one or more components of processor units 902 and 904, such as processor cores 908 and 910. Caches 912 and 914 may be part of a memory hierarchy for computing system 900. For example, caches 912 may locally store data that is also stored in memory 916 to allow faster access to the data by processor unit 902. In some embodiments, cache memories 912 and 914 may include multiple cache levels, such as Level 1 (L1), Level 2 (L2), Level 3 (L3), Level 4 (L4), and / or other caches or cache levels.In some embodiments, one or more levels of cache memory (e.g., L2, L3, L4) may be shared by multiple cores in a processor unit or multiple processor units in an IC component. In some embodiments, the last level of cache memory on an IC component may be referred to as a last-level cache (LLC). One or more of the higher levels of cache memory (the smaller and faster caches) in the memory hierarchy may be located on the same integrated circuit die as a processor core, and one or more of the lower levels of cache memory (the larger and slower caches) may be located on integrated circuit dies that are physically separate from the integrated circuit dies of the processor core.

[0059] Although data processing system 900 is shown with two processor units, data processing system 900 may include any number of processor units. Further, a processor unit may include any number of processor cores. A processor unit may take various forms, such as a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), an accelerated processing unit (APU), a field-programmable gate array (FPGA), a neural network processing unit (NPU), a data processing unit (DPU), an accelerator (e.g., a graphics accelerator, a digital signal processor (DSP), a compression accelerator, an artificial intelligence (AI) accelerator), a control unit, or other types of processing units. Thus, the processor unit may be referred to as an XPU (or xPU).Furthermore, a processor unit may include one or more of these different types of processing units. In some embodiments, the computing system includes a processor unit with multiple cores, and in other embodiments, the computing system includes a single processor unit with a single core. As used herein, the terms "processor unit" and "processing unit" may refer to any processor, processor core, component, module, engine, circuit, or any other processing element described or mentioned herein.

[0060] In some embodiments, data processing system 900 may include one or more processing units that are heterogeneous or asymmetric with another processing unit in the data processing system. A variety of differences may exist between the processing units in a system with respect to a spectrum of performance metrics, including architectural, microarchitectural, thermal, power consumption characteristics, and the like. These differences may effectively manifest as asymmetry and heterogeneity between the processing units in a system.

[0061] The processor units 902 and 904 may be located in a single integrated circuit component (such as a multi-chip package (MCP) or a multi-chip module (MCM)), or they may be located in separate integrated circuit components. An integrated circuit component that includes one or more processor units may include additional components, such as embedded DRAM, high-bandwidth stacked memories (HBMs), shared cache memories (e.g., L3, L4, LLC), input / output controllers (I / O controllers), or memory controllers. Each of the additional components may be located on the same integrated circuit die as a processor unit or on one or more integrated circuit dies that are separate from the integrated circuit dies that include the processor units.In some embodiments, these separate integrated circuit dies may be referred to as "chiplets." In some embodiments where heterogeneity or asymmetry exists between the processing units in a computing system, the heterogeneity or asymmetry may exist between processing units located in the same integrated circuit component. In embodiments where an integrated circuit component includes multiple IC dies, interconnections between the dies may be provided by the package substrate, by one or more silicon interposers, by one or more silicon bridges embedded in the package substrate (such as Intel® Embedded Multi-Die Interconnect Bridges (EMIBs)), or combinations thereof.

[0062] The processor units 902 and 904 further include memory controller logic (MC) 920 and 922. As in Fig. 9, MCs 920 and 922 control memories 916 and 918, which are coupled to processor units 902 and 904, respectively. Memories 916 and 918 may include various types of volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) and / or non-volatile memory (e.g., flash memory, non-volatile chalcogenide-based phase-change memory), and may comprise one or more layers of the computing system's memory hierarchy. Although MCs 920 and 922 are illustrated as being integrated with processor units 902 and 904, in alternative embodiments, the MCs may be external to a processor unit.

[0063] Processor units 902 and 904 are coupled to an input / output (I / O) subsystem 930 via point-to-point interconnects 932 and 934. Point-to-point interconnect 932 connects a point-to-point interface 936 of processor unit 902 to a point-to-point interface 938 of I / O subsystem 930, and point-to-point interconnect 934 connects a point-to-point interface 940 of processor unit 904 to a point-to-point interface 942 of I / O subsystem 930. Input / output subsystem 930 further includes an interface 950 for coupling I / O subsystem 930 to a graphics engine 952. The I / O subsystem 930 and the graphics engine 952 are coupled via a bus 954.

[0064] The input / output subsystem 930 is further coupled to a first bus 960 via an interface 962. The first bus 960 may be a PCIe (Peripheral Component Interconnect Express) bus or any other type of bus. Various I / O devices 964 may be coupled to the first bus 960. A bus bridge 970 may couple the first bus 960 to a second bus 980. In some embodiments, the second bus 980 may be a LPC (Low Pin Count) bus. Various devices, including, for example, a keyboard / mouse 982, audio I / O devices 988, and a storage device 990, such as a hard disk drive, a solid state drive, or other storage device for storing computer-executable instructions (code) 992 or data, may be coupled to the second bus 980.Code 992 may include computer-executable instructions for performing the methods described herein. Additional components that may be coupled to second bus 980 include communication devices 984 that may provide communication between computing system 900 and one or more wired or wireless networks 986 (e.g., Wi-Fi, cellular, or satellite networks) via one or more wired or wireless communication links (e.g., a wire, a cable, an Ethernet connection, a radio frequency (RF) channel, an infrared channel, a Wi-Fi channel) using one or more communication standards (e.g., IEEE 802.11 standard and its amendments).

[0065] In embodiments where the communication devices 984 support wireless communication, the communication devices 984 may include wireless communication components coupled to one or more antennas to support communication between the computing system 900 and external devices. The wireless communication components may support various wireless communication protocols and technologies, such as near-field communication (NFC), IEEE 802.11 variants (Wi-Fi variants), WiMax, Bluetooth, Zigbee, 4G LTE (4G Long Term Evolution), CDMA (Code Division Multiplexing Access), UMTS (Universal Mobile Telecommunication System), and GSM (Global System for Mobile Telecommunication), as well as 5G broadband cellular technologies.In addition, the wireless modems can support communication with one or more cellular networks for data and voice communications within a single cellular network, between cellular networks, or between the computing system and a public switched telephone network (PSTN).

[0066] System 900 may include removable memory, such as flash memory cards (e.g., Secure Digital (SD) cards), memory sticks, and Subscriber Identity Module (SIM) cards. The memory of system 900 (including cache memories 912 and 914, memory 916 and 918, and storage device 990) may store data and / or computer-executable instructions for running an operating system 994 and application programs 996. Example data would include web pages, text messages, images, audio files, and video data, biometric thresholds for particular users, or other data sets to be sent and / or received by system 900 over one or more wired or wireless networks 986 to and / or from one or more network servers or other devices, or for use by system 900.The system 900 may also have access to external memory or external data storage (not shown), such as external hard drives or cloud-based data storage.

[0067] The operating system 994 (here also simply called “OS”) can control the allocation and use of the Fig. 6 and support the one or more application programs 996. The application programs 996 may include common computing system applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications) as well as other computing applications.

[0068] In some embodiments, a hypervisor (or a virtual machine manager) runs on top of the operating system 994, and the application programs 996 run within one or more virtual machines running on the hypervisor. Hypervisor 124, in these embodiments, is a Type 2 or hosted hypervisor running on top of the operating system 994. In other hypervisor-based embodiments, the hypervisor is a Type 1 or "bare metal" hypervisor running directly on the platform resources of the computing system 994 without an intermediate operating system layer.

[0069] In some embodiments, the application programs 996 may operate within one or more containers. A container is a running instance of a container image, which is a package of binary images for one or more of the application programs 996 and any libraries, configuration settings, and any other information that one or more application programs 996 require to execute. A container image may conform to any container image format, such as Docker®, APP, or LXC container image formats. In container-based embodiments, a container runtime engine, such as Docker Engine, LXU, or an Open Container Initiative (OCI)-compliant container runtime (e.g., Railcar, CRI-O), runs on top of the operating system (or virtual machine monitor) to provide an interface between the containers and the operating system 994.An orchestrator may be responsible for managing the computing system 900 and various container-related tasks, such as deploying container images to the computing system 994, monitoring the performance of deployed containers, and monitoring the utilization of the computing system 994's resources.

[0070] The data processing system 900 may support various additional input devices, generally represented as user interfaces 998, such as a touchscreen, a microphone, a monoscopic camera, a stereoscopic camera, a trackball, a touchpad, a trackpad, a proximity sensor, a light sensor, an electrocardiogram (ECG) sensor, a PPG (photoplethysmogram) sensor, a galvanic skin response sensor, and one or more output devices, such as one or more speakers or displays. Other possible input and output devices include piezoelectric and other haptic I / O devices. Each of the input or output devices may be located inside or outside the system or removably attached to the system 900. External input and output devices may communicate with the system 900 via wired or wireless connections.

[0071] Additionally, one or more of the user interfaces 998 may be natural user interfaces (NUIs). For example, the operating system 994 or the applications 996 may include speech recognition logic as part of a voice user interface that allows a user to operate the system 900 via voice commands. Furthermore, the computing system 900 may include input devices and logic that allow a user to interact with the computing system 900 via body, hand, or facial gestures. For example, a user's hand gestures may be detected and interpreted to provide input for a gaming application.

[0072] The system 2000 may further include at least one input / output port comprising physical connectors (e.g., USB, IEEE 1394 (FireWire), Ethernet, RS-232), a power supply (e.g., a battery), a Global Satellite Navigation System (GNSS) receiver (e.g., a GPS receiver), a gyroscope, an accelerometer, and / or a compass. A GNSS receiver may be coupled to a GNSS antenna. The computing system 900 may further include one or more additional antennas coupled to one or more additional receivers, transmitters, and / or transceivers to enable additional functionality.

[0073] In addition to those already discussed, IC components, IC component components, and other components in the computing system 994 may communicate using interconnect technologies such as Intel® QuickPath Interconnect (QPI), Intel® Ultra Path Interconnect (UPI), Computer Express Link (CXL), Cache Coherent Interconnect for Accelerators (CCIX®), Serializer / Deserializer (SERDES), Nvidia® NVLink, ARM Infinity Link, Gen-Z, or Open Coherent Accelerator Processor Interface (OpenCAPI). Other interconnect technologies may be used, and a computing system 994 may utilize more or fewer interconnect technologies.

[0074] It goes without saying that Fig.9 illustrates only one exemplary data processing system architecture. Data processing systems based on alternative architectures may be used to implement the technologies described herein. For example, instead of the processor units 902 and 904 and the graphics engine 952 being located on discrete integrated circuits, a data processing system may include an integrated circuit in a SoC (System on a Chip) that includes multiple processors, a graphics engine, and additional components. Furthermore, a data processing system may connect its component parts via bus or point-to-point configurations that are different from the ones described in Fig. 9 are different. In addition, the Fig. 9 are not required or all-inclusive, as components shown in alternative embodiments may be removed and other components added.

[0075] Fig. 10 is a block diagram of an exemplary processor unit 1100 for executing computer-executable instructions as part of implementing technologies described herein. Processor unit 1100 may be a single-threaded core or a multi-threaded core in that it may include more than one hardware thread context (or "logical processor") per processor unit.

[0076] Fig. 10 also shows a memory 1010 coupled to the processor unit 1000. The memory 1010 may be any memory described herein or any other memory known to those skilled in the art. The memory 1010 may store computer-executable instructions 1015 (code) executable by the processor unit 1000.

[0077] The processor unit includes front-end logic 1020 that receives instructions from memory 1010. An instruction may be processed by one or more decoders 1030. The decoder 1030 may generate as its output a micro-operation, such as a fixed-width micro-operation in a predefined format, or generate other instructions, micro-instructions, or control signals that reflect the original code instruction. The illustrated front-end logic 1020 also includes register renaming logic 1035 and scheduling logic 1040, which generally allocates resources and queues operations corresponding to converting an instruction for execution.

[0078] Processor unit 1000 further includes execution logic 1050, which includes one or more execution units (EUs) 1065-1 through 1065-N. Some embodiments of processor units may include several execution units dedicated to specific functions or sets of functions. Other embodiments may include only one execution unit or one execution unit capable of performing a specific function. Execution logic 1050 executes the operations specified by code instructions. Upon completion of the execution of the operations specified by the code instructions, back-end logic 1070 retires the instructions using retirement logic 1075. In some embodiments, processor unit 1000 allows out-of-order execution but requires in-order retirement of instructions. Retirement logic 1075 may take a variety of forms, as known to those skilled in the art (e.g.,B. reordering buffers or the like).

[0079] Processor unit 1000 is transformed during instruction execution with respect to at least the output generated by decoder 1030, hardware registers and tables used by register renaming logic 1035, and any registers (not shown) modified by execution logic 1050.

[0080] Although at least one embodiment has been presented in the foregoing detailed description, it should be understood that a vast number of variations exist. It should also be understood that the disclosed embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of the disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a practical roadmap for implementing the disclosed aspects of the present disclosure. Various changes may be made in the function and arrangement of elements without departing from the scope of the disclosure as defined in the appended claims and their legal equivalents.

[0081] As used herein, a "computer," "data processing system," or "computing device" refers to any of a variety of data processing devices and includes systems that include multiple discrete physical components capable of executing instructions. In some embodiments, the computing systems are located in a data center, such as an enterprise data center (e.g., a data center owned and operated by a company and typically located on the company's premises), a managed service data center (e.g., a data center managed by a third party on behalf of a company), a shared data center (e.g., a data center where the data center infrastructure is provided by the data center owner and a company provides its own data center components (servers, etc.).)), a cloud data center (e.g., a data center operated by a cloud service provider that houses an organization's applications and data), and an edge data center (e.g., a data center that typically has a smaller footprint than other data center types and is located close to the geographic area it serves).

[0082] Thus, the term "computer-executable instruction" refers to instructions that can be executed by any data processing system, apparatus, or machine described or mentioned herein, as well as any other data processing system, apparatus, or machine capable of executing instructions. Any of the disclosed methods (or a portion thereof) may be implemented as computer-executable instructions or a computer program product. Such instructions may cause a computing system or one or more processor units capable of executing computer-executable instructions to perform any of the disclosed methods.

[0083] The computer-executable instructions or computer program products, as well as any data generated and / or used during an implementation of the disclosed technologies, may be stored on one or more tangible or non-transient computer-readable storage media, such as volatile memory (e.g., DRAM, SRAM), non-volatile memory (e.g., flash memory, non-volatile chalcogenide-based phase-change optical memory), optical media discs (e.g., DVDs, CDs), and magnetic storage (e.g., magnetic tape storage, hard disk drives). Computer-readable storage media may be embodied in computer-readable storage devices, such as solid-state drives, USB flash drives, and memory modules. Alternatively, any of the methods disclosed herein (or a portion thereof) may be performed by hardware components comprising non-programmable circuitry.In some embodiments, each of the present methods may be performed by a combination of non-programmable hardware components and one or more processing units that execute computer-executable instructions stored on computer-readable storage media.

[0084] The computer-executable instructions may, for example, be part of an operating system of the computing system, an application stored locally on the computing system, or a remote application accessible to the computing system (e.g., via a web browser). Any of the methods described herein may be performed by computer-executable instructions executed by a single computing system or by one or more networked computing systems operating in a network environment. Computer-executable instructions and updates to the computer-executable instructions may be downloaded to a computing system from a remote server.

[0085] Furthermore, it should be understood that implementation of the disclosed technologies is not limited to any specific computer language or computer program. For example, the disclosed technologies may be implemented by software written in C++, C#, Java, Perl, Python, JavaScript, Adobe Flash, C#, assembly language, or any other programming language. Likewise, the disclosed technologies are not limited to any particular computer system or type of hardware.

[0086] Furthermore, any of the software-based embodiments (including, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or remotely accessed through any suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, cable (including fiber optic cable), magnetic communication, electromagnetic communication (including RF, microwave, ultrasonic, and infrared communications), electronic communication, or other such communication means.

[0087] Furthermore, any operating theories, scientific principles, or other theoretical descriptions presented herein with reference to the devices or methods of this disclosure are provided for convenience of understanding and are not intended to limit the scope of protection. The devices and methods in the appended claims are not limited to those devices and methods that operate in the manner described by such operating theories.

[0088] As used herein, terms such as "embodiments," "one aspect of the present disclosure," "various aspects of the present disclosure," "some aspects of the present disclosure," and the like indicate that some aspects of the present disclosure include some, all, or none of the features described for other aspects of the present disclosure. "First," "second," "third," and the like describe a common object and indicate various instances of similar objects being referred to; unless explicitly stated, they do not imply a particular ordering, whether temporal, spatial, ranking, or otherwise.According to patent application jargon, "connected" indicates elements that are in direct physical or electrical contact with each other, and "coupled" indicates elements that cooperate or interact with each other, where coupled elements may or may not be in direct physical or electrical contact. Furthermore, the terms "comprising," "including," "with," and the like are used interchangeably to denote non-exclusive inclusions.

[0089] As used in this application and in the claims, a list of items joined by the phrase "at least one of" can mean any combination of the listed terms. For example, the phrase "at least one of A, B, or C" can mean A; B; C; A and B; A and C; B and C; or A, B, and C. As used herein, a list of items joined by the phrase "one or more of" can mean any combination of the listed terms. For example, the phrase "one or more of A, B, and / or C" can mean A; B; C; A and B; A and C; B and C; or A, B, and C.

[0090] The following examples relate to additional aspects of the present disclosure of technologies disclosed herein. Examples

[0091] Example 1 is a system, a system comprising: a display component on a user device; and a control circuit coupled to the display component, the control circuit configured to receive a mixed-media data signal from an external device; generate a video image from the mixed-media data signal; delimit a person in the video image from a background with a bounding box; determine that the background is not significant; wherein the video image includes a respective original pixel intensity for a plurality of pixels;Identifying pixels of the plurality of pixels associated with the background as background pixels; determining a respective dimming factor for individual background pixels that reduces the respective original pixel intensity, wherein the dimming factor is a function of the distance from the bounding box and ranges from a minimum amount adjacent to the bounding box to a maximum amount at an edge of the video image; generating a dimmed background by applying the respective dimming factor to the individual background pixels; and causing the display component to display a converted video image with the dimmed background.

[0092] Example 2 includes the subject matter of Example 1, with the minimum amount being in a range of 0.001% to 0.01% and the maximum amount being in a range of 20% to 50%.

[0093] Example 3 includes the subject matter of Example 1 or Example 2, wherein the dimming factor further has a rate of change that is at least 20% higher near a periphery of the video image than near the bounding box.

[0094] Example 4 includes the subject matter of any of Examples 1-3, wherein the control circuit is further configured to perform object detection on the background; and determine that the background is not significant by classifying an object detection output or comparing an object detection output with contents in a lookup table.

[0095] Example 5 includes the subject matter of any of Examples 1-3, further comprising: an AI (artificial intelligence) model trained with training data defining objects or text with meaning; wherein the control circuit is further configured to provide the background to the AI ​​model; and determine that the background has no meaning based on an output of the AI ​​model.

[0096] Example 6 includes the subject matter of any of Examples 1-5, wherein the control circuit is further configured to determine that the background has significance; stop applying the respective dimming factor to the individual background pixels; and cause the display component to display the video image at the respective original pixel intensity for the plurality of pixels.

[0097] Example 7 includes the subject matter of any of Examples 1-6, wherein the display component comprises an OLED (organic light emitting diode) display screen.

[0098] Example 8 includes the subject matter of any of Examples 1-6, wherein the display component comprises an LED display screen (light-emitting diodes).

[0099] Example 9 includes the subject matter of any of Examples 1-8, wherein the mixed media data signal is associated with a video conferencing application.

[0100] Example 10 includes the subject matter of any of Examples 1-9, wherein the control circuit is further configured to perform a person detection and segmentation operation; and delimit the person in the video image from the background with the bounding box in response to performing the person detection and segmentation operation.

[0101] Example 11 includes the subject matter of any of Examples 1-10, wherein the control circuit is further configured to detect when an audio or video conferencing application is not running in a foreground mode; and cause the display component to stop displaying when the audio or video conferencing application is not running in the foreground mode.

[0102] Example 12 is a non-transitory computer-readable medium comprising instructions that, when executed by processing circuitry, are configured to receive a video image from a video conferencing application on an external device; receive metadata associated with the video image, wherein the metadata delimits a person from a background in the video image using a bounding box, and wherein the bounding box can be an n-sided polygon, a curved shape generated by depth information, or a combination thereof; and wherein the metadata associated with the video image further provides a background blur flag that is set when the background is not significant; wherein the video image includes, for a plurality of pixels, a respective original pixel intensity;Identifying pixels of the plurality of pixels associated with the background as background pixels; determining a respective dimming factor for individual background pixels that reduces the respective original pixel intensity, and wherein the dimming factor is a function of the distance from the bounding box and ranges from a minimum amount adjacent to the bounding box to a maximum amount at an edge of the video image; generating a dimmed background by applying the respective dimming factor to the individual background pixels; and causing a display component to display a converted video image with the dimmed background.

[0103] Example 13 includes the subject matter of Example 12, wherein the instructions are further to implement the minimum amount in a range of 0.001% to 0.01% and the maximum amount in a range of 20% to 50%.

[0104] Example 14 includes the subject matter of Example 12 or Example 13, wherein the instructions are further to change the dimming factor further closer to the bounding box more frequently than at a periphery of the video image.

[0105] Example 15 includes the subject matter of any of Examples 12-14, wherein the instructions are further configured to detect when the background blur flag is reset; and cause the display component to display the video image at the respective original pixel intensity for the plurality of pixels.

[0106] Example 16 includes the subject matter of any of Examples 12-14, wherein the instructions are further configured to detect when the background blur flag is reset; and identify pixels of the plurality of pixels associated with the background as background pixels; determine a respective dimming factor for individual background pixels that reduces the respective original pixel intensity, and wherein the dimming factor is a function of distance from the bounding box and ranges from a minimum amount adjacent to the bounding box to a maximum amount at an edge of the video image; generate a dimmed background by applying the respective dimming factor to the individual background pixels; and cause the display component to display a converted video image with the dimmed background.

[0107] Example 17 includes the subject matter of any of Examples 12-16, wherein the instructions are further configured to detect when an audio or video conferencing application is not running in a foreground mode; and cause the display component to stop displaying when the audio or video conferencing application is not running in the foreground mode.

[0108] Example 18 is a system comprising: a display component comprising display electronics; and means for managing the display electronics, wherein the means for managing the display electronics is configured to receive a mixed-media data signal from an external device; generate a video image from the mixed-media data signal; delimit a person in the video image from a background with a bounding box; determine that the background is not significant; wherein the video image comprises a respective original pixel intensity for a plurality of pixels;Identifying pixels of the plurality of pixels associated with the background as background pixels; determining a respective dimming factor for individual background pixels that reduces the respective original pixel intensity, and wherein the dimming factor is a function of the distance from the bounding box and ranges from a minimum amount adjacent to the bounding box to a maximum amount at an edge of the video image; generating a dimmed background by applying the respective dimming factor to the individual background pixels; and causing the display component to display a converted video image with the dimmed background.

[0109] Example 19 includes the subject matter of Example 18, wherein the means for managing the display electronics further varies the dimming factor at a rate of change that is at least 20% higher at a periphery of the video image than adjacent to the bounding box.

[0110] Example 20 includes the subject matter of Example 19, further comprising: means for classifying the background, wherein the means for classifying the background is to determine whether the background has significance; and wherein the means for managing the display electronics is further configured to stop applying the respective dimming factor to the individual background pixels when it is determined that the background has significance.

[0111] Example 21 includes the subject matter of Example 1, wherein the bounding box may be an n-sided polygon, a curved shape generated by depth information, or a combination thereof; and wherein the respective dimming factor varies nonlinearly as a function of distance from the bounding box.

Claims

[1] System comprising: a display component associated with a user device; and a control circuit coupled to the display component, wherein the control circuit is configured to Receiving a mixed media data signal from an external device; generating a video image from the mixed media data signal; Separating a person in the video image from a background with a bounding box; Determine that the background is irrelevant; wherein the video image comprises a respective original pixel intensity for a plurality of pixels; Identifying pixels of the plurality of pixels associated with the background as background pixels, Determining a respective dimming factor for individual background pixels that reduces the respective original pixel intensity, the dimming factor being a function of the distance from the bounding box and ranging from a minimum amount adjacent to the bounding box to a maximum amount at an edge of the video image; Creating a dimmed background by applying the respective dimming factor to the individual background pixels, and Causes the display component to display a converted video image with the background dimmed. [2] The system of claim 1, wherein the bounding box may be an n-sided polygon, a curved shape generated by depth information, or a combination thereof; and wherein the respective dimming factor varies nonlinearly as a function of distance from the bounding box. [3] The system of claim 1, wherein the dimming factor further comprises a rate of change that is at least 20% higher near a periphery of the video image than near the bounding box. [4] System according to one of the preceding claims, wherein the control circuit is further designed to Performing object detection on the background; and Determine that the background is irrelevant by classifying an object detection output or comparing an object detection output with contents in a lookup table. [5] System according to any one of claims 1-3, further comprising: an AI (artificial intelligence) model trained with training data that defines objects or text with meaning; wherein the control circuit is further designed to Providing the background to the AI ​​model; and determining that the background has no meaning based on an output of the AI ​​model. [6] System according to one of claims 1-5, wherein the control circuit is further configured to Determine that the background has meaning; Stop applying the respective dimming factor to the individual background pixels, and Cause the display component to display the video image with the respective original pixel intensity for the multiple pixels. [7] The system of claim 6, wherein the display component comprises an OLED (organic light-emitting diode) display screen. [8] The system of any of claims 1-6, wherein the display component comprises an LED display screen. [9] A system according to any preceding claim, wherein the mixed media data signal is associated with a video conferencing application. [10] System according to one of the preceding claims, wherein the control circuit is further configured to Performing a person detection and segmentation operation; and Delimiting the person in the video image from the background with the bounding box in response to performing the person detection and segmentation operation. [11] The system of claim 1, wherein the control circuit is further configured to detect when an audio or video conferencing application is not running in a foreground mode; and causing the display component to stop displaying when the audio or video conferencing application is not running in the foreground mode. [12] Non-transitory computer-readable medium comprising instructions that, when executed by processing circuitry, are arranged to Receiving a video image from a video conferencing application on an external device; Receiving metadata associated with the video image, wherein the metadata delimits a person from a background in the video image using a bounding box, and wherein the bounding box may be an n-sided polygon, a curved shape generated by depth information, or a combination thereof; and wherein the metadata associated with the video image further provides a background blur flag that is set when the background is not significant; wherein the video image comprises a respective original pixel intensity for a plurality of pixels; Identifying pixels of the plurality of pixels associated with the background as background pixels, Determining a respective dimming factor for individual background pixels that reduces the respective original pixel intensity, and wherein the dimming factor is a function of the distance from the bounding box and ranges from a minimum amount adjacent to the bounding box to a maximum amount at an edge of the video image; Creating a dimmed background by applying the respective dimming factor to the individual background pixels, and Cause a display component to show a converted video image with the background dimmed. [13] The non-transitory computer-readable medium of claim 12, wherein the instructions are further for non-linearly varying the respective dimming factor as a function of distance from the bounding box. [14] The non-transitory computer-readable medium of claim 12 or claim 13, wherein the instructions are further to further implement the dimming factor by changing it more frequently closer to the bounding box than at a periphery of the video image. [15] The non-transitory computer-readable medium of claim 12, wherein the operations are further configured to Detecting when the background blur flag is reset; and Cause the display component to display the video image with the respective original pixel intensity for the multiple pixels. [16] The non-transitory computer-readable medium of claim 12, wherein the operations are further configured to Detecting when the background blur flag is reset; and Identifying pixels of the plurality of pixels associated with the background as background pixels, Determining a respective dimming factor for individual background pixels that reduces the respective original pixel intensity, and wherein the dimming factor is a function of the distance from the bounding box and ranges from a minimum amount adjacent to the bounding box to a maximum amount at an edge of the video image; Creating a dimmed background by applying the respective dimming factor to the individual background pixels, and Causes the display component to display a converted video image with the background dimmed. [17] The non-transitory computer-readable medium of claim 12, wherein the operations are further configured to Detecting when an audio or video conferencing application is not running in a foreground mode; and Make the display component stop displaying when the audio or video conferencing application is not running in foreground mode. [18] The non-transitory computer-readable medium of claim 13, wherein the operations are further configured to Detecting when an audio or video conferencing application is not running in a foreground mode; and Make the display component stop displaying when the audio or video conferencing application is not running in foreground mode. [19] The non-transitory computer-readable medium of claim 14, wherein the operations are further configured to Detecting when an audio or video conferencing application is not running in a foreground mode; and Make the display component stop displaying when the audio or video conferencing application is not running in foreground mode. [20] The non-transitory computer-readable medium of claim 15, wherein the operations are further configured to Detecting when an audio or video conferencing application is not running in a foreground mode; and Make the display component stop displaying when the audio or video conferencing application is not running in foreground mode. [21] The non-transitory computer-readable medium of claim 16, wherein the Operations are also designed to Detecting when an audio or video conferencing application is not running in a foreground mode; and Causes the display component to stop displaying when the audio or video conferencing application is not running in foreground mode. [22] System comprising: a display component comprising display electronics; and Means for managing the display electronics, wherein the means for managing the display electronics is designed to Receiving a mixed media data signal from an external device; generating a video image from the mixed media data signal; Separating a person in the video image from a background with a bounding box; Determine that the background is irrelevant; wherein the video image comprises a respective original pixel intensity for a plurality of pixels; Identifying pixels of the plurality of pixels associated with the background as background pixels, Determining a respective dimming factor for individual background pixels that reduces the respective original pixel intensity, and wherein the dimming factor is a function of the distance from the bounding box and ranges from a minimum amount adjacent to the bounding box to a maximum amount at an edge of the video image; Creating a dimmed background by applying the respective dimming factor to the individual background pixels, and Causes the display component to display a converted video image with the background dimmed. [23] The system of claim 22, wherein the means for managing the display electronics further varies the dimming factor at a rate of change that is at least 20% higher at a periphery of the video image than adjacent to the bounding box. [24] The system of claim 23, further comprising: a means for classifying the background, wherein the means for classifying the background is to determine whether the background has meaning; and wherein the means for managing the display electronics is further configured to stop applying the respective dimming factor to the individual background pixels when it is determined that the background has significance. [25] A system according to any one of claims 20-24, wherein the bounding box may be an n-sided polygon, a curved shape generated by depth information, or a combination thereof; and wherein the respective dimming factor varies non-linearly as a function of distance from the bounding box.

Images