Multimedia system with optimized performance

Abstract

The present invention relates to a multimedia system in a vehicle, the system being capable of running more than one user application at the same time, the system comprising: at least one processor; and a computer-readable storage medium, the computer-readable storage medium comprising a non-transitory computer-readable storage medium storing computer-readable instructions, the system being configured to execute the computer-readable instructions by the processor, wherein the computer-readable instructions include a display controller, the display controller being configured to determine which user application is the foreground application being displayed on the display, and when the system is running more than one user application at the same time, the display controller is configured to reduce the size of the window defining at least one user application running in the background. The present invention also relates to a method for optimizing the performance of a multimedia system when more than one user application is running at the same time.

Description

Technical Field

[0001] The present invention relates generally to multimedia systems in vehicles, and in particular to optimizing the performance of multimedia systems. Background Art

[0002] With the widespread use of high-definition displays in in-vehicle multimedia entertainment systems, the bus bandwidth connected to the memory of such systems has become critical when considering performance. This is because high bandwidth is required to render content to a high-definition display when reading and writing to memory. In particular, the simultaneous use of applications that require such high memory bus bandwidth often results in a 100% bus load, causing slow system response, which is manifested in, for example, slow execution of user input, sharp drops in frame rate, and delays in rendering to the display. Bus load is the percentage of total bus traffic passing through the bus and includes the overhead of bus arbitration. Bus utilization is the percentage of total bus traffic that is useful - that is, useful data passing through the bus, such as data entering the application rather than unneeded data that will be refreshed. When the memory bus has a high bus load percentage, such as 90% or more, the bus utilization may be only a small fraction of the bus load percentage, such as less than 60%, depending on the system. In other words, although the memory bus is occupied, for example, 90% of the time, the actual useful data or throughput passing through the memory bus may be less than 60%. In addition, as systems become more complex, the number of bus masters increases, which increases the bus load and further reduces the bus utilization, for example, by a further 30%. Therefore, it is imperative to reduce the load on the memory bus.

[0003] Many prior art systems focus on optimizing network bandwidth to receive content from background applications or lower priority applications at a limited rate or quantity over the Internet. However, optimizing the bandwidth required to obtain information over the Internet does not necessarily optimize the bandwidth requirements on the memory bus. In particular, after obtaining information from the Internet, the processing of such prior art systems may not optimize the utilization of system resources, such as memory and CPU load, and therefore may not optimize the load on the memory bus.

[0004] The existing way to reduce the load on the memory bus is to stop or pause the application running in the background so that the system can allocate resources to the foreground application. However, it takes time to resume the stopped or paused application, resulting in a delay in the user experiencing the restart of the application after calling the application.

[0005] In order to avoid such delays while saving bus or memory bandwidth, known methods propose to calculate which data is displayed on the display or blocked by overlapping data and select to read only the data that can be seen on the display. However, additional resources are required to determine which data to read.

[0006] Therefore, there is a need for new or improved systems and methods that overcome or at least ameliorate one or more of the above-mentioned disadvantages. Summary of the invention

[0007] Therefore, an object is to provide a multimedia system and related methods that solve the above problems. In particular, an object is to provide a multimedia system and related methods that optimize performance and reduce latency when more than one user application is running simultaneously. Another object is to provide a multimedia system and related methods that can simultaneously run multiple user applications that require high bandwidth of a memory bus.

[0008] To achieve these and other purposes, in a first aspect, a multimedia system in a vehicle is provided, the system being capable of running more than one user application simultaneously, the system comprising: at least one processor; and a computer-readable storage medium comprising a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, the system being configured to execute the computer-readable instructions by the processor, wherein the computer-readable instructions include a display controller configured to determine which user application is a foreground application being displayed on a display, and when the system is running more than one user application simultaneously, the display controller is configured to reduce the size of a window defining at least one user application being run in the background. The display controller may be further configured to call a background user application to the foreground upon request and enlarge the window size of the background user application for display on the display.

[0009] In a second aspect, a method for optimizing the performance of a multimedia system in a vehicle when multiple user applications are running simultaneously is provided, the method comprising: determining, by a display controller of the system, which user application is a foreground application being displayed on a display; when more than one user application is running on the system simultaneously, instructing, by the display controller, a processor of the system to reduce the size of a window defining at least one user application running in the background. The method may also include: when a request is made to call a background user application to the foreground, instructing, by the display controller, a processor to enlarge the window size of the background user application for display on the display.

[0010] Advantageously, the applications are prioritized so that the load on the memory bus can be balanced or distributed. Applications that are not of interest to the user, such as background applications, are configured to run continuously but scaled down to reduce the resources required to operate such applications.

[0011] The display controller can be configured to instruct the processor to reduce the window size of the background user application and amplify the window size of the background user application called to display on the display. In such an embodiment, the size scaling of the user application can be performed by the processor and is therefore hardware-assisted. Therefore, the size scaling of the user application is advantageously fast, typically a few milliseconds, and does not require additional calculations or resources to reduce the background user application. Advantageously, the size scaling of the background user application can not increase the calculation or computing workload associated with reducing the resolution or identifying the data to be displayed as in the prior art. When the user request has been reduced to a specific user application, the amplification of the user application as provided herein is advantageously faster than recovering such a user application from a pause or stop mode.

[0012] The system may be configured to execute computer readable instructions when a bus coupled to the computer readable storage medium has insufficient bandwidth to simultaneously execute more than one user application.

[0013] When the bandwidth of a bus connected to the computer-readable storage medium of the system is insufficient to run more than one user application simultaneously, the display controller may instruct the processor to reduce the window size of the background user application and to enlarge the window size of the background user application.

[0014] The vehicle multimedia system disclosed herein may be an embedded system. An embedded system is typically designed to perform one or several dedicated functions, such as the functions disclosed herein. The embedded system may not be configured to include functions that are unrelated to or separate from the dedicated functions. An embedded system may read software or computer-readable instructions from a built-in chip, rather than from a disk like a general-purpose computer system such as a desktop PC. Therefore, the system resources for an embedded system to run concurrent user applications are limited. An embedded system may have real-time computing constraints. Although the system may be designed to run more than one user application at the same time, the system resources of the embedded system may not be sufficient to support the running user applications and / or any new actions. Therefore, in some embodiments, the present disclosure advantageously provides a solution for optimizing the performance of an embedded system. In some embodiments, at least one reduced user application running in the background requires synchronization of audio frames with the presented frames. In some embodiments, at least one reduced user application running in the background is not displayed on the display. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A hardware block diagram of an exemplary multimedia system 100 is shown.

[0016] Figure 2The operations involved in playing a high definition video file 102 on a multimedia system 100 according to one embodiment of the present disclosure are shown.

[0017] Figure 3 A graph illustrating the total consumption of the bandwidth of the memory bus of the multimedia system 100 at different stages is shown.

[0018] Figure 4 Shows Figure 3 A graph of the percentage load of the memory bus versus bandwidth consumption.

[0019] In the drawings, the same reference numerals denote the same components. DETAILED DESCRIPTION

[0020] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description of the present invention will be provided for the purpose of explaining the principles of the present invention and its practical application, so that those skilled in the art can understand the present invention with respect to various exemplary embodiments and various modifications suitable for the intended specific use. The detailed description is not intended to be exhaustive or to limit the present invention to the specific embodiments disclosed. Modifications and equivalents will be apparent to those skilled in the art and are included within the spirit and scope of the appended claims.

[0021] A multimedia system in a vehicle is provided. The system is capable of running more than one user application at the same time. The system includes: at least one processor; and a computer-readable storage medium, which includes a non-transitory computer-readable storage medium storing computer-readable instructions, and the system is configured to be executed by the processor. The computer-readable instructions include a display controller, which is configured to determine which user application is the foreground application being displayed on the display. When the system runs more than one user application at the same time, the display controller is configured to reduce the size of the window defining at least one user application running in the background. The display controller can be further configured to call the background user application to the foreground upon request and enlarge the window size of the background user application for display on the display.

[0022] A method for optimizing the performance of a vehicle multimedia system when more than one user application is running simultaneously is also provided. The method includes determining, by a display controller of the system, which user application is a foreground application being displayed on a display. When more than one user application is running simultaneously on the system, the method also includes instructing, by the display controller, a processor of the system to reduce the size of a window defining at least one user application running in the background. The method may also include instructing, by the display controller, a processor to enlarge the window size of the background user application for display on the display when requesting to call the background user application to the foreground.

[0023] Multimedia systems are systems used to play different types of digital media such as text, audio, video, computer graphics, photos and animations. In the context of vehicle multimedia systems, the most common applications include but are not limited to navigation, audio, radio, auxiliary media playback, video playback, voice recognition, such as through Apple CarPlay TM , Baidu CarLife, Android Auto, etc., and connections with mobile devices. Other applications that the in-vehicle multimedia system may have include hands-free voice calls and reading messages and emails aloud. The vehicle multimedia system may also be referred to as or may include a head unit or an infotainment unit. The user of the multimedia system may be the driver or owner or a passenger or occupant of the vehicle.

[0024] To play different types of media, the multimedia system may be, for example, an embedded system or a general-purpose computing device. Figure 1 1 shows a hardware block diagram of an exemplary multimedia system 100. The multimedia system 100 may include one or more processing devices 10, such as a system on a chip or a microcontroller, to implement the functions of the multimedia system. The multimedia system 100 generally includes a plurality of processors and other components connected to a computer-readable storage medium or memory module (generally 118) via a communication bus (generally 30). Although the memory module 118 is generally Figure 1 10, but the memory module may alternatively or additionally be included in the processing device. The processor may include one or more central processing units, such as ARM cores, and other processors located on the processing device, such as controllers, 2D and / or 3D graphics processors, video processors, image processors, etc. Figure 1 In the embodiment, the processing device 10 includes a central processing unit 120, a video processor 108, an image processor 110, and a graphics processor 112, all of which communicate with a memory module 118. Figure 1One central processing unit 120 is shown, but the multimedia system may include more than one central processing unit. For example, a multimedia system including two central processing units is generally considered to be a dual-core system, while a multimedia system including four central processing units is generally considered to be a quad-core system. Typically, the central processing unit 120 executes instructions to initialize and run the video processor 108, the image processor 110, and / or the graphics processor 112. Once the central processing unit 120 instructs the processors 108, 110, and / or 112, the processors 108, 110, and / or 112 execute the instructions independently. That is, the processors 108, 110, and 112 can individually write data to and read data from memory modules such as cache memory, RAM (e.g., DDR RAM), and flash memory (e.g., NAND flash memory and NOR flash memory). The computer-readable storage medium or memory module 118 may include temporary and non-temporary computer-readable storage media. For example, the central processing unit 120 can read computer-readable instructions from a non-temporary memory module and write data to a temporary memory module in response to the instructions. In response to the instructions, the central processing unit 120 may write instructions for the video processor 108, the image processor 110, and / or the graphics processor 112 to read and execute. One, some, or all of the processors may be configured to or participate in presenting the content of each user application or running user applications simultaneously, and the memory module connected to the processor may be configured to store the presented content. The memory module may also be used by other peripheral devices such as USB devices, personal mobile devices, DVD or Blu-ray TM CD drive access, although the drivers for peripheral devices usually write to the memory module through direct memory access to improve efficiency. Direct memory access can be a potential memory bus master, which will cause the memory bus load to increase. The multimedia system usually also includes a device for receiving user input, such as a user interface or human-machine interface, a knob, an actuator on a steering wheel, a mouse, a joystick, a touch screen display, a microphone, a tactile actuator and / or other input mechanisms such as voice recognition, gesture recognition, etc. The multimedia system usually also includes a display screen or display for outputting multimedia or any information. The input device can communicate with a display controller, which receives user input from the input device and directs the execution of the user input to, for example, a processor or a processor group, which processes the input to provide output according to the request. The peripheral device, the device for receiving user input, the device for outputting multimedia or information (such as a display) and other peripheral devices of the multimedia system are generally 116.

[0025] The multimedia system may include a software architecture. The software architecture may generally include computer-readable instructions stored on a non-transitory computer-readable storage medium to enable the multimedia system to perform its functions. As known to those skilled in the art, software or code or instructions may be implemented in various layers of a computing system. The lowest layer of code (physical or hardware layer) transmits and receives raw bits from the hardware to serve the layers above it. The highest layer of code interacts directly with the end user. Computer-readable instructions may include code across several layers. The software architecture may include, but is not limited to, an operating system for the multimedia system, application software for each user application, and other software required to interface between the operating system and the application software or to perform other functions required by the multimedia system.

[0026] A user application of a multimedia system is a software program or a set of instructions that works with the hardware components of the system to allow the user to use and experience multimedia. The operations involved in running a multimedia application may include audio encoding and decoding, video encoding and decoding, content presentation, volume change, color space conversion, overlay, and other operations.

[0027] The display controller can be part of the user application software, typically as part of one of the top two layers of code or across the top two layers of code. The display controller can be a protocol that specifies the communication between a display server and its client (e.g., a user application) that controls the display of rendered content. Alternatively, the display controller can include a rendering module to control the rendering of data and the display of rendered data. The display controller can include the functionality of a display server and / or the functionality of combining faces from each user application to create a graphical user interface that is output to a display. The display controller can control the display of content, transmit communications between a processor and an input device, and / or output images to a display.

[0028] User applications can be manipulated by user actions or user inputs, for example, a user can start an application, switch multimedia sources, such as switching to a video source or an audio source, switch application interfaces, or manipulate application interfaces, such as scrolling to display another map portion or zooming a map portion. These operations and actions contribute to the bandwidth requirements of the memory bus.

[0029] When the user provides input, one or more processors are instructed to perform the action. These processors can control the bus to the memory module, thereby becoming a memory bus master. For example, when the user provides input to play a high-definition video, the operations involved in playing the video may include reading a media file, decompressing or decoding compressed video frames from the file, decoding audio frames, presenting each video frame in a form suitable for display, converting each presented frame into a format suitable for display on a display screen (color space conversion), synchronizing the audio frame with each presented frame, scaling each presented frame to a size suitable for display on a display screen, writing each presented frame to a dedicated surface or memory module, and if necessary, covering multiple dedicated surfaces, where each dedicated surface represents a user application, such as superimposing an HMI animation on a video frame, writing each frame to a frame buffer, and displaying the frame on a high-definition display screen. These operations involve a video processing unit (VPU), a graphics processing unit (GPU), and an image processing unit (IPU), so these processors become memory bus masters. In another example, presenting and displaying a navigation map involves a GPU, an ARM core, and an IPU as a memory bus master. Therefore, any user action, such as selecting a video to play, requesting navigation route guidance, changing the audio source, changing the volume, and switching screens, requires one or more bus masters.

[0030] The bus or buses to memory, i.e., the memory bus, should have sufficient bandwidth to transfer data to and from memory, even data for high bandwidth use cases or user applications or resource intensive operations (e.g., user actions). However, in situations where hardware, such as a memory bus, is insufficient to support increasingly complex applications and increasing application requirements, the disclosed invention is intended to alleviate such situations where the memory bus does not have sufficient bandwidth to perform all actions and any new actions simultaneously, especially when the system is multitasking and running multiple user applications simultaneously. In general, the capacity of the bus determines how quickly the components of a computer system can communicate with each other. Therefore, the speed of the connection to the memory (e.g., RAM) directly controls how fast the computer system can access instructions and data.

[0031] While the disclosed invention generally optimizes the performance of any multimedia system, the disclosed invention is particularly advantageous in scenarios where a user can only focus on one use case or user application at a time. This may be the case when there is only one display screen, or when there is no split screen functionality on the display screen.

[0032] In a multitasking scenario, for example, when a second user application is triggered, for example, by a user input, the display controller identifies the second user application as a foreground application. At this point, when the second user application is called onto the display, the first user application is not the user's focus and / or has been completely or partially obscured. It has been found that the presentation of content for display on a full display screen utilizes a large amount of bandwidth compared to other operations such as audio / video decoding, color space conversion, and overlay. This is because each pixel of the display requires a certain amount of data, and a typical display has thousands of pixels, so the amount of data required to present content on a full display screen becomes very large. Therefore, reducing or minimizing the window of the background application so that the presented content is not suitable for display on the full display screen but is displayed on a reduced portion of the display screen reduces the amount of data to be presented. Therefore, the amount of data that needs to be written to and read from the memory is reduced, thereby saving bandwidth through the memory bus.

[0033] In addition, in the case where the background application requires content to be presented and other components to be synchronized with the presented content, the presence of the reduced window indicates to the processor to continue running the application, but at a reduced bandwidth so that the components continue to be synchronized, thereby avoiding any resynchronization delays when called to the foreground. Such components rely on the presented content, so it is necessary to synchronize these components with the presented content. For example, in a scenario where the background application includes audio and video components, the presence of the window (albeit a reduced window) can achieve the continuation of the audio and video output, so that the audio stream does not need to be resynchronized with the video stream when the background application is called to the foreground. Users may be sensitive to any desynchronization of the audio output and its associated video output, so ensuring such synchronization is beneficial from the user's perspective.

[0034] At least one application selected to be reduced in size when in the background and enlarged when called to the foreground may be an application that requires content presentation and / or synchronization of other components with the presented content. At least one reduced user application running in the background may require synchronization of audio frames with presented frames. Such applications may be identified when designing a multimedia system and programmed to be reduced or enlarged when in the background or foreground, respectively. Alternatively, the disclosed invention may include, for example, identifying applications that require content presentation and / or synchronization of components with the presented content through a display controller, and then reducing the size of the window defining such applications when such applications retreat to the background.

[0035] Accordingly, in scenarios where the background application may not encounter problems with resynchronizing components such as audio and video streams, or where resynchronization does not consume a lot of bandwidth, or where resynchronization does not have undesirable missynchronization side effects, the rendering activity of the reduced background application window may be stopped, or the background application may be stopped. For example, a navigation application that does not require map caching, route calculation, visual guidance, voice guidance, etc. to be synchronized with the rendered map content may be stopped when in the background.

[0036] In a scenario where an application that generates audio is moved to the background, the audio of the background application can be attenuated so that the user can focus on the foreground application. Alternatively, if the foreground application does not generate audio, the audio of the background application can be maintained. In a scenario where both the foreground application and the background application generate audio, the volume of audio output from the background application may be reduced relative to the volume of the foreground application. In such a scenario, the system may include an audio manager that is configured to attenuate or otherwise control any audio output of each background application.

[0037] There may be one or more background applications, and at least one, some, or all of these applications may be scaled down to reduce the resources required to run them. Thus, the background applications advantageously contribute a smaller amount of data that needs to be transferred to and from the memory, thereby freeing up bandwidth on the memory bus.

[0038] The surface or area designated for the presentation data to be displayed on the display is called a "window". The window surrounds or delimits the data of the output user application. The window can occupy the entire display screen or a portion of the display screen. The window of the background application can be reduced to any predefined size or size, so that the background application requires less data to run. Advantageously, if the user takes further actions, the user will not experience lag because the memory resources of the multimedia system are optimized and not running at 100%. The display controller can use hardware or a processor, such as an image processing unit, to reduce the size of the window of the background user application to a predetermined size. Since the reduction of the background user application can be advantageously performed by hardware, that is, assisted by hardware, the reduction step can be free of the calculation or operation work associated with reducing the resolution or identifying the data to be displayed as in the prior art. Hardware-assisted scaling means that once the processor obtains the scaled data from the memory, the subsequent steps of performing the scaling are completed in the processor itself. The system's central processing unit or ARM core does not occupy resources for scaling, so there is no need to access the system's memory. Therefore, hardware-assisted scaling is not constrained by the memory bus or other hardware. Once the reduced window is generated in the processor, the reduced window is written to the memory through the memory bus.

[0039] The predefined size may be a hardware-defined, system-defined, or application-defined minimum. For example, the minimum predefined size may be a window size specified by the specifications of an image processor. The window may be reduced to a minimum size predefined by the processor. In one embodiment, a window of a background application may be reduced to a minimum predefined size of 1 pixel by 1 pixel.

[0040] The reduced window can be hidden behind the foreground application and not displayed on the display screen. The reduced window of the background application can be configured not to be displayed on the display screen or output to the display. The reduced background window can not be displayed on the display. The display controller can be configured to handle the operation of the background application, but is configured not to be displayed or output to the display. For example, a reduced window of 1 pixel × 1 pixel can be displayed as a dot, so such a window can be configured not to be displayed on the display. Alternatively, the reduced window can be displayed as a thumbnail on the display screen. In such embodiments, the display controller can be configured to build a dedicated surface, including the reduced window and the window that defines the foreground application. The display controller can overlay various dedicated surfaces to create an interface for display or output to the display. In such embodiments, the thumbnail can have its frame rate attenuated to further reduce its bandwidth contribution.

[0041] When the user triggers the first user application to return to the foreground and receive the user's renewed attention, the display controller instructs, for example, the processor to enlarge the window size of the first user application for display on the display. Advantageously, in such embodiments, the enlargement of the background window is hardware-assisted, for example undertaken by the processor, and therefore requires minimal time and memory resources to enlarge. In particular, the time required to enlarge the background window can advantageously be less than the time required to resume a stopped or paused application. In an embodiment, the time required to enlarge the background window can be less than half, less than one-third, or even less than one-fifth of the time required to resume a stopped or paused application. In one embodiment, the processor configured to perform window scaling is an image processing unit.

[0042] In multimedia entertainment systems, performance tuning is imperative to ensure a good user experience. As screens get larger and playback resolutions get higher, the bus to and from memory consumes more and more bandwidth, creating severe bandwidth challenges when the system must support multiple applications running simultaneously. To ensure seamless transitions between screen changes, the time delay of transitions is critical to meeting end-user expectations. The present invention not only solves the problem of slow system response due to high memory bus load, but also reduces the delay in resuming background applications.

[0043] In automotive multimedia systems such as disclosed herein, in some countries / regions, it is legally required to stop video applications such as video playback and / or associated audio when the vehicle is moving. Therefore, the resynchronization of video frames with their audio frames when restarting a stopped or paused video application is a problem that can be solved by the present disclosure. In particular, the reduced background window enables the video application to continue running, so that resynchronization is not required. Since the background window can be not displayed, legal requirements can be met, while also meeting system requirements because the background video application continues to run with a reduced bandwidth.

[0044] While the invention is applicable to any application running simultaneously on a multimedia system, the disclosed invention advantageously aids a multimedia system configured to run video playback, particularly high definition video playback, and at least one other user application simultaneously. The invention will now be described with respect to this embodiment.

[0045] A conventional way to save time for reading and writing high-definition video data to memory is to continue audio decoding and playback, but stop high-definition video rendering, and navigation map guidance when the video source retreats to the background, and the navigation map guidance comes to the foreground. This is because video rendering takes up a lot of bandwidth, while audio and video decoding activities and audio playback take up minimal bandwidth. However, when the user switches back to the video source, the media software component or application needs to synchronize the stopped decoded video frame with the ongoing audio playback before presenting to the surface display. The time required for synchronization and video recovery can typically be 1 second or more, depending on the system resource consumption status. Audio and video playback may even be out of sync. From the user's perspective, a blank screen or frozen frame will be seen during the time waiting for video playback to resume and display.

[0046] To avoid the delay required for resynchronization, the rendering of the HD video continues, but the video playback display window is reduced to a 1×1 pixel window to reduce the consumption of memory bus bandwidth caused by the rendering of the HD video to the standard size display. This is an advantage because the user will not see the HD video playing in the background when the background application is not displayed on the display. The resizing of the HD video playback display window uses the graphics processing unit, which is also an advantage because it is hardware-assisted. Therefore, the system is not deprived of memory access and is able to operate and respond.

[0047] Figure 2 1 shows the operations involved in playing a high-definition video file 102 from a USB storage drive 104 inserted into a multimedia system 100 according to one embodiment of the present disclosure. The high-definition video file 102 is copied or written to the DDR memory 106 via direct memory access, as shown by arrow 202. The CPU ( Figure 2The IPU 110 (not shown) reads the high-definition video file 102 from the DDR memory 106 and instructs the VPU 108 to retrieve the compressed video data from the DDR memory 106 and decode the video data using the video codec of the VPU. The compressed audio data can be decoded in this way or by software decoding. The decoded video / audio data is written to the DDR memory 106 via the memory bus 204. The decoded video data may not be in a format suitable for display on the display 116, so the IPU 110 is instructed to perform a color space conversion to convert the decoded data into a format suitable for display. The IPU 110 can also be instructed to scale the decoded data to a size suitable for display. The synchronization of the decoded audio and video data can be performed at this stage by means of software or hardware. The IPU 110 then writes the converted and / or scaled video data to the DDR memory 106 via the memory bus 206.

[0048] At this stage, the video data is usually still a fragment. Therefore, if the high-definition video is playing in the background, the GPU 112 is instructed to present the video data and generate a video frame suitable for display on the display 116 or in a reduced window. The presented frame is sent via the memory bus 208 to write to the memory block allocated to the high-definition video application in the DDR memory 106, which is called the dedicated surface of the application. It has been found that when a video frame suitable for display on the display 116 is generated, reading and writing via the memory buses 204 and 206 does not take up too much bandwidth, while reading and writing via the memory bus 208 takes up a lot of bandwidth. Therefore, if the high-definition video is in the background, the amount of data passing through the memory bus 208 can be advantageously reduced by reducing the window defining the high-definition video. The memory buses 204, 206, and 208 can be different buses connected to the DDR memory 106, in which case the total bandwidth and bus load are the sum of these memory buses. The memory buses 204, 206, and 208 can be a single bus or different buses that eventually form a single bus to the DDR memory 106. Here, the memory bus may refer to an internal bus, ie, a connection within a chip, component, or unit, or an external bus, ie, a connection between chips, components, or units.

[0049] The rendering of the GPU 112 may be controlled by a display controller 114 located above the physical layer. The display controller 114 is also responsible for overlaying the dedicated surfaces to output a final image or frame for display on the display 116, typically by using the IPU 110, and then transmitting the final image or frame to the display 116 via the display port 210 (e.g., LVDS, HDMI, MIPI). That is, the display controller 114 instructs the IPU 110 to retrieve the various dedicated surfaces to be overlaid from the DDR memory 106 before the IPU 110 transmits the final image or frame to the display 116.

[0050] By virtue of its responsibility for controlling the final image or frame output to the display 116, the display controller 114 is able to determine the user application whose frame is being displayed on the display 116, i.e., the foreground application, and which applications are background applications. By virtue of its responsibility for controlling the rendering or creation of each dedicated surface, the controller 114 is also able to specify the size of each dedicated surface. Thus, for one, some, or all background applications, the display controller 114 is able to specify the size of the window defining the background application, e.g., 1×1 pixels, and instruct the GPU 112 to render a video frame suitable for the reduced background window. The display controller 114 can choose to output or not output a window of the background high-definition video to the display 116. If the reduced background window is output to the display 116, for example, as a thumbnail, the display controller 114 instructs the IPU 110 to create the final image or frame to be output. The IPU 110 then transmits the final image or frame to the display 116 via the display port 210.

[0051] A common HD video display size uses 720 lines of 1280 pixels per line (1280×720). Each color pixel typically uses 32 bits / pixel, or 4 bytes, and a typical HD display has a refresh rate of 30 frames per second. Therefore, for one read cycle and one write cycle, 30 video frames are presented to such a HD video display in one second (in Figure 2 The required bandwidth (indicated by display 116) is:

[0052] Bandwidth = 1280 pixels × 720 pixels × 4 bytes (true color pixel size) × 30fps × 2 (read and write cycles) = 220MB / second

[0053] Figure 3 A graph illustrating the total consumption of the memory bus bandwidth of a multimedia system 100 at different stages, the multimedia system 100 comprising a 2D and 3D graphics processing unit, a video processing unit, a digital signal processor and an ARM core is shown. Figure 4 A graph of memory bus load versus bandwidth consumption is shown. In the first stage, the HD video is playing in the foreground, and each sampled bandwidth shows an average bus load of about 80%. The second stage shows the increase in sampling bandwidth due to navigation map guidance and related rendering activities running simultaneously in the foreground, while the HD video is playing in the background at its standard, unchanged window size. The third stage shows the drop in sampling bandwidth when the background HD video window is resized from 1280 pixels × 720 pixels to 1 × 1 pixels. The bandwidth required for such a 1 × 1 pixel background window is:

[0054] Bandwidth = 1 pixel × 1 pixel × 4 bytes (true color pixel size) × 30fps × 2 (read and write cycles) = 240 bytes / second

[0055] In the third phase, it is observed that the bus load does not decrease immediately because the system was overloaded in the second phase before the resizing and thus built up a backlog of ARM core processing. After the resizing, the ARM cores had to catch up with the backlog, so the bus load did not decrease immediately. The bandwidth drop is mainly due to the bandwidth savings when the 2D GPU is acting as the memory bus master. This is because the 2D GPU is mainly responsible for video rendering. Figure 3 and 4 In the third stage, it can be seen that the read bandwidth is saved by about 200MB / s and the write bandwidth is saved by 100MB / s. When the HD video window is minimized, the total memory bus bandwidth drops from 1800MB / s to 1400MB / s, and the overall memory bus load drops by 15%.

[0056] If the third phase is allowed to continue longer, the bus load will be further reduced until the ARM core backlog is cleared, thereby reducing its contribution to the bus load percentage. Importantly, despite the relatively high bus load in the third phase of the figure, the system is relatively more responsive because the ARM core now has more access to the memory bus and is able to execute instructions in a timely manner.

[0057] When the HD video is called to the foreground, it only takes a few milliseconds for the HD video window size to be restored from 1 pixel × 1 pixel to 1280 pixels × 720 pixels. In addition, since the video and audio continue to run in the 1×1 pixel background window, there is no need for any resynchronization of the video frames with the audio playback. The steps that were not completed when the HD video was in the background, namely, outputting the HD video to the display 116, are now resumed. Therefore, the bandwidth of the memory bus can be optimized, thereby preventing the performance of the multimedia system from being degraded. In addition, operations such as decoding, color space conversion, and synchronizing audio and video frames do not need to be stopped and resumed, thereby preventing any lag.

Claims

1. An in-vehicle multimedia system, the system being capable of simultaneously running multiple user applications, the system comprising: at least one processor, each of the at least one processor comprising a bus; and a computer-readable storage medium, the computer-readable storage medium comprising a non-transitory computer-readable storage medium storing computer-readable instructions, The system is configured to execute the computer readable instructions via a processor, Characterized in that the system further comprises a display controller, The display controller is configured to determine: when the system runs more than one user application simultaneously, a foreground user application being displayed on the display and a background user application being displayed on the display, and, In response to the bandwidth of the bus being insufficient to run more than one user application simultaneously, the display controller is further configured to instruct at least one processor to reduce the size of the window of the determined background user application.

2. The system according to claim 1, wherein: The display controller is further configured to call a background user application to the foreground according to a request and enlarge a window size of the background user application to display on the display.

3. The system according to claim 1 or 2, wherein: The window is reduced to a minimum predefined size.

4. The system according to claim 1 or 2, wherein: The window is reduced to 1 pixel by 1 pixel.

5. The system according to claim 1 or 2, wherein: The processor configured to perform the scaling of the window size of the background user application is a graphics processing unit.

6. The system according to claim 1 or 2, wherein: The computer readable instructions also include an audio manager configured to attenuate any audio output of each background user application.

7. The system of claim 1 or 2, wherein the one or more processors are configured to present content of concurrently running user applications.

8. The system according to claim 1 or 2, wherein: At least one scaled-down user application running in the background requires synchronization of audio frames with rendered frames.

9. The system according to claim 8, wherein: At least one reduced user application running in the background is not displayed on the display.

10. The system according to claim 1 or 2, wherein: The system is configured to run video playback and at least one other user application simultaneously.

11. A method for optimizing vehicle multimedia system performance when more than one user application is running simultaneously, the method comprising: When more than one user application is running on the system at the same time, determining, by a display controller of the system, a foreground user application being displayed on the display and a background user application being displayed on the display; and, Each of the at least one processor of the system includes a bus, and in response to insufficient bandwidth of the bus to run more than one user application simultaneously, the display controller instructs the at least one processor of the system to reduce the size of the window of the determined background user application.

12. The method according to claim 11, further comprising: When requesting to call the background user application to the foreground, the processor is instructed by the display controller to enlarge the window size of the background user application so as to display it on the display.

13. The method according to claim 11 or 12, wherein: The window is reduced to a minimum predefined size.

14. The method according to claim 11 or 12, wherein: The window is reduced to 1 pixel by 1 pixel.

15. The method according to claim 11 or 12, further comprising: Any audio output of each background user application is attenuated by the system's audio manager.

16. The method according to claim 11 or 12, further comprising: Content of concurrently running user applications is presented by one or more processors of the system.

17. The method according to claim 11 or 12, wherein: At least one scaled-down user application running in the background requires synchronization of audio frames with rendered frames.

18. The method according to claim 17, wherein: At least one reduced user application running in the background is not displayed on the display.

19. The method according to claim 11 or 12, wherein: User applications running simultaneously include navigation map guidance and video playback.

Images