An inter-process multi-layer image fusion control method based on shared memory
By using a kernel display driver and layer blending management process with shared memory in the embedded system, the problem of high memory consumption in multi-layer image blending is solved, and efficient image blending display between processes is achieved, improving interface rendering and response speed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INGENIC SEMICON CO LTD
- Filing Date
- 2023-10-20
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies are not suitable for multi-layer image fusion in the embedded field, resulting in high memory resource consumption and not involving multi-layer fusion between processes, which affects interface rendering and response speed and cannot meet users' needs for efficient graphics display.
The shared memory is divided into two parts: the shared memory reserved by the kernel display driver for the layer and the shared memory area of the layer fusion management process. The image is processed and fused in multiple processes through a memory mapping mechanism, and the producer-consumer model and queue mechanism are used to coordinate the synchronization of image data between processes.
It achieves efficient image fusion display across multiple processes, reduces memory resource consumption, improves interface rendering and response speed, and meets users' needs for efficient graphics display.
Smart Images
Figure CN119863374B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and specifically relates to an inter-process multi-layer image fusion control method based on shared memory. Background Technology
[0002] With the development of modern embedded computing technology and the advent of the artificial intelligence era, users in various smart devices and terminal scenarios have an increasingly strong demand for the integration of software application interfaces and video interfaces. Embedded interfaces not only need to meet users' visualization needs by changing interface elements and drawing application styles, but also need to be able to smoothly play and display videos. In embedded environments where hardware resources are limited, the speed of interface rendering and response needs to be improved as much as possible to avoid interface lag and reduce the user experience. As embedded processors gradually develop into multi-core processors, their performance in multi-process environments is getting better and better. By fully utilizing system characteristics, smoother graphics display can be achieved.
[0003] However, existing technologies are generally not suitable for embedded systems, as they consume a lot of memory resources or do not involve multi-layer fusion between processes. For example, using a GPU for image layer fusion requires additional memory resources.
[0004] In addition, commonly used technical terms in this field include:
[0005] Process: A program instance running in the operating system, with its own address space, consisting of the code executed by the storage system, storage variables, memory requested during program execution, and instructions executed by the program.
[0006] Shared memory: One method for data communication between multiple processes, allowing multiple processes to access the same address space. It is the fastest inter-process communication method. The use of shared memory can reduce overhead during large-scale data processing, making it particularly suitable for graphics and image processing.
[0007] Hardware alignment: For logical design or performance reasons, hardware devices usually require the physical memory to be read to be aligned by bytes, so that the hardware will not have to access memory multiple times to cross byte boundaries when reading data, thereby greatly improving the data access speed.
[0008] Physical address: The address of a memory unit linearly numbered by the CPU to correctly store and retrieve information when accessing memory. When the CPU needs to read or write data, it sends the physical address to the memory controller, which then performs the corresponding operation on the physical memory based on the physical address. The physical address is also called the actual address of memory.
[0009] Virtual addresses: In an operating system, memory addresses mapped from physical addresses are also called logical addresses. A single physical address can correspond to multiple virtual addresses simultaneously, and the mapping and management between them are handled by a dedicated memory management unit. Virtual addresses are provided for use by processes and kernel modules. When a process or kernel module calls the virtual memory request interface to request memory, the operating system allocates the corresponding memory to them, but does not immediately occupy physical memory. Instead, the actual physical memory is used only when the process or kernel module actually needs it. The virtual memory mechanism enhances the utilization of system memory.
[0010] Multi-layer image fusion: A commonly used image fusion technique that combines multiple images into a single image, resulting in better visual effects. Multi-layer image fusion programming facilitates the separation of information at different levels, enabling smoother logical processing of graphics and images.
[0011] Producer-Consumer Model: A classic concurrent programming model used to describe and solve the problem of cooperation between multiple threads or processes. Producers are responsible for generating data and placing it into a shared buffer; consumers are responsible for retrieving data from the shared buffer and processing it.
[0012] Queues: A common linear data structure in programming, characterized by First-In, First-Out (FIFO). This means that data is stored and retrieved in a queue in a specific order; data added to the queue is retrieved first. Queues are often used to solve producer-consumer problems. Producers add goods to the queue, and consumers retrieve them for consumption. Queues effectively coordinate the speed differences between producers and consumers, avoiding too many or too few goods and ensuring system stability and efficiency. Memory mapping: A method of mapping file nodes in a computer operating system to the virtual address space of processes. Different processes can share data by using memory mapping of the same file node; that is, memory mapping is an implementation of shared memory. Summary of the Invention
[0013] To address the aforementioned issues, the purpose of this application is as follows: The shared memory used in this application is divided into two parts: one part refers to the layer shared memory reserved by the kernel display driver, and the other part refers to the shared memory area requested by the layer fusion management process for controlling inter-process image data synchronization. This application fully utilizes multi-layer image functionality, processing one layer of image in multiple processes and then fusion displaying them. It provides a method that allows different processes to focus on only a single layer to draw and update images, enabling inter-process multi-layer fusion based on shared memory queues.
[0014] Specifically, the present invention provides an inter-process multi-layer image fusion control method based on shared memory, the method comprising the following steps:
[0015] S1, the kernel display driver reserves shared memory for multiple layers, with contiguous physical addresses and hardware alignment; the function for reserving memory is the system function dma_alloc_coherent, with the size of the memory as the parameter; the memory reservation function is executed multiple times depending on the number of layers;
[0016] S2, User space creates a layer blending management process to map shared memory and initializes the queue; that is, it creates an image blending management process, obtains the virtual address of shared memory in user space through a memory mapping mechanism, and initializes the shared memory queue.
[0017] The process creation process involves executing the corresponding binary file in the file system. The image fusion management process creates a shared memory handle using the system function shm_open, with the handle file path as the function parameter. Then, it requests shared memory using the system function mmap, with the size of the shared memory as the function parameter. The shared memory queue initialization process involves requesting a queue data structure and calling the system function memset for initialization, with 0 as the function parameter, to clear the queue.
[0018] S3: Each process obtains shared memory through a memory mapping mechanism. In different processes, they can obtain operable memory addresses through the producer queue stored in shared memory to draw and update the graphics. After the update is completed, the memory address where the current operation is completed is stored in the consumer queue.
[0019] S4, the fusion process merges data from various layers; that is, the image fusion management process reads the consumer queue in shared memory through virtual address for consumption, and performs image fusion on the memory updated by different process producers;
[0020] S5 updates image data to the hardware device, that is...
[0021] The image fusion result is input into the hardware display screen for display.
[0022] Step S3 further includes:
[0023] Shared memory is obtained through a memory mapping mechanism, which involves creating a shared memory handle using the system function shm_open with the handle file path as the function parameter, and then requesting shared memory using the system function mmap with the size of the shared memory as the function parameter.
[0024] The operable memory address is obtained by using the producer queue stored in shared memory. That is, after obtaining the shared memory through the system function mmap, the queue stores the index value of the memory array. The corresponding operable memory address FbAddr can be calculated from the index value through the memory offset. addr = mmap_header + index * sizeof(FbAddr). Here, the system function sizeof is used to calculate the memory size.
[0025] The graph is drawn and updated. After the update is completed, the memory address where the current operation ends is stored in the consumer queue, which is the data copying implemented by calling the system function memcpy as mentioned above, and the counter is updated. The consumer queue here is the integer calculator.
[0026] In step S3, it is assumed that
[0027] Process A draws and updates layer 1 of the image, pushing it into the consumer queue;
[0028] Process B draws and updates layer 2 of the image, pushing it into the consumer queue;
[0029] Process A is responsible for the interface controls. It copies the data of graphic buttons, text, and static images to the shared memory of the corresponding layer of Process A. The copy function is the system call function memcpy. The source data is the graphic buttons, text, and static images drawn by the process, and the destination data is the shared memory of the layer. After the copy is completed, Process A will synchronously control the counter in the shared memory corresponding to Process A to increment by 1.
[0030] The B process is responsible for video playback and copies the video data to the shared memory of the layer corresponding to the B process. After the copying is completed, the B process synchronously controls the corresponding counter in the shared memory to increment by 1.
[0031] The method assumes three processes, A, B, and C.
[0032] In step S1, the kernel displays that the three layers of the driver reserve shared memory;
[0033] In step S2, the layer blending management process requests shared memory to be used for inter-process image data synchronization control;
[0034] And establish counters corresponding to the three processes A, B, and C, with the counter type being integers;
[0035] In step S3, process A is responsible for the interface controls, copying the data of graphic buttons, text, and static images to the shared memory of the layer corresponding to process A; the copy function is the system call function memcpy, the source data is the graphic buttons, text, and static images drawn by the process, and the destination data is the shared memory of the layer; after the copy is completed, process A will synchronously control the counter in the shared memory corresponding to process A to increment by 1 number.
[0036] Process B is responsible for video playback, copying the video data to the shared memory of the layer corresponding to Process B. After the copying is completed, Process B synchronously increments the counter in the shared memory by one number.
[0037] The C process, similar to the B process, copies video data and also completes its own data copy, synchronously controlling the corresponding counter in the shared memory to increment by one number.
[0038] Step S4 further includes:
[0039] The layer blending management process is also responsible for monitoring the layer counter. When the counter is detected to be increasing, the shared memory of the three layers is used as a parameter to call the JZFB_PUT_FRM_CFG instruction. This instruction will allocate the memory to the DMA address of the display device to achieve layer blending. After the instruction is executed successfully, the counter is set to 0.
[0040] If there are only two layers, then both layers will be used as input parameters. The JZFB_PUT_FRM_CFG command supports a maximum of four layers as input parameters.
[0041] Step S5 further includes:
[0042] The JZFB_PUT_FRM_CFG instruction is invoked, which allocates memory to the DMA address of the display device. The display device then reads the image data through the DMA address and displays it on the hardware display screen.
[0043] Therefore, the advantage of this application is that by applying the technical solution of this application, only a single layer needs to be focused on in different processes to draw and update the image, and multi-layer fusion display between processes can be achieved based on a shared memory queue. Attached Figure Description
[0044] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0045] Figure 1 This is a flowchart illustrating the method.
[0046] Figure 2This is a flowchart illustrating an embodiment of the method. Detailed Implementation
[0047] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0048] This application provides a method for controlling multi-layer image fusion across processes using shared memory. The implementation logic can be found by referring to... Figure 1 As shown, the description is as follows:
[0049] Step S1 involves reserving shared memory for multiple layers in the kernel display driver. This memory is physically contiguous and hardware-aligned. The function for reserving this memory is the system function `dma_alloc_coherent`, with the size of the memory allocated as the parameter. This memory reservation function is executed multiple times depending on the number of layers.
[0050] Step S2 involves creating an image fusion management process. In user space, a shared memory virtual address is obtained through a memory mapping mechanism, and the shared memory queue is initialized. Process creation involves executing the corresponding binary file in the file system. The image fusion management process creates a shared memory handle using the system function `shm_open`, with the handle file path as the parameter. Then, it allocates shared memory using the system function `mmap`, with the size of the shared memory as the parameter. Initializing the shared memory queue involves allocating a queue data structure and calling the system function `memset` to initialize it with a parameter of 0, used to clear the queue.
[0051] In step S3, each process obtains shared memory through a memory mapping mechanism. In different processes, they can obtain operable memory addresses through the producer queue stored in the shared memory to draw and update the graphics. After the update is completed, the memory address where the current operation is completed is stored in the consumer queue.
[0052] Shared memory is obtained through a memory mapping mechanism. Specifically, a shared memory handle is created using the system function shm_open, with the handle file path as the function parameter. Then, the shared memory is requested using the system function mmap, with the size of the shared memory as the function parameter.
[0053] The operable memory address is obtained through a producer queue stored in shared memory. Specifically, after obtaining the shared memory using the system function `mmap`, the queue stores the index of the memory array. The corresponding operable memory address `FbAddr` can be calculated using the index and memory offset: `addr = mmap_header + index * sizeof(FbAddr)`. Here, the system function `sizeof` is used to calculate the memory size.
[0054] The process involves drawing and updating the graph. After the update is complete, the memory address where the current operation ended is stored in the consumer queue, which is achieved by calling the system function memcpy to copy the data, as mentioned above, and then updating the counter. Here, the consumer queue is the integer calculator.
[0055] For example, process A draws and updates image layer one, pushing it into the consumer queue; process B draws and updates image layer two, pushing it into the consumer queue.
[0056] Process A is responsible for the UI controls, copying the data of graphic buttons, text, and static images to the shared memory of its corresponding layer. The copy function is the system call function memcpy, with the source data being the graphic buttons, text, and static images drawn by the process, and the destination data being the shared memory of the layer. After the copy is completed, process A synchronously increments the counter corresponding to process A in the shared memory by one.
[0057] Process B is responsible for video playback, copying the video data to the shared memory of the layer corresponding to Process B. After the copy is completed, Process B synchronously increments the counter in the shared memory by one.
[0058] In step S4, the image fusion management process reads the consumer queue in the shared memory through the virtual address and consumes the updated memory from different process producers to perform image fusion.
[0059] The layer blending management process also monitors the layer counter. When it detects an increase in the counter, it calls the JZFB_PUT_FRM_CFG instruction, using the shared memory of the three layers as a parameter. This instruction allocates memory to the display device's DMA address, thus achieving layer blending. After successful instruction execution, the counter is set to 0.
[0060] If there are only two layers, then both layers will be used as input parameters. The JZFB_PUT_FRM_CFG command supports a maximum of four layers as input parameters.
[0061] Execute step S5 to input the image fusion result into the hardware display screen for display, that is, call the JZFB_PUT_FRM_CFG instruction. This instruction will allocate memory to the DMA address of the display device, and the display device will read the image data through the DMA address and display it on the hardware display screen.
[0062] In summary, in short... Figure 2 As shown:
[0063] S1, the kernel display driver reserves shared memory for multiple layers, with contiguous physical addresses and hardware alignment;
[0064] S2, User space creates layer fusion management process, maps shared memory, initializes queue;
[0065] S3, process A draws and updates layer 1 of the image, pushing it into the consumer queue;
[0066] Process B draws and updates layer 2 of the image, pushing it into the consumer queue;
[0067] S4, the fusion process merges data from each layer;
[0068] S5 updates image data to the hardware device.
[0069] The shared memory used in this application is divided into two parts: one part refers to the layer shared memory reserved by the kernel display driver, and the other part refers to the shared memory area allocated by the layer fusion management process for controlling inter-process image data synchronization. This application makes full use of multi-layer image functionality, processing one layer of image in multiple processes, and then fusion and displaying them. The specific process is as follows:
[0070] The following explanation uses three processes, A, B, and C, as an example. The kernel display driver reserves three layers of shared memory. The layer fusion management process requests shared memory to be responsible for controlling the synchronization of image data between processes, and establishes counters for the three processes A, B, and C. The counter type is integer.
[0071] Process A is responsible for the UI controls, copying the data of graphic buttons, text, and static images to the shared memory of its corresponding layer. The copy function is the system call function memcpy, with the source data being the graphic buttons, text, and static images drawn by the process, and the destination data being the shared memory of the layer. After the copy is completed, process A synchronously increments the counter corresponding to process A in the shared memory by one.
[0072] Process B is responsible for video playback, copying the video data to the shared memory of the layer corresponding to Process B. After the copy is completed, Process B synchronously increments the counter in the shared memory by one.
[0073] Similar to the video data copying process B, process C also completes its own data copying, incrementing the corresponding counter in the shared memory for image data synchronization control by one number.
[0074] The layer blending management process also monitors the layer counter. When it detects an increase in the counter, it uses the shared memory of the three layers as a parameter to call the JZFB_PUT_FRM_CFG instruction. This instruction allocates memory to the DMA address of the display device, thus achieving layer blending. After successful instruction execution, the counter is set to 0.
[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A shared memory based inter-process multi-layer image fusion control method, characterized in that, The method includes the following steps: S1, the kernel display driver reserves shared memory with contiguous physical addresses and in line with hardware alignment requirements for 2 to 4 layers; the function for reserving memory is the system function dma_alloc_coherent, with the size of the memory as the parameter; the memory reservation function is executed multiple times depending on the number of layers; S2, the user space creates a layer fusion management process, maps shared memory, and initializes the queue; that is... Create an image fusion management process, obtain the shared memory virtual address in user space through a memory mapping mechanism, and initialize the shared memory queue; The process creation process involves executing the corresponding binary file in the file system. The image fusion management process creates a shared memory handle using the system function shm_open, with the handle file path as the function parameter. Then, it requests shared memory using the system function mmap, with the size of the shared memory as the function parameter. The shared memory queue initialization process involves requesting a queue data structure and calling the system function memset for initialization, with 0 as the function parameter, to clear the queue. S3: Each process obtains shared memory through a memory mapping mechanism. In different processes, they can obtain operable memory addresses through the producer queue stored in shared memory to draw and update the graphics. After the update is completed, the memory address where the current operation is completed is stored in the consumer queue. S4, the fusion process merges data from various layers; that is... The image fusion management process reads the consumer queue in shared memory via virtual address for consumption, and performs image fusion on the updated memory from different process producers; further including: The layer blending management process is also responsible for monitoring the layer counter. When the counter is detected to be increasing, the shared memory of the three layers is used as a parameter to call the JZFB_PUT_FRM_CFG instruction. This instruction will allocate the memory to the DMA address of the display device to achieve layer blending. After the instruction is executed successfully, the counter is set to 0. If there are only two layers, then the two layers will be used as input parameters. The JZFB_PUT_FRM_CFG command supports a maximum of four layers as input parameters. S5 updates image data to the hardware device, that is... The image fusion result is input into the hardware display screen for display.
2. The shared memory based inter-process multi-layer image fusion control method according to claim 1, wherein, Step S3 further includes: Shared memory is obtained through a memory mapping mechanism, which involves creating a shared memory handle using the system function shm_open with the handle file path as the function parameter, and then requesting shared memory using the system function mmap with the size of the shared memory as the function parameter. The operable memory address is obtained by using the producer queue stored in shared memory. That is, after obtaining the shared memory through the system function mmap, the queue stores the index value of the memory array. The corresponding operable memory address can be calculated from the index value and the memory offset. FbAddr addr = mmap_header + index * sizeof(FbAddr), where the system function sizeof is used to calculate the memory size. The graph is drawn and updated. After the update is completed, the memory address where the current operation ends is stored in the consumer queue, which is the data copying implemented by calling the system function memcpy as mentioned above, and the counter is updated. Here, the consumer queue is the integer calculator.
3. The shared memory based inter-process multi-layer image fusion control method according to claim 2, wherein, In step S3, it is assumed that Process A draws and updates layer 1 of the image, pushing it into the consumer queue; Process B draws and updates layer 2 of the image, pushing it into the consumer queue; Process A is responsible for the interface controls, copying the data of graphic buttons, text, and static images to the shared memory of the corresponding layer of process A; the copy function is the system call function memcpy, the source data is the graphic buttons, text, and static images drawn by the process, and the destination data is the shared memory of the layer; after the copy is completed, process A will synchronously control the counter in the shared memory corresponding to process A to increment by 1 number. Process B is responsible for video playback and copies the video data to the shared memory of the layer corresponding to process B. After the copying process is complete, process B will increment the corresponding counter in the shared memory for image data synchronization by one number.
4. The shared memory based inter-process multi-layer image fusion control method according to claim 2, wherein, The method assumes three processes, A, B, and C. In step S1, the kernel displays that the three layers of the driver reserve shared memory; In step S2, the layer blending management process requests shared memory to be used for inter-process image data synchronization control; And establish counters for the three processes A, B, and C, with the counter type being integers.
5. The shared memory based inter-process multi-layer image fusion control method according to claim 1, wherein, Step S5 further includes: The JZFB_PUT_FRM_CFG instruction is invoked, which allocates memory to the DMA address of the display device. The display device then reads the image data through the DMA address and displays it on the hardware display screen.