Interprocess multilayer image fusion control method based on shared memory

Abstract

The invention provides an inter-process multi-layer image fusion control method based on a shared memory. The method comprises the steps that S1, a kernel display driver reserves a multi-layer shared memory, physical addresses are continuous, and hardware is aligned; s2, creating a layer fusion management process mapping shared memory in a user space, and initializing a queue; s3, each process obtains a shared memory through a memory mapping mechanism, in different processes, operable memory addresses can be obtained through producer queues stored in the shared memory, graph drawing and updating can be carried out, and after updating is finished, the memory address with the current operation finished is stored in a consumer queue; s4, fusing the data of each layer in the fusion process; and S5, updating the image data to hardware equipment. The multi-layer image function is fully utilized, one layer of image is processed in multiple processes, and then fusion display is carried out. Different processes only need to pay attention to a single layer to draw and update an image, and multi-layer fusion between the processes can be realized based on a queue of a shared memory.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an inter-process multi-layer image fusion control method based on shared memory. Background Art

[0002] With the development of modern embedded computer technology and the advent of the era of artificial intelligence, users have an increasingly strong demand for the integration of software application interfaces and video interfaces in various smart devices and terminal scenarios. The embedded interface not only needs to meet the user's visualization needs by changing interface elements and drawing application styles, but also needs to be able to smoothly play and display videos. In an embedded environment where hardware resources are the priority, the speed of interface rendering and response needs to be increased as much as possible to avoid interface freezes that reduce the user's experience. As embedded processors gradually develop into multi-core processors, the performance in a multi-process environment is getting better and better. By making full use of system characteristics, a smoother graphic display can be achieved.

[0003] However, existing technologies are usually not suitable for embedded applications, have high memory usage, or do not involve multi-layer fusion between processes. For example, using GPU for image layer fusion requires additional memory resources.

[0004] In addition, technical terms commonly used in this field include:

[0005] Process: A program instance running in an operating system with its own address space, which consists of the code executed by the storage system, storage variables, memory requested during program execution, and instructions for program execution.

[0006] Shared memory: One of the methods for data communication between multiple processes, allowing multiple processes to access the same address space, is the fastest method of inter-process communication. The use of shared memory can reduce consumption in large-scale data processing and is very suitable for graphics and image processing.

[0007] Hardware alignment: For logical design or performance reasons, hardware devices usually limit the physical memory to be read to byte alignment so that the hardware does not need to access the memory multiple times due to the need to cross byte boundaries when reading data, thereby greatly improving the data access speed.

[0008] Physical address: The address that the CPU uses to linearly number memory units in order to correctly store and read information when accessing memory. When the CPU needs to read or write data, it sends the physical address to the memory controller, and the memory controller performs corresponding operations on the physical memory based on the physical address. The physical address is also called the actual address of the memory.

[0009] Virtual address: In the operating system, the memory address mapped from the physical address is also called the logical address. The same physical address can correspond to multiple virtual addresses at the same time, and the mapping and management between them are handled by a dedicated memory management unit. Virtual addresses are provided to processes and kernel modules. When a process or kernel module calls the virtual memory application interface to apply for memory, the operating system allocates the corresponding memory to them, but does not occupy the physical memory immediately. Instead, it uses the specific physical memory only when the process or kernel module actually uses the memory. The virtual memory mechanism enhances the utilization of system memory.

[0010] Multi-layer image fusion: A commonly used image fusion technology that can fuse multiple images into one image to achieve better visual effects. The programming of multi-layer images facilitates users to separate information at different levels and can achieve smoother logical processing of graphic images.

[0011] Producer-consumer model: A classic concurrent programming model used to describe and solve the problem of cooperation between multiple threads or processes. The producer is responsible for generating data and putting the data into a shared buffer; the consumer is responsible for taking the data out of the shared buffer and processing it.

[0012] Queue: A common linear data structure in programming, with the characteristics of first-in-first-out, that is, there is a sequence relationship between the storage and retrieval of data in the queue, and the data stored in the queue will be taken out first. It is often used to solve the problem of the producer-consumer model. The producer puts the goods into the queue, and the consumer takes them out for consumption. The queue can effectively coordinate the speed difference between producers and consumers, avoid too many or too few goods, and ensure the stability and efficiency of the system. Memory mapping: A method of mapping file nodes in a computer operating system to the virtual address space of a process. In different processes, data information can be shared by memory mapping the same file node, that is, memory mapping is a way to implement shared memory. Summary of the invention

[0013] In order to solve the above problems, the purpose of this application is: the shared memory used in this application is divided into two parts, one part refers to the layer shared memory reserved by the kernel display driver, and the other part refers to the shared memory area applied for by the layer fusion management process for the synchronization control of image data between processes. This application makes full use of the multi-layer image function, processes one layer of image in multiple processes respectively, and then fuses and displays them. A method is provided so that different processes only need to focus on a single layer to draw and update the image, and multi-layer fusion between processes can be realized based on the shared memory queue.

[0014] Specifically, the present invention provides a method for controlling multi-layer image fusion between processes based on shared memory, the method comprising the following steps:

[0015] S1, the kernel display driver reserves multiple layers of shared memory, with continuous physical addresses and hardware alignment; the function for reserving memory is the system function dma_alloc_coherent, and the parameter is the size of the memory; the memory reservation function is executed multiple times according to the number of layers;

[0016] S2, the user space creates a layer fusion management process to map the shared memory and initialize the queue; that is, the image fusion management process is created, the shared memory virtual address is obtained through the memory mapping mechanism in the user space, and the shared memory queue is initialized;

[0017] The action of creating a process is to execute the binary file corresponding to the process in the file system. The image fusion management process will create a shared memory handle through the system function shm_open, and the function parameter is the handle file path, and then apply for shared memory through the system function mmap, and the function parameter is the shared memory size; the action of initializing the shared memory queue is to apply for a queue data structure, call the system function memset for initialization, and the function parameter is 0, which is used to clear the queue;

[0018] S3, each process obtains shared memory through the memory mapping mechanism. In different processes, the producer queue stored in the shared memory can obtain the operable memory address to draw and update the graphics. After the update is completed, the memory address where the current operation is completed is stored in the consumer queue;

[0019] S4, the fusion process fuses the data of each layer; that is, the image fusion management process reads the consumer queue in the shared memory through the virtual address for consumption, and performs image fusion on the memory updated by the producers of different processes;

[0020] S5, update the image data to the hardware device, i.e.

[0021] The fusion result of the image fusion is input to the hardware display screen for display.

[0022] The step S3 further comprises:

[0023] Get shared memory through the memory mapping mechanism, that is, create a shared memory handle through the system function shm_open, the function parameter is the handle file path, and then apply for shared memory through the system function mmap, the function parameter is the shared memory size;

[0024] Obtain the operable memory address through the producer queue stored in the shared memory. That is, after obtaining the shared memory through the system function mmap, the subscript value of the memory array is saved in the queue. The corresponding operable memory address FbAddr addr = mmap_header + index * sizeof (FbAddr) can be calculated through the memory offset through the subscript value. Here, the system function sizeof is used to calculate the memory size.

[0025] Draw and update the graphics. After the update is completed, the memory address where the current operation ends is stored in the consumer queue, that is, the system function memcpy is called as mentioned above to copy the data and update the counter. The consumer queue here is the integer calculator.

[0026] In step S3, it is assumed that

[0027] Process A draws the image and updates layer 1, pushing it into the consumer queue;

[0028] Process B draws the image and updates layer 2, pushing it into the consumer queue;

[0029] The A process is responsible for the interface control, and copies the data of the graphic buttons, texts, and static pictures to the layer shared memory corresponding to the A process; the copy function is the system call function memcpy, the source data is the graphic buttons, texts, and static pictures drawn by the process, and the destination data is the layer shared memory; after the copy is completed, the A process will synchronously control the image data to increase the counter corresponding to the A process in the shared memory by 1 number;

[0030] The B process is responsible for video playback and copies the video data to the layer shared memory corresponding to the B process; after the copy is completed, the B process synchronously controls the image data to increase the corresponding counter in the shared memory by 1 number.

[0031] In the method described, it is assumed that there are three processes A, B, and C.

[0032] In step S1, the kernel displays the three layers of shared memory reserved by the driver;

[0033] In step S2, the layer fusion management process applies for a shared memory for synchronous control of image data between processes;

[0034] And establish counters corresponding to the three processes A, B, and C, and the counter type is an integer number;

[0035] In step S3, the A process is responsible for the interface control, and copies the data of the graphic button, text, and static picture to the layer shared memory corresponding to the A process; the copy function is the system call function memcpy, the source data is the graphic button, text, and static picture drawn by the process, and the destination data is the layer shared memory; after the copy is completed, the A process synchronizes the image data to control the counter corresponding to the A process in the shared memory to increase by 1 number;

[0036] The B process is responsible for video playback and copies the video data to the layer shared memory corresponding to the B process. After the copy is completed, the B process synchronously controls the image data to increase the corresponding counter in the shared memory by 1.

[0037] The C process also completes its own data copying similar to the video data copying by the B process, and synchronously controls the image data to increase the corresponding counter in the shared memory by 1 number.

[0038] The step S4 further comprises:

[0039] The layer fusion management process is also responsible for monitoring the layer calculator. When the counter is detected to increase, the three layers of shared memory are used as parameters to call the JZFB_PUT_FRM_CFG instruction, which configures the memory to the DMA address of the display device to achieve layer fusion. After the instruction is successfully executed, the counter is set to 0.

[0040] If there are only two layers, input the two layers as parameters. The JZFB_PUT_FRM_CFG instruction supports up to four layers as parameter input.

[0041] The step S5 further comprises:

[0042] Call the JZFB_PUT_FRM_CFG instruction, which will configure the memory to the DMA address of the display device. The display device will read the image data through the DMA address and display it on the hardware display screen.

[0043] Therefore, the advantage of the present application is that: by applying the technical solution of the present application, only a single layer needs to be focused on to draw and update the image in different processes, and multi-layer fusion display between processes can be achieved based on a shared memory queue. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0045] Figure 1 is a schematic diagram of the process of this method.

[0046] Figure 2It is a flow chart of an embodiment of the method. DETAILED DESCRIPTION

[0047] In order to more clearly understand the technical content and advantages of the present invention, the present invention is now further described in detail in conjunction with the accompanying drawings.

[0048] This application provides a method to use shared memory between processes to control multi-layer image fusion. Its implementation logic can be referenced Figure 1 As shown, the description is as follows:

[0049] Execute step S1 to reserve multiple layers of shared memory in the kernel display driver, with continuous physical addresses and hardware alignment. The function for reserving memory is the system function dma_alloc_coherent, and the parameter is the size of the memory. The memory reservation function is executed multiple times according to the number of layers.

[0050] Execute step S2, create an image fusion management process, obtain the shared memory virtual address through the memory mapping mechanism in the user space, and initialize the shared memory queue. The action of creating a process is to execute the binary file corresponding to the process in the file system. The image fusion management process will create a shared memory handle through the system function shm_open, and the function parameter is the handle file path, and then apply for shared memory through the system function mmap, and the function parameter is the shared memory size. The action of initializing the shared memory queue is to apply for a queue data structure, call the system function memset for initialization, and the function parameter is 0, which is used to clear the queue.

[0051] Execute step S3, each process obtains shared memory through the memory mapping mechanism. In different processes, the producer queue stored in the shared memory can be used to obtain the operable memory address to draw and update the graphics. After the update is completed, the memory address where the current operation is completed is stored in the consumer queue.

[0052] Get shared memory through the memory mapping mechanism, that is, create a shared memory handle through the system function shm_open, the function parameter is the handle file path, and then apply for shared memory through the system function mmap, the function parameter is the shared memory size.

[0053] The operable memory address is obtained through the producer queue stored in the shared memory. That is, after the shared memory is obtained through the system function mmap, the index value of the memory array is saved in the queue. The corresponding operable memory address FbAddr addr = mmap_header + index * sizeof (FbAddr) can be calculated through the index value through the memory offset. The system function sizeof here is used to calculate the memory size.

[0054] Draw and update the graphics. After the update is completed, the memory address where the current operation ends is stored in the consumer queue, that is, the system function memcpy is called as mentioned above to copy the data and update the counter. The consumer queue here is the integer calculator.

[0055] For example, process A updates layer 1 by drawing an image and pushes it into the consumer queue; process B updates layer 2 by drawing an image and pushes it into the consumer queue.

[0056] Process A is responsible for the interface controls, and copies the data of graphic buttons, text, and static images to the layer shared memory corresponding to process A. The copy function is the system call function memcpy, the source data is the graphic buttons, text, and static images drawn by the process, and the destination data is the layer shared memory. After the copy is completed, process A synchronizes the image data to control the counter corresponding to process A in the shared memory and increases by 1.

[0057] Process B is responsible for video playback and copies the video data to the layer shared memory corresponding to process B. After the copy is completed, process B synchronizes the image data and controls the corresponding counter in the shared memory to increase by 1.

[0058] Execute step S4, the image fusion management process reads the consumer queue in the shared memory through the virtual address for consumption, and performs image fusion on the memory updated by producers from different processes.

[0059] The layer fusion management process is also responsible for monitoring the layer counter. When the counter is detected to increase, the three layers of shared memory are used as parameters to call the JZFB_PUT_FRM_CFG instruction, which will configure the memory to the DMA address of the display device to achieve layer fusion. After the instruction is successfully executed, the counter is set to 0:

[0060] If there are only two layers, input the two layers as parameters. The JZFB_PUT_FRM_CFG instruction supports up to four layers as parameter input.

[0061] Execute step S5 to input the fusion result of the image fusion into the hardware display screen for display, that is, call the JZFB_PUT_FRM_CFG instruction, which will configure the memory to the DMA address of the display device, and the display device will read the image data through the DMA address and display it on the hardware display screen.

[0062] To sum up, in short, Figure 2 As shown:

[0063] S1, the kernel display driver reserves multiple layers of shared memory, with continuous physical addresses and hardware alignment;

[0064] S2, the user space creates a layer fusion management process, maps shared memory, and initializes the queue;

[0065] S3, process A draws the image and updates layer 1, pushing it into the consumer queue;

[0066] Process B draws the image and updates layer 2, pushing it into the consumer queue;

[0067] S4, the fusion process fuses the data of each layer;

[0068] S5, updating the image data to the hardware device.

[0069] The shared memory used in this application is divided into two parts. One part refers to the layer shared memory reserved by the kernel display driver, and the other part refers to the shared memory area applied by the layer fusion management process for the synchronization control of image data between processes. This application makes full use of the multi-layer image function, processes one layer of image in multiple processes respectively, and then fuses and displays them. The specific process is as follows:

[0070] The following uses processes A, B, and C as an example to illustrate that the kernel display driver reserves three layers of shared memory. The layer fusion management process applies for shared memory responsible for synchronous control of image data between processes, and establishes counters corresponding to processes A, B, and C. The counter type is an integer.

[0071] Process A is responsible for the interface controls, and copies the data of graphic buttons, text, and static images to the layer shared memory corresponding to process A. The copy function is the system call function memcpy, the source data is the graphic buttons, text, and static images drawn by the process, and the destination data is the layer shared memory. After the copy is completed, process A synchronizes the image data to control the counter corresponding to process A in the shared memory and increases by 1.

[0072] Process B is responsible for video playback and copies the video data to the layer shared memory corresponding to process B. After the copy is completed, process B synchronizes the image data and controls the corresponding counter in the shared memory to increase by 1.

[0073] Similar to the video data copying of process B, process C also completes its own data copying and synchronously controls the corresponding counter in the shared memory to increase by 1.

[0074] The layer fusion management process is also responsible for monitoring the layer counter. When the counter is detected to increase, the three layers of shared memory are used as parameters to call the JZFB_PUT_FRM_CFG instruction, which will configure the memory to the DMA address of the display device to achieve layer fusion. After the instruction is successfully executed, the counter is set to 0.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A shared memory based multi-layer image fusion control method between processes, characterized in that: The method comprises the following steps: S1, the kernel display driver reserves multiple layers of shared memory, with continuous physical addresses and hardware alignment; the function for reserving memory is the system function dma_alloc_coherent, and the parameter is the size of the memory; the memory reservation function is executed multiple times according to the number of layers; S2, the user space creates a layer fusion management process to map the shared memory and initialize the queue; that is, the image fusion management process is created, the shared memory virtual address is obtained through the memory mapping mechanism in the user space, and the shared memory queue is initialized; The action of creating a process is to execute the binary file corresponding to the process in the file system. The image fusion management process will create a shared memory handle through the system function shm_open, and the function parameter is the handle file path, and then apply for shared memory through the system function mmap, and the function parameter is the shared memory size; the action of initializing the shared memory queue is to apply for a queue data structure, call the system function memset for initialization, and the function parameter is 0, which is used to clear the queue; S3, each process obtains shared memory through the memory mapping mechanism. In different processes, the producer queue stored in the shared memory can obtain the operable memory address to draw and update the graphics. After the update is completed, the memory address where the current operation is completed is stored in the consumer queue; S4, the fusion process fuses the data of each layer; The image fusion management process reads the consumer queue in the shared memory through the virtual address for consumption, and performs image fusion on the memory updated by producers from different processes; S5, update the image data to the hardware device, i.e. The fusion result of the image fusion is input to the hardware display screen for display.

2. The method for controlling multi-layer image fusion between processes based on shared memory according to claim 1, characterized in that: The step S3 further comprises: Get shared memory through the memory mapping mechanism, that is, create a shared memory handle through the system function shm_open, the function parameter is the handle file path, and then apply for shared memory through the system function mmap, the function parameter is the shared memory size; Obtain the operable memory address through the producer queue stored in the shared memory. That is, after obtaining the shared memory through the system function mmap, the subscript value of the memory array is saved in the queue. The corresponding operable memory address FbAddr addr = mmap_header + index * sizeof (FbAddr) can be calculated through the memory offset through the subscript value. Here, the system function sizeof is used to calculate the memory size. Draw and update the graphics. After the update is completed, the memory address where the current operation ends is stored in the consumer queue, that is, the system function memcpy is called as mentioned above to copy the data and update the counter. The consumer queue here is the integer calculator.

3. The method for controlling multi-layer image fusion between processes based on shared memory according to claim 2, characterized in that: In step S3, it is assumed that Process A draws the image and updates layer 1, pushing it into the consumer queue; Process B draws the image and updates layer 2, pushing it into the consumer queue; The A process is responsible for the interface control, and copies the data of the graphic buttons, texts, and static pictures to the layer shared memory corresponding to the A process; the copy function is the system call function memcpy, the source data is the graphic buttons, texts, and static pictures drawn by the process, and the destination data is the layer shared memory; after the copy is completed, the A process will synchronously control the image data to increase the counter corresponding to the A process in the shared memory by 1 number; The B process is responsible for video playback and copies the video data to the layer shared memory corresponding to the B process; After the copy is completed, process B will synchronize the image data and control the corresponding counter in the shared memory to increase by 1.

4. The method for controlling multi-layer image fusion between processes based on shared memory according to claim 2, characterized in that: In the method, it is assumed that there are three processes A, B, and C. Then, in step S1, the kernel displays the three layers of shared memory reserved by the driver; In step S2, the layer fusion management process applies for a shared memory for synchronous control of image data between processes; And establish counters corresponding to the three processes A, B, and C, and the counter type is an integer number; In step S3, the A process is responsible for the interface control, and copies the data of the graphic button, text, and static picture to the layer shared memory corresponding to the A process; the copy function is the system call function memcpy, the source data is the graphic button, text, and static picture drawn by the process, and the destination data is the layer shared memory; after the copy is completed, the A process synchronizes the image data to control the counter corresponding to the A process in the shared memory to increase by 1 number; The B process is responsible for video playback and copies the video data to the layer shared memory corresponding to the B process. After the copy is completed, the B process synchronously controls the image data to increase the corresponding counter in the shared memory by 1; The C process also completes its own data copying similar to the video data copying by the B process, and increases the corresponding counter in the shared memory by 1 by synchronously controlling the image data.

5. The method for controlling multi-layer image fusion between processes based on shared memory according to claim 4, characterized in that: The step S4 further comprises: The layer fusion management process is also responsible for monitoring the layer calculator. When the counter is detected to increase, the three layers of shared memory are used as parameters to call the JZFB_PUT_FRM_CFG instruction, which configures the memory to the DMA address of the display device to achieve layer fusion. After the instruction is successfully executed, the counter is set to 0. If there are only two layers, input the two layers as parameters. The JZFB_PUT_FRM_CFG instruction supports up to four layers as parameter input.

6. The method for controlling multi-layer image fusion between processes based on shared memory according to claim 5, characterized in that: The step S5 further comprises: Call the JZFB_PUT_FRM_CFG instruction, which will configure the memory to the DMA address of the display device. The display device will read the image data through the DMA address and display it on the hardware display screen.

Images