Picture processing method and system for realizing binocular fusion, and storage medium

Abstract

The invention discloses a picture processing method and system for realizing binocular fusion and a storage medium, and the method comprises the steps: creating a drawing surface, drawing the content of a picture on the drawing surface, obtaining a picture layer formed after the drawing is completed, synthesizing a plurality of picture layers into a display texture, and splitting the display texture into a left eye image texture and a right eye image texture. And sending the spliced image to a frame buffer area, displaying the content of the frame buffer area, and finally realizing a binocular fusion effect. According to the application, an application end does not need to consider difference of image textures of two eyes, a complete image is used during drawing, drawing of different image textures of the left eye and the right eye is achieved through an SF, the workload of application development is reduced, a universal Android application can normally run on XR equipment without being modified, and construction and expansion of application ecology are facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of power supply control and management, and in particular to an image processing method, system and storage medium for realizing binocular fusion. Background Art

[0002] Due to the difference in the field of vision of the two eyes, in extended reality (XR) glasses, in order to simulate the field of vision of the two eyes, the XR glasses need to draw image textures of different fields of vision on the left and right screens, so that a complete image can be formed in the user's brain. This is binocular fusion.

[0003] In the existing technology, binocular fusion is achieved through the application side of the XR device, that is, the application depicts the image textures seen by the user's left and right eyes on the left and right screens respectively. This method requires that each application running on the XR device needs to process the drawing of the left and right eye image textures, which undoubtedly causes a lot of duplication of work when developing different APPs, and will also cause a large number of existing Android applications to fail to run and display normally on XR devices.

[0004] Therefore, it is urgent to propose a binocular fusion image processing method to solve the above problems existing in the prior art. Summary of the invention

[0005] The purpose of the present invention is to provide an image processing method, system and storage medium for realizing binocular fusion, so as to solve the problem that the application end needs to depict the texture of the left and right eye images during binocular fusion.

[0006] In order to solve the above technical problems, the present invention provides an image processing method for realizing binocular fusion, comprising:

[0007] Create a drawing surface;

[0008] Drawing the contents of the picture on the drawing surface;

[0009] Get the layer formed after drawing is completed;

[0010] compositing a plurality of said layers into a display texture;

[0011] Splitting the display texture into image textures for a left eye and a right eye, splicing them and sending them to a frame buffer; and

[0012] Display the content of the frame buffer.

[0013] Optionally, the content of the image includes at least one of graphics, text, and user interface elements.

[0014] Optionally, the process of creating a drawing surface includes: creating the drawing surface in a shared memory between an application end and a processing unit, and the processing unit includes SurfaceFlinger.

[0015] Optionally, the process of drawing the contents of the picture on the drawing surface includes:

[0016] Convert the collected drawing operations in the application end into a drawing operation list;

[0017] Synchronize the drawing operation list to the Render thread;

[0018] The Render thread merges and optimizes the drawing operation list;

[0019] The merged and optimized list of drawing operations is submitted to the GPU; and

[0020] The GPU performs the drawing work and generates new layers.

[0021] Optionally, the process of obtaining the layer formed after drawing includes:

[0022] The application side notifies the processing unit that a new layer has been generated;

[0023] The layer formed by the drawing is obtained through the shared memory.

[0024] Optionally, the process of synthesizing the multiple layers into a display texture includes: merging the multiple layers according to a display order and an overlay relationship.

[0025] Optionally, the process of sending the stitched images to the frame buffer includes: drawing the image texture of the left eye on the left side, and drawing the image texture of the right eye on the right side.

[0026] Optionally, the process of displaying the content of the frame buffer includes: displaying the content of the frame buffer on a physical display screen through a display driver.

[0027] The present application also provides a binocular fusion image processing system, which is used to implement the steps of the binocular fusion image processing method as described above, including an application end, a processing unit and a display output unit;

[0028] The application end includes a drawing surface creation unit and the image drawing unit, and the processing unit includes a layer acquisition unit, a layer synthesis unit and an image texture splitting unit;

[0029] The drawing surface creation unit is used to create a drawing surface;

[0030] The image drawing unit is used to draw the content of the picture on the drawing surface;

[0031] The layer acquisition unit is used to acquire the layer formed by drawing through the shared memory;

[0032] The layer synthesis unit is used to synthesize the multiple layers into a display texture;

[0033] The image texture splitting unit is used for splitting the display texture into image textures for a left eye and a right eye;

[0034] The display output unit is used to display the content of the frame buffer.

[0035] Optionally, the image drawing unit includes:

[0036] A drawing operation conversion unit, used to convert the collected drawing operations in the application end into a drawing operation list;

[0037] A drawing operation synchronization unit, used for synchronizing the drawing operation list to the Render thread;

[0038] A drawing operation merging and optimizing unit, used for merging and optimizing the drawing operation list;

[0039] A drawing operation submission unit, used to submit the merged and optimized drawing operation list to the GPU;

[0040] The GPU drawing unit is used to perform drawing work and generate new layers.

[0041] The present invention also provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are executed.

[0042] Compared with the prior art, the present invention has at least the following technical effects:

[0043] The present invention provides an image processing method, system and storage medium for realizing binocular fusion. Binocular fusion is realized through the bottom layer service SurfaceFlinger (hereinafter referred to as SurfaceFlinger) of the Android operating system. When the application side performs image rendering, it only needs to pay attention to the entire image scene observed by the person, and does not need to pay attention to the different image textures of the left and right eyes. After the image rendered by the application side is sent to SurfaceFlinger, SurfaceFlinger re-divides the entire image into texture areas for the left and right eyes, and then renders the image texture of the left eye to the left screen, and the image texture of the right eye to the right screen. Through the method provided by the present invention, the development of applications for XR devices can be consistent with that of ordinary Android devices. The development of applications does not need to consider the separate drawing of binocular image textures, which greatly reduces the workload and improves work efficiency. At the same time, existing general Android applications can also run and display normally on XR devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A block diagram of a method for processing an image to achieve binocular fusion in an embodiment of the present invention;

[0045] Figure 2 A complete image that needs to be displayed by an XR device in an embodiment of the present invention;

[0046] Figure 3 is the image after binocular fusion processing in the embodiment of the present invention;

[0047] Figure 4 The figure is a functional block diagram of a binocular fusion image processing system in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following is a description of an image processing method, system and storage medium for realizing binocular fusion of the present invention in conjunction with a schematic diagram, wherein a preferred embodiment of the present invention is shown, and it should be understood that those skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art, and not as a limitation of the present invention.

[0049] The present invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in illustrating the purpose of the embodiments of the present invention.

[0050] This embodiment provides an image processing method for achieving binocular fusion. Figure 1 ,include:

[0051] S1. Create a drawing surface;

[0052] S2, drawing the content of the picture on the drawing surface;

[0053] S3, obtaining the layer formed after drawing;

[0054] S4, synthesizing the plurality of layers into a display texture;

[0055] S5, splitting the display texture into image textures for a left eye and a right eye, splicing them and sending them to a frame buffer; and

[0056] S6. Displaying the content of the frame buffer.

[0057] In this embodiment, by splitting the display texture into image textures for the left eye and the right eye, and correctly stitching and displaying them, a stereoscopic visual effect can be achieved, the depth and stereoscopic sense of the image can be enhanced, and there is no need to pay attention to the complexity of binocular rendering on the application side. The application side can be developed in the same way as monocular images, which reduces the difficulty of development; since the application side does not need to process the separate drawing of binocular images, the development time can be reduced and the development efficiency can be improved; since the application side does not need to perform complex binocular rendering calculations, the requirements for the application side hardware can be reduced, so that more devices can support binocular fusion display. The method provided by this application allows existing general Android applications to run and display on XR devices without modification or with only a small amount of modification, which improves the compatibility of applications and the applicability of devices.

[0058] Furthermore, the content of the picture includes at least one of graphics, text and user interface elements.

[0059] The present embodiment can process various visual images, such as photos, illustrations, 3D model renderings, etc., which is essential for applications that require rich visual expressions. Supporting text ensures that the method can be applied to scenarios that require the display of text information, such as reading applications, educational software, office tools, etc. In addition, the inclusion of user interface elements enables the method to handle controls such as buttons, sliders, menus, etc., which is necessary for all graphical user interface applications.

[0060] Furthermore, in step S1, the process of creating the drawing surface includes: creating the drawing surface in a shared memory between the application end and the processing unit, and the processing unit includes SurfaceFlinger.

[0061] In this embodiment, through the shared memory mechanism, the image data can be efficiently shared and transmitted between the application end and the processing unit, thereby reducing resource consumption and improving data processing speed.

[0062] SurfaceFlinger is the underlying rendering service process in the Android system. The application draws images on the device's display screen through the service functions provided by SurfaceFlinger.

[0063] Furthermore, in a specific example, step S2 specifically includes:

[0064] S21, converting the collected drawing operations in the application end into a drawing operation list;

[0065] S22, synchronizing the drawing operation list to the Render thread;

[0066] S23, the Render thread merges and optimizes the drawing operation list;

[0067] S24, submitting the merged and optimized drawing operation list to the GPU; and

[0068] S25. GPU performs drawing work and generates a new layer.

[0069] Furthermore, in a specific example, step S3 specifically includes:

[0070] S31, the application end notifies the processing unit that a new layer has been generated;

[0071] S32, obtaining a layer formed by drawing through the shared memory.

[0072] In this embodiment, after the application completes the drawing of the layer, it sends a notification to SurfaceFlinger through the system-defined interface or event mechanism, informing that a new layer is ready and can be further processed. This ensures that the processing unit can be aware of the generation of the new layer in a timely manner, so that subsequent layer synthesis and display operations can be performed quickly. Through the notification mechanism, the operations between the application and the processing unit are more synchronized, avoiding the situation where the processing unit processes the layer before it is completed, reducing errors and resource waste.

[0073] After receiving the notification of the generation of a new layer, the processing unit accesses the layer data drawn by the application through the shared memory mechanism. Shared memory reduces the overhead of data replication because the layer data does not need to be transmitted multiple times between the application and the processing unit, thereby improving the overall rendering performance, reducing the bus load, and optimizing the use of system resources; in addition, by directly accessing the layer data in the shared memory, the processing unit can perform layer synthesis faster, speed up the display speed of the image, and improve the user experience.

[0074] Furthermore, the process of synthesizing the plurality of layers into a display texture includes: merging the plurality of layers according to a display order and an overlay relationship.

[0075] In this embodiment, by accurately controlling the display order and superposition relationship of the layers, the image processing module can provide a more realistic stereoscopic visual effect.

[0076] Specifically, in a specific example, step S4 includes the following process:

[0077] Layer sorting: Determine the visual order of each layer based on its Z-axis position (depth information), ensuring that the foreground layer covers the background layer.

[0078] Layer Merge: Merge each layer in a certain order.

[0079] Transparency processing: For layers with transparency or semi-transparency, their blending effect with the underlying layers needs to be processed to ensure that the transparency effect is displayed correctly.

[0080] Color correction and adjustment: During the merging process, adjust the color, brightness, contrast, etc. of the layer to ensure the visual consistency and quality of the final composite image.

[0081] Furthermore, the process of sending the stitched image to the frame buffer includes: drawing the left eye image texture on the left, and drawing the right eye image texture on the right. Please refer to the images before and after stitching. Figure 2 - Figure 3 shown.

[0082] After the display texture is synthesized, it is split according to the designed left and right eye views to generate two independent image textures, one for the user's left eye and the other for the user's right eye. The image texture for the left eye is accurately drawn in the corresponding left area of ​​the frame buffer to ensure that the left eye can only see this part of the image when it is finally viewed. Similarly, the image texture for the right eye is drawn in the corresponding right area of ​​the frame buffer to ensure that the right eye can only see this part of the image when it is viewed. In addition, ensuring that the left and right eye image textures are accurately aligned in the frame buffer to avoid visual misalignment or overlap is critical to providing a comfortable stereoscopic visual effect.

[0083] Furthermore, the process of displaying the content of the frame buffer includes: displaying the content of the frame buffer on a physical display screen through a display driver.

[0084] In this embodiment, during steps S1-S2, the image drawn by the application end is as follows: Figure 2As shown, the application side draws a complete image seen by both eyes of the user, and there is no need to consider the difference in texture between the two eyes. There is no need to start from step S1, and the application side processes the drawing of the left and right eye image textures; in this application, the difference in image texture caused by binocular vision is processed by SurfaceFlinger in the synthesis thread. SurfaceFlinger splits a complete image into left and right eye image textures in the synthesis thread, and sends them to FrameBuffer after splicing. Please refer to Figure 3 As shown, and finally displayed on the display screen, binocular fusion effect is achieved on the XR device.

[0085] Embodiment 2

[0086] This embodiment provides a binocular fusion image processing system, please refer to Figure 4 , used to implement the steps of the image processing method for realizing binocular fusion provided in Example 1, including an application end, a processing unit and a display output unit; the application end includes a drawing surface creation unit and the image drawing unit, and the processing unit includes a layer acquisition unit, a layer synthesis unit and an image texture splitting unit.

[0087] The drawing surface creation unit is used to create a drawing surface; the image drawing unit is used to draw the content of the picture on the drawing surface; the layer acquisition unit is used to acquire the layer formed by the drawing through the shared memory; the layer synthesis unit is used to synthesize multiple layers into a display texture; the image texture splitting unit is used to split the display texture into image textures for the left eye and the right eye; the display output unit is used to display the content of the frame buffer.

[0088] Furthermore, the image drawing unit includes:

[0089] A drawing operation conversion unit, used to convert the collected drawing operations in the application end into a drawing operation list;

[0090] A drawing operation synchronization unit, used for synchronizing the drawing operation list to the Render thread;

[0091] A drawing operation merging and optimizing unit, used for merging and optimizing the drawing operation list;

[0092] A drawing operation submission unit, used to submit the merged and optimized drawing operation list to the GPU;

[0093] The GPU drawing unit is used to perform drawing work and generate new layers.

[0094] Accordingly, other embodiments of the present application may also provide a computer-readable storage medium in which computer executable instructions are stored, and when the computer executable instructions are executed by the processor, the various method embodiments of the present application are implemented. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0095] In summary, the present invention provides an image processing method, system and storage medium for realizing binocular fusion. The application end does not need to consider the difference in image textures of both eyes. The application end uses the complete image when drawing, and SurfaceFlinger realizes the drawing of different image textures for the left and right eyes, which greatly reduces the workload of application development. In addition, general Android applications can run normally on XR devices without modification, which is conducive to the construction and expansion of the application ecosystem.

[0096] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for processing an image to achieve binocular fusion, characterized in that: include: Create a drawing surface; Drawing the contents of the picture on the drawing surface; Get the layer formed after drawing is completed; compositing a plurality of said layers into a display texture; Splitting the display texture into image textures for a left eye and a right eye, splicing them and sending them to a frame buffer; as well as Display the content of the frame buffer.

2. The image processing method for realizing binocular fusion according to claim 1, characterized in that: The content of the picture includes at least one of graphics, text and user interface elements.

3. The image processing method for realizing binocular fusion according to claim 1, characterized in that: The process of creating a drawing surface includes: creating the drawing surface through a shared memory between an application end and a processing unit, wherein the processing unit includes SurfaceFlinger.

4. The image processing method for realizing binocular fusion according to claim 3, characterized in that: The process of drawing the content of the picture on the drawing surface includes: Convert the collected drawing operations in the application end into a drawing operation list; Synchronize the drawing operation list to the Render thread; The Render thread merges and optimizes the drawing operation list; The merged and optimized list of drawing operations is submitted to the GPU; and The GPU performs the drawing work and generates new layers.

5. The image processing method for realizing binocular fusion according to claim 4, characterized in that: The process of obtaining the layer formed by drawing includes: The application side notifies the processing unit that a new layer has been generated; The layer formed by the drawing is obtained through the shared memory.

6. The image processing method for realizing binocular fusion according to claim 1, characterized in that: The process of synthesizing the plurality of layers into a display texture includes: merging the plurality of layers according to a display order and an overlay relationship.

7. The image processing method for realizing binocular fusion according to claim 1, characterized in that: The process of sending the spliced ​​textures into the frame buffer includes: drawing the image texture of the left eye on the left side, and drawing the image texture of the right eye on the right side.

8. The image processing method for realizing binocular fusion according to claim 1, characterized in that: The process of displaying the content of the frame buffer includes: displaying the content of the frame buffer on a physical display screen through a display driver.

9. A binocular fusion image processing system, characterized in that: The method for processing an image to realize binocular fusion according to any one of claims 1 to 8 comprises an application end, a processing unit and a display output unit; The application end includes a drawing surface creation unit and the image drawing unit, and the processing unit includes a layer acquisition unit, a layer synthesis unit and an image texture splitting unit; The drawing surface creation unit is used to create a drawing surface; The image drawing unit is used to draw the content of the picture on the drawing surface; The layer acquisition unit is used to acquire the layer formed by drawing through the shared memory; The layer synthesis unit is used to synthesize the multiple layers into a display texture; The image texture splitting unit is used for splitting the display texture into image textures for a left eye and a right eye; The display output unit is used to display the content of the frame buffer.

10. The binocular fusion image processing system according to claim 9, characterized in that: The image drawing unit comprises: A drawing operation conversion unit, used to convert the collected drawing operations in the application end into a drawing operation list; A drawing operation synchronization unit, used for synchronizing the drawing operation list to the Render thread; A drawing operation merging and optimizing unit, used for merging and optimizing the drawing operation list; A drawing operation submission unit, used to submit the merged and optimized drawing operation list to the GPU; The GPU drawing unit is used to perform drawing work and generate new layers.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are executed.

Images