Image processing method and device and display equipment

Abstract

The invention relates to the technical field of multimedia post-processing, in particular to an image processing method and device and display equipment. The method comprises the following steps: when setting information sent by a split screen projection application is received, switching a video channel to a target channel; wherein the setting information comprises a coordinate set corresponding to a segmented image actually displayed by the display equipment in a frame image, the segmented image is a part of the frame image, and the target channel comprises any one of a multimedia channel, an analog signal channel and a high-definition multimedia interface channel; when starting information which is sent by the split screen projection application and is used for indicating starting of special effect processing is received, capturing a frame image from the target channel; performing special effect processing on the frame image to obtain a segmented image corresponding to the coordinate set on the frame image; wherein the special effect processing at least comprises image cutting; and rendering the segmented image, and displaying the rendered segmented image.

Description

Technical Field

[0001] The present disclosure relates to the technical field of multimedia post - processing, and in particular, to an image processing method, apparatus, and display device. Background Art

[0002] Currently, when a user needs to split - screen display a video image through multiple displays, an HDMI splitter can be used to connect multiple displays, so that each display can display a part of the video image. However, currently, when a user needs to split - screen display a video image through multiple displays, the user needs to process the original video image so that the display images respectively displayed on multiple displays can be combined into a complete video image.

[0003] It can be seen that the existing operation process of splitting - screen display a video image through multiple displays is complex, and the user experience is poor. Summary of the Invention

[0004] To solve the above - mentioned technical problems, the present disclosure provides an image processing method, apparatus, and display device.

[0005] The technical solution of the present disclosure is as follows:

[0006] In a first aspect, the present disclosure provides a display device, including: a display; an external device interface; and a processor; the external device interface is used to connect the display device to an external device, so that the display device body receives the multimedia playback content of the external device and presents it on the display; the processor is configured to: when receiving the setting information sent by the split - screen projection application, switch the video channel to the target channel; where the setting information includes a coordinate set corresponding to the split image actually displayed by the display device in the frame image, the split image is a part of the frame image, and the target channel includes any one of a multimedia channel, an analog signal channel, and a high - definition multimedia interface channel; when receiving the start information sent by the split - screen projection application for instructing to start special effect processing, grab the frame image from the target channel; perform special effect processing on the frame image to obtain the split image corresponding to the coordinate set on the frame image; where the special effect processing at least includes image cropping; render the split image to obtain the rendered split image, and control the display to display the rendered split image.

[0007] Second aspect, the present disclosure provides an image processing method, including: when receiving the setting information sent by the split screen projection application, switching the video channel to the target channel; wherein, the setting information includes the coordinate set corresponding to the split image actually displayed on the display device in the frame image, the split image is a part of the frame image, and the target channel includes any one of a multimedia channel, an analog signal channel, and a high-definition multimedia interface channel; when receiving the start information sent by the split screen projection application for indicating the start of special effect processing, capturing the frame image from the target channel; performing special effect processing on the frame image to obtain the split image corresponding to the coordinate set on the frame image; wherein, the special effect processing at least includes image cropping; rendering the split image to obtain the rendered split image, and displaying the rendered split image.

[0008] Third aspect, the present disclosure provides a display device, including: a memory and a processor, the memory is used for storing a computer program; the processor is used for enabling the display device to implement any one of the image processing methods provided in the first aspect above when executing the computer program.

[0009] Fourth aspect, the present disclosure provides a computer-readable storage medium, including: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a computing device, enabling the computing device to implement any one of the image processing methods provided in the first aspect above.

[0010] Fifth aspect, the present invention provides a computer program product, when the computer program product runs on a computer, enabling the computer to execute any one of the image processing methods provided in the first aspect.

[0011] It should be noted that the above computer instructions can be stored in whole or in part on the first computer-readable storage medium. Wherein, the first computer-readable storage medium can be packaged together with the processor of the image processing device, or can be separately packaged from the processor of the image processing device, and the present disclosure does not make any limitation in this regard.

[0012] The descriptions of the second aspect, the third aspect, the fourth aspect, and the fifth aspect in the present disclosure can refer to the detailed description of the first aspect; and, the beneficial effects of the descriptions of the second aspect, the third aspect, the fourth aspect, and the fifth aspect can refer to the analysis of the beneficial effects of the first aspect, and will not be elaborated here.

[0013] In the present disclosure, the names of the above image processing devices do not constitute a limitation to the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of the present disclosure, they fall within the scope of the claims of the present disclosure and their equivalent technologies.

[0014] These aspects or other aspects of the present disclosure will be more concise and understandable in the following description.

[0015] The technical solution provided by the present disclosure has the following advantages compared with the prior art:

[0016] For the display device provided by the embodiment of the present disclosure, when performing image segmentation display, each display device switches the video channel to the target channel when receiving the setting information sent by the segmentation screen mirroring application; when the display device receives the start information sent by the segmentation screen mirroring application for indicating the start of special effect processing, it captures a frame image from the target channel; the display device performs special effect processing on the frame image to obtain a segmented image corresponding to the coordinate set on the frame image; the display device renders the segmented image to obtain a rendered segmented image, and displays the rendered segmented image. In this way, each display device can display the segmented image corresponding to the frame image according to the setting information sent by the segmentation screen mirroring application. Furthermore, the rendered segmented images displayed by all display devices can form a complete frame image, eliminating the need for manual pre-setting of the frame images to be displayed for each display device, thus improving the user experience and solving the problems of complex operation processes for splitting and displaying video images on multiple displays and poor user interaction experience in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure and, together with the specification, are used to explain the principles of the present disclosure.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is a schematic flowchart of image processing in the prior art;

[0020] Figure 2 It is a schematic diagram of the image processing method provided by the embodiment of the present application applied to a segmentation display system;

[0021] Figure 3 It is one of the schematic structural diagrams of the display device in the image processing method provided by the embodiment of the present application;

[0022] Figure 4 It is the other schematic structural diagram of the display device in the image processing method provided by the embodiment of the present application;

[0023] Figure 5 It is one of the schematic flowcharts of the image processing method provided by the embodiment of the present application;

[0024] Figure 6It is the second flowchart of the image processing method provided by the embodiment of the present application;

[0025] Figure 7 It is the third flowchart of the image processing method provided by the embodiment of the present application;

[0026] Figure 8 It is the structural schematic diagram of the display device provided by the embodiment of the present application;

[0027] Figure 9 It is the schematic diagram of a chip system provided by the embodiment of the present application;

[0028] Figure 10 It is the structural schematic diagram of a computer program product of an image processing method provided by the embodiment of the present disclosure. Detailed implementation manners

[0029] In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0030] Many specific details are set forth in the following description in order to fully understand the present disclosure, but the present disclosure may be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all the embodiments.

[0031] The display device provided by the embodiment of the present application may have various implementation forms. For example, it may be a television, a smart TV, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 3 It is a specific implementation manner of the display device of the present application.

[0032] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, the element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0033] In some examples, the Surface in the embodiments of the present disclosure is a basic component for graphics rendering in the Android system. An application can draw image content or display views through the Surface.

[0034] In some examples, the OSD layer image in the embodiments of the present disclosure generally corresponds to the image drawn by the user interface (UI), and has no picture quality effect.

[0035] In some examples, the EGL in the embodiments of the present disclosure is an intermediate interface layer between the OpenGL ES rendering API and the native window system.

[0036] In some examples, the Framebuffer in the embodiments of the present disclosure uses a video output device to drive a video display device from a memory buffer containing complete frame data.

[0037] In some examples, the prior art image processing method is as Figure 1 shown, including an OSD layer, a VIDEO layer, an image processing module, a display processing module, and a screen display. Among them, the OSD layer generally corresponds to the image drawn by the UI and has no picture quality effect, while the VIDEO layer corresponds to the video images played under channels such as multimedia (MM), DTV, and HDMI, and PQ picture quality processing can be performed on it, such as SNR (signal-to-noise ratio), TNR (temporal noise reduction), ACM (adaptive contrast), etc., to display clearer and higher-quality images to the user; after being scaled to the display resolution size through a Scalar and then passing through the display processing module, it is superimposed with the image of the OSD layer, and the superimposed signal is transmitted to the screen end for screen display. For the display device using the above solution, only the image displayed on the OSD layer can be filtered, and the customized feature processing of the user cannot be realized, resulting in a poor user experience. Therefore, the embodiments of the present disclosure provide an image processing method to solve the above problems, and the specific implementation process is as follows:

[0038] Figure 2 It is a schematic diagram of the scenario of a display device according to one or more embodiments of the present application. As Figure 2 shown, the user can control at least two display devices to display the same video signal through the electronic device 1. The display device and the electronic device 1 can establish a communication connection in a wired or wireless manner. When connected by wire, it can be connected through the High Definition Multimedia Interface (HDMI) 3.

[0039] Exemplarily, taking the display device as 2 for example, namely display device 2-1 and display device 2-2, the electronic device 1 is connected to the external device interface (such as: HDMI interface) of the display device 2-1 through an HDMI cable, and at the same time, the electronic device 1 is connected to the external device interface (such as: HDMI interface) of the display device 2-2 through an HDMI cable. In this way, the electronic device 1 can transmit the same 2 multimedia signals to the display device 2-1 and the display device 2-2. Then, the display device 2-1 and the display device 2-2 project their respective display screens onto the display background 4. As Figure 2 shown, since both the display device 2-1 and the display device 2-2 display the same multimedia signal, in order to make the displayed picture larger, the projected pictures of the display device 2-1 and the display device 2-2 can be spliced, so as to obtain the effect as shown in Figure 2 shown. In order to ensure the integrity of the display picture, usually the display pictures of the display device 2-1 and the display device 2-2 need to be spliced, so that display pictures with different display ratios (such as: 16:9, etc.) can be obtained.

[0040] In some embodiments, when the user needs to split and display a frame image through multiple display devices, at this time, the user can set the number of display devices that need to be split and displayed on the split screen projection application. Then, by setting the display area of each display device in the split display (for example, when the number of display devices that need to be split and displayed is 2, at this time, the display area of the display device in the split display can be set to include any one of the left side and the right side, or when the number of display devices that need to be split and displayed is 4, at this time, the display area of the display device in the split display can be set to include any one of the upper left side, the lower left side, the upper right side and the lower right side) and the video resolution in the display area, the setting information of each display device (at least including the display area and the video resolution) can be obtained. At this time, after the split screen projection application obtains the setting information of each display device, it sends the setting information to the corresponding display device. When the display device receives the setting information sent by the split screen projection application, it switches the video channel to the target channel; wherein, the setting information includes the coordinate set corresponding to the split image actually displayed by the display device in the frame image, the split image is a part of the frame image, and the target channel includes any one of the multimedia channel, the analog signal channel and the high-definition multimedia interface channel. Then, when the user can perform a start operation for indicating the start of special effect processing on the split display application, when the display device receives the start information for indicating the start of special effect processing sent by the split screen projection application, it grabs the frame image from the target channel; the display device performs special effect processing on the frame image to obtain the split image corresponding to the coordinate set on the frame image; wherein, the special effect processing at least includes image cropping; the display device renders the split image to obtain the rendered split image, and displays the rendered split image.

[0041] In some examples, the display device may be a laser TV or a projector, etc., which are devices that can be used for projection display.

[0042] It can be seen that in the image processing method provided by the embodiments of the present disclosure, when performing image segmentation display, each display device switches the video channel to the target channel when receiving the setting information sent by the segmentation screen mirroring application; when the display device receives the start information sent by the segmentation screen mirroring application for instructing to start special effect processing, it grabs a frame image from the target channel; the display device performs special effect processing on the frame image to obtain a segmented image corresponding to the coordinate set on the frame image; the display device renders the segmented image to obtain a rendered segmented image, and displays the rendered segmented image. In this way, each display device can display the segmented image corresponding to the frame image according to the setting information sent by the segmentation screen mirroring application. Furthermore, the rendered segmented images displayed by all the display devices can form a complete frame image, without the need for manual pre-setting of the frame images to be displayed for each display device, thereby improving the user experience.

[0043] Figure 3 Shows a block diagram of the hardware configuration of a display device according to an exemplary embodiment. As Figure 3 shown, the display device includes at least one of a tuner demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first interface to an nth interface for input / output. The display 260 may be a display with touch function, such as a touch display. The tuner demodulator 210 receives broadcast television signals by wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as EPG data signals. The detector 230 is used to collect signals of the external environment or interact with the outside. The controller 250 and the tuner demodulator 210 may be located in different split devices, that is, the tuner demodulator 210 may also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0044] In some examples, the external device interface 240 may include, but is not limited to, the following: a high-definition multimedia interface (HDMI), an analog or data high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc., any one or more interfaces. It may also be a composite input / output interface formed by the above multiple interfaces.

[0045] In some examples, the display 260 can be a liquid crystal display, an OLED display, and a projection display, and can also be a projection device and a projection screen. For example, when the display device is a projector, the display 260 can be a projection screen, or when the display device is a laser TV, the display 260 can be a projection display.

[0046] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device.

[0047] In some examples, taking the display device of one or more embodiments as the projector 1 and the operating system of the projector 1 as the Android system as an example, as Figure 4 shown, the projector 1 can be logically divided into an applications layer (referred to as the "application layer") 21, an application framework layer (referred to as the "framework layer") 22, an Android runtime and system library layer (referred to as the "system runtime library layer") 23, and a kernel layer 24.

[0048] Among them, the application layer 21 includes one or more applications. The application can be a system application or a third-party application. For example, the application layer 21 includes a first application, and the first application can execute the image processing method provided by the embodiments of the present disclosure. The framework layer 22 provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer 21. The system runtime library layer 23 provides support for the upper layer, i.e., the framework layer 22. When the framework layer 22 is used, the Android operating system will run the C / C++ libraries included in the system runtime library layer 23 to implement the functions to be implemented by the framework layer 22. The kernel layer 24 serves as software middleware between the hardware layer and the application layer 21, and is used to manage and control hardware and software resources.

[0049] In some examples, when a user needs to split and display a frame image through multiple projectors 1, the user can set the number of projectors 1 that need to be split and displayed on the split screen mirroring application. Then, by setting the position of each projector 1 in the split display, the setting information of each projector 1 is obtained. At this time, after the split screen mirroring application obtains the setting information of each projector 1, it sends the setting information to the corresponding projector 1. When the processing unit 201 of the first application of the projector 1 receives the setting information sent by the split screen mirroring application through the receiving unit 202, it controls the VIDEO layer of the framework layer 22 through the OSD layer of the framework layer 22 to switch the video channel to the target channel; the image quality processing module of the framework layer 22 preprocesses the image frame of the target channel, such as the preprocessing includes PQ processing, Scalar processing, etc., and inputs the preprocessed frame image into the display processing module of the framework layer 22. The display processing module of the framework layer 22 inputs the preprocessed frame image into the background display module of the framework layer 22 for background display. During this period, when the user performs a start operation on the split display application to indicate the start of special effect processing, when the processing unit 201 of the projector 1 receives the start information sent by the split screen mirroring application indicating the start of special effect processing through the receiving unit 202, the processing unit 201 of the projector controls the frame image capture module of the VIDEO layer of the image processing module of the framework layer 22 to capture a frame image from the target channel through the VIDEO layer graphic recording of the image processing module of the framework layer 22 (for example: copying (or image recording) the preprocessed frame image output by the image quality processing module of the framework layer 22, so as to complete capturing the frame image from the target channel); then, the processing unit 201 of the first application controls the VIDEO layer graphic recording of the image processing module of the framework layer 22 to buffer the frame image captured by the frame image capture module of the VIDEO layer of the framework layer 22 into the image frame buffer queue of the image processing module of the framework layer 22. Then, the yuvfilter of the image processing module of the framework layer 22 reads the buffered frame image in the image frame buffer queue and transfers the read frame image to the customfilter of the image processing module of the framework layer 22. The processing unit 201 of the first application of the projector 1 controls the customfilter of the image processing module of the framework layer 22 to perform special effect processing on the frame image to obtain a split image corresponding to the coordinate set on the frame image; the processing unit 201 of the first application of the projector 1 controls the framebuffer of the image processing module of the framework layer 22 to render the split image output by the customfilter to obtain a rendered frame image. Then, the processing unit 201 of the first application of the projector 1 transfers the rendered frame image to the foreground display module of the framework layer 22 through the EGL of the image processing module of the framework layer 22, so that the foreground display module can control the display unit 203 to display the rendered split image in the display area.

[0050] In some examples, the OSD layer is used to provide a fusion-processed surface, which can be processed with different special effects based on different user needs. The split screen application provides a Surface for graphic display after the final special effect processing, and switches to the VIDEO channel to be processed in the background, such as the HDMI channel. Customized image special effects processing is performed through the "special effects processing module", such as the fusion display scene of multiple display devices.

[0051] In some examples, the target channel includes any one of: Multimedia (MM), Digital Television (DTV), and HDMI.

[0052] In some examples, the VIDEO layer capture module is used to perform image recording services according to the setting parameters set by the user (such as video resolution, frame rate, etc.).

[0053] In some examples, VIDEO layer graphics recording: mainly used to start the image recording service of the VIDEO layer, capture the frame image of the VIDEO layer based on the set parameters (such as video resolution, frame rate, etc.), that is, capture the frame image from the target channel.

[0054] In some examples, the image frame buffer queue mainly serves as a buffer for buffering the graphics captured by the VIDEO layer graphics recording and handing them over to the split screen projection application (ie, the graphics consumer) for special effects processing.

[0055] In some examples, OPENGLES image processing: receiving the frame image captured from the target channel stored in the external captured image frame buffer queue through yuvfilter, such as the frame image can be a YUV420 image, customfilter performs image special effects processing on the YUV420 image, such as adding target effects (such as vertigo), adding stickers, text, image fusion, etc. Finally, the processed YUV420 image is stored in the framebuffer.

[0056] In some examples, EGL is used to bind the Surface set by the split screen projection application, and the frame image processed by OPENGLES is finally displayed on the Surface. Finally, the image processed with special effects can be displayed through the foreground window, that is, the OSD layer, but the picture displayed by the OSD layer includes the image quality processing effect of the VIDEO layer on the one hand, and the customized image special effect processing effect of the application on the other hand.

[0057] In some examples, the special effect processing includes one or more of image cropping and image fusion. It can be seen that the difference from the traditional solution is that in the image processing method provided by the embodiments of the present invention, a VIDEO layer screenshot module is added to the VIDEO layer path, and this VIDEO layer screenshot module can also retain the effect of image quality processing at the same time. The main function of the VIDEO layer screenshot module is to capture images based on parameters such as the resolution and frame rate set by the user, and provide an image recording service function. This function is relatively single and relatively easy to implement. And the more flexible image special effect processing module is controlled by the split screen projection application, and the screenshot module is only used as the input of the special effect processing module, that is, the graphics producer.

[0058] Specifically, the storage unit 204 of the projector 1 is used to store data such as the computer program of the image processing method provided by the embodiments of the present disclosure.

[0059] In the following embodiments, taking the execution subject of the image processing method provided by the embodiments of the present disclosure as the above display device, and the display device being the projector 1 as an example, the method of the embodiments of the present application will be described.

[0060] The embodiments of the present application provide an image processing method, as Figure 5 shown, this image processing method may include S11 - S14.

[0061] S11. When receiving the setting information sent by the split screen projection application, switch the video channel to the target channel; wherein, the setting information includes the coordinate set corresponding to the split image actually displayed by the display device in the frame image, the split image is a part of the frame image, and the target channel includes any one of the multimedia channel, the analog signal channel, and the high - definition multimedia interface channel.

[0062] In some examples, the projector 1 can receive the setting information sent by the split screen projection application, as well as the start information for instructing the start of special effect processing sent by the split screen projection application, through the external device interface 240, such as a high - definition multimedia interface (HDMI).

[0063] In some examples, the target channel is a channel for transmitting the frame image that the user needs to perform special effect processing on, such as: any one of the multimedia channel, the analog signal channel, and the high - definition multimedia interface channel.

[0064] In some examples, the OSD layer image generally corresponds to the image drawn by the UI and has no picture quality effect, while the VIDEO layer image corresponds to the video images played under channels such as MM, DTV, and HDMI, and PQ picture quality processing can be performed on it, such as Signal to Interference plus Noise Ratio (SNR), Temporal Noise Reduction (TNR), Auto Colour Management (ACM), etc., to display clearer and higher-quality images to the user; after being scaled to the display resolution size by the Scalar and then passing through the display processing module, it is superimposed with the OSD layer image, and the superimposed signal is transmitted to the screen end for display.

[0065] In some examples, the user can set the number of split screen mirroring required on the display interface of the electronic device 1, such as: 2, 4, 6, 8, etc. Then, the user can set the overlapping ratio of the pictures displayed on each display device within the display area, that is, the proportion of the overlapping displayed image in the split image actually displayed by the display device. For example, when the number of split screen mirroring is 2, assume that the split image displayed by the display device 1 on the left shows the left side of the frame image The split image displayed by the display device 2 on the right shows the right side of the frame image At this time, the overlapping ratio of both the display device 1 and the display device 2 is 50%.

[0066] In some examples, assume that the length of the frame image is 640 and the width is 480. Taking the lower left corner of the frame image as the origin to establish a rectangular coordinate system, when the number of split screen mirroring is 2, assume that the split image displayed by the display device 1 on the left shows the left side of the frame image The four vertices of the split image actually displayed by the display device 1 in the frame image are (0, 0), (0, 640), (320 0), (320, 640). Then, the coordinate set corresponding to the split image displayed by the display device 1 is the set of coordinate points enclosed within the area surrounded by the four vertices (0, 0), (0, 640), (320, 0), and (320, 640). The four vertices of the split image actually displayed by the display device 2 in the frame image are (160 (0), (480, 160), (160, 640), (480, 640), the set of coordinates corresponding to the segmented image displayed by the display device 2 is the set of coordinate points enclosed within the region surrounded by the four vertices (160, 0), (480, 160), (160, 640), (480, 640). Thus, it can be seen that the four vertices of the images overlappingly displayed by the display device 1 and the display device 2 are (160, 0), (320, 0), (160, 640), (320, 640). The ratio of the area of the image with the four vertices (160, 0), (320, 0), (160, 640), (320, 640) to the area of the segmented image actually displayed by the display device 1 is 50%, that is, the overlapping ratio of the display device 1 is 50%. At the same time, the ratio of the area of the image with the four vertices (160, 0), (320, 0), (160, 640), (320, 640) to the area of the segmented image actually displayed by the display device 2 is 50%, that is, the overlapping ratio of the display device 2 is 50%.

[0067] In some examples, the length of the frame image refers to the number of pixels from top to bottom of the picture, and the width of the frame image refers to the number of pixels from left to right of the picture.

[0068] In some examples, the overlapping ratio is greater than or equal to 0.5 and less than or equal to 1.

[0069] S12. When receiving the start information sent by the split screen mirroring application indicating the start of special effect processing, capture the frame image from the target channel.

[0070] In some examples, when capturing the frame image from the target channel upon receiving the start information sent by the split screen mirroring application indicating the start of special effect processing, the data stream of the target channel can be copied (such as image recording) to obtain the data stream of the target channel. Then, the data stream of the target channel is parsed to obtain the corresponding frame image.

[0071] S13. Perform special effect processing on the frame image to obtain the segmented image corresponding to the coordinate set on the frame image; wherein, the special effect processing at least includes image cropping.

[0072] In some examples, a rectangular coordinate system can be established with the lower left corner of the frame image as the origin. Then, when performing image cropping on the frame image, according to the coordinate set of the display device, the region image corresponding to the coordinate set on the frame image can be determined in the rectangular coordinate system, and the region image in the frame image is cropped, and the region image is used as the segmented image corresponding to the coordinate set on the frame image.

[0073] In some examples, since the image size of the actual frame image is different from the video resolution set by the user, it is necessary to crop the frame image so that the image size of the segmented image is the same as the video resolution set by the user.

[0074] In some examples, in order to make the image ratio of the rendered segmented image more diverse, the special effect processing also includes image fusion. In this way, by splicing the images output by the projector, the image ratio of the spliced image can be changed. However, when splicing the images output by the projector, in order to prevent the problem of incomplete display, it is necessary to overlap the images output by the projector to ensure the integrity of the display. However, for the overlapping part of the images output by the projector, image fusion is required to make the display effect of the overlapping part of the image consistent with that of the non-overlapping part. For this reason, the set parameters also include the overlapping ratio.

[0075] Exemplarily, taking the upper left corner of the frame image as the origin, a rectangular coordinate system is established. The coordinates of the four vertices of the frame image are (0, 0), (0, 1), (1, 0), and (1, 1) respectively. When the number of split screen projections is 2 and the overlapping ratio is 0.6, at this time, the non-image fusion area (x_left, y_left, w_left, h_left) of the image output by the left projector can be determined. x_left is equal to 0, y_right is equal to 0, w_left is equal to x represents the overlapping ratio, and h_left is equal to 1. The non-image fusion area (x_right, y_right, w_right, h_right) of the image output by the right projector, x_right is equal to y_right is equal to 0, w_right is equal to h_right is equal to 1. Then, for the area that needs image fusion, adjust the image parameters of the image (such as brightness, contrast, etc.) to make the display effect of the overlapping part of the image consistent with that of the non-overlapping part.

[0076] S14. Render the segmented image to obtain the rendered segmented image and display the rendered segmented image.

[0077] In some examples, the display device needs to have the ability to record images in the VIDEO layer so that the split screen projection application can control the recording of the frame images in the VIDEO layer in the OSD layer, and then perform special effect processing on the recorded frame images in the VIDEO layer. The split screen projection application provides an entry for user parameter settings, such as the current terminal video display area, the resolution, frame rate, and stickers for image special effect processing of the video recording in the VIDEO layer, and provides the Surface that needs to be finally displayed after the graphic special effect processing.

[0078] Split the channel where the video to be processed with special effects for the screen mirroring application switch is located, such as the HDMI channel, and perform the sound muting operation. In this way, although the content of the HDMI channel is already playing in the background, since the application interface is displayed in the foreground, the user is unaware. When the user clicks Start, perform the special effects processing and jump to a new page to display the graph after the special effects processing.

[0079] First, start the graphic reading thread for image recording, open the "image recording service of the VIDEO layer", loop to obtain the video data for image recording based on the frame rate set by the user, and put it into the image frame buffer queue, and send a frame available signal. For example, if the frame rate set by the user is 25fps, the graphic data is read every 40ms.

[0080] Second, start the graphic rendering thread for image recording, determine whether there is YUV data in the image frame buffer queue. If not, wait for the frame available signal sent by the graphic reading thread for image recording. Otherwise, after receiving the signal, read the yuv data in the image frame buffer queue and perform customized special effects processing based on the opengles programmable pipeline. At the same time, determine whether it is the first frame to be rendered. If so, cancel the sound muting operation.

[0081] Finally, output the processed frame image to the Surface provided by the split screen mirroring application through EGL.

[0082] Through the above steps, the graphic data in the graphic frame buffer queue can be cyclically read, and customized special effects processing can be performed, and finally displayed on the screen.

[0083] As can be seen from the above, in the image processing method provided by the embodiments of the present disclosure, when performing image segmentation display, each display device switches the video channel to the target channel when receiving the setting information sent by the split screen mirroring application; when the display device receives the start information sent by the split screen mirroring application for instructing the start of special effects processing, grabs the frame image from the target channel; the display device performs special effects processing on the frame image to obtain the segmented image corresponding to the coordinate set on the frame image; the display device renders the segmented image and displays the rendered segmented image in the display area. In this way, each display device can display the segmented image corresponding to the frame image in the corresponding display area according to the setting information sent by the split screen mirroring application, and then the rendered segmented images displayed by all display devices can form a complete frame image, without the need for manual pre-setting of the frame images to be displayed for each display device, thereby improving the user experience.

[0084] In some feasible examples, in combination with Figure 5 , such as Figure 6 shown, the image processing method adopted by the embodiments of the present disclosure further includes S15, and the above S14 can be specifically implemented by the following S140.

[0085] S15. When receiving the setting information sent by the split screen projection application, switch the video channel to the target channel and mute the output audio of the target channel;

[0086] S140. Render the split image. When it is determined that the split image is the first image frame, unmute the output audio of the target channel and display the rendered split image.

[0087] In some implementable examples, the special effect processing further includes image fusion and special effect addition. The special effect addition includes adding a target animation effect and adding a target special effect. The target special effect includes one or more of stickers and text.

[0088] In some implementable examples, in combination Figure 5 , such as Figure 7 shown, the above S14 can be specifically implemented by the following S141 and S142.

[0089] S141. Bind the split image to the Surface corresponding to the display area, and render the split image through the Surface to obtain the rendered split image;

[0090] S142. Display the rendered split image in the Surface.

[0091] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of methods. To implement the above functions, it includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0092] The embodiments of the present application can divide the function modules of the image processing device according to the above method examples. For example, each function module can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated modules can be implemented in the form of hardware or in the form of software function modules. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0093] Such as Figure 8As shown in the figure, an embodiment of the present application provides a schematic structural diagram of a display device, including an external device interface 101, a processor 102, and a display 103.

[0094] The display 103;

[0095] The external device interface 101 is used to connect the display device to an external device, so that the display device body receives the multimedia playback content of the external device and presents it on the display;

[0096] And the processor 102;

[0097] The processor 102 is configured to:

[0098] When receiving the setting information sent by the split screen mirroring application, switch the video channel to the target channel; where the setting information includes the coordinate set corresponding to the split image actually displayed by the display device in the frame image, the split image is a part of the frame image, and the target channel includes any one of the multimedia channel, the analog signal channel, and the high-definition multimedia interface channel;

[0099] When receiving the start information sent by the split screen mirroring application for indicating the start of special effect processing, capture the frame image from the target channel;

[0100] Perform special effect processing on the frame image to obtain the split image corresponding to the coordinate set on the frame image; where the special effect processing at least includes image cropping;

[0101] Render the split image to obtain the rendered split image, and control the display 103 to display the rendered split image.

[0102] In some feasible examples, the processor 102 is further configured to:

[0103] When receiving the setting information, switch the video channel to the target channel and perform a mute operation on the output audio of the target channel;

[0104] When rendering the split image and determining that the split image is the first image frame, cancel the mute operation on the output audio of the target channel, and control the display 103 to display the rendered split image.

[0105] In some feasible examples, the special effect processing further includes image fusion and special effect addition, and the special effect addition includes adding a target animation effect and adding a target special effect, and the target special effect includes one or more of stickers and text.

[0106] In some feasible examples, the processor 102 is further configured to:

[0107] Bind the Surface corresponding to the segmented image to the display area, and render the segmented image through the Surface to obtain the rendered segmented image; control the display 103 to display the rendered segmented image in the Surface.

[0108] Among them, all relevant contents of each step involved in the above method embodiments can be cited in the function descriptions of the corresponding functional modules, and their functions will not be elaborated here.

[0109] Of course, the display device provided in the embodiments of the present application includes but is not limited to the above modules. For example, the display device may further include a memory 104. The memory 104 can be used to store the program code of the display device and can also be used to store the data generated during the operation of the display device, such as the data in the write request, etc.

[0110] As an example, in combination with Figure 4 , the functions implemented by the acquisition unit 202 in the projector 1 are the same as those of the external device interface 101, the functions implemented by the processing unit 201 are the same as those of the processor 102, the functions implemented by the display unit 203 are the same as those of the display 103, and the functions implemented by the storage unit 204 are the same as those of the memory 104.

[0111] Figure 9 As shown, the embodiments of the present application further provide a chip system, which can be applied to the display device in the foregoing embodiments. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 can be the processor in the above display device. The processor 1501 and the interface circuit 1502 can be interconnected by a line. The processor 1501 can receive and execute computer instructions from the memory of the above display device through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the display device can execute each step executed by the display device in the above embodiments. Of course, the chip system may further include other discrete devices, and the embodiments of the present application do not make specific limitations on this.

[0112] The embodiments of the present application further provide a computer-readable storage medium for storing the computer instructions for the operation of the above display device.

[0113] As Figure 10 shown, the embodiments of the present application further provide a computer program product including the computer instructions for the operation of the above display device.

[0114] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A display device, characterized in that: include: monitor; An external device interface, used to connect the display device with an external device, so that the display device body receives multimedia playback content from the external device and presents it on the display; and processors; The processor is configured to: Upon receiving the setting information sent by the split screen projection application, switching the video channel to the target channel; wherein the setting information includes a coordinate set corresponding to the split image actually displayed by the display device in the frame image, the split image is a part of the frame image, and the target channel includes any one of a multimedia channel, an analog signal channel, and a high-definition multimedia interface channel; When receiving the start information sent by the split screen projection application for instructing to start special effect processing, capturing a frame image from the target channel; Performing special effect processing on the frame image to obtain a segmented image corresponding to the coordinate set on the frame image; wherein the special effect processing at least includes image cropping; The segmented image is rendered to obtain the rendered segmented image, and the display is controlled to display the rendered segmented image.

2. The display device according to claim 1, characterized in that The processor is further configured to: When receiving the setting information sent by the split screen projection application, the video channel is switched to the target channel, and the output audio of the target channel is muted; The segmented image is rendered, and when it is determined that the segmented image is the first image frame, a muting operation is canceled on the output audio of the target channel, and the display is controlled to display the rendered segmented image.

3. The display device according to claim 1, characterized in that The processor is further configured to: The segmented image is bound to a Surface corresponding to the display area, and the segmented image is rendered through the Surface to obtain the rendered segmented image; and the display is controlled to display the rendered segmented image in the Surface.

4. The display device according to claim 1, characterized in that The special effects processing also includes image fusion and special effects adding, and the special effects adding includes adding target animation effects and adding target special effects, and the target special effects include one or more of stickers and texts.

5. An image processing method, characterized in that: include: Upon receiving the setting information sent by the split screen projection application, switching the video channel to the target channel; wherein the setting information includes a coordinate set corresponding to the split image actually displayed by the display device in the frame image, the split image is a part of the frame image, and the target channel includes any one of a multimedia channel, an analog signal channel, and a high-definition multimedia interface channel; When receiving the start information sent by the split screen projection application for instructing to start special effect processing, capturing a frame image from the target channel; Performing special effect processing on the frame image to obtain a segmented image corresponding to the coordinate set on the frame image; wherein the special effect processing at least includes image cropping; The segmented image is rendered to obtain the rendered segmented image, and the rendered segmented image is displayed.

6. The image processing method according to claim 5, characterized in that: When the setting information sent by the split screen projection application is received, after switching the video channel to the target channel, the method further includes: When receiving the setting information sent by the split screen projection application, the video channel is switched to the target channel, and the output audio of the target channel is muted; The step of rendering the segmented image and displaying the rendered segmented image comprises: The segmented image is rendered, and when it is determined that the segmented image is the first image frame, a muting operation is canceled on the output audio of the target channel, and the rendered segmented image is displayed in the display area.

7. The image processing method according to claim 5, characterized in that: The rendering of the segmented image and displaying the rendered segmented image in the display area includes: Binding the segmented image to a Surface corresponding to the display area, and rendering the segmented image through the Surface to obtain the rendered segmented image; The rendered segmented image is displayed in the Surface.

8. The image processing method according to claim 5, characterized in that: The special effects processing also includes image fusion and special effects adding, and the special effects adding includes adding target animation effects and adding target special effects, and the target special effects include one or more of stickers and texts.

9. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the image processing method according to any one of claims 5 to 8.

10. A computer program product, characterized in that include: The computer program product stores a computer program, and when the computer program is executed by a computing device, the computing device implements the image processing method according to any one of claims 5 to 8.

Images