Display processing method and device of projection screen, storage medium and electronic equipment

Abstract

The invention discloses a screen projection picture display processing method and device, a storage medium and electronic equipment, and relates to the technical field of screen projection, the method comprises the following steps: carrying out text content extraction processing on a screen projection picture to obtain text content; constructing a first display area and a second display area on the screen; displaying the screen projection picture in the first display area; and displaying the text content in the second display area. The text content display effect of the screen projection display end can be effectively improved, and the user reading experience is improved.

Description

Technical Field

[0001] This application relates to the field of screen projection technology, specifically to a method, apparatus, storage medium, and electronic device for displaying projected images. Background Technology

[0002] In wireless screen mirroring scenarios, when the source end encodes text content into a video stream and sends it to the sink end, although the large-screen display by the sink end can enlarge the text, the existing screen mirroring process often involves compression, resolution changes, and image processing (scaling, filtering, etc.) at the sink end. This results in blurred edges, pixelation, or compression artifacts on the text content displayed on the sink end, leading to poor text display quality and severely impacting the reading experience at long distances. Summary of the Invention

[0003] This application provides a solution that can effectively improve the text content display effect on the screen projection terminal and enhance the user's reading experience.

[0004] The embodiments of this application provide the following technical solutions: According to one embodiment of this application, a method for displaying a projected screen includes: extracting text content from the projected screen to obtain text content; constructing a first display area and a second display area on the screen; displaying the projected screen in the first display area; and displaying the text content in the second display area.

[0005] In some embodiments of this application, the step of extracting text content from the projected screen to obtain text content includes: extracting the original video frame corresponding to the projected screen from the video stream processing pipeline; and performing text content extraction processing based on the original video frame to obtain the text content.

[0006] In some embodiments of this application, the step of extracting text content based on the original video frame to obtain the text content includes: enhancing the original video frame to obtain an enhanced video frame; and extracting text content from the enhanced video frame to obtain the text content.

[0007] In some embodiments of this application, displaying the text content in the second display area includes: determining the original distribution positions of different sub-contents in the text content on the projection screen; and displaying the different sub-contents in the text content in the second display area according to the original distribution positions.

[0008] In some embodiments of this application, after displaying the text content in the second display area, the method further includes: performing interactive processing on the text content in the second display area according to a first interactive operation in the second display area to obtain interactively processed content; and displaying the interactively processed content in the second display area.

[0009] In some embodiments of this application, after displaying the text content in the second display area, the method further includes: performing display enhancement processing on the content area corresponding to the sub-content in the first display area according to a second interactive operation in the second display area for the sub-content in the text content.

[0010] In some embodiments of this application, before performing text content extraction processing on the projected screen to obtain text content, the method further includes: determining whether the user has enabled the text partition display mode; if it is determined that the user has enabled the text partition display mode, then the text partition display process is triggered.

[0011] According to one embodiment of this application, a display processing device for a projected screen includes: a text extraction module, used to: extract text content from the projected screen to obtain text content; a region construction module, used to: construct a first display region and a second display region on the screen; a first display module, used to: display the projected screen in the first display region; and a second display module, used to: display the text content in the second display region.

[0012] According to another embodiment of this application, a storage medium stores a computer program thereon, which, when executed by a processor of an electronic device, causes the electronic device to perform the methods described in the embodiments of this application.

[0013] According to another embodiment of this application, an electronic device may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the methods described in the embodiments of this application.

[0014] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations described in the embodiments of this application.

[0015] In this embodiment, the text content of the projected screen is extracted to obtain the text content; a first display area and a second display area are constructed on the screen; the projected screen is displayed in the first display area; and the text content is displayed in the second display area.

[0016] In this embodiment of the application, the screen projection display terminal extracts text content from the screen projection screen, displays the screen projection screen in the first display area, and displays the text content in the second display area. The text content displayed independently in the second display area is not affected by the various processing steps during the screen projection process. The text content displayed in the second display area is clearer, and users can more clearly refer to and read the text content in the screen projection screen from the second display area. Overall, this effectively improves the text content display effect of the screen projection display terminal and enhances the user's reading experience. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a screen projection display processing method according to an embodiment of this application is shown.

[0019] Figure 2 A schematic diagram of a screen mirroring interface according to an embodiment of this application is shown.

[0020] Figure 3 An interactive processing flowchart according to an embodiment of this application is shown.

[0021] Figure 4 A block diagram of a screen projection display processing apparatus according to an embodiment of this application is shown.

[0022] Figure 5 A block diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0023] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present disclosure and are not intended to limit the present disclosure. Furthermore, the embodiments provided below are some embodiments for implementing the present disclosure, and not all embodiments for implementing the present disclosure. Unless otherwise specified, the technical solutions described in the embodiments of the present disclosure can be implemented in any combination. It should be noted that, in the embodiments of this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus; for example, a unit may be a portion of circuitry, a portion of a processor, a portion of a program or software, etc.) in the method or apparatus that includes that element. For example, the screen projection display processing method provided in this embodiment includes a series of steps, but the screen projection display processing method provided in this embodiment is not limited to the steps described. Similarly, the screen projection display processing device provided in this embodiment includes a series of units, but the device provided in this embodiment is not limited to the units explicitly described, and may also include units that need to be set for obtaining relevant information or processing based on information. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. It is understood that in the specific implementation of this application, relevant data is involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0024] In wireless screen mirroring scenarios, when the source end encodes text content into a video stream and sends it to the sink end, although the large-screen display by the sink end can enlarge the text, the existing screen mirroring process often involves compression, resolution changes, and image processing (scaling, filtering, etc.) at the sink end. This results in blurred edges, pixelation, or compression artifacts on the text content displayed on the sink end, leading to poor text display quality and severely impacting the reading experience at long distances.

[0025] To address these issues, this application provides a display processing solution for screen projection, which can effectively improve the display effect of text content on the screen projection display end and enhance the user's reading experience.

[0026] The following is a detailed description of the relevant embodiments of the screen projection display processing scheme provided in this application.

[0027] Figure 1The flowchart illustrating a screen projection display processing method according to an embodiment of this application is shown. The entity executing this screen projection display processing method can be an electronic device serving as the screen projection display end (Sink end), such as a television, display screen, computer, etc.

[0028] like Figure 1 As shown, the display processing method for the projected screen may include steps S110 to S140.

[0029] Step S110: Extract text content from the projected screen to obtain the text content; Step S120: Construct a first display area and a second display area on the screen; Step S130: Display the projected screen in the first display area; Step S140: Display the text content in the second display area.

[0030] The screen mirroring display terminal extracts text content from the projected screen to obtain all or some key text content. The text extraction process can be based on traditional OCR (Optical Character Recognition) technology, a deep learning-based OCR engine, or open-source OCR tools; this application does not impose any specific limitations on this method. In some approaches, a lightweight deep learning-based OCR engine can be trained for use with TV UI fonts and commonly used PPT fonts, enabling rapid deployment on ARM, CPU, or NPU platforms.

[0031] The projection display further constructs a first display area and a second display area on the screen. One approach involves using a split-screen mode to create two separate screens, one serving as the first display area and the other as the second. Another approach involves generating a floating window on the screen, with the screen itself serving as the first display area and the floating window as the second. The layout of the first and second display areas can adapt to the screen resolution to ensure text readability.

[0032] The projected image is displayed in the first display area. The projected image in the first display area will also contain text content. However, due to compression, resolution changes and image processing (scaling, filtering, etc.) at the projection display end, the text content in the projected image in the first display area may have problems such as blurred edges, pixelation or compression artifacts.

[0033] The text content displayed independently in the second display area is unaffected by the processing during the projection process, resulting in greater clarity. Furthermore, by displaying the projected image in the first display area and the text content in the second, users can more clearly refer to and read the text content from the projected image in the second display area.

[0034] See Figure 2 The projection source 210 encodes the original image containing text content into a video stream and sends it to the projection display 220. The projection display 220 decodes the video stream to generate the projection image. The projection display 220 extracts the text content from the projection image. The projection display 220 constructs a first display area 221 and a second display area 222 on the screen, displays the projection image in the first display area 221, and displays the text content in the second display area 222.

[0035] In this embodiment of the application, the screen projection display terminal extracts text content from the screen projection screen, displays the screen projection screen in the first display area, and displays the text content in the second display area. The text content displayed independently in the second display area is not affected by the various processing steps during the screen projection process. The text content displayed in the second display area is clearer, and users can more clearly refer to and read the text content in the screen projection screen from the second display area. Overall, this effectively improves the text content display effect of the screen projection display terminal and enhances the user's reading experience.

[0036] The following description Figure 1 Further optional specific embodiments are provided for each step performed during the display processing of the projected screen in the example.

[0037] In one embodiment, before "step S110 extracts text content from the projected screen to obtain text content", the process may further include: determining whether the user has enabled the text partition display mode; if it is determined that the user has enabled the text partition display mode, then the text partition display process is triggered.

[0038] Users can turn the text partition display mode on or off as needed through operations such as "remote control buttons, voice commands, screen settings," etc. Once the user sets the text partition display mode, the text partition display process is triggered (the text partition display process may include steps S110 to S140 and other optional steps after step S110).

[0039] This implementation allows for the text partitioning display process to be executed only when needed by the user, displaying the projected screen and text content in multiple areas. This not only flexibly enhances the user experience based on demand but also reduces the system processing load during projection when the user does not need it, thus saving energy and reducing power consumption.

[0040] Optionally, in other embodiments, before “step S110 extracts text content from the projected screen to obtain text content”, it may also include: when the projected display terminal receives a text partition display instruction sent by the projected source terminal, it triggers a text partition display process (the text partition display process may include steps S110 to S140 and other optional steps after step S110).

[0041] In this implementation method, the projection source can flexibly control the projection display to display the projection screen and text content in multiple areas, further enhancing the projection experience.

[0042] In one embodiment, step S110, which involves extracting text content from the projected screen to obtain text content, may include: extracting the original video frame corresponding to the projected screen from the video stream processing pipeline; and extracting text content based on the original video frame to obtain text content.

[0043] The source end encodes the original image containing text content into a video stream and sends it to the projection display end. The video stream processing pipeline on the projection display end decodes the video stream to obtain the original video frames (such as YUV format image data). The display controller further outputs the original video frames to the screen to form the projection image.

[0044] In this embodiment, before the display controller outputs the original video frame to the screen, the original screen data corresponding to the projection screen is captured. Then, the original video frame can be obtained in the video decoding pipeline with zero copy for text content extraction processing, further reducing processing delay and further improving the synchronization of the projection screen and the multi-area display of text content.

[0045] Furthermore, in one embodiment, text content extraction processing is performed based on the original video frames to obtain text content, including: enhancing the original video frames to obtain enhanced video frames; and extracting text content from the enhanced video frames to obtain text content.

[0046] Enhanced video frames are obtained by performing enhancement processing on the original video frames (such as denoising, binarization, super-resolution, etc.). Then, text content extraction processing is performed on the enhanced video frames to further improve the accuracy of text content extraction. Before extracting the text content from the enhanced video frames, they can be converted into RGB or grayscale images suitable for OCR text content extraction.

[0047] Optionally, in other embodiments, step S110, performing text content extraction processing on the projected screen to obtain text content, may include: displaying the projected screen in the first display area, taking a screenshot of the projected screen from the first display area, and then performing text content extraction processing on the captured projected screen to obtain text content.

[0048] In one embodiment, step S140, displaying the text content in the second display area, may include: determining the original distribution positions of different sub-contents in the text content on the projection screen; and displaying the different sub-contents in the text content in the second display area according to their original distribution positions.

[0049] The text content can contain different sub-contents, and the original distribution position of these sub-contents on the projected screen is also the display position of these sub-contents on the projected screen. The original distribution position of different sub-contents on the projected screen can be detected synchronously during the text content extraction and processing.

[0050] Different sub-contents in the text are displayed in the second display area according to their original distribution positions. As a result, the sub-contents in the projection screen in the first display area and the sub-contents in the second display area will have the same distribution, making it easier for users to quickly find the sub-contents corresponding to the sub-contents in the first display area from the second display area.

[0051] Optionally, in other embodiments, step S140, displaying the text content in the second display area, may include: displaying the text content in the second display area in other custom layouts or predetermined layouts.

[0052] See Figure 3 In one embodiment, after "step S140 displays the text content in the second display area", the method may further include: step S310, performing interactive processing on the text content in the second display area according to the first interactive operation in the second display area to obtain the interactively processed content; step S320, displaying the interactively processed content in the second display area.

[0053] Users can perform initial interactive operations in the second display area, processing the text content there (such as zooming, scrolling, copying, and editing) to obtain the processed content. In other words, the second display area allows users to further process the text content through the initial interactive operation, thus enhancing the user experience.

[0054] The first interactive operation may include operations such as clicking, selecting, inputting, and swiping, and this application does not impose any special limitations on this.

[0055] In one embodiment, after "step S140 displays the text content in the second display area", the method may further include: performing display enhancement processing on the content area corresponding to the sub-content in the first display area according to the second interactive operation in the second display area for the sub-content in the text content.

[0056] Users can perform a second interactive operation on a specific sub-content A in the second display area. Through this second interactive operation, the display of the content area corresponding to sub-content A in the first display area can be enhanced. For example, the content area corresponding to sub-content A can be highlighted or marked with a box, allowing users to quickly locate the content area corresponding to sub-content A and further improving the user's reading experience.

[0057] To facilitate better implementation of the screen projection display processing method provided in this application embodiment, this application embodiment also provides a screen projection display processing device based on the above-described screen projection display processing method. The meanings of the terms used are the same as in the screen projection display processing method described above, and specific implementation details can be found in the descriptions in the method embodiments. Figure 4 A block diagram of a screen projection display processing apparatus according to an embodiment of this application is shown.

[0058] like Figure 4 As shown, the screen projection display processing device 400 may include: a text extraction module 410 for extracting text content from the screen projection to obtain text content; an area construction module 420 for constructing a first display area and a second display area on the screen; a first display module 430 for displaying the screen projection in the first display area; and a second display module 440 for displaying the text content in the second display area.

[0059] In some embodiments of this application, when performing text content extraction processing on the projected screen to obtain text content, the text extraction module 410 can be used to: extract the original video frame corresponding to the projected screen from the video stream processing pipeline; and perform text content extraction processing based on the original video frame to obtain the text content.

[0060] In some embodiments of this application, when the text content extraction processing is performed based on the original video frame to obtain the text content, the text extraction module 410 can be used to: enhance the original video frame to obtain an enhanced video frame; and perform text content extraction processing on the enhanced video frame to obtain the text content.

[0061] In some embodiments of this application, when the text content is displayed in the second display area, the second display module 440 can be used to: determine the original distribution positions of different sub-contents in the text content in the projection screen; and display the different sub-contents in the text content in the second display area according to the original distribution positions.

[0062] In some embodiments of this application, after the text content is displayed in the second display area, the device further includes a first interaction module that can be used to: perform interactive processing on the text content in the second display area according to a first interaction operation in the second display area to obtain interactively processed content; and display the interactively processed content in the second display area.

[0063] In some embodiments of this application, after the text content is displayed in the second display area, the device further includes a second interaction module that can be used to: perform display enhancement processing on the content area corresponding to the sub-content in the first display area according to a second interaction operation in the second display area for the sub-content in the text content.

[0064] In some embodiments of this application, before the text content extraction processing of the projected screen is performed to obtain the text content, the device further includes a process triggering module that can be used to: determine whether the user has enabled the text partition display mode; if it is determined that the user has enabled the text partition display mode, then the text partition display process is triggered.

[0065] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0066] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 5 As shown, Figure 5 A block diagram of an electronic device according to an embodiment of this application is shown, specifically: The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 501 is the control center of the electronic device, connecting various parts of the computer device via various interfaces and lines. It executes software programs and / or modules stored in the memory 502, and calls data stored in the memory 502, to perform various functions of the computer device and process data. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and application programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0067] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0068] The electronic device also includes a power supply 503 that supplies power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0069] The electronic device may also include an input unit 504, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0070] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device can load the executable files corresponding to the processes of one or more computer programs into the memory 502 according to the following instructions, and the processor 501 runs the computer programs stored in the memory 502, thereby realizing the various functions in the foregoing embodiments of this application.

[0071] For example, processor 501 can perform the following operations: extract text content from the projected screen to obtain text content; construct a first display area and a second display area on the screen; display the projected screen in the first display area; and display the text content in the second display area.

[0072] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0073] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the methods provided in embodiments of this application.

[0074] The storage medium can be a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0075] Since the computer program stored in the storage medium can execute the steps of any of the methods provided in the embodiments of this application, the beneficial effects that the methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0076] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations described in the embodiments of this application.

[0077] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0078] It should be understood that this application is not limited to the embodiments described above and shown in the accompanying drawings, but various modifications and changes can be made without departing from its scope.

Claims

1. A method for displaying projected images, characterized in that, include: The text content is extracted from the projected screen. Construct a first display area and a second display area on the screen; The projected image is displayed in the first display area; The text content is displayed in the second display area.

2. The method according to claim 1, characterized in that, The text content extraction process of the projected screen to obtain the text content includes: Extract the original video frames corresponding to the projected screen from the video stream processing pipeline; The text content is obtained by extracting text content from the original video frames.

3. The method according to claim 2, characterized in that, The text content extraction process based on the original video frames to obtain the text content includes: The original video frame is enhanced to obtain an enhanced video frame; The enhanced video frame is processed to extract the text content, thus obtaining the text content.

4. The method according to claim 1, characterized in that, Displaying the text content in the second display area includes: Determine the original distribution positions of different sub-contents within the text content on the projected screen; The different sub-contents in the text are displayed in the second display area according to their original distribution positions.

5. The method according to any one of claims 1 to 4, characterized in that, After displaying the text content in the second display area, the method further includes: Based on the first interactive operation in the second display area, the text content in the second display area is interactively processed to obtain the interactively processed content; The interactive content is displayed in the second display area.

6. The method according to any one of claims 1 to 4, characterized in that, After displaying the text content in the second display area, the method further includes: Based on the second interactive operation in the second display area for the sub-content in the text content, the content area corresponding to the sub-content in the first display area is subjected to display enhancement processing.

7. The method according to any one of claims 1 to 4, characterized in that, Before performing text content extraction processing on the projected screen to obtain the text content, the method further includes: Determine if the user has enabled text partition display mode; If it is determined that the user has enabled the text partition display mode, the text partition display process is triggered.

8. A display processing device for projected images, characterized in that, include: The text extraction module is used to extract text content from the projected screen to obtain the text content. The region construction module is used to: construct a first display region and a second display region on the screen; The first display module is used to: display the projected screen in the first display area; The second display module is used to display the text content in the second display area.

9. A storage medium, characterized in that, It stores a computer program that, when executed by the processor of the electronic device, causes the electronic device to perform the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores computer programs; A processor reads a computer program stored in memory to perform the method described in any one of claims 1 to 7.

Images