Method and system for collecting screen picture and related equipment

Abstract

The invention provides a method and system for collecting a screen picture and related equipment. The method is applied to a first terminal, the first terminal comprises a first display screen, and the method specifically comprises the following steps: creating a virtual screen according to the size of the first display screen; wherein the size of the virtual screen is the same as that of the first display screen; obtaining display information, drawing to obtain display picture data according to the display information, and writing the display picture data into the virtual screen; reading display picture data in the virtual screen; based on the display picture data, controlling the first display screen to display; and obtaining streaming media data through hard coding based on the display picture data, and sending the streaming media data to a second terminal, so that the streaming media data can be monitored at the second terminal. The technical effect of remotely monitoring and broadcasting the screen picture of the first terminal in real time is achieved, and the method has the advantages of low cost, simple deployment and capability of discovering the fault of the screen picture in time.

Description

Technical Field

[0001] The present disclosure relates to the field of multimedia publishing technology, and in particular to a method, system and related equipment for capturing screen images. Background Art

[0002] With the advancement of science and technology, the way of information release has gradually become electronic, and information release display screens have been used more and more widely. In order to ensure the quality of information release, it is essential to monitor the playback screen of the information release display screen. At present, manual offline inspections check each information release display screen one by one. This monitoring method is not only costly but also has poor timeliness, affects the effect of information publishing, and is prone to economic losses. In addition, through the hardware camera video monitoring solution, you can use the remote screen video stream to view the monitoring, and you can detect screen failures in time for repair, but the cost is greatly increased. At the same time, the deployment environment requirements of the information release display screen are high, which makes it difficult to deliver on the ground. Summary of the invention

[0003] In view of this, the purpose of the present disclosure is to provide a method, system and related equipment for capturing screen images.

[0004] Based on the above purpose, the present disclosure provides a method for collecting screen images, which is applied to a first terminal, where the first terminal includes a first display screen; the method includes:

[0005] Creating a virtual screen according to the size of the first display screen; wherein the size of the virtual screen is the same as the size of the first display screen;

[0006] Acquire display information, and draw display screen data according to the display information and write the data into the virtual screen;

[0007] Reading display picture data in the virtual screen;

[0008] Based on the display picture data, controlling the first display screen to display; and

[0009] Based on the display screen data, streaming media data is obtained through hard coding and sent to the second terminal, so that the streaming media data can be monitored at the second terminal.

[0010] Based on the same inventive concept, an embodiment of the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the program.

[0011] Based on the same inventive concept, the embodiment of the present disclosure also provides a system for collecting screen images, including:

[0012] The first terminal, including a first display screen, is configured as follows:

[0013] Creating a virtual screen according to the size of the first display screen; wherein the size of the virtual screen is the same as the size of the first display screen;

[0014] Acquire display information, and draw display screen data according to the display information and write the data into the virtual screen;

[0015] Reading display picture data in the virtual screen;

[0016] Based on the display picture data, controlling the first display screen to display; and

[0017] Based on the display screen data, obtain streaming media data through hard coding and send the streaming media data to the second terminal, so that the streaming media data can be monitored on the second terminal; and

[0018] The second terminal includes a second display screen, is connected to the first terminal via a network, and is configured as follows:

[0019] Receiving the streaming media data, and decoding the streaming media data to obtain streaming media decoded data;

[0020] Based on the streaming media decoded data, a video picture is determined and displayed on the second display screen.

[0021] Based on the same inventive concept, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable a computer to execute any of the above-described methods.

[0022] Based on the same inventive concept, an embodiment of the present disclosure further provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer is enabled to execute any of the above-described methods.

[0023] From the above description, it can be seen that a method, system and related equipment for collecting screen images provided by the embodiments of the present disclosure create a virtual screen with the same size as the first display screen; display image data is drawn according to the display information and stored in the virtual screen, so as to control the first display screen to display according to the display image data, and at the same time, based on the display image data, streaming media data is obtained by hard coding and sent to the second terminal, so that the streaming media data can be monitored on the second terminal, achieving the technical effect of remote real-time monitoring of the screen image of the first terminal, and has the advantages of low cost, simple deployment and timely detection of screen image failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A schematic diagram showing an application scenario of information release provided by an embodiment of the present disclosure;

[0026] Figure 2 A schematic diagram showing the structure of an information publishing system provided by an embodiment of the present disclosure;

[0027] Figure 3 A schematic diagram of the software and hardware architecture of a first terminal provided by an embodiment of the present disclosure is shown;

[0028] Figure 4 A schematic diagram showing the structure of a system for collecting screen images provided by an embodiment of the present disclosure is shown;

[0029] Figure 5 An interactive schematic diagram of a system for collecting screen images provided by an embodiment of the present disclosure is shown;

[0030] Fig. 6A A schematic diagram showing a screen image provided by an embodiment of the present disclosure;

[0031] Figure 6B A schematic diagram showing another screen image provided by an embodiment of the present disclosure;

[0032] Figure 6C A schematic diagram showing another screen image provided by an embodiment of the present disclosure;

[0033] Figure 7 A schematic diagram of a process of collecting screen images provided by an embodiment of the present disclosure is shown;

[0034] Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0036] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0037] In order to facilitate understanding of the technical solutions of the present disclosure, some technical terms involved in the present disclosure are introduced below.

[0038] OpenGL (Open Graphics Library) is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. This interface consists of nearly 350 different function calls, which are used to draw simple graphic elements to complex three-dimensional scenes.

[0039] WebGL is a 3D drawing standard. This drawing technology standard allows JavaScript and OpenGL ES2.0 to be combined together. By adding a JavaScript binding to OpenGL ES2.0, hardware 3D accelerated rendering is provided for HTML5 Canvas, so that Web developers can use the system graphics card to display 3D scenes and models more smoothly in the browser, and can also create complex navigation and data visualization. The WebGL technology standard eliminates the trouble of developing special rendering plug-ins for web pages and can be used to create website pages with complex 3D structures. It should be noted that OpenGL ES2.0 is the embedded platform version of the open graphics library OpenGL.

[0040] MediaCodec is a codec component in Android, which is used to access the codec provided by the underlying layer and is usually used with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack. Typically, MediaCodec includes encoders and decoders.

[0041] The Video Processing Unit (VPU) is a highly customized chip for visual tasks, with hard encoding and decoding functions and the ability to reduce CPU load. There are two main types of inputs it processes, real-time image data from sensors, and pre-recorded video or image data. VPU is involved in any task related to these two types of data. From signal processing tasks such as imaging and coding, to classic computer vision tasks such as image deformation and deformation correction, deep information processing (stereo depth), to deep learning tasks for semantic extraction. Therefore, VPU is a heterogeneous and complex structure dedicated to image tasks, processing the intersection of signal processing, computer vision, and deep learning.

[0042] Streaming media refers to a technology and process that compresses a series of media data, sends the data in segments over the Internet, and transmits audio and video in real time for viewing. Streaming is the key technology to achieve streaming media. There are two ways to achieve streaming: real-time streaming and progressive streaming. Among them, real-time streaming refers to ensuring that the bandwidth of the media signal matches the network connection so that the media can be viewed in real time. Real-time streaming adjusts the quality of the output audio and video according to the network conditions to achieve continuous real-time transmission of the media. Users can fast forward or rewind to watch the previous or next content. Usually, real-time streaming requires a specific streaming media server and also requires the support of specific network protocols. Among them, progressive streaming is sequential downloading. Users can watch online media while downloading files. At a given time, users can only watch the downloaded part and cannot jump to the front part that has not been downloaded. During the transmission, the download order is not adjusted according to the speed of the user's connection. Since standard HTTP servers can send this form of streaming media without the support of other special protocols, no specific server is required.

[0043] The information publishing system is composed of a server, a network, a player, and a display device. The server information is sent to the player through the network (applicable to wide area networks / local area networks / dedicated networks, including wireless networks). The player then combines audio, video, pictures, text and other information (including playback position and playback content, etc.) and transmits it to LCD TVs and other display devices that can accept audio and video input to form audio and video file playback. In this way, a system is formed that can send all server information to the terminal through the network.

[0044] As described in the background art, with the advancement of science and technology, the way of information release has gradually become electronic, and information release display screens have been used more and more widely.

[0045] Figure 1 A schematic diagram of an application scenario 100 for information release provided by an embodiment of the present disclosure is shown. There are numerous buildings and roads in outdoor places. In order to increase the customer reach and efficiency of information, information release terminals 300A and 300B are often set near the buildings and roads. For example, a large-size display terminal 300A is set in front of a building; as another example, an electronic sign 300B is set near a road. Information release terminals 300A and 300B have electronic display screens that can play various information as needed for user 101 to watch. In this way, information release is flexible and the information presentation form is richer. It should be noted that information release terminals can also be set in buildings, scenic spots, and other scenes where information needs to be disseminated. Figure 1 The application scenario 100 is only exemplary.

[0046] In order to better manage the playback information of the information release terminal, an information release system is often used to efficiently manage the information release based on the information release system. Figure 2 FIG. 2 is a schematic diagram showing the structure of an information publishing system 200 provided by an embodiment of the present disclosure. Figure 2 It can be seen that the information release system 200 includes a backstage terminal 400 and information release terminals 300A, 300B, and 300C connected to the network. Here, the network connection can be a wired or wireless communication, which is not limited in the present disclosure. The backstage terminal 400 can send a sound video playback plan and audio and video information to the information release terminals 300A, 300B, and 300C. After receiving the audio and video playback plan and the audio and video information, the information release terminals 300A, 300B, and 300C can play the audio and video information on the display screen according to the audio and video playback plan.

[0047] It should be noted that the information release terminals 300A, 300B, and 300C can have various forms. Exemplarily, the information release terminals 300A, 300B, and 300C can be in the form of an extra-large display screen with an information release box, wherein the display screen and the information release box can be split or integrated, and the present disclosure does not limit this. Exemplarily, the information release terminals 300A, 300B, and 300C can be electronic signs. The backend terminal 400 can include but is not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA), or other electronic devices capable of realizing the above functions.

[0048] Figure 3 FIG. 1 is a schematic diagram showing the software and hardware architecture of an information publishing terminal, namely, a first terminal 300, provided in an embodiment of the present disclosure. Figure 3, the first terminal 300 includes a hardware layer 301 and a software layer 302. It should be noted that the software layer 302 of the first terminal is described by taking the Android system as an example. Exemplarily, the first terminal 300 can be an Android information publishing box and a matching display screen.

[0049] In some embodiments, the hardware layer 301 includes memory, CPU (Central Processing Unit) and VPU. The CPU and VPU can share the memory space of the motherboard. The CPU is mainly used for the calculation of applications at the operating system level, and the VPU is responsible for the calculation of graphics and audio and video encoding and decoding. It should be noted that the hardware layer 301 may also include a display screen (not shown in the figure), which is not described in detail in this disclosure. Exemplarily, the board card of the hardware layer 301 may be RK3288 or RK3399.

[0050] In some embodiments, the software layer 302 includes a system library 3021, an application framework layer 3022, an application layer 3023, and a kernel layer 3024. Optionally, the application layer 3023 can call a technical module provided by the application framework layer 3022 through an interface provided by the application framework layer 3022. The application framework layer 3022 can call a functional module provided by the system library 3021 through an interface provided by the system library 3021. The functional module provided by the system library 3021 can complete the corresponding function through hardware (e.g., CPU, VPU).

[0051] Exemplarily, the system library 3021 may include an open image library OpenGL. In an embodiment of the present disclosure, OpenGL may be used to support the drawing of display screen data. Exemplarily, the system library 3021 may include an Android runtime (AndroidRuntime), which is responsible for the scheduling and management of the Android system. The Android runtime includes a core library and a virtual machine. The core library contains two parts: one part is the function function that needs to be called by the Java language, and the other part is the core library of Android. The application framework layer 3022 and the application layer 3023 run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0052] Exemplarily, the application framework layer 3022 provides an application programming interface (API) and a programming framework for the application of the application layer 3023. The application framework layer 3022 includes a media framework. The media framework provides a plurality of tools for editing video and audio.

[0053] In some embodiments, the media framework includes MediaCodec. MediaCodec is a class provided by Android for encoding and decoding audio and video. It includes encoders, decoders, and surface type caches. The encoder provided by MediaCodec can convert a video or audio in one format input to the encoder into another format through compression technology, and the decoder performs the reverse process of encoding, and can convert a video or audio in one format input to the decoder into another format through decompression technology. For example, the encoder can encapsulate frame-by-frame display screen data into a video; then, the decoder can deframe the encapsulated video, thereby dividing the encapsulated video into frame-by-frame display screen data.

[0054] Optionally, MediaCodec can also apply for a memory of type surface. Surface can be used to cache video data. For example, after the first terminal 300 performs a display screen drawing operation and obtains display screen data, the first terminal 300 can input the above display screen data into the surface cache. Then, the application can obtain the drawn display screen data from the above surface for storage, encoding or display, etc.

[0055] In some embodiments, the application framework layer 3022 also includes a Java Native Interface (JNI for short).

[0056] In some embodiments, the application layer 3023 may include applications such as information publishing and calendaring.

[0057] In some embodiments, the kernel layer 3024 is a layer between hardware and software. The kernel layer 3024 includes at least a display driver, an audio driver, etc. Other layers of the software can call corresponding functions of the kernel layer using a set functional interface.

[0058] The above-mentioned first terminal 300 is used in combination with the background terminal 400 to complete the function of information release.

[0059] As described in the background art, in order to monitor whether the information is released normally, it is necessary to monitor the display screen of the first terminal 300. The existing monitoring methods have problems such as cost and efficiency, and it is difficult to meet the needs of users.

[0060] In view of this, an embodiment of the present disclosure provides a method for capturing screen images, by creating a virtual screen of the same size as the first display screen; writing display image data drawn according to display information into the virtual screen, thereby realizing the display of the first display screen, and at the same time, based on the display image data, obtaining streaming media data through hard coding and sending it to a second terminal, so that the streaming media data can be monitored on the second terminal, achieving the technical effect of remotely monitoring the screen image of the first terminal in real time, and having the advantages of low cost, simple deployment and the ability to promptly detect screen image failures.

[0061] Figure 2 The information publishing system 200 shown can be used to implement the above-mentioned method of collecting screen images and realize the downstream transmission of streaming media data. Figure 4 The following is a schematic diagram showing the structure of a system for collecting screen images provided by an embodiment of the present disclosure. Figure 4 The screen image acquisition system shown can not only realize information release, but also can realize real-time streaming and sequential streaming of streaming media data, meeting the diverse monitoring needs of the second terminal (ie, the backend terminal 400).

[0062] like Figure 4 As shown, the system for collecting screen images includes a first terminal 300A, 300B, 300C, a streaming media server 500, and a second terminal 400A, 400B. The first terminal 300A, 300B, 300C pushes the encoded streaming media data to the streaming media server 500, and the streaming media server 500 forwards it to the second terminal 400A, 400B that has subscribed to the streaming media data, so that the second terminal 400A, 400B monitors the display screen images of the first terminal 300A, 300B, 300C. The second terminal 400A, 400B can directly send the audio and video playback plan and audio and video information to the first terminal 300A, 300B, 300C, or send it through the streaming media server 500, which is not limited in this disclosure. With the help of the streaming media server 500, it is possible to achieve multiple inputs and multiple outputs of streaming media data, which is helpful for being applicable to the situation with multiple first terminals and multiple second terminals. In addition, such a system for capturing screen images has strong scalability and can adapt to the increase of first terminals and second terminals by adding new streaming media servers without changing the original hardware equipment, thereby meeting the monitoring needs of more first terminals in the system.

[0063] Below, in conjunction with the accompanying drawings, the method for capturing screen images provided by the embodiment of the present disclosure is described in detail.

[0064] Figure 5FIG. 1 is a schematic diagram showing an interaction of a system for collecting screen images provided by an embodiment of the present disclosure. Specifically, Figure 5 As shown, the first terminal (information publishing terminal) 300 creates a virtual screen 601 according to the size of the first display screen; wherein the size of the virtual screen is the same as the size of the first display screen. Here, the virtual screen can be a frame buffer with a certain size opened up in the kernel, for example, the resolution is 800×480. It should be noted that the frame buffer is an interface provided by Linux for the display device, which describes the display device as a buffer. For developers, the frame buffer is a display buffer, and writing data in a specific format to the display buffer means outputting content to the display screen. By continuously writing data to the frame buffer, the display controller automatically takes data from the frame buffer and displays it.

[0065] By setting the size of the virtual screen to be the same as the size of the first display screen, it can be ensured that the virtual screen can directly mirror the display screen of the first display screen.

[0066] In some embodiments, the second terminal 400 may send a start message 6021 to the first terminal 300. The start message here is an instruction to control the first terminal 300 to execute the step of encoding the display screen data to obtain the streaming media data 6051. Of course, the second terminal 400 may also send a stop message (not shown in the figure) to the first terminal 300. The stop message here is an instruction to control the first terminal 300 to stop executing the step of encoding the display screen data to obtain the streaming media data 6051. The second terminal 400 can control whether to monitor the display screen of the first terminal 300 through the start message and the stop message. Such an operation can be performed remotely, and the operation is simple and has a wide range of applications.

[0067] In some alternative embodiments, the first terminal 300 may always execute the step of encoding the display screen data to obtain the streaming media data 6051.

[0068] The second terminal 400 may send media display information 6022 to the first terminal 300. Here, the media display information may include an audio and video playback plan and audio and video information. It should be noted that if the audio and video information to be played is already stored in the first terminal 300, only the audio and video playback plan may be sent. Here, when the media display information sent by the second terminal 400 includes only the audio and video playback plan, the first terminal 300 may search for the corresponding audio and video information based on the audio and video playback plan to obtain the complete media display information to be played.

[0069] It should be noted that the second terminal 400 sending the media display information 6022 and the first terminal 400 creating the virtual screen 601 can be completed before drawing the display screen data 603, and there is no time sequence between them. The second terminal 400 sending the start information 6021 can be completed before editing the display screen data to obtain the streaming media data, and there is no time sequence relationship between sending the media display information 6022 and creating the virtual screen 601.

[0070] In some embodiments, the first terminal 300 may also obtain system display information, where the system display information may be display information provided by the Android system, such as a status bar, shortcut buttons, and the like.

[0071] Next, the first terminal 300 can draw and obtain display screen data based on the system display information and the media display information and write it into the virtual screen 603. Here, the display screen data refers to the display data of the image frame. Those skilled in the art can understand that the first terminal 300 can draw multiple frames of images, that is, multiple display screen data can be obtained.

[0072] In some embodiments, the first terminal 300 may include OpenGL, and OpenGL may be used to draw a system display screen using system display information, and to obtain a media display screen using media display information. Optionally, the system display screen and the media display screen may be synthesized into one frame of image using SurfaceFlinger. It should be noted that the screen synthesis of SurfaceFlinger is performed according to certain rules. When the media display screen is displayed in full screen, the system display screen is not displayed. In some embodiments, the display screen data synthesized by SurfaceFlinger (corresponding to one frame of image) may be stored in an image buffer, written to a virtual screen by the image buffer, or may be directly written to the virtual screen. In some embodiments, new display screen data is written to the virtual screen by refreshing the virtual screen.

[0073] Then, the display screen data 604 in the virtual screen is read. Next, based on the read display screen data, the first display screen is controlled to display and display the display screen data 6052. On the other hand, the display screen data read by the virtual screen can be hard-coded to obtain the streaming media data 6051. Here, the first display screen refers to the display screen of the first terminal 300.

[0074] In other words, the technical solution of the embodiment of the present disclosure adopts two processing methods for one display screen data, and encodes the streaming media data while realizing the display of the first display screen, ensuring the consistency of the display screen data, and avoiding the error of monitoring caused by the difference of the obtained display screen data. With such a solution, the collected screen image can directly use the display screen data used for the first display screen display, which can effectively reduce the consumption of system resources for the collected screen image. It should be noted that those skilled in the art can select appropriate technology to obtain the display screen data according to the performance of the first terminal, and this is not listed in the present disclosure.

[0075] The following is an example of how to encode the display screen data:

[0076] The encoder creates an input data buffer and an output data buffer; here, the encoder can be a component of MediaCodec;

[0077] Next, the display screen data of the virtual screen is read in real time and pushed into the cache queue of the input data buffer. In some embodiments, the first terminal 300 includes a display controller, and the display controller corresponding to the first display screen can read the display screen data from the virtual screen for display, and the encoder can read the display screen data of the virtual screen for pushing into the cache queue.

[0078] Finally, the hard-coded interface of the video processing unit is called to read the display screen data from the cache queue of the input data buffer to perform streaming media data encoding, and the obtained streaming media data is pushed into the output data buffer so as to be sent to the second terminal 400.

[0079] Exemplarily, the streaming media data encoding format may be h264 / h265 YUV4:2:0.

[0080] Compared with the method of collecting screen images based on the CPU, the use of the video processing unit for encoding can effectively reduce the occupation of CPU resources by screen image collection, effectively ensure the normal release of media information, and ensure that the first terminal 300 plays audio and video normally. In particular, for the case where the computing power of the Android board's own CPU is weak, while the clarity of media display information is constantly improving, the VPU method used in the embodiment of the present disclosure can greatly reduce the dependence of screen image collection on the CPU, ensuring the normal publication and broadcasting of media information.

[0081] Next, the first terminal 300 may push the streaming data 606 to the streaming server 500. The streaming server 500 forwards the streaming data 607 to the second terminal 400 that has subscribed to the streaming data.

[0082] It should be noted that if the screen image collection system does not have a streaming media server 500, the screen image can be directly sent to the corresponding second terminal 400 ( Figure 5 not shown).

[0083] In some embodiments, the streaming media data includes a registration address that matches the second terminal, which is used to determine the subscribed second terminal 400.

[0084] In some embodiments, the streaming protocol for pushing the streaming media data 606 may be RTSP, RTM, or http-flv.

[0085] In some embodiments, the first terminal 300 can use FFmpeg technology to push streaming media data. Here, FFmpeg is an open source audio and video processing tool library. Through FFmpeg, functions such as format conversion, encoding and decoding, editing, and merging of audio and video can be realized. FFmpeg supports streaming media protocols and can perform real-time audio and video stream processing. Through FFmpeg, audio and video files can be pushed to a streaming media server, and audio and video data can also be pulled from a streaming media server.

[0086] Then, after receiving the streaming media data 607, the second terminal 400 may decode the streaming media data, and after decoding, may draw a video picture and display it on the second display screen 608. Here, the second display screen is the display screen corresponding to the second terminal 400.

[0087] It should be noted that by cyclically executing the steps of drawing display screen data 603 to decoding streaming media data, and after decoding is completed, drawing and displaying the video screen 608, real-time monitoring of the video screen of the first display screen of the first terminal 300 can be achieved.

[0088] In some embodiments, the second terminal 400 uses WebGL technology to draw the video screen and display it on a web page for the user to view.

[0089] In some embodiments, Fig. 6A As shown, if the first terminal 300 fails to obtain the media display information, the first terminal 300 can draw the display screen data based on the system display information. Fig. 6A A schematic diagram of a screen provided by an embodiment of the present disclosure is shown, wherein screen 701 is a display screen of the system itself. In this way, when the second terminal 400 monitors and finds that the first terminal 300 only displays the display screen of the system, it indicates that there is a fault in the first terminal 300 or the information transmission path, and the next maintenance measure can be initiated.

[0090] Figure 6BA schematic diagram of another screen image provided by an embodiment of the present disclosure is shown. In some embodiments, Figure 6B As shown, when the media information is displayed in full screen, the second terminal 400 can see the media video 702.

[0091] Figure 6C In some embodiments, the media information is set to be displayed in full screen, and the second terminal 400 can see the display screen 703, which means that the first screen display of the first terminal 300 has a fault and does not meet the set requirements, and the next step of maintenance or adjustment can be performed.

[0092] This public embodiment provides a method for capturing screen images. Figure 7 The following is a schematic diagram showing a process of collecting screen images provided by an embodiment of the present disclosure. Figure 7 The method for collecting screen images is applied to a first terminal 300, wherein the first terminal includes a first display screen; the method includes:

[0093] Step S802: creating a virtual screen according to the size of the first display screen; wherein the size of the virtual screen is the same as the size of the first display screen;

[0094] Step S804: acquiring display information, and drawing display screen data according to the display information and writing the data into the virtual screen;

[0095] Step S806: reading the display image data in the virtual screen;

[0096] Step S8081: Based on the display picture data, control the first display screen to display; and

[0097] Step S8082: Based on the display screen data, streaming media data is obtained through hard coding and sent to the second terminal, so that the streaming media data can be monitored on the second terminal 400.

[0098] In some embodiments, step S804 includes:

[0099] Retrieving system display information and acquiring media display information sent by the second terminal 400;

[0100] In response to determining that the acquisition of the media display information fails, display screen data is obtained by drawing according to the system display information.

[0101] In some embodiments, it also includes:

[0102] Acquire control information sent by the second terminal 400;

[0103] According to the control information, hard coding of the display screen data is started or stopped.

[0104] In some embodiments, the first terminal includes an open graphics library OpenGL, and the drawing to obtain display screen data according to the display information includes:

[0105] The OpenGL draws a media display screen based on the media display information in the display information.

[0106] In some embodiments, the OpenGL draws a system display screen based on the system display information in the display information.

[0107] In some embodiments, the first terminal 300 includes a video processing unit and an encoder, and the obtaining of streaming media data by hard coding based on the display screen data includes:

[0108] The encoder creates an input data buffer and an output data buffer;

[0109] Acquire the display screen data in real time and push it into a cache queue of an input data buffer;

[0110] The hard-coded interface of the video processing unit is called to read the display screen data from the cache queue of the input data buffer to perform streaming media data encoding, and the obtained streaming media data is pushed into the output data buffer so as to be sent to the second terminal.

[0111] In some embodiments, obtaining the streaming media data by hard coding and sending it to the second terminal includes:

[0112] The streaming media data is sent to the second terminal via a predetermined streaming media protocol.

[0113] In some embodiments, step S8082 includes:

[0114] The streaming media data is pushed to a streaming media server, so that the streaming media server forwards the streaming media data to a second terminal that subscribes to the streaming media data.

[0115] In some embodiments, the streaming media data includes a registration address matching the second terminal, which is used by the streaming media server to determine the second terminal.

[0116] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0117] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] Based on the same inventive concept, the embodiment of the present disclosure also provides a system for collecting screen images. Figure 2 and Figure 4 As shown, the system includes:

[0119] The first terminal 300 includes a first display screen and is configured as follows:

[0120] Creating a virtual screen according to the size of the first display screen; wherein the size of the virtual screen is the same as the size of the first display screen;

[0121] Acquire display information, and draw display screen data according to the display information and write the data into the virtual screen;

[0122] Reading display picture data in the virtual screen;

[0123] Based on the display picture data, controlling the first display screen to display; and

[0124] Based on the display screen data, obtain streaming media data through hard coding and send the streaming media data to the second terminal, so that the streaming media data can be monitored on the second terminal; and

[0125] The second terminal 400 includes a second display screen, is connected to the first terminal via a network, and is configured as follows:

[0126] Receiving the streaming media data, and decoding the streaming media data to obtain streaming media decoded data;

[0127] Based on the streaming media decoded data, a video picture is determined and displayed on the second display screen.

[0128] In some embodiments, the first terminal is further configured to:

[0129] Acquiring system display information and acquiring media display information sent by the second terminal;

[0130] In response to determining that the acquisition of the media display information fails, display screen data is obtained by drawing according to the system display information.

[0131] In some embodiments, the first terminal 300 is further configured to:

[0132] Acquire control information sent by the second terminal;

[0133] According to the control information, hard coding of the display screen data is started or stopped.

[0134] In some embodiments, the first terminal 300 includes an open graphics library OpenGL, and the first terminal is configured such that: the OpenGL draws a media display screen based on the media display information in the display information.

[0135] In some embodiments, the OpenGL draws a system display screen based on the system display information in the display information.

[0136] In some embodiments, the first terminal 300 includes a video processing unit and an encoder, and the first terminal is configured to:

[0137] The encoder creates an input data buffer and an output data buffer;

[0138] Acquire the display screen data in real time and push it into a cache queue of an input data buffer;

[0139] The hard-coded interface of the video processing unit is called to read the display screen data from the cache queue of the input data buffer to perform streaming media data encoding, and the obtained streaming media data is pushed into the output data buffer so as to be sent to the second terminal.

[0140] In some embodiments, the first terminal 300 is configured to:

[0141] The streaming media data is sent to the second terminal via a predetermined streaming media protocol.

[0142] In some embodiments, the system further comprises a streaming media server; the streaming media server is respectively connected to the first terminal and the second terminal via a network;

[0143] The first terminal 300 is configured as follows:

[0144] Pushing the streaming media data to the streaming media server; and

[0145] The streaming media server is configured as follows:

[0146] The streaming media data is forwarded to a second terminal that subscribes to the streaming media data.

[0147] In some embodiments, the streaming media data includes a registration address matching the second terminal, which is used by the streaming media server to determine the second terminal.

[0148] In some embodiments, the second terminal 400 draws the video screen based on WebGL technology.

[0149] The system of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0150] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.

[0151] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0152] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0153] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0154] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0155] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0156] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0157] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0158] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0159] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.

[0160] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0161] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0162] Based on the same inventive concept, corresponding to the method described in any of the above embodiments, the present disclosure further provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the color correction method. Corresponding to the execution subject corresponding to each step in each embodiment of the color correction method, the processor that executes the corresponding step can belong to the corresponding execution subject.

[0163] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0164] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0165] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0166] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0167] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A method for collecting screen images, characterized in that: Applied to a first terminal, the first terminal includes a first display screen; the method includes: Creating a virtual screen according to the size of the first display screen; wherein the size of the virtual screen is the same as the size of the first display screen; Acquire display information, and draw display screen data according to the display information and write the data into the virtual screen; Reading display picture data in the virtual screen; Based on the display picture data, controlling the first display screen to display; and Based on the display screen data, streaming media data is obtained through hard coding and sent to the second terminal, so that the streaming media data can be monitored at the second terminal.

2. The method according to claim 1, characterized in that The step of obtaining display information and drawing display screen data according to the display information includes: Retrieving system display information and acquiring media display information sent by the second terminal; In response to determining that the acquisition of the media display information fails, display screen data is obtained by drawing according to the system display information.

3. The method according to claim 1, characterized in that Also includes: Acquire control information sent by the second terminal; According to the control information, hard coding of the display screen data is started or stopped.

4. The method according to claim 1, characterized in that: The first terminal includes an open graphics library OpenGL, and the drawing to obtain display screen data according to the display information includes: The OpenGL draws a media display screen based on the media display information in the display information; and / or The OpenGL draws a system display screen based on the system display information in the display information.

5. The method according to claim 1, characterized in that The first terminal includes a video processing unit and an encoder, and the streaming media data is obtained by hard coding based on the display screen data, including: The encoder creates an input data buffer and an output data buffer; Acquire the display screen data in real time and push it into a cache queue of an input data buffer; The hard-coded interface of the video processing unit is called to read the display screen data from the cache queue of the input data buffer to perform streaming media data encoding, and the obtained streaming media data is pushed into the output data buffer so as to be sent to the second terminal.

6. The method according to claim 1, characterized in that The step of obtaining the streaming media data by hard coding and sending the streaming media data to the second terminal includes: The streaming media data is sent to the second terminal via a predetermined streaming media protocol.

7. The method according to claim 6, characterized in that The sending the streaming media data to the second terminal includes: The streaming media data is pushed to a streaming media server, so that the streaming media server forwards the streaming media data to a second terminal that subscribes to the streaming media data.

8. The method according to claim 7, characterized in that The streaming media data includes a registration address matching the second terminal, which is used by the streaming media server to determine the second terminal.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, characterized in that: The processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. A system for collecting screen images, characterized in that: include: The first terminal, including a first display screen, is configured as follows: Creating a virtual screen according to the size of the first display screen; wherein the size of the virtual screen is the same as the size of the first display screen; Acquire display information, and draw display screen data according to the display information and write the data into the virtual screen; Reading display picture data in the virtual screen; Based on the display picture data, controlling the first display screen to display; and Based on the display screen data, obtain streaming media data through hard coding and send the streaming media data to the second terminal, so that the streaming media data can be monitored on the second terminal; and The second terminal includes a second display screen, is connected to the first terminal via a network, and is configured as follows: Receiving the streaming media data, and decoding the streaming media data to obtain streaming media decoded data; Based on the streaming media decoded data, a video picture is determined and displayed on the second display screen.

11. The system according to claim 10, characterized in that The first terminal is further configured to: Acquiring system display information and acquiring media display information sent by the second terminal; In response to determining that the acquisition of the media display information fails, display screen data is obtained by drawing according to the system display information.

12. The system according to claim 10, characterized in that The first terminal is further configured to: Acquire control information sent by the second terminal; According to the control information, hard coding of the display screen data is started or stopped.

13. The method according to claim 10, characterized in that The first terminal includes an open graphics library OpenGL, and the first terminal is configured as follows: The OpenGL draws a media display screen based on the media display information in the display information; and / or The OpenGL draws a system display screen based on the system display information in the display information.

14. The system according to claim 10, characterized in that The first terminal includes a video processing unit and an encoder, and the first terminal is configured as follows: The encoder creates an input data buffer and an output data buffer; Acquire the display screen data in real time and push it into a cache queue of an input data buffer; The hard-coded interface of the video processing unit is called to read the display screen data from the cache queue of the input data buffer to perform streaming media data encoding, and the obtained streaming media data is pushed into the output data buffer so as to be sent to the second terminal.

15. The system according to claim 10, characterized in that The first terminal is configured as follows: The streaming media data is sent to the second terminal via a predetermined streaming media protocol.

16. The system according to claim 15, characterized in that The system also includes a streaming media server; The streaming media server is respectively connected to the first terminal and the second terminal via a network; The first terminal is configured as follows: Pushing the streaming media data to the streaming media server; as well as The streaming media server is configured as follows: The streaming media data is forwarded to a second terminal that subscribes to the streaming media data.

17. The system according to claim 16, characterized in that The streaming media data includes a registration address matching the second terminal, which is used by the streaming media server to determine the second terminal.

18. The system according to claim 10, characterized in that The second terminal draws the video screen based on WebGL technology.

19. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the method according to any one of claims 1 to 8.

20. A computer program product, characterized in that The method comprises computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 8.

Images