Picture processing method and device, equipment and computer storage medium

Abstract

The embodiment of the invention provides a picture processing method and device, equipment and a computer storage medium, and the method comprises the steps: confirming the hierarchy type of virtual display equipment based on an equipment state, and when the hierarchy type indicates that the virtual display equipment is first hierarchy equipment, displaying the first hierarchy equipment; and when the hierarchy type indicates that the virtual display device is the first hierarchy device, first picture data of the first hierarchy device is sent to the second hierarchy device, when the hierarchy type indicates that the virtual display device is the second hierarchy device, second picture data of the second hierarchy device and the first picture data are merged to obtain third picture data, and the third picture data is sent to a receiving device. And the receiving device displays the picture data of any virtual display device based on the third picture data. The image data of the multiple virtual display devices are combined through the virtual display devices, so that the images of the multiple virtual display devices are displayed on one receiving device, and the transmission consumption and the load pressure generated when the multiple virtual display devices are projected to one receiving device are reduced.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a picture processing method, device, equipment and computer storage medium. Background Art

[0002] Virtual Reality (VR) is a computer simulation system that can create and experience a virtual world. It uses computers to generate simulated environments to achieve interactive three-dimensional vision and dynamic entity behaviors. Its multi-source information fusion feature allows users to experience an immersive sense of reality. In existing solutions, the display content of multiple VR devices can be displayed on one VR device, achieving many-to-one real-time sharing of virtual reality scenes.

[0003] However, the existing multi-VR device projection solution consumes a lot of performance of the VR receiving device, and requires decoding and rendering multiple images at the same time. Therefore, it is urgent to propose a processing method that can reduce the load pressure caused by decoding and rendering of the receiving device. Summary of the invention

[0004] The main purpose of this specification is to provide a picture processing method, device, equipment and computer storage medium, aiming to reduce the resource consumption of video receiving equipment. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present specification provides a picture processing method, which is applied to a virtual display device, comprising:

[0006] Determine the level type of the virtual display device based on the device status;

[0007] When the level type indicates that the virtual display device is a first-level device, sending the first picture data of the first-level device to a second-level device;

[0008] When the hierarchy type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to a receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the third screen data.

[0009] In a second aspect, an embodiment of the present specification provides a picture processing method, which is applied to a receiving device, including:

[0010] receiving third picture data sent by a second-level device; the third picture data is obtained by the second-level device merging the second picture data of the second-level device and the first picture data sent by the first-level device to the second-level device;

[0011] Based on the identification information in the third picture data, the picture data of the virtual display device corresponding to the identification information is displayed.

[0012] In a third aspect, an embodiment of the present specification provides a picture processing method, which is applied to a virtual display device, including:

[0013] Determine the level type of the virtual display device based on the device status;

[0014] When the hierarchy type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the interactive tablet, so that the interactive tablet merges the first screen data to obtain sixth screen data, and the sixth screen data is sent to the receiving device, so that the receiving device can display the screen data of any one of the virtual display devices based on the sixth screen data; there are at least two first-level devices.

[0015] In a fourth aspect, an embodiment of the present specification provides a screen processing method, which is applied to an interactive tablet, including:

[0016] When receiving first picture data sent by a first-level device, merging the first picture data to generate a merged data stream; there are at least two first-level devices;

[0017] Confirm the identification information corresponding to each of the first screen data, write the identification information into the merged data stream, generate sixth screen data, and send the sixth screen data to a receiving device so that the receiving device can display the screen data of any of the virtual display devices based on the sixth screen data.

[0018] In a fifth aspect, an embodiment of the present specification provides a picture processing method, which is applied to a receiving device, including:

[0019] receiving sixth screen data sent by the interactive tablet; the sixth screen data is obtained by the interactive tablet merging the first screen data received from each of the first-level devices;

[0020] Based on the identification information in the sixth screen data, screen data of the virtual display device corresponding to the identification information is displayed.

[0021] In a sixth aspect, an embodiment of the present specification provides a picture processing device, including:

[0022] A first confirmation module, used to confirm the level type of the virtual display device based on the device status;

[0023] a sending module, configured to send first picture data of the first-level device to a second-level device when the level type indicates that the virtual display device is a first-level device;

[0024] A first merging module is used to merge the second screen data of the second-level device with the first screen data to obtain third screen data when the level type indicates that the virtual display device is a second-level device, and send the third screen data to a receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the third screen data.

[0025] In a seventh aspect, an embodiment of the present specification provides a picture processing device, including:

[0026] A first receiving module is used to receive third picture data sent by a second-level device; the third picture data is obtained by the second-level device merging the second picture data of the second-level device and the first picture data sent by the first-level device to the second-level device;

[0027] The first display module is used to display the screen data of the virtual display device corresponding to the identification information based on the identification information in the third screen data.

[0028] In an eighth aspect, an embodiment of the present specification provides a picture processing device, including:

[0029] A second confirmation module, used to confirm the level type of the virtual display device based on the device status;

[0030] A second merging module is used to send the first screen data of the first-level device to the interactive tablet when the level type indicates that the virtual display device is a first-level device, so that the interactive tablet merges the first screen data to obtain sixth screen data, and sends the sixth screen data to the receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the sixth screen data; there are at least two first-level devices.

[0031] In a ninth aspect, an embodiment of the present specification provides a picture processing device, including:

[0032] A second receiving module is used for merging the first picture data sent by the first layer device to generate a merged data stream when receiving the first picture data sent by the first layer device; the first layer device is at least two;

[0033] The third merging module is used to confirm the identification information corresponding to each of the first screen data, write the identification information into the merged data stream, generate sixth screen data, and send the sixth screen data to the receiving device, so that the receiving device can display the screen data of any of the virtual display devices based on the sixth screen data.

[0034] In a tenth aspect, an embodiment of the present specification provides a picture processing device, including:

[0035] A third receiving module is used to receive sixth screen data sent by the interactive tablet; the sixth screen data is obtained by the interactive tablet after merging the first screen data received from each of the first-level devices;

[0036] The second display module is used to display the screen data of the virtual display device corresponding to the identification information based on the identification information in the sixth screen data.

[0037] In an eleventh aspect, an embodiment of the present specification provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above method when executed by the processor.

[0038] In a twelfth aspect, an embodiment of the present specification provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0039] In an embodiment of the present specification, by confirming the level type of the virtual display device based on the device status, when the level type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the second-level device, and when the level type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to the receiving device, so that the receiving device can display the screen data of any virtual display device based on the third screen data. By using a designated virtual display device to merge the screen data of one or more virtual display devices, and the designated virtual display device transmits the screen data to the receiving device, the screens of multiple virtual display devices are displayed (projected) on one receiving device, the number of devices directly connected to the receiving device is reduced, the transmission consumption, communication consumption and the load pressure caused by decoding and rendering of multiple virtual display devices on one receiving device are reduced, and the battery life of the receiving device is increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 is an example schematic diagram of a picture processing method provided in an embodiment of this specification;

[0042] Figure 2 It is a flowchart of a picture processing method provided by an embodiment of this specification;

[0043] Figure 3 is an example schematic diagram of a picture processing method provided in an embodiment of this specification;

[0044] Figure 4 is an example schematic diagram of a picture processing method provided in an embodiment of this specification;

[0045] Figure 5 It is a flowchart of a picture processing method provided by an embodiment of this specification;

[0046] Figure 6 is an example schematic diagram of a picture processing method provided in an embodiment of this specification;

[0047] Figure 7 It is a flowchart of a picture processing method provided by an embodiment of this specification;

[0048] Figure 8 is an example schematic diagram of a picture processing method provided in an embodiment of this specification;

[0049] Fig. 9 It is a flowchart of a picture processing method provided by an embodiment of this specification;

[0050] Fig.10 is an example schematic diagram of a picture processing method provided in an embodiment of this specification;

[0051] Fig.11 is a structural schematic diagram of a picture processing device provided by an embodiment of this specification;

[0052] Fig.12 is a structural schematic diagram of a picture processing device provided by an embodiment of this specification;

[0053] Fig.13 is a structural schematic diagram of a picture processing device provided by an embodiment of this specification;

[0054] Fig.14 is a structural schematic diagram of a picture processing device provided by an embodiment of this specification;

[0055] Fig.15 is a structural schematic diagram of a picture processing device provided by an embodiment of this specification;

[0056] Fig.16 is a structural schematic diagram of a picture processing device provided by an embodiment of this specification;

[0057] Fig.17 is a structural schematic diagram of a picture processing device provided by an embodiment of this specification;

[0058] Fig.18 is a structural schematic diagram of a picture processing device provided by an embodiment of this specification;

[0059] Fig.19 is a structural schematic diagram of a picture processing device provided by an embodiment of this specification;

[0060] Fig. 20 It is a structural schematic diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0062] The image processing device can be a terminal device such as a VR (Virtual Reality) device, an AR (Augmented Reality) device, an MR (Mixed Reality) device, a mobile phone, a computer, or a tablet computer, or it can be a module in the terminal device for implementing the image processing method. The image processing device can confirm the level type of the virtual display device based on the device status. When the level type indicates that the virtual display device is a first-level device, the first image data of the first-level device is sent to the second-level device. When the level type indicates that the virtual display device is a second-level device, the second image data of the second-level device is merged with the first image data to obtain third image data, and the third image data is sent to the receiving device so that the receiving device can display the image data of any virtual display device based on the third image data.

[0063] Please also see Figure 1, is a schematic diagram of an example of a picture processing method provided in an embodiment of the present specification, wherein the picture processing device may be a virtual display device, Figure 1 In the figure, virtual display device 1 and virtual display device N (referring to N first-level devices) are first-level devices, virtual display device 2 and virtual display device 3 are second-level devices. After the level type is confirmed based on the device status, according to the device type indicated by the level type, the first-level device sends the first screen data to the second-level device, and the second-level device merges the second screen data of the second-level device with the first screen data to obtain the third screen data, and sends the third screen data to the receiving device, so that the receiving device can display the screen data of any of the virtual display devices based on the third screen data, that is, virtual display device 1 sends its screen data 1 to virtual display device 2, virtual display device 2 merges the received screen data 1 with its own screen data 2 to obtain screen data 1+2, and sends the screen data 1+2 to the receiving device, virtual display device N sends screen data N to virtual display device 3, virtual display device 3 merges the received screen data N with screen data 3 to obtain screen data 3+N, and sends the screen data 3+N to the receiving device, wherein the receiving device can be a virtual display device.

[0064] By adopting the image processing method provided in the embodiments of this specification, the image data of one or more virtual display devices can be merged by using a designated virtual display device, and the designated virtual display device transmits the image data to a receiving device, thereby realizing the display (screen projection) of multiple virtual display devices on one receiving device, reducing the number of devices directly connected to the receiving device, reducing the transmission consumption, communication consumption and load pressure caused by decoding and rendering of multiple virtual display devices on one receiving device, and increasing the battery life of the receiving device.

[0065] The picture processing method provided in this specification is described in detail below in conjunction with specific embodiments.

[0066] See also Figure 2 , is a flowchart of a method for processing a picture provided by an embodiment of this specification. Figure 2 As shown, the embodiment of this specification describes the specific process of the image processing method from the virtual display device side and the receiving device side. The method may include the following steps S101-S105.

[0067] S101, the virtual display device confirms the level type of the virtual display device based on the device state;

[0068] In one embodiment, the corresponding layer type can be confirmed according to the device state of the virtual display device, and the layer type is the layer type when the virtual display device performs data transmission, which is used to constrain the data transmission content of the virtual display device. The device state of the virtual display device can be confirmed according to CPU occupancy, memory occupancy, communication protocol, etc.

[0069] Specifically, virtual display devices can be divided into first-level devices and second-level devices according to the device status. The screen processing method provided in the embodiments of this specification can be applied to one or more virtual display devices. When the receiving device needs to view the screen data in the virtual display device, the content that the virtual display device needs to execute can be confirmed according to the level type of the current virtual display device. If the virtual display device is a first-level device, the first-level device sends the first screen data currently displayed to the second-level device, which is then merged and sent to the receiving device. If the virtual display device is a second-level device, after receiving the first screen data sent by the first-level device, the second-level device merges the second screen data currently displayed with the first screen data and sends it to the receiving device.

[0070] S102, when the level type indicates that the virtual display device is a first-level device, sending first picture data of the first-level device to a second-level device;

[0071] Specifically, the first screen data is the screen data currently displayed by the first-level device, which may include image data in the real environment and the virtual environment. In one embodiment, the virtual display device in the embodiment of this specification may be a VR device in a classroom teaching scenario, and students may watch the learning content in the head display through the VR device. The learning content may be a teaching video recorded by the teacher, an experimental demonstration video, an interactive learning video, etc., without specific limitation. When the level type indicates that the virtual display device is a first-level device, the virtual display device sends the first screen data to the second-level device. Among them, when sending the first screen data, there may be multiple second-level devices, and the second-level device to which the first-level device is to send the data may be a second-level device with a pre-confirmed association relationship, or may be any one of the second-level devices.

[0072] S103, when the level type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to a receiving device, so that the receiving device can display the screen data of any one of the virtual display devices based on the third screen data.

[0073] Specifically, when the level type indicates that the virtual display device is a second-level device, the virtual display device receives the first screen data from the first-level device, and merges the second screen data and the first screen data displayed by the second-level device to obtain the third screen data. The third screen data is a merged data stream including the first screen data and the second screen data. Among them, there can be one or more second-level devices. Preferably, the number of second-level devices directly connected to the receiving device is as small as possible to reduce the transmission burden of the receiving device as much as possible. A second-level device can receive the first screen data sent from one or more first-level devices. It can be understood that since there is a need to view the screen content of the virtual display device in real time at the receiving device end, the real-time first screen data of the first-level device and the real-time second screen data of the second-level device can be merged to obtain the merged third screen data, and the merged third screen data is transmitted to the receiving device, so that the receiving device can view the screen content of multiple virtual display devices by receiving only one stream. Among them, multiple virtual display devices can be networked through WiFi P2P (Wi-Fi Peer-to-Peer). This LAN connection method does not occupy the school WiFi bandwidth, reduces the network load pressure in the classroom, and transmits quickly and stably. At the same time, the video is merged and processed through the processing power of the second-level device. The receiving device only needs to receive the merged third-screen data transmitted by the second-level device, and does not need to receive the video stream transmitted by each virtual display device, decode, and render multiple screens. Therefore, the decoding load pressure of the receiving device is also shared.

[0074] S104, the receiving device receives third picture data sent by the second-level device; the third picture data is obtained by the second-level device merging the second picture data of the second-level device and the first picture data sent by the first-level device to the second-level device;

[0075] Specifically, the receiving device receives the third screen data sent by the second-level device to display the screen content of the corresponding virtual display device according to the third screen data. It should be noted that the distance between students is relatively fixed and close, and a P2P local area network connection can be used. In actual application scenarios, the receiving device is mainly a virtual display device worn by a teacher. Usually, the teacher will walk around in the classroom. If a P2P connection is used, the connection effect is not particularly stable. Therefore, a WIFI connection is used between the second-level device and the receiving device.

[0076] S105: The receiving device displays the screen data of the virtual display device corresponding to the identification information based on the identification information in the third screen data.

[0077] Specifically, after receiving the third screen data, the receiving device can decode, render and display the screen data of the corresponding virtual display device according to the identification information in the third screen data. The identification information is information that identifies the source of the video, for example, it can be the student identity information identified in the screen data, or it can be identification data that does not exist in the screen data and is generated separately and is associated with the screen data. For example, see Figure 3 , Figure 3 An example schematic diagram of a picture processing method is provided for the embodiment of this specification. Figure 3 The illustrated example is an example of identification information. The identification information may be the student name written in the lower right corner of the video in the screen data, or may be student information stored in association with the screen data. For example, when the second-level device generates the third screen data, the first-level device from which each first screen data in the third screen data comes may be identified, or the name of the student from which it comes. Figure 4 , Figure 4 An example schematic diagram of a screen processing method is provided for the embodiment of this specification. In the classroom application scenario, the effect achieved by the screen processing method of the embodiment of this specification is to aggregate the screen contents that multiple students are learning, and send them together by the second-level device to the teacher's receiving device. The teacher can view the screen data of all students in the receiving device, or select the screen data of one of the students to be viewed and enlarge it for display. For example, the gesture recognition function provided by the VR device can be used to select the screen of the student "Wang Wu" in the third screen data displayed through a virtual gesture to view the details. Of course, the screen data to be viewed can also be selected through eye tracking and other methods, without specific limitation.

[0078] In an embodiment of the present specification, by confirming the level type of the virtual display device based on the device status, when the level type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the second-level device, and when the level type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to the receiving device, so that the receiving device can display the screen data of any virtual display device based on the third screen data. By merging the screen data through the virtual display device, the receiving device only receives one stream transmitted by the second-level device in the virtual display device, which reduces the network load pressure in the multi-virtual display device scenario and alleviates the decoding load pressure of the receiving device.

[0079] See also Figure 5 , is a flowchart of a method for processing a picture provided by an embodiment of this specification. Figure 5As shown, the embodiment of this specification describes the specific process of the image processing method from the virtual display device side, and the method may include the following steps S201-S204.

[0080] S201, confirming the level type of the virtual display device based on the device status;

[0081] In one embodiment, the virtual display device can be divided into a first-level device, a second-level device, and a third-level device according to the device state. The third-level device can receive the first screen data sent by the first-level device and merge it with the fourth screen data of the third-level device to obtain the fifth screen data. The third-level device sends the fifth screen data to the second-level device, and the second-level device merges the fifth screen data with the second screen data of the second-level device to obtain the third screen data. The third-level device is used to transfer between the first-level device and the second-level device to reduce the video processing burden of the second-level device. It can be understood that since the number of second-level devices is usually small, when the number of virtual display devices is large, if a large number of first-level devices are to be merged through the second-level device, the amount of screen data to be merged by each second-level device may be too large, which may cause low efficiency. Therefore, the third-level device can also be selected from the virtual display device, and the third-level device can also participate in the screen data merging process, so as to reasonably utilize the processing capacity of each virtual display device and improve the processing efficiency.

[0082] Further, in one embodiment, the step of determining the level type of the virtual display device based on the device state includes:

[0083] S2011, acquiring a device state, and judging the device state based on a predefined communication protocol to obtain a judgment result;

[0084] It can be understood that the level type of the virtual display device can be confirmed according to a predefined communication protocol. For example, it can be agreed in the communication protocol that virtual display device 1 needs to send its first screen data to virtual display device 2, and virtual display device 3 needs to send its first screen data to virtual display device 4. Then the device state of the virtual display device that only sends screen data is as a sender, and its corresponding level type is the first level, while the device state of the virtual display device that receives the first screen data is as a receiver, and its corresponding level type is the second level. Therefore, according to the predefined communication protocol, it is first determined whether the virtual display device is a sender or a receiver of screen data to obtain a determination result.

[0085] S2012: Determine the level type of the virtual display device according to the determination result.

[0086] Specifically, if the judgment result indicates that the virtual display device is a transmitter, the level type is the first level, and if the judgment result indicates that the virtual display device is a receiver, the level type is the second level. Optionally, if the judgment result indicates that the virtual display device is both a transmitter and a receiver, the level type is the third level.

[0087] Further, in one embodiment, the step of determining the level type of the virtual display device based on the device state includes:

[0088] S2013, confirming the device idleness of the virtual display device based on the device status;

[0089] S2014, based on the device idleness, confirming the level type of the virtual display device;

[0090] Specifically, the device with a large current remaining space (device idleness) is used as the second-level device or third-level device for merging the screen data. It is understandable that since screen merging requires the use of the CPU, memory, etc. of the virtual display device, the device idleness can be confirmed based on the CPU and memory occupancy. The lower the CPU and memory occupancy, the greater the device idleness. If the virtual display device idleness is high, it can be used as the second-level device for processing screen merging. If the virtual display device idleness is low, it can be used as the first-level device for sending screen data.

[0091] S2015, if the device idleness is less than or equal to the idleness threshold, confirming that the level type of the virtual display device is a first level type;

[0092] Specifically, the level type of the virtual display device can be confirmed based on the size relationship between the device idleness and the idleness threshold. If the device idleness of the virtual display device is less than or equal to the idleness threshold, it means that its current device load is high and it is unable to perform the video merging task. Then it can send the current first screen data to the second-level device or the third-level device for merging. Optionally, the device idleness of the second-level device can be lower than the device idleness of the third-level device.

[0093] S2016, if the device idleness is greater than the idleness threshold, confirming that the level type of the virtual display device is a second level type;

[0094] Specifically, if the device idleness of the virtual display device is greater than the idleness threshold, it means that its current device load is low and it is capable of processing the video merging task, and it can be set to the second level type for video merging processing. It can be understood that when the device idleness of the virtual display device is greater than the idleness threshold, its level type can also be the third level type, and it can be used as a third level device for video merging processing.

[0095] S202, when the level type indicates that the virtual display device is a first-level device, sending the first picture data of the first-level device to the third-level device; the third-level device is used to merge the first picture data with the fourth picture data of the third-level device, and send the fifth picture data obtained by the merged processing to the second-level device;

[0096] Specifically, when the virtual display device is a first-level device, the first screen data of the first-level device can also be sent to the third-level device, so that the third-level device can merge and process the first screen data, and then send the fifth screen data including the first screen data to the second-level device.

[0097] S203, when the level type indicates that the virtual display device is a third-level device, merging the fourth screen data of the third-level device with the first screen data to obtain fifth screen data, and sending the fifth screen data to the second-level device;

[0098] Specifically, when the level type indicates that the virtual display device is a third-level device, the third-level device can merge the fourth screen data currently displayed with the first screen data to obtain the fifth screen data. It should be noted that there can be multiple third-level devices, and the screen data can be merged again between the third-level devices to reduce the processing load of the second-level device for merging multiple screen data.

[0099] S204: When the level type indicates that the virtual display device is a second-level device, merge the second screen data of the second-level device with the fifth screen data to obtain third screen data, and send the third screen data to a receiving device.

[0100] Specifically, when the level type indicates that the virtual display device is a second level device, the second level device combines the second screen data of the second level device with the fifth screen data sent by the third level device to obtain third screen data, and sends the third screen data to the receiving device.

[0101] Further, in one embodiment, when the level type indicates that the virtual display device is a second-level device, merging the second screen data of the second-level device with the first screen data to obtain the third screen data includes:

[0102] S2041, when the level type indicates that the virtual display device is a second-level device, merging the second picture data of the second-level device with the first picture data to generate a merged data stream;

[0103] Specifically, when performing the merging process, the second-level device can merge its current second picture data with the received first picture data to obtain a merged data stream. The first picture data and the second picture data can be first processed into picture data with the same video encoding format and the same bitstream configuration parameters, and then the bitstreams are merged. The bitstream configuration parameters that change when merging the video bitstream corresponding to the first picture data and the video bitstream corresponding to the second picture data can be written into the merged data stream.

[0104] S2042, obtaining first identification information corresponding to the first screen data and second identification information corresponding to the second screen data;

[0105] Specifically, the first identification information corresponding to the first screen data and the second identification information of the second screen data can also be obtained. For example, the first identification information of the first screen data can be the device number of the first-level device, and the second identification information of the second screen data can be the device number of the second-level device. Exemplarily, the second screen data and the first screen data can be merged and processed using ffmpeg software to generate a merged data stream. In addition, the first identification information is generated according to the source of the first screen data, and the second identification information is generated according to the source of the second screen data.

[0106] S2043: Write the first identification information and the second identification information into the merged data stream to obtain third picture data.

[0107] Specifically, the first identification information and the second identification information are written into the combined data stream, and the third picture data carrying the identification information can be obtained, so that the receiving device can confirm the virtual display device corresponding to each picture data based on the third picture data.

[0108] See also Figure 6 , Figure 6 An example schematic diagram of a picture processing method is provided for the embodiment of this specification. Figure 6In the embodiment, virtual display device 1 and virtual display device 4 are first-level devices, virtual display device 2 and virtual display device 3 are third-level devices, and virtual display device 5 is second-level device. After the level type is confirmed based on the device status, the first-level device sends the first screen data to the third-level device according to the device type indicated by the level type. The third-level device merges the first screen data of the first-level device with the fourth screen data currently displayed by the third-level device to obtain the fifth screen data, and sends the fifth screen data to the second-level device. The second-level device merges the received fifth screen data with the second screen data of the second-level device to obtain the third screen data, and sends the third screen data to the receiving device for the receiving device to receive the third screen data based on the first level type. The three-screen data display the screen data of any virtual display device, that is, virtual display device 1 sends its screen data 1 to virtual display device 2, virtual display device 2 merges the received screen data 1 with its own screen data 2 to obtain screen data 1+2, and sends screen data 1+2 to virtual display device 5, virtual display device 4 sends screen data 4 to virtual display device 3, virtual display device 3 merges the received screen data 4 and screen data 3 to obtain screen data 3+4, and sends screen data 3+4 to virtual display device 5, virtual display device 5 merges its own screen data 5 with the received screen data 1+2 and screen data 3+4 to obtain screen data 1+2+3+4+5 and sends it to the receiving device.

[0109] In the embodiments of the present specification, the hierarchical type of the virtual display device can be confirmed by the judgment results of the device idleness and the device status, so that different hierarchical types of devices can be confirmed in the virtual display device to realize different device functions. Specifically, three hierarchical types can be divided. When the hierarchical type indicates that the virtual display device is of the first hierarchical type, the first screen data of the first hierarchical device is sent to the third hierarchical device, and the third hierarchical device merges the first screen data with the fourth screen data of the third hierarchical device, and sends the fifth screen data obtained by the merged processing to the second hierarchical device, and the second hierarchical device merges the fifth screen data with the second screen data of the second hierarchical device to obtain the third screen data. By setting the third hierarchical device to act as a transfer between the first hierarchical device and the second hierarchical device, the video processing burden of the second hierarchical device can be reduced and the image processing efficiency can be improved.

[0110] See also Figure 7 , is a flowchart of a method for processing a picture provided by an embodiment of this specification. Figure 7 As shown, the embodiment of this specification jointly describes the specific process of the image processing method from the virtual display device side, the interactive tablet side and the receiving device side. The method may include the following steps S301-S306.

[0111] S301, the virtual display device confirms the level type of the virtual display device based on the device state;

[0112] In one embodiment, the virtual display device can confirm its level type according to the device status. When the level type indicates that the virtual display device is a first-level device, the first screen data of the first-level device can be sent to the interactive tablet, and the interactive tablet merges the first screen data to obtain the sixth screen data. Specifically, the interactive tablet and the virtual display device can also perform data transmission through a WiFi P2P connection. The interactive tablet merges the first screen data of multiple virtual display devices to obtain the sixth screen data, and sends the sixth screen data to the receiving device, so that the receiving device only needs to receive one stream sent by the interactive tablet, that is, it can obtain the screen data of each first-level device, which reduces the performance consumption of the receiving device decoding the screen data and reduces the bandwidth usage. Among them, the step of confirming the level type of the virtual display device based on the device status can refer to the description of step S201 in the above embodiment, which will not be repeated here.

[0113] S302, when the hierarchical type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the interactive tablet, so that the interactive tablet merges the first screen data to obtain sixth screen data, and the sixth screen data is sent to the receiving device, so that the receiving device can display the screen data of any one of the virtual display devices based on the sixth screen data; there are at least two first-level devices.

[0114] It should be noted that, in the case where only the interactive tablet is directly connected to the receiving device, when there is only one first-level device that needs to transmit screen data, the first screen data of the first-level device can also be sent to the interactive tablet, and the first screen data is directly sent to the receiving device through the interactive tablet. When there are at least two first-level devices, the interactive tablet needs to merge the multiple first screen data obtained to obtain the sixth screen data.

[0115] S303, when the interactive tablet receives the first screen data sent by the first-level device, it merges the first screen data to generate a merged data stream; there are at least two first-level devices;

[0116] Specifically, when the interactive tablet receives the first screen data sent by the first-level device, each of the first screen data is merged and processed to generate a merged data stream. Exemplarily, a merged data stream can be generated based on the video encoding format and configuration parameters of each first screen data. Optionally, the first screen data can be synthesized based on the layout parameters set in the layout mode, so that each screen data in the merged data stream can be spliced ​​in a layout manner, so that the screen data of each virtual display device can be displayed in a layout manner on the receiving device. The interactive tablet can be a display screen with a touch function with a host module and a whole machine module. The hardware configuration of the whole machine module is relatively low. In addition to configuring the operating system and basic processing software (such as whiteboard writing software, annotation application software), it does not support the installation of other application software. The host module is a pluggable (OPS, Open Pluggable Specification) module on the interactive tablet, and the application software running on the interactive tablet is installed on the host module. The host module can run operating systems such as Windows and high-performance Android. For example, the host module can adopt a PC module, a high-performance Android module, etc. It can be understood as a computer. The corresponding functions are realized by installing relevant application software on the host module. For example, if files need to be displayed, relevant display software needs to be installed on the host module. If video conferencing function needs to be realized, video conferencing software needs to be installed on the host module, and so on.

[0117] S304, the interactive tablet confirms the identification information corresponding to each of the first screen data, writes the identification information into the merged data stream, generates sixth screen data, and sends the sixth screen data to the receiving device, so that the receiving device can display the screen data of any of the virtual display devices based on the sixth screen data.

[0118] Specifically, the interactive tablet can also confirm the first identification information corresponding to each first screen data, write the first identification information into the merged data stream, and thus obtain the sixth screen data carrying the identification information. The sixth screen data is the screen data obtained by merging the first screen data sent by each first-level device by the interactive tablet. The interactive tablet sends the sixth screen data to the receiving device, so that the receiving device can display the screen data of any virtual display device according to the sixth device.

[0119] S305, the receiving device receives the sixth screen data sent by the interactive tablet; the sixth screen data is obtained after the interactive tablet merges the first screen data of each of the first-level devices;

[0120] S306: The receiving device displays the screen data of the virtual display device corresponding to the identification information based on the identification information in the sixth screen data.

[0121] Specifically, the receiving device displays the screen data of the virtual display device corresponding to the identification information according to the identification information in the received sixth screen data. The receiving device can decode, render and display the screen data of the corresponding virtual display device according to the identification information in the sixth screen data.

[0122] See also Figure 8 , Figure 8 An example schematic diagram of a picture processing method is provided for the embodiment of this specification. Figure 8 In the figure, virtual display device 1, virtual display device 2, virtual display device 3 and virtual display device 4 are first-level devices. After the level type is confirmed based on the device status, the first-level device sends the first screen data to the interactive tablet according to the device type indicated by the level type. The interactive tablet merges the first screen data of the first-level device to obtain sixth screen data, and sends the sixth screen data to the receiving device, so that the receiving device can display the screen data of any virtual display device based on the sixth screen data. That is, virtual display device 1 sends its screen data 1 to the interactive tablet, virtual display device 2 sends its screen data 2 to the interactive tablet, virtual display device 3 sends its screen data 3 to the interactive tablet, and virtual display device 4 sends its screen data 4 to the interactive tablet. The interactive tablet merges the received screen data 1, screen data 2, screen data 3 and screen data 4 to obtain screen data 1+2+3+4 and sends it to the receiving device.

[0123] In the embodiment of the present specification, by confirming the level type of the virtual display device based on the device status, when the level type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the interactive tablet, so that the interactive tablet merges and processes each of the first screen data to obtain the sixth screen data, and the sixth screen data is sent to the receiving device, so that the receiving device can display the screen data of any virtual display device based on the sixth screen data. The interactive tablet is used as a device that directly communicates with the receiving device, so that the receiving device only needs to receive the sixth screen data obtained by the interactive tablet merging the first screen data of each first-level device to view the screen data of each virtual display device, thereby reducing the load pressure caused by decoding and rendering of the receiving device and increasing the battery life.

[0124] See also Fig. 9 , is a flowchart of a method for processing a picture provided by an embodiment of this specification. Fig. 9 As shown, the embodiment of this specification jointly describes the specific process of the image processing method from the virtual display device side, the interactive tablet side and the receiving device side. The method may include the following steps S401-S406.

[0125] S401, when the hierarchical type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the second-level device, the second-level device is used to merge the first screen data and the second screen data of the second-level device, and send the merged third screen data to the interactive tablet, so that the interactive tablet merges the third screen data to obtain seventh screen data, and sends the seventh screen data to the receiving device, so that the receiving device can display the screen data of any one of the virtual display devices based on the seventh screen data.

[0126] In one embodiment, the virtual display device can be divided into a first-level device and a second-level device according to the level type. The first-level device sends the first screen data of the first-level device to the second-level device. The second-level device merges the first screen data and the second screen data of the second-level device, and sends the obtained third screen data to the interactive tablet. It can be understood that if the virtual display device has spare load, it will first merge between the virtual display devices to reduce the processing pressure of the interactive tablet and improve efficiency. For example, if there are 30 virtual display device terminals, the second-level devices can be selected according to the actual situation to merge into 10, 15 or 20 third screen data, and the third screen data is sent to the interactive tablet. The interactive tablet then merges each third screen data to obtain the seventh screen data. The interactive tablet sends the seventh screen data to the receiving device, and the receiving device displays the screen data of the received virtual display device based on the seventh screen data.

[0127] S402, when the level type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to the interactive tablet so that the interactive tablet merges the third screen data to obtain seventh screen data, and the seventh screen data is sent to the receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the seventh screen data.

[0128] S403, when the interactive tablet receives the third screen data sent by the second-level device, it merges the third screen data to generate a merged data stream;

[0129] S404, the interactive tablet confirms the identification information corresponding to each of the third screen data, writes the first identification information into the merged data stream, generates seventh screen data, and sends the seventh screen data to the receiving device, so that the receiving device can display the screen data of any of the virtual display devices based on the seventh screen data.

[0130] In one embodiment, the interactive tablet receives third screen data from the second-level device, merges the third screen data, and obtains seventh screen data. Specifically, the third screen data can be merged to obtain a merged data stream, and then the identification information corresponding to each third screen data is confirmed, and the identification information is written into the merged data stream to generate the seventh screen data. The third screen data includes screen data from the first-level device and the second-level device, so the annotation information can mark the source of each screen data in the third screen data.

[0131] S405, the receiving device receives the seventh screen data sent by the interactive tablet; the seventh screen data is obtained by the interactive tablet after merging and processing the third screen data sent by the second-level device; the third screen data is obtained by each of the second-level devices by merging and processing the second screen data of the second-level device and the first screen data;

[0132] Specifically, the receiving device receives the seventh screen data sent by the interactive tablet. In a feasible implementation, the receiving device and the interactive tablet can be connected via WiFi for data transmission, and the receiving device only receives the screen data from the interactive tablet.

[0133] S406: The receiving device displays the screen data of the virtual display device corresponding to the identification information based on the identification information in the seventh screen data.

[0134] Specifically, the receiving device displays the screen data of the virtual display device corresponding to the identification information according to the identification information in the seventh screen data. According to the identification information in the seventh screen data, the virtual display device to which each screen data in the seventh screen data belongs can be confirmed, and the receiving device can confirm the screen data of the virtual display device to be displayed according to the identification information in the seventh screen data.

[0135] It should be noted that steps S301-S306 and S401-S406 in the above embodiments can be selectively executed according to the device status of the virtual display device. It is understandable that the processing capability of the virtual display device is usually worse than that of the interactive tablet. Therefore, when the load of the virtual display device is low, the virtual display device can be used to merge the screen data once. However, if the load of the current virtual display device is relatively large, there will be no confluence between the virtual display devices. The data will be directly sent to the interactive tablet, which will merge the screen data and then send it to the receiving device.

[0136] See also Fig.10 , Fig.10 An example schematic diagram of a picture processing method is provided for the embodiment of this specification. Fig.10In the figure, virtual display device 1 and virtual display device 4 are first-level devices, virtual display device 2 and virtual display device 3 are second-level devices. When the level type is confirmed based on the device status, according to the device type indicated by the level type, the first-level device sends the first screen data to the second-level device, the second-level device merges the first screen data and the second screen data to obtain the third screen data and sends it to the interactive tablet, the interactive tablet merges the third screen data to obtain the seventh screen data, and sends the seventh screen data to the receiving device, so that the receiving device can display the screen data of any virtual display device based on the seventh screen data, that is, virtual display device 1 sends its screen data 1 to virtual display device 2, virtual display device 4 sends its screen data 2 to virtual display device 3, virtual display device 3 sends its screen data 3+4 to the interactive tablet, virtual display device 2 sends screen data 1+2 to the interactive tablet, and the interactive tablet merges the received screen data 1+2 and screen data 3+4 to obtain screen data 1+2+3+4 and sends it to the receiving device.

[0137] In the embodiment of the present specification, according to the device type indicated by the level type, the first level device sends the first screen data to the second level device, the second level device combines the first screen data and the second screen data to obtain the third screen data and sends it to the interactive tablet, the interactive tablet combines the third screen data to obtain the seventh screen data, and sends the seventh screen data to the receiving device, so that the receiving device can display the screen data of any virtual display device based on the seventh screen data. By setting the second level device, the processing pressure of the interactive tablet is reduced and the efficiency is improved. At the same time, the receiving device receives the seventh screen data sent by the interactive tablet to display the screen data of each virtual display device.

[0138] The following will be combined with the attached Figure 11-19 , the picture processing device provided by the embodiment of this specification is introduced in detail. It should be noted that the attached Figure 11-19 The screen processing device in this manual is used to execute Figure 2-Figure 10 For the convenience of explanation, only the part related to the embodiment of this specification is shown. For the specific technical details not disclosed, please refer to this specification. Figure 2-Figure 10 The embodiment shown.

[0139] See also Fig.11 , which shows a schematic diagram of the structure of a picture processing device provided by an exemplary embodiment of the present specification. The picture processing device can be implemented as all or part of the device through software, hardware or a combination of both. The device 1 includes a first confirmation module 11, a sending module 12, and a first merging module 13.

[0140] A first confirmation module 11 is used to confirm the level type of the virtual display device based on the device status;

[0141] A sending module 12, configured to send first picture data of the first-level device to a second-level device when the level type indicates that the virtual display device is a first-level device;

[0142] The first merging module 13 is used to merge the second picture data of the second-level device with the first picture data to obtain third picture data when the level type indicates that the virtual display device is a second-level device, and send the third picture data to the receiving device so that the receiving device can display the picture data of any one of the virtual display devices based on the third picture data.

[0143] Optionally, the sending module 12 is specifically used to send the first picture data of the first-level device to the third-level device when the level type indicates that the virtual display device is a first-level device; the third-level device is used to merge the first picture data with the fourth picture data of the third-level device, and send the fifth picture data obtained by the merge to the second-level device.

[0144] Optionally, the first merging module 13 is specifically used to merge the second screen data of the second-level device with the fifth screen data to obtain third screen data when the level type indicates that the virtual display device is a second-level device, and send the third screen data to a receiving device.

[0145] Optional, see Fig.12 , which shows a structural schematic diagram of a picture processing device provided by an exemplary embodiment of the present specification, wherein the device 1 also includes a fourth merging module 14, which is used to merge the fourth picture data of the third-level device with the first picture data when the level type indicates that the virtual display device is a third-level device to obtain fifth picture data, and send the fifth picture data to the second-level device.

[0146] Optionally, the first confirmation module 11 is specifically used to obtain a device state, and judge the device state based on a predefined communication protocol to obtain a judgment result;

[0147] The level type of the virtual display device is determined according to the determination result.

[0148] Optionally, the first confirmation module 11 is specifically used to confirm the device idleness of the virtual display device based on the device status;

[0149] Based on the device idleness, a level type of the virtual display device is determined.

[0150] Optionally, the first confirmation module 11 is specifically configured to confirm that the level type of the virtual display device is a first level type if the device idleness is less than or equal to an idleness threshold;

[0151] If the device idleness is greater than the idleness threshold, it is determined that the level type of the virtual display device is a second level type.

[0152] Optionally, the first merging module 13 is specifically configured to merge the second screen data of the second-level device with the first screen data to generate a merged data stream when the level type indicates that the virtual display device is a second-level device.

[0153] Acquire first identification information corresponding to the first screen data and second identification information corresponding to the second screen data;

[0154] The first identification information and the second identification information are written into the merged data stream to obtain third picture data.

[0155] See also Fig.13 , which shows a schematic diagram of the structure of a picture processing device provided by an exemplary embodiment of the present specification. The picture processing device can be implemented as all or part of the device through software, hardware or a combination of both. The device 2 includes a first receiving module 21 and a first display module 22.

[0156] The first receiving module 21 is used to receive third picture data sent by the second-level device; the third picture data is obtained by the second-level device merging the second picture data of the second-level device and the first picture data sent by the first-level device to the second-level device;

[0157] The first display module 22 is used to display the picture data of the virtual display device corresponding to the identification information based on the identification information in the third picture data.

[0158] See also Fig.14 , which shows a schematic diagram of the structure of a picture processing device provided by an exemplary embodiment of the present specification. The picture processing device can be implemented as all or part of the device through software, hardware or a combination of both. The device 3 includes a second confirmation module 31 and a second merging module 32.

[0159] A second confirmation module 31 is used to confirm the level type of the virtual display device based on the device status;

[0160] The second merging module 32 is used to send the first screen data of the first-level device to the interactive tablet when the level type indicates that the virtual display device is a first-level device, so that the interactive tablet merges the first screen data to obtain sixth screen data, and sends the sixth screen data to the receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the sixth screen data; there are at least two first-level devices.

[0161] Optional, see Fig.15 , which shows a schematic diagram of the structure of a picture processing device provided by an exemplary embodiment of this specification, wherein the device 3 further includes:

[0162] A second sending module 33 is used for sending the first screen data of the first-level device to the second-level device when the level type indicates that the virtual display device is a first-level device, and the second-level device is used for merging the first screen data and the second screen data of the second-level device, and sending the merged third screen data to the interactive tablet, so that the interactive tablet merges the third screen data to obtain seventh screen data, and sends the seventh screen data to the receiving device, so that the receiving device can display the screen data of any of the virtual display devices based on the seventh screen data;

[0163] The fifth merging module 34 is used to merge the second screen data of the second-level device with the first screen data to obtain third screen data when the level type indicates that the virtual display device is a second-level device, and send the third screen data to the interactive tablet so that the interactive tablet merges the third screen data to obtain seventh screen data, and sends the seventh screen data to the receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the seventh screen data.

[0164] See also Fig.16 , which shows a schematic diagram of the structure of a picture processing device provided by an exemplary embodiment of the present specification. The picture processing device can be implemented as all or part of the device through software, hardware or a combination of both. The device 4 includes a second receiving module 41 and a third merging module 42.

[0165] The second receiving module 41 is used for merging the first picture data sent by the first layer device to generate a merged data stream when receiving the first picture data sent by the first layer device; there are at least two first layer devices;

[0166] The third merging module 42 is used to confirm the identification information corresponding to each of the first screen data, write the identification information into the merged data stream, generate sixth screen data, and send the sixth screen data to the receiving device so that the receiving device can display the screen data of any of the virtual display devices based on the sixth screen data.

[0167] Optional, see Fig.17 , which shows a structural schematic diagram of a picture processing device provided by an exemplary embodiment of the present specification, wherein the device 4 also includes a fourth receiving module 43, which is used to merge the third picture data when receiving the third picture data sent by the second-level device to generate a merged data stream; confirm the identification information corresponding to each of the third picture data, write the identification information into the merged data stream, generate seventh picture data, and send the seventh picture data to the receiving device, so that the receiving device can display the picture data of any of the virtual display devices based on the seventh picture data.

[0168] See also Fig.18 , which shows a schematic diagram of the structure of a picture processing device provided by an exemplary embodiment of the present specification. The picture processing device can be implemented as all or part of the device through software, hardware or a combination of both. The device 1 includes a third receiving module 51 and a second display module 52.

[0169] The third receiving module 51 is used to receive the sixth screen data sent by the interactive tablet; the sixth screen data is obtained by the interactive tablet after merging the first screen data received from each of the first-level devices;

[0170] The second display module 52 is used to display the picture data of the virtual display device corresponding to the identification information based on the identification information in the sixth picture data.

[0171] Optional, see Fig.19 , which shows a structural schematic diagram of a screen processing device provided by an exemplary embodiment of the present specification, the device 5 also includes a fifth receiving module 53, which is used to receive the seventh screen data sent by the interactive tablet; the seventh screen data is obtained by the interactive tablet after merging and processing the third screen data sent by the second-level device; the third screen data is obtained by merging and processing the second screen data of the second-level device and the first screen data of each second-level device; based on the seventh screen data, the screen data of the virtual display device corresponding to the identification information is displayed.

[0172] It should be noted that, when the picture processing device provided in the above embodiment executes the picture processing method, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the picture processing device provided in the above embodiment and the picture processing method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0173] The serial numbers of the embodiments of the present specification are for description only and do not represent the advantages and disadvantages of the embodiments. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0174] The present specification also provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned Figure 1-Figure 10 The picture processing method of the embodiment shown in the figure can be specifically executed by referring to Figure 1-Figure 10 The specific description of the illustrated embodiment will not be repeated here.

[0175] Please refer to Fig. 20 , which shows a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.

[0176] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions and processes data of the terminal 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user pages, and applications; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0177] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable medium (Non-Transitory Computer-Readable Storage Medium). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system developed by Apple, including a system deeply developed based on the IOS system or other systems.

[0178] The memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve good operating results, the operating system allocates corresponding system resources to different third-party applications. However, the requirements for system resources in different application scenarios in the same third-party application are also different. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and third-party applications are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.

[0179] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0180] The input device 130 is used to receive input commands or data, and includes but is not limited to a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 140 is used to output commands or data, and includes but is not limited to a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are touch screen displays.

[0181] The touch display screen can be designed as a full screen, a curved screen or a special-shaped screen. The touch display screen can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the embodiments of this specification.

[0182] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above drawings does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the electronic device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a WiFi module, a power supply, and a Bluetooth module, which will not be described in detail here.

[0183] exist Fig. 20 In the electronic device shown, the processor 110 may be used to call a computer application stored in the memory 120 and specifically perform the following operations:

[0184] Determine the level type of the virtual display device based on the device status;

[0185] When the level type indicates that the virtual display device is a first-level device, sending the first picture data of the first-level device to a second-level device;

[0186] When the hierarchy type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to a receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the third screen data.

[0187] In one embodiment, when the processor 110 sends the first picture data of the first-level device to the second-level device when the level type indicates that the virtual display device is a first-level device, the processor 110 specifically performs the following operations:

[0188] When the hierarchy type indicates that the virtual display device is a first-level device, the first picture data of the first-level device is sent to the third-level device; the third-level device is used to merge the first picture data with the fourth picture data of the third-level device, and send the fifth picture data obtained by the merged processing to the second-level device.

[0189] In one embodiment, when the processor 110 performs, when the level type indicates that the virtual display device is a second-level device, merging the second screen data of the second-level device with the first screen data to obtain third screen data, and sending the third screen data to the receiving device, the processor 110 specifically performs the following operations:

[0190] When the level type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is combined with the fifth screen data to obtain third screen data, and the third screen data is sent to a receiving device.

[0191] In one embodiment, the processor 110 may further perform the following operations:

[0192] When the level type indicates that the virtual display device is a third-level device, fourth screen data of the third-level device is combined with the first screen data to obtain fifth screen data, and the fifth screen data is sent to the second-level device.

[0193] In one embodiment, when the processor 110 determines the level type of the virtual display device based on the device state, the processor 110 specifically performs the following operations:

[0194] Acquire the device status, and judge the device status based on a predefined communication protocol to obtain a judgment result;

[0195] The level type of the virtual display device is determined according to the determination result.

[0196] In one embodiment, when the processor 110 determines the level type of the virtual display device based on the device state, the processor 110 specifically performs the following operations:

[0197] confirming the device idleness of the virtual display device based on the device status;

[0198] Based on the device idleness, a level type of the virtual display device is determined.

[0199] In one embodiment, when the processor 110 determines the level type of the virtual display device based on the device idleness, the processor 110 specifically performs the following operations:

[0200] If the device idleness is less than or equal to the idleness threshold, confirming that the level type of the virtual display device is a first level type;

[0201] If the device idleness is greater than the idleness threshold, it is determined that the level type of the virtual display device is a second level type.

[0202] In one embodiment, when the processor 110 performs the step of merging the second screen data of the second-level device with the first screen data to obtain the third screen data when the level type indicates that the virtual display device is a second-level device, the processor 110 specifically performs the following operations:

[0203] When the level type indicates that the virtual display device is a second-level device, merging the second picture data of the second-level device with the first picture data to generate a merged data stream;

[0204] Acquire first identification information corresponding to the first screen data and second identification information corresponding to the second screen data;

[0205] The first identification information and the second identification information are written into the merged data stream to obtain third picture data.

[0206] In one embodiment, the processor 110 may also be used to call a computer application stored in the memory 120 and specifically perform the following operations:

[0207] receiving third picture data sent by a second-level device; the third picture data is obtained by the second-level device merging the second picture data of the second-level device and the first picture data sent by the first-level device to the second-level device;

[0208] Based on the identification information in the third picture data, the picture data of the virtual display device corresponding to the identification information is displayed.

[0209] In one embodiment, the processor 110 may also be used to call a computer application stored in the memory 120 and specifically perform the following operations:

[0210] Determine the level type of the virtual display device based on the device status;

[0211] When the hierarchy type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the interactive tablet, so that the interactive tablet merges the first screen data to obtain sixth screen data, and the sixth screen data is sent to the receiving device, so that the receiving device can display the screen data of any one of the virtual display devices based on the sixth screen data; there are at least two first-level devices.

[0212] In one embodiment, the processor 110 may further perform the following operations:

[0213] When the hierarchical type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the second-level device, the second-level device is used to merge the first screen data and the second screen data of the second-level device, and send the merged third screen data to the interactive tablet, so that the interactive tablet merges the third screen data to obtain seventh screen data, and sends the seventh screen data to the receiving device, so that the receiving device can display the screen data of any one of the virtual display devices based on the seventh screen data.

[0214] In one embodiment, the processor 110 may further perform the following operations:

[0215] When the hierarchy type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to the interactive tablet so that the interactive tablet merges the third screen data to obtain seventh screen data, and the seventh screen data is sent to the receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the seventh screen data.

[0216] In one embodiment, the processor 110 may also be used to call a computer application stored in the memory 120 and specifically perform the following operations:

[0217] When receiving first picture data sent by a first-level device, merging the first picture data to generate a merged data stream; there are at least two first-level devices;

[0218] Confirm the identification information corresponding to each of the first screen data, write the identification information into the merged data stream, generate sixth screen data, and send the sixth screen data to a receiving device so that the receiving device can display the screen data of any of the virtual display devices based on the sixth screen data.

[0219] In one embodiment, the processor 110 may further perform the following operations:

[0220] When receiving the third picture data sent by the second-level device, merging the third picture data to generate a merged data stream;

[0221] Confirm the identification information corresponding to each of the third screen data, write the identification information into the merged data stream, generate seventh screen data, and send the seventh screen data to a receiving device so that the receiving device can display the screen data of any of the virtual display devices based on the seventh screen data.

[0222] In one embodiment, the processor 110 may also be used to call a computer application stored in the memory 120 and specifically perform the following operations:

[0223] receiving sixth screen data sent by the interactive tablet; the sixth screen data is obtained by the interactive tablet merging the first screen data received from each of the first-level devices;

[0224] Based on the identification information in the sixth screen data, screen data of the virtual display device corresponding to the identification information is displayed.

[0225] In one embodiment, the processor 110 may further perform the following operations:

[0226] receiving seventh screen data sent by the interactive tablet; the seventh screen data is obtained by the interactive tablet after merging and processing the third screen data sent by the second-level device; the third screen data is obtained by each of the second-level devices by merging and processing the second screen data of the second-level device and the first screen data;

[0227] Based on the seventh screen data, screen data of the virtual display device corresponding to the identification information is displayed.

[0228] In an embodiment of the present specification, by confirming the level type of the virtual display device based on the device status, when the level type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the second-level device, and when the level type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to the receiving device, so that the receiving device can display the screen data of any virtual display device based on the third screen data. By merging the screen data through the virtual display device, the receiving device only receives one stream transmitted by the second-level device in the virtual display device, which reduces the network load pressure in the multi-virtual display device scenario and alleviates the decoding load pressure of the receiving device. The hierarchical type of the virtual display device can be confirmed by judging the device idleness and device status, so that different hierarchical types of devices can be confirmed in the virtual display device to realize different device functions. Specifically, three hierarchical types can be divided. When the hierarchical type indicates that the virtual display device is of the first hierarchical type, the first screen data of the first hierarchical device is sent to the third hierarchical device, and the third hierarchical device merges the first screen data with the fourth screen data of the third hierarchical device, and sends the fifth screen data obtained by the merged processing to the second hierarchical device, and the second hierarchical device merges the fifth screen data with the second screen data of the second hierarchical device to obtain the third screen data. By setting the third hierarchical device to act as a transfer between the first hierarchical device and the second hierarchical device, the video processing burden of the second hierarchical device can be reduced and the image processing efficiency can be improved. By confirming the hierarchical type of the virtual display device based on the device status, when the hierarchical type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the interactive tablet, so that the interactive tablet merges and processes each of the first screen data to obtain the sixth screen data, and sends the sixth screen data to the receiving device, so that the receiving device can display the screen data of any virtual display device based on the sixth screen data. The interactive tablet is used as a device that directly communicates with the receiving device, so that the receiving device only needs to receive the sixth screen data obtained by the interactive tablet merging and processing the first screen data of each first-level device to view the screen data of each virtual display device, which reduces the load pressure caused by decoding and rendering of the receiving device and increases the battery life. According to the device type indicated by the hierarchical type, the first-level device sends the first screen data to the second-level device, and the second-level device merges the first screen data and the second screen data to obtain the third screen data and sends it to the interactive tablet. The interactive tablet merges and processes the third screen data to obtain the seventh screen data, and sends the seventh screen data to the receiving device, so that the receiving device can display the screen data of any virtual display device based on the seventh screen data.The processing pressure of the interactive tablet is reduced by setting the second-level device, and the efficiency is improved. At the same time, the receiving device receives the seventh screen data sent by the interactive tablet to display the screen data of each virtual display device.

[0229] In addition, an embodiment of the present specification provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor of an electronic device, the processor can at least implement the above-mentioned Figures 1 to 10 The picture processing method provided in the illustrated embodiment.

[0230] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0231] The above disclosure is only the preferred embodiment of this specification, which certainly cannot be used to limit the scope of rights of this specification. Therefore, equivalent changes made according to the claims of this specification are still within the scope covered by this specification.

Claims

1. A picture processing method, characterized in that: Applied to a virtual display device, the method comprises: Determine the level type of the virtual display device based on the device status; When the level type indicates that the virtual display device is a first-level device, sending the first picture data of the first-level device to a second-level device; When the hierarchy type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to a receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the third screen data.

2. The method according to claim 1, characterized in that When the level type indicates that the virtual display device is a first-level device, sending the first picture data of the first-level device to the second-level device includes: When the level type indicates that the virtual display device is a first-level device, the first picture data of the first-level device is sent to the third-level device; the third-level device is used to merge the first picture data with the fourth picture data of the third-level device, and send the fifth picture data obtained by the merged processing to the second-level device; When the level type indicates that the virtual display device is a second-level device, merging the second screen data of the second-level device with the first screen data to obtain third screen data, and sending the third screen data to a receiving device, comprises: When the level type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is combined with the fifth screen data to obtain third screen data, and the third screen data is sent to a receiving device.

3. The method according to claim 2, characterized in that The method further comprises: When the level type indicates that the virtual display device is a third-level device, fourth screen data of the third-level device is combined with the first screen data to obtain fifth screen data, and the fifth screen data is sent to the second-level device.

4. The method according to claim 1, characterized in that The step of confirming the level type of the virtual display device based on the device state includes: Acquire the device status, and judge the device status based on a predefined communication protocol to obtain a judgment result; The level type of the virtual display device is determined according to the determination result.

5. The method according to claim 1, characterized in that The step of confirming the level type of the virtual display device based on the device state includes: confirming the device idleness of the virtual display device based on the device status; Based on the device idleness, a level type of the virtual display device is determined.

6. The method according to claim 5, characterized in that The step of determining the level type of the virtual display device based on the device idleness includes: If the device idleness is less than or equal to the idleness threshold, confirming that the level type of the virtual display device is a first level type; If the device idleness is greater than the idleness threshold, it is determined that the level type of the virtual display device is a second level type.

7. The method according to claim 1, characterized in that When the level type indicates that the virtual display device is a second-level device, merging the second screen data of the second-level device with the first screen data to obtain third screen data includes: When the level type indicates that the virtual display device is a second-level device, merging the second picture data of the second-level device with the first picture data to generate a merged data stream; Acquire first identification information corresponding to the first screen data and second identification information corresponding to the second screen data; The first identification information and the second identification information are written into the merged data stream to obtain third picture data.

8. A picture processing method, characterized in that: Applied to a receiving device, the method comprises: receiving third picture data sent by a second-level device; the third picture data is obtained by the second-level device merging the second picture data of the second-level device and the first picture data sent by the first-level device to the second-level device; Based on the identification information in the third picture data, the picture data of the virtual display device corresponding to the identification information is displayed.

9. A picture processing method, characterized in that: Applied to a virtual display device, the method comprises: Determine the level type of the virtual display device based on the device status; When the hierarchy type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the interactive tablet, so that the interactive tablet merges the first screen data to obtain sixth screen data, and the sixth screen data is sent to the receiving device, so that the receiving device can display the screen data of any one of the virtual display devices based on the sixth screen data; there are at least two first-level devices.

10. The method according to claim 9, characterized in that The method further comprises: When the hierarchical type indicates that the virtual display device is a first-level device, the first screen data of the first-level device is sent to the second-level device, the second-level device is used to merge the first screen data and the second screen data of the second-level device, and send the merged third screen data to the interactive tablet, so that the interactive tablet merges the third screen data to obtain seventh screen data, and sends the seventh screen data to the receiving device, so that the receiving device can display the screen data of any one of the virtual display devices based on the seventh screen data.

11. The method according to claim 9, characterized in that The method further comprises: When the hierarchy type indicates that the virtual display device is a second-level device, the second screen data of the second-level device is merged with the first screen data to obtain third screen data, and the third screen data is sent to the interactive tablet so that the interactive tablet merges the third screen data to obtain seventh screen data, and the seventh screen data is sent to the receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the seventh screen data.

12. A picture processing method, characterized in that: Applied to an interactive tablet, the method comprises: When receiving first picture data sent by a first-level device, merging the first picture data to generate a merged data stream; there are at least two first-level devices; Confirm the identification information corresponding to each of the first screen data, write the identification information into the merged data stream, generate sixth screen data, and send the sixth screen data to a receiving device so that the receiving device can display the screen data of any of the virtual display devices based on the sixth screen data.

13. The method according to claim 12, characterized in that The method further comprises: When receiving the third picture data sent by the second-level device, merging the third picture data to generate a merged data stream; Confirm the identification information corresponding to each of the third screen data, write the identification information into the merged data stream, generate seventh screen data, and send the seventh screen data to a receiving device so that the receiving device can display the screen data of any of the virtual display devices based on the seventh screen data.

14. A picture processing method, characterized in that: Applied to a receiving device, the method comprises: receiving sixth screen data sent by the interactive tablet; the sixth screen data is obtained by the interactive tablet merging the first screen data received from each of the first-level devices; Based on the identification information in the sixth screen data, screen data of the virtual display device corresponding to the identification information is displayed.

15. The method according to claim 14, characterized in that The method further comprises: receiving seventh screen data sent by the interactive tablet; the seventh screen data is obtained by the interactive tablet after merging and processing the third screen data sent by the second-level device; the third screen data is obtained by each of the second-level devices by merging and processing the second screen data of the second-level device and the first screen data; Based on the seventh screen data, screen data of the virtual display device corresponding to the identification information is displayed.

16. A picture processing device, characterized in that: The device comprises: A first confirmation module, used to confirm the level type of the virtual display device based on the device status; a sending module, configured to send first picture data of the first-level device to a second-level device when the level type indicates that the virtual display device is a first-level device; A first merging module is used to merge the second screen data of the second-level device with the first screen data to obtain third screen data when the level type indicates that the virtual display device is a second-level device, and send the third screen data to a receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the third screen data.

17. A picture processing device, characterized in that: The device comprises: A first receiving module is used to receive third picture data sent by a second-level device; the third picture data is obtained by the second-level device merging the second picture data of the second-level device and the first picture data sent by the first-level device to the second-level device; The first display module is used to display the screen data of the virtual display device corresponding to the identification information based on the identification information in the third screen data.

18. A picture processing device, characterized in that: The device comprises: A second confirmation module, used to confirm the level type of the virtual display device based on the device status; A second merging module is used to send the first screen data of the first-level device to the interactive tablet when the level type indicates that the virtual display device is a first-level device, so that the interactive tablet merges the first screen data to obtain sixth screen data, and sends the sixth screen data to the receiving device so that the receiving device can display the screen data of any one of the virtual display devices based on the sixth screen data; there are at least two first-level devices.

19. A picture processing device, characterized in that: The device comprises: A second receiving module is used for merging the first picture data sent by the first layer device to generate a merged data stream when receiving the first picture data sent by the first layer device; there are at least two first layer devices; The third merging module is used to confirm the identification information corresponding to each of the first screen data, write the identification information into the merged data stream, generate sixth screen data, and send the sixth screen data to the receiving device, so that the receiving device can display the screen data of any of the virtual display devices based on the sixth screen data.

20. A picture processing device, characterized in that: The device comprises: A third receiving module is used to receive sixth screen data sent by the interactive tablet; the sixth screen data is obtained by the interactive tablet after merging the first screen data received from each of the first-level devices; The second display module is used to display the screen data of the virtual display device corresponding to the identification information based on the identification information in the sixth screen data.

21. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer program, wherein the computer program is suitable for being loaded by the processor and executing the steps of the method according to any one of claims 1 to 15.

22. A computer storage medium, characterized in that The computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 15 are implemented.

Images