Picture processing method, device, apparatus and computer storage medium

Abstract

The embodiment of the present specification provides a picture processing method, device, equipment and computer storage medium, the method comprises: confirming the hierarchical type of a virtual display device based on the device state, when the hierarchical type indicates that the virtual display device is a first hierarchical device, sending first picture data of the first hierarchical device to a second hierarchical device, when the hierarchical type indicates that the virtual display device is a second hierarchical device, performing merging processing on second picture data of the second hierarchical device and the first picture data to obtain third picture data, and sending the third picture data to a receiving device, so that the receiving device displays picture data of any virtual display device based on the third picture data. Through the virtual display device, the picture data of multiple virtual display devices is merged, so that the picture of multiple virtual display devices is displayed on one receiving device, and the transmission consumption and load pressure of multiple virtual display devices being projected onto one receiving device are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a picture processing method, device, equipment and computer storage medium. BACKGROUND

[0002] Virtual reality (VR) is a computer simulation system that can create and experience virtual world, which uses computer to generate simulation environment to realize interactive three-dimensional view and dynamic entity behavior. Its multi-source information fusion characteristic makes users feel the real sense of being there. In the existing scheme, the display contents of multiple VR devices can be displayed on one VR device to realize real-time sharing of multi-to-one virtual reality scene.

[0003] However, the existing multi-VR device screen projection scheme consumes a lot of performance of the VR receiving device, which needs to decode and render multiple pictures at the same time. Therefore, it is urgent to propose a processing method capable of reducing the load pressure caused by decoding and rendering of the receiving device. SUMMARY

[0004] The main purpose of the present specification is to provide a picture processing method, device, equipment and computer storage medium, which aims to reduce the resource consumption of the video receiving device. The technical solution is as follows:

[0005] In a first aspect, the embodiment of the present specification provides a picture processing method applied to a virtual display device, comprising:

[0006] confirming the hierarchical type of the virtual display device based on the device state;

[0007] when the hierarchical type indicates that the virtual display device is a first hierarchical device, sending first picture data of the first hierarchical device to a second hierarchical device;

[0008] when the hierarchical type indicates that the virtual display device is a second hierarchical device, performing merging processing on second picture data of the second hierarchical device and the first picture data to obtain third picture data, and sending the third picture data to a receiving device, so that the receiving device displays picture data of any virtual display device based on the third picture data.

[0009] In a second aspect, the embodiment of the present specification provides a picture processing method applied to a receiving device, comprising:

[0010] receiving third picture data sent by a second hierarchical device; the third picture data is obtained by performing merging processing on second picture data of the second hierarchical device and first picture data sent by a first hierarchical device to the second hierarchical device by the second hierarchical device;

[0011] based on the identification information in the third picture data, display picture data of a virtual display device corresponding to the identification information.

[0012] In a third aspect, an embodiment of the present specification provides a picture processing method, applied to a virtual display device, comprising:

[0013] confirm a hierarchical type of the virtual display device based on a device state;

[0014] when the hierarchical type indicates that the virtual display device is a first hierarchical device, send first picture data of the first hierarchical device to an interactive panel, so that the interactive panel performs merging processing on each of the first picture data to obtain sixth picture data, and send the sixth picture data to a receiving device, so that the receiving device displays picture data of any virtual display device based on the sixth picture data; the first hierarchical device is at least two.

[0015] In a fourth aspect, an embodiment of the present specification provides a picture processing method, applied to an interactive panel, comprising:

[0016] when receiving first picture data sent by a first hierarchical device, perform merging processing on each of the first picture data to generate a merging data stream; the first hierarchical device is at least two;

[0017] confirm identification information corresponding to each of the first picture data, write the identification information into the merging data stream to generate sixth picture data, and send the sixth picture data to a receiving device, so that the receiving device displays picture data of any virtual display device based on the sixth picture data.

[0018] In a fifth aspect, an embodiment of the present specification provides a picture processing method, applied to a receiving device, comprising:

[0019] receive sixth picture data sent by the interactive panel; the sixth picture data is obtained by performing merging processing on each of first picture data of a first hierarchical device received by the interactive panel;

[0020] based on identification information in the sixth picture data, display picture data of a virtual display device corresponding to the identification information.

[0021] In a sixth aspect, an embodiment of the present specification provides a picture processing apparatus, comprising:

[0022] a first confirmation module configured to confirm a hierarchical type of a virtual display device based on a device state;

[0023] The sending module is configured to send first picture data of the first hierarchical device to a second hierarchical device when the hierarchical type indicates that the virtual display device is the first hierarchical device.

[0024] The first merging module is configured to merge second picture data of the second hierarchical device and the first picture data to obtain third picture data when the hierarchical type indicates that the virtual display device is the second hierarchical device, and send the third picture data to a receiving device, so that the receiving device displays picture data of any virtual display device based on the third picture data.

[0025] In a seventh aspect, an embodiment of the present specification provides a picture processing apparatus, comprising:

[0026] The first receiving module is configured to receive third picture data sent by a second hierarchical device; the third picture data is obtained by merging second picture data of the second hierarchical device and first picture data sent by a first hierarchical device to the second hierarchical device.

[0027] The first display module is configured to display picture data of a virtual display device corresponding to identification information based on the identification information in the third picture data.

[0028] In an eighth aspect, an embodiment of the present specification provides a picture processing apparatus, comprising:

[0029] The second confirming module is configured to confirm a hierarchical type of a virtual display device based on a device state.

[0030] The second merging module is configured to send first picture data of the first hierarchical device to an interactive panel when the hierarchical type indicates that the virtual display device is the first hierarchical device, so that the interactive panel merges each of the first picture data to obtain sixth picture data, and send the sixth picture data to a receiving device, so that the receiving device displays picture data of any virtual display device based on the sixth picture data; the first hierarchical device is at least two.

[0031] In a ninth aspect, an embodiment of the present specification provides a picture processing apparatus, comprising:

[0032] The second receiving module is configured to merge each of the first picture data to generate a merged data stream when receiving first picture data sent by a first hierarchical device; the first hierarchical device is at least two.

[0033] The third combining module is configured to confirm the identification information corresponding to each of the first picture data, write the identification information into the combined data stream, generate sixth picture data, and send the sixth picture data to the receiving device, so that the receiving device displays the picture data of any of the virtual display devices based on the sixth picture data.

[0034] In a tenth aspect, an embodiment of the present specification provides a picture processing apparatus, comprising:

[0035] The third receiving module is configured to receive the sixth picture data sent by the interactive panel; the sixth picture data is obtained by the interactive panel after combining and processing the first picture data of each of the first-level devices;

[0036] The second displaying module is configured to display the picture data of the virtual display device corresponding to the identification information based on the identification information in the sixth picture data.

[0037] In an eleventh aspect, an embodiment of the present specification provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is executed by the processor to implement the steps of the above method.

[0038] In a twelfth aspect, an embodiment of the present specification provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0039] In the embodiments of the present specification, the level type of the virtual display device is confirmed based on the device state, when the level type indicates that the virtual display device is a first-level device, the first picture data of the first-level device is sent to a second-level device, when the level type indicates that the virtual display device is a second-level device, the second picture data of the second-level device and the first picture data are combined and processed to obtain third picture data, and the third picture data is sent to a receiving device, so that the receiving device displays the picture data of any of the virtual display devices based on the third picture data. By using a specified virtual display device to combine the picture data of one or more virtual display devices, and transmitting the picture data to the receiving device by the specified virtual display device, the picture of multiple virtual display devices is displayed (screened) on one receiving device, the number of devices directly connected to the receiving device is reduced, the transmission consumption, communication consumption, and load pressure caused by decoding and rendering of the receiving device are reduced, and the endurance of the receiving device is increased. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present specification, and for those skilled in the art, other drawings can also be obtained from these accompanying drawings without creative labor.

[0041] Figure 1 is an example schematic diagram of a picture processing method provided by an embodiment of the present specification;

[0042] Figure 2 is a flowchart of a picture processing method provided by an embodiment of the present specification;

[0043] Figure 3 is an example schematic diagram of a picture processing method provided by an embodiment of the present specification;

[0044] Figure 4 is an example schematic diagram of a picture processing method provided by an embodiment of the present specification;

[0045] Figure 5 is a flowchart of a picture processing method provided by an embodiment of the present specification;

[0046] Figure 6 is an example schematic diagram of a picture processing method provided by an embodiment of the present specification;

[0047] Figure 7 is a flowchart of a picture processing method provided by an embodiment of the present specification;

[0048] Figure 8 is an example schematic diagram of a picture processing method provided by an embodiment of the present specification;

[0049] Figure 9 is a flowchart of a picture processing method provided by an embodiment of the present specification;

[0050] Figure 10 is an example schematic diagram of a picture processing method provided by an embodiment of the present specification;

[0051] Figure 11 is a structural schematic diagram of a picture processing device provided by an embodiment of the present specification;

[0052] Figure 12 is a structural schematic diagram of a picture processing device provided by an embodiment of the present specification;

[0053] Figure 13 is a structural schematic diagram of a picture processing device provided by an embodiment of the present specification;

[0054] Figure 14 is a structural schematic diagram of a picture processing device provided by an embodiment of the present specification;

[0055] Figure 15 is a structural schematic diagram of a picture processing device provided by an embodiment of the present specification;

[0056] Figure 16 is a structural schematic diagram of a picture processing device provided by an embodiment of the present specification;

[0057] Figure 17 is a structural schematic diagram of a picture processing device provided by an embodiment of the present specification;

[0058] Figure 18 is a structural schematic diagram of a picture processing device provided by an embodiment of the present specification;

[0059] Figure 19 is a structural schematic diagram of a picture processing device provided by an embodiment of the present specification;

[0060] Figure 20 is a structural schematic diagram of an electronic device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present specification will be described clearly and completely below in combination with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only part of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present specification.

[0062] The picture processing device can be a VR (Virtual Reality) device, an AR (Augmented Reality) device, an MR (Mixed Reality) device, a mobile phone, a computer, or a tablet computer, etc. terminal device, or a module for implementing a picture processing method in a terminal device. The picture processing device can confirm the hierarchical type of a virtual display device based on a device state. When the hierarchical type indicates that the virtual display device is a first hierarchical device, the picture processing device sends first picture data of the first hierarchical device to a second hierarchical device. When the hierarchical type indicates that the virtual display device is a second hierarchical device, the picture processing device performs merging processing on second picture data of the second hierarchical device and the first picture data to obtain third picture data, and sends the third picture data to a receiving device, so that the receiving device displays picture data of any virtual display device based on the third picture data.

[0063] Please see Figure 1An example schematic diagram of a picture processing method is provided for an embodiment of the present specification. The picture processing device can be a virtual display device, Figure 1 The virtual display device 1 and the virtual display device N (denoting N first-level devices) are first-level devices, and the virtual display device 2 and the virtual display device 3 are second-level devices. After confirming the level type based on the device state, the first-level device sends the first picture data to the second-level device according to the device type indicated by the level type. The second-level device performs merging processing on the second picture data and the first picture data to obtain third picture data, and sends the third picture data to a receiving device for the receiving device to display the picture data of any of the virtual display devices based on the third picture data. That is, the virtual display device 1 sends its picture data 1 to the virtual display device 2, the virtual display device 2 merges the received picture data 1 and its own picture data 2 to obtain picture data 1+2, and sends the picture data 1+2 to the receiving device. The virtual display device N sends the picture data N to the virtual display device 3, the virtual display device 3 merges the received picture data N and the picture data 3 to obtain picture data 3+N, and sends the picture data 3+N to the receiving device. The receiving device can be a virtual display device.

[0064] By using the picture processing method provided by the present specification, the picture data of one or more virtual display devices can be merged by a specified virtual display device, and the picture data is transmitted to a receiving device by the specified virtual display device, so as to realize the display (screen projection) of the pictures of multiple virtual display devices on one receiving device, reduce the number of devices directly connected to the receiving device, reduce the transmission consumption, communication consumption, and load pressure caused by decoding and rendering of the receiving device when multiple virtual display devices are projected on one receiving device, and increase the endurance of the receiving device.

[0065] The picture processing method provided by the present specification will be described in detail below in combination with specific embodiments.

[0066] Please refer to Figure 2 An example schematic diagram of a picture processing method is provided for an embodiment of the present specification. As shown in Figure 2 The specific process of the picture processing method is jointly described from the virtual display device side and the receiving device side. The method can include the following steps S101-S105.

[0067] S101, the virtual display device confirms the level type of the virtual display device based on the device state;

[0068] In an embodiment, the device state of the virtual display device can be used to determine the corresponding hierarchical type, which is the hierarchical type of the virtual display device in data transmission, and is used to restrict the data transmission content of the virtual display device. The device state of the virtual display device can be determined according to the CPU occupancy rate, the memory occupancy rate, the communication protocol, and the like.

[0069] Specifically, according to the device state, the virtual display device can be divided into a first-level device and a second-level device. The picture processing method provided in the present specification can be applied to one or more virtual display devices. When a receiving device needs to view the picture data in the virtual display device, the content required to be executed by the virtual display device can be determined according to the hierarchical type of the current virtual display device. If the virtual display device is a first-level device, the first-level device sends the first picture data currently displayed by the first-level device to a second-level device, and the second-level device sends the combined picture data to the receiving device. If the virtual display device is a second-level device, the second-level device combines the second picture data currently displayed by the second-level device with the first picture data sent by the first-level device, and then sends the combined picture data to the receiving device.

[0070] S102, when the hierarchical type indicates that the virtual display device is a first-level device, sending the first picture data of the first-level device to a second-level device;

[0071] Specifically, the first picture data is the picture data currently displayed by the first-level device, which can include image data in the real environment and the virtual environment. In an embodiment, the virtual display device in the present specification can be a VR device in a classroom teaching scenario. Students can watch learning content in the VR device through a head-mounted display. The learning content can be a teaching video, an experimental demonstration video, an interactive learning video, and the like recorded by a teacher, and the like, which is not limited in particular. When the hierarchical type indicates that the virtual display device is a first-level device, the virtual display device sends the first picture data to a second-level device. When sending the first picture data, the second-level device can be multiple, and the second-level device to which the first-level device sends can be a second-level device with a pre-confirmed association relationship, or can be any second-level device.

[0072] S103, when the hierarchical type indicates that the virtual display device is a second-level device, combining the second picture data of the second-level device with the first picture data to obtain third picture data, and sending the third picture data to a receiving device, so that the receiving device displays the picture data of any virtual display device based on the third picture data.

[0073] Specifically, when the level type indicates that the virtual display device is a second level device, the virtual display device receives first picture data from a first level device, and merges second picture data displayed by the second level device and the first picture data to obtain third picture data. The third picture data is a merged data stream including the first picture data and the second picture data. There can be one or more second level devices, preferably as few second level devices as possible connected directly to the receiving device to reduce the transmission burden of the receiving device as much as possible. One second level device can receive first picture data sent by one or more first level devices. It can be understood that, due to the need for real-time viewing of virtual display device picture content at the receiving device end, the first picture data of the first level device in real time and the second picture data of the second level device in real time can be merged to obtain merged third picture data, and the merged third picture data is transmitted to the receiving device, so that the receiving device can view the picture content of multiple virtual display devices by receiving only one stream. Among them, multiple virtual display devices can be connected through WiFi P2P (Wi-Fi Peer-to-Peer) networking, and this way of local area network connection does not occupy the bandwidth of the school WiFi, reduces the network load pressure in the classroom, and the transmission is fast and stable. At the same time, the video is merged by the processing capacity of the second level device, and the receiving device only needs to receive the merged third picture data transmitted by the second level device, and does not need to receive the video stream transmitted by each virtual display device, decode and render multiple pictures, so that the decoding load pressure of the receiving device is also shared.

[0074] S104, the receiving device receives third picture data sent by the second level device; the third picture data is obtained by merging second picture data of the second level device and first picture data sent by the first level device to the second level device by the second level device;

[0075] Specifically, the receiving device receives third picture data sent by the second level device to display picture content of the corresponding virtual display device according to the third picture data. It should be noted that the distance between students is relatively fixed and close, and P2P local area network connection can be used. The receiving device in the actual application scenario is mainly a virtual display device worn by a teacher, and the teacher usually walks in the classroom. If P2P connection is used, the connection effect is not very stable, therefore, WIFI connection is used between the second level device and the receiving device.

[0076] S105, the receiving device displays picture data of the virtual display device corresponding to the identification information based on the identification information in the third picture data.

[0077] Specifically, after receiving the third picture data, the receiving device can decode and render the picture data corresponding to the virtual display device according to the identification information in the third picture data. The identification information is information identifying the video source, which can be, for example, student identity information identified in the picture data, or identification data associated with the picture data and generated in addition. For example, see Figure 3 , Figure 3 An example schematic diagram of a picture processing method is provided for the embodiments of the present specification, Figure 3 An example of identification information is shown, which can be the student name written in the lower right corner of the video in the picture data, and can also be student information stored in association with the picture data, for example, when the second-level device generates the third picture data, the first-level device of each first picture data source in the third picture data can be identified, or the student name of the source. Please continue to see Figure 4 , Figure 4 An example schematic diagram of a picture processing method is provided for the embodiments of the present specification, and the effect achieved by the picture processing method of the present specification in the classroom application scenario is to collect the picture content of multiple students studying together by the second-level device and send it to the receiving device of the teacher. The teacher can view the picture data of all students in the receiving device, or can select the picture data of a student to be enlarged and displayed, for example, through the gesture recognition function provided by the VR device, through a virtual gesture to select the picture of student "Wang Wu" in the displayed third picture data to view details. Of course, the picture data to be viewed can also be selected by eye tracking and the like, which is not limited in detail.

[0078] In the embodiments of the present specification, the level type of the virtual display device is confirmed based on the device state, when the level type indicates that the virtual display device is a first-level device, the first picture data of the first-level device is sent to a second-level device, when the level type indicates that the virtual display device is a second-level device, the second picture data of the second-level device and the first picture data are merged to obtain third picture data, and the third picture data is sent to a receiving device for the receiving device to display the picture data of any virtual display device based on the third picture data. By merging the picture data by the virtual display device, the receiving device only receives one stream transmitted by the second-level device in the virtual display device, reducing the network load pressure in the multi-virtual display device scenario, and at the same time reducing the decoding load pressure of the receiving device.

[0079] Please see Figure 5 A flowchart of a picture processing method is provided for the embodiments of the present specification. As Figure 5As shown, the embodiment of the present specification sets forth the specific flow of the picture processing method from the virtual display device side, which can include the following steps S201-S204.

[0080] S201, confirming the hierarchical type of the virtual display device based on the device state;

[0081] In an embodiment, according to the device state, the virtual display device can be divided into a first hierarchical device, a second hierarchical device, and a third hierarchical device. The third hierarchical device can receive the first picture data sent by the first hierarchical device and merge the fourth picture data of the third hierarchical device to obtain the fifth picture data, and the third hierarchical device sends the fifth picture data to the second hierarchical device, and the second hierarchical device merges the fifth picture data with the second picture data of the second hierarchical device to obtain the third picture data. The third hierarchical device is used to transit between the first hierarchical device and the second hierarchical device to reduce the video processing burden of the second hierarchical device. It can be understood that, since the number of second hierarchical devices is usually small, when the number of virtual display devices is large, if the second hierarchical device is used to perform merging processing on a large number of first hierarchical devices, the amount of picture data to be merged by each second hierarchical device may be too large, which may cause low efficiency. Therefore, the third hierarchical device can also be selected in the virtual display device, and the third hierarchical device can participate in the picture data merging processing, so as to reasonably utilize the processing capacity of each virtual display device and improve the processing efficiency.

[0082] Further, in an embodiment, the confirming the hierarchical type of the virtual display device based on the device state includes:

[0083] S2011, obtaining the device state, judging the device state based on a pre-defined communication protocol to obtain a judgment result;

[0084] It can be understood that the hierarchical type of the virtual display device can be confirmed according to the pre-defined communication protocol. For example, in the communication protocol, it can be agreed that the virtual display device 1 needs to send its first picture data to the virtual display device 2, and the virtual display device 3 needs to send its first picture data to the virtual display device 4. Therefore, the device state of the virtual display device that only sends picture data is the sending end, and the corresponding hierarchical type is the first hierarchical type. The device state of the virtual display device that receives the first picture data is the receiving end, and the corresponding hierarchical type is the second hierarchical type. Therefore, according to the pre-defined communication protocol, it is first judged whether the virtual display device is the sending end or the receiving end, and a judgment result is obtained.

[0085] S2012, determining the hierarchical type of the virtual display device according to the judgment result.

[0086] Specifically, if the judgment result indicates that the virtual display device is a sending end, the level type is a first level, if the judgment result indicates that the virtual display device is a receiving end, the level type is a second level. Optionally, if the judgment result indicates that the virtual display device is both a sending end and a receiving end, the level type is a third level.

[0087] Further, in an embodiment, the level type of the virtual display device is confirmed based on the device state, including:

[0088] S2013, confirming a device idle degree of the virtual display device based on the device state;

[0089] S2014, confirming a level type of the virtual display device based on the device idle degree;

[0090] Specifically, a device with a large current remaining space (device idle degree) is a second level device or a third level device for performing the merging processing of the picture data. It can be understood that, since the picture merging needs to occupy the CPU, memory, etc. of the virtual display device, the device idle degree can be confirmed according to the CPU and memory occupation degree, and the lower the CPU and memory occupation degree, the larger the device idle degree. If the virtual display device has a high device idle degree, it can be used as a second level device for processing picture merging, and if the virtual display device has a low idle degree, it can be used as a first level device for sending picture data.

[0091] S2015, if the device idle degree is less than or equal to an idle degree threshold, confirming that the level type of the virtual display device is a first level type;

[0092] Specifically, according to the size relationship between the device idle degree and the idle degree threshold, the level type of the virtual display device can be confirmed. If the device idle degree of the virtual display device is less than or equal to the idle degree threshold, it means that the current device load is high and it cannot perform the video merging task, and then it can send the current first picture data to a second level device or a third level device for merging processing. Optionally, the device idle degree of the second level device can be lower than that of the third level device.

[0093] S2016, if the device idle degree is greater than the idle degree threshold, confirming that the level type of the virtual display device is a second level type;

[0094] Specifically, if the device idle degree of the virtual display device is greater than the idle degree threshold, it indicates that the current device load of the virtual display device is low, and the virtual display device has the ability to process the video merging task, and the virtual display device can be set as the second level type, and used for video merging processing. It can be understood that when the device idle degree of the virtual display device is greater than the idle degree threshold, the level type of the virtual display device can also be the third level type, and the virtual display device can be used as the third level device for video merging processing.

[0095] S202, when the level type indicates that the virtual display device is the first level device, the first picture data of the first level device is sent to the third level device; the third level device is used to merge the first picture data and fourth picture data of the third level device, and send fifth picture data obtained by merging to the second level device;

[0096] Specifically, when the virtual display device is the first level device, the first picture data of the first level device can also be sent to the third level device for merging processing, and then the fifth picture data containing the first picture data is sent to the second level device.

[0097] S203, when the level type indicates that the virtual display device is the third level device, the fourth picture data of the third level device is merged with the first picture data to obtain the fifth picture data, and the fifth picture data is sent to the second level device;

[0098] Specifically, when the level type indicates that the virtual display device is the third level device, the third level device can merge the fourth picture data currently displayed by the third level device with the first picture data to obtain the fifth picture data. It should be noted that there can be multiple third level devices, and the third level devices can further merge the picture data to reduce the processing amount of the second level device for merging multiple picture data.

[0099] S204, when the level type indicates that the virtual display device is the second level device, the second picture data of the second level device is merged with the fifth picture data to obtain the third picture data, and the third picture data is sent to the receiving device.

[0100] Specifically, when the level type indicates that the virtual display device is the second level device, the second level device merges the second picture data of the second level device with the fifth picture data sent by the third level device to obtain the third picture data, and sends the third picture data to the receiving device.

[0101] Further, in an embodiment, when the hierarchy type indicates that the virtual display device is a second hierarchy device, the second picture data of the second hierarchy device is merged with the first picture data to obtain third picture data, including:

[0102] S2041, when the hierarchy type indicates that the virtual display device is a second hierarchy device, the second picture data of the second hierarchy device is merged with the first picture data to obtain third picture data, including:

[0103] Specifically, when performing the merging process, the second hierarchy device can merge its current second picture data and the received first picture data to obtain the merged data stream. Wherein, the first picture data and the second picture data can be processed into picture data of the same video coding format and with the same code stream configuration parameters first, and then the code streams are merged. The code stream configuration parameters that change when merging the video code stream corresponding to the first picture data and the video code stream corresponding to the second picture data can be written into the merged data stream.

[0104] S2042, obtaining the first identification information corresponding to the first picture data and the second identification information corresponding to the second picture data;

[0105] Specifically, the first identification information corresponding to the first picture data and the second identification information of the second picture data can also be obtained, for example, the first identification information of the first picture data can be the device number of the first hierarchy device, and the second identification information of the second picture data can be the device number of the second hierarchy device. For example, the ffmpeg software can be used to merge the second picture data and the first picture data to generate the merged data stream, and the first identification information is generated according to the source of the first picture data, and the second identification information is generated according to the source of the second picture data.

[0106] S2043, writing the first identification information and the second identification information into the merged data stream to obtain third picture data.

[0107] Specifically, writing the first identification information and the second identification information into the merged data stream can obtain the third picture data carrying the identification information, so that the receiving device can confirm the virtual display device corresponding to each picture data based on the third picture data.

[0108] Please refer to Figure 6 , Figure 6 An example schematic diagram of a picture processing method is provided for the embodiments of the present specification, Figure 6The virtual display device 1 and the virtual display device 4 are first-level devices, the virtual display device 2 and the virtual display device 3 are third-level devices, and the virtual display device 5 is a second-level device. After the device state is confirmed based on the device state, according to the device type indicated by the level type, the first-level device sends the first picture data to the third-level device, the third-level device merges the first picture data of the first-level device with the fourth picture data currently displayed by the third-level device to obtain fifth picture data, and sends the fifth picture data to the second-level device. The second-level device merges the received fifth picture data with the second picture data of the second-level device to obtain third picture data, and sends the third picture data to the receiving device, so that the receiving device displays the picture data of any virtual display device based on the third picture data. That is, the virtual display device 1 sends the picture data 1 thereof to the virtual display device 2, the virtual display device 2 merges the received picture data 1 with the picture data 2 thereof to obtain picture data 1+2, and sends the picture data 1+2 to the virtual display device 5. The virtual display device 4 sends the picture data 4 to the virtual display device 3, the virtual display device 3 merges the received picture data 4 with the picture data 3 to obtain picture data 3+4, and sends the picture data 3+4 to the virtual display device 5. The virtual display device 5 merges the picture data 1+2 and the picture data 3+4 received by itself with the picture data 5 thereof to obtain picture data 1+2+3+4+5, and sends the picture data 1+2+3+4+5 to the receiving device.

[0109] In the embodiment of the present application, the level type of the virtual display device can be confirmed by the judgment result of the device idle degree and the device state, so that the devices of different level types are confirmed in the virtual display device, different device functions are realized, and specifically, three level types can be divided. When the level type indicates that the virtual display device is a first-level type, the first picture data of the first-level device is sent to the third-level device, the first picture data is merged with the fourth picture data of the third-level device, the fifth picture data obtained by the merging is sent to the second-level device, and the fifth picture data is merged with the second picture data of the second-level device by the second-level device to obtain the third picture data. By setting the third-level device as a relay between the first-level device and the second-level device, the video processing burden of the second-level device can be reduced, and the picture processing efficiency can be improved.

[0110] Please refer to Figure 7 A flowchart of a picture processing method is provided in the embodiment of the present application. As shown in Figure 7 The specific flow of the picture processing method is jointly described from the virtual display device side, the interactive tablet side and the receiving device side. The method can include the following steps S301-S306.

[0111] S301, the virtual display device confirms the hierarchical type of the virtual display device based on the device state;

[0112] In an embodiment, the virtual display device can confirm its hierarchical type according to the device state, and when the hierarchical type indicates that the virtual display device is a first-level device, the first picture data of the first-level device can be sent to the interactive tablet, and the interactive tablet can perform merging processing on the first picture data to obtain sixth picture data. Specifically, the interactive tablet and the virtual display device can also perform data transmission through WiFi P2P connection, and the interactive tablet can obtain the sixth picture data by merging the first picture data of multiple virtual display devices, and send the sixth picture data to the receiving device, so that the receiving device only needs to receive one stream sent by the interactive tablet, that is, the picture data of each first-level device can be obtained, the performance consumption of the receiving device for decoding picture data is reduced, and the occupation of bandwidth is also reduced. The step of confirming the hierarchical type of the virtual display device based on the device state can refer to the description of step S201 in the above embodiment, and will not be repeated here.

[0113] S302, when the hierarchical type indicates that the virtual display device is a first-level device, the first picture data of the first-level device is sent to the interactive tablet, so that the interactive tablet performs merging processing on each first picture data to obtain sixth picture data, and the sixth picture data is sent to the receiving device, so that the receiving device displays the picture data of any virtual display device based on the sixth picture data; the first-level device is at least two.

[0114] It should be noted that in the case of directly connecting the interactive tablet and the receiving device, when the first-level device that needs to transmit picture data is one, the first picture data of the first-level device can also be sent to the interactive tablet, and the interactive tablet directly sends the first picture data to the receiving device. When the first-level device is at least two, the interactive tablet needs to merge the multiple first picture data obtained to obtain the sixth picture data.

[0115] S303, the interactive tablet merges each first picture data when receiving the first picture data sent by the first-level device to generate a merged data stream; the first-level device is at least two;

[0116] Specifically, the interactive tablet receives the first picture data sent by the first level device, and performs merging processing on each first picture data to generate a merged data stream. For example, the merged data stream can be generated based on the video coding format and configuration parameters of each first picture data. Alternatively, the first picture data can be synthesized based on layout parameters set according to the layout mode, so that each picture data in the merged data stream can be spliced according to the layout mode, so that the picture data of each virtual display device can be displayed on the receiving device in the layout mode. The interactive tablet can be a display screen with a host module and a whole machine module, and the hardware configuration of the whole machine module is relatively low. In addition to configuring the operating system and the basic processing software (such as whiteboard writing software and annotation application software), it does not support the installation of other application software. The host module is an open pluggable specification (OPS) module on the interactive tablet, and the application software running on the interactive tablet is installed on the host module. The host module can run Windows, high-performance Android operating system, etc. For example, the host module can adopt a PC module, a high-performance Android module, etc. It can be understood as a computer. By installing relevant application software on the host module, the corresponding functions can be realized. For example, if file display is needed, relevant display software needs to be installed on the host module; if video conference function is needed, video conference software needs to be installed on the host module, etc.

[0117] S304, the interactive tablet confirms the identification information corresponding to each first picture data, writes the identification information into the merged data stream, generates sixth picture data, and sends the sixth picture data to the receiving device, so that the receiving device displays the picture data of any virtual display device based on the sixth picture data.

[0118] Specifically, the interactive tablet can also confirm the first identification information corresponding to each first picture data, and write the first identification information into the merged data stream to obtain sixth picture data carrying the identification information. The sixth picture data is the picture data obtained by the interactive tablet after merging the first picture data sent by each first level device. The interactive tablet sends the sixth picture data to the receiving device, so that the receiving device can display the picture data of any virtual display device according to the sixth device.

[0119] S305, the receiving device receives the sixth picture data sent by the interactive tablet; the sixth picture data is obtained by the interactive tablet after merging the first picture data of each first level device;

[0120] S306, the receiving device displays the picture data of the virtual display device corresponding to the identification information based on the identification information in the sixth picture data.

[0121] Specifically, the receiving device displays the picture data of the virtual display device corresponding to the identification information according to the identification information in the received sixth picture data. The receiving device can decode and render the picture data of the corresponding virtual display device according to the identification information in the sixth picture data.

[0122] Please refer to Figure 8 , Figure 8 An example schematic diagram of a picture processing method is provided for the embodiments of the present specification. Figure 8 In the example, the virtual display device 1, the virtual display device 2, the virtual display device 3, and the virtual display device 4 are first-level devices. After confirming the level type based on the device state, the first-level devices send the first picture data to the interactive panel according to the device type indicated by the level type. The interactive panel performs merging processing on the first picture data of the first-level devices to obtain the sixth picture data, and sends the sixth picture data to the receiving device for the receiving device to display the picture data of any virtual display device based on the sixth picture data. That is, the virtual display device 1 sends its picture data 1 to the interactive panel, the virtual display device 2 sends its picture data 2 to the interactive panel, the virtual display device 3 sends its picture data 3 to the interactive panel, and the virtual display device 4 sends its picture data 4 to the interactive panel. The interactive panel merges the received picture data 1, picture data 2, picture data 3, and picture data 4 to obtain picture data 1+2+3+4 and sends it to the receiving device.

[0123] In the embodiments of the present specification, the level type of the virtual display device is confirmed based on the device state. When the level type indicates that the virtual display device is a first-level device, the first picture data of the first-level device is sent to the interactive panel, so that the interactive panel performs merging processing on each of the first picture data to obtain the sixth picture data, and sends the sixth picture data to the receiving device for the receiving device to display the picture data of any virtual display device based on the sixth picture data. The interactive panel is used as a device that directly communicates with the receiving device, so that the receiving device only needs to receive the sixth picture data obtained by the interactive panel by merging and processing the first picture data of each first-level device, and can view the picture data of each virtual display device, thereby reducing the load pressure caused by decoding and rendering of the receiving device and increasing the endurance.

[0124] Please refer to Figure 9 A flowchart of a picture processing method is provided for the embodiments of the present specification. As shown in Figure 9 the embodiments of the present specification jointly elaborate the specific flow of the picture processing method from the virtual display device side, the interactive panel side, and the receiving device side. The method can include the following steps S401-S406.

[0125] S401, when the level type indicates that the virtual display device is a first level device, sending first picture data of the first level device to a second level device, the second level device being configured to perform merging processing on the first picture data and second picture data of the second level device, and send third picture data obtained by the merging processing to an interactive panel, so that the interactive panel performs merging processing on each of the third picture data to obtain seventh picture data, and send the seventh picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data.

[0126] In an embodiment, the virtual display devices can be divided into first level devices and second level devices according to the level type. The first level device sends first picture data of the first level device to the second level device, and the second level device performs merging processing on the first picture data and second picture data of the second level device, and sends third picture data obtained by the merging processing to the interactive panel. It can be understood that if the virtual display devices have spare loads, the virtual display devices are merged first to reduce the processing pressure of the interactive panel and improve the efficiency. For example, there are 30 virtual display devices, and the second level devices can be selected to merge into 10, 15 or 20 third picture data according to the actual situation, the third picture data is sent to the interactive panel, and the interactive panel performs merging processing on each of the third picture data to obtain seventh picture data. The interactive panel sends the seventh picture data to the receiving device, and the receiving device displays picture data of the received virtual display devices based on the seventh picture data.

[0127] S402, when the level type indicates that the virtual display device is a second level device, performing merging processing on second picture data of the second level device and the first picture data to obtain third picture data, sending the third picture data to an interactive panel, so that the interactive panel performs merging processing on each of the third picture data to obtain seventh picture data, and sending the seventh picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data.

[0128] S403, when the interactive panel receives the third picture data sent by the second level device, performing merging processing on each of the third picture data to generate a merging data stream;

[0129] S404, the interactive panel confirms identification information corresponding to each of the third picture data, writes the first identification information into the merging data stream to generate seventh picture data, and sends the seventh picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data.

[0130] In an embodiment, the interactive tablet receives third picture data from the second level device, merges the third picture data to obtain seventh picture data. Specifically, the third picture data can be merged to obtain a merged data stream, and then the identification information corresponding to each third picture data is confirmed and written into the merged data stream to generate the seventh picture data. The third picture data includes picture data from the first level device and the second level device, and therefore the identification information can indicate the source of each picture data in the third picture data.

[0131] S405, the receiving device receives the seventh picture data sent by the interactive tablet; the seventh picture data is obtained by merging the third picture data sent by the second level device by the interactive tablet; the third picture data is obtained by merging the second picture data of the second level device and the first picture data by each second level device;

[0132] Specifically, the receiving device receives the seventh picture data sent by the interactive tablet. In a feasible implementation, the receiving device and the interactive tablet can use WiFi connection to transmit data, and the receiving device only receives picture data from the interactive tablet.

[0133] S406, the receiving device displays the picture data of the virtual display device corresponding to the identification information based on the identification information in the seventh picture data.

[0134] Specifically, the receiving device displays the picture data of the virtual display device corresponding to the identification information according to the identification information in the seventh picture data. According to the identification information in the seventh picture data, the virtual display device to which each picture data in the seventh picture data belongs can be confirmed, and the receiving device can confirm the picture data of the virtual display device to be displayed according to the identification information in the seventh picture data.

[0135] It should be noted that steps S301-S306 and S401-S406 in the above embodiments can be executed according to the device state of the virtual display device. It can be understood that the processing capacity of the virtual display device is usually worse than that of the interactive tablet, and therefore when the load of the virtual display device is low, the virtual display device can be used to merge the picture data once, but if the load of the virtual display device is large, the virtual display device will not merge the picture data, and the interactive tablet will merge the picture data and then send the picture data to the receiving device.

[0136] Please refer to Figure 10 , Figure 10 An example schematic diagram of a picture processing method is provided for the embodiments of the present specification, Figure 10Virtual display devices 1 and 4 are first-level devices, and virtual display devices 2 and 3 are second-level devices. After the level type is determined based on the device status, the first-level device sends the first screen data to the second-level device according to the device type indicated by the level type. The second-level device merges the first screen data and the second screen data to obtain the third screen data and sends it to the interactive flat panel. The interactive flat panel merges the third screen data to obtain the seventh screen data and sends it to the receiving device so that the receiving device can display the screen data of any virtual display device based on the seventh screen data. That is, virtual display device 1 sends its screen data 1 to virtual display device 2, virtual display device 4 sends its screen data 2 to virtual display device 3, virtual display device 3 sends its screen data 3+4 to the interactive flat panel, virtual display device 2 sends its screen data 1+2 to the interactive flat panel, and the interactive flat panel merges the received screen data 1+2 and screen data 3+4 to obtain screen data 1+2+3+4 and sends it to the receiving device.

[0137] In the embodiments of this specification, according to the device type indicated by the layer type, the first-layer device sends the first screen data to the second-layer device. The second-layer device merges the first and second screen data to obtain third screen data, which is then sent to the interactive flat panel. The interactive flat panel performs merging processing on the third screen data to obtain seventh screen data, which is then sent to the receiving device. The receiving device then displays the screen data of any virtual display device based on the seventh screen data. By setting up a second-layer device, the processing load on the interactive flat panel is reduced, improving efficiency. Simultaneously, the receiving device receives the seventh screen data sent by the interactive flat panel to display the screen data of each virtual display device.

[0138] The following will be combined with the appendix Figures 11-19 This specification provides a detailed description of the image processing apparatus provided in the embodiments. It should be noted that the appendix... Figures 11-19 The image processing device in this manual is used to execute the image processing device described herein. Figures 2-10 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 2-10 The example shown.

[0139] Please see Figure 11 This diagram illustrates the structure of a screen processing apparatus provided in an exemplary embodiment of this specification. The screen processing apparatus can be implemented as all or part of a device through software, hardware, or a combination of both. The device 1 includes a first confirmation module 11, a sending module 12, and a first merging module 13.

[0140] The first confirmation module 11 is used to confirm the layer type of the virtual display device based on the device status.

[0141] The sending module 12 is configured to send the first picture data of the first hierarchical device to a second hierarchical device when the hierarchical type indicates that the virtual display device is the first hierarchical device.

[0142] The first merging module 13 is configured to merge the second picture data of the second hierarchical device and the first picture data to obtain third picture data when the hierarchical type indicates that the virtual display device is the second hierarchical device, and send the third picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the third picture data.

[0143] Optionally, the sending module 12 is specifically configured to send the first picture data of the first hierarchical device to a third hierarchical device when the hierarchical type indicates that the virtual display device is the first hierarchical device, and the third hierarchical device is configured to merge the first picture data and fourth picture data of the third hierarchical device, and send fifth picture data obtained by the merging to the second hierarchical device.

[0144] Optionally, the first merging module 13 is specifically configured to merge the second picture data of the second hierarchical device and the fifth picture data to obtain third picture data when the hierarchical type indicates that the virtual display device is the second hierarchical device, and send the third picture data to the receiving device.

[0145] Optionally, referring to Figure 12 which shows a structural schematic diagram of a picture processing apparatus provided by one example embodiment of the present specification, the apparatus 1 further comprises a fourth merging module 14 configured to merge the fourth picture data of the third hierarchical device and the first picture data to obtain fifth picture data when the hierarchical type indicates that the virtual display device is the third hierarchical device, and send the fifth picture data to the second hierarchical device.

[0146] Optionally, the first confirming module 11 is specifically configured to acquire a device state, judge the device state based on a pre-defined communication protocol to obtain a judgment result.

[0147] The hierarchical type of the virtual display device is determined according to the judgment result.

[0148] Optionally, the first confirming module 11 is specifically configured to confirm a device idle degree of the virtual display device based on the device state.

[0149] The hierarchical type of the virtual display device is confirmed based on the device idle degree.

[0150] Optionally, the first confirming module 11 is specifically configured to confirm the level type of the virtual display device as a first level type if the device idle degree is less than or equal to an idle degree threshold.

[0151] confirm the level type of the virtual display device as a second level type if the device idle degree is greater than the idle degree threshold.

[0152] Optionally, the first merging module 13 is specifically configured to perform merging processing on the second picture data of the second level device and the first picture data when the level type indicates that the virtual display device is a second level device, to generate a merged data stream.

[0153] obtain first identification information corresponding to the first picture data and second identification information corresponding to the second picture data;

[0154] write the first identification information and the second identification information into the merged data stream to obtain third picture data.

[0155] Please refer to Figure 13 which shows a structural schematic diagram of a picture processing apparatus provided by an example embodiment of the present specification. The picture processing apparatus can be realized as all or part of the apparatus by software, hardware, or a combination of both. The apparatus 2 includes a first receiving module 21 and a first display module 22.

[0156] The first receiving module 21 is configured to receive third picture data sent by a second level device; the third picture data is obtained by performing merging processing on second picture data of the second level device and first picture data sent by a first level device to the second level device by the second level device.

[0157] The first display module 22 is configured to display picture data of a virtual display device corresponding to identification information based on the identification information in the third picture data.

[0158] Please refer to Figure 14 which shows a structural schematic diagram of a picture processing apparatus provided by an example embodiment of the present specification. The picture processing apparatus can be realized as all or part of the apparatus by software, hardware, or a combination of both. The apparatus 3 includes a second confirming module 31 and a second merging module 32.

[0159] The second confirming module 31 is configured to confirm a level type of a virtual display device based on a device state.

[0160] The second combining module 32 is configured to, when the level type indicates that the virtual display device is a first level device, send first picture data of the first level device to the interactive panel, so that the interactive panel combines each of the first picture data to obtain sixth picture data, and send the sixth picture data to the receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the sixth picture data; and the first level device is at least two.

[0161] Optionally, referring to Figure 15 which shows a structural schematic diagram of a picture processing apparatus provided by an example embodiment of the present specification. The apparatus 3 further includes:

[0162] The second sending module 33 is configured to, when the level type indicates that the virtual display device is a first level device, send first picture data of the first level device to a second level device, and the second level device is configured to combine the first picture data and second picture data of the second level device, send third picture data obtained by the combination to the interactive panel, so that the interactive panel combines each of the third picture data to obtain seventh picture data, and send the seventh picture data to the receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data.

[0163] The fifth combining module 34 is configured to, when the level type indicates that the virtual display device is a second level device, combine the second picture data of the second level device and the first picture data to obtain third picture data, send the third picture data to the interactive panel, so that the interactive panel combines each of the third picture data to obtain seventh picture data, and send the seventh picture data to the receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data.

[0164] Optionally, referring to Figure 16 which shows a structural schematic diagram of a picture processing apparatus provided by an example embodiment of the present specification. The picture processing apparatus can be realized as all or part of the apparatus by software, hardware or a combination of both. The apparatus 4 includes a second receiving module 41 and a third combining module 42.

[0165] The second receiving module 41 is configured to, when receiving first picture data sent by a first level device, combine each of the first picture data to generate a combined data stream; and the first level device is at least two.

[0166] The third combining module 42 is configured to confirm the identification information corresponding to each of the first picture data, write the identification information into the combined data stream, generate sixth picture data, and send the sixth picture data to the receiving device, so that the receiving device displays the picture data of any virtual display device based on the sixth picture data.

[0167] Optionally, referring to Figure 17 Fig. 4 shows a structural schematic diagram of a picture processing apparatus according to an example embodiment of the present specification. The apparatus 4 further includes a fourth receiving module 43 configured to perform combining processing on each of the third picture data sent by the second level device when the third picture data is received, generate a combined data stream, confirm the identification information corresponding to each of the third picture data, write the identification information into the combined data stream, generate seventh picture data, and send the seventh picture data to the receiving device, so that the receiving device displays the picture data of any virtual display device based on the seventh picture data.

[0168] Optionally, referring to Figure 18 Fig. 4 shows a structural schematic diagram of a picture processing apparatus according to an example embodiment of the present specification. The apparatus 4 further includes a fourth receiving module 43 configured to perform combining processing on each of the third picture data sent by the second level device when the third picture data is received, generate a combined data stream, confirm the identification information corresponding to each of the third picture data, write the identification information into the combined data stream, generate seventh picture data, and send the seventh picture data to the receiving device, so that the receiving device displays the picture data of any virtual display device based on the seventh picture data.

[0169] The third receiving module 51 is configured to receive the sixth picture data sent by the interactive panel; the sixth picture data is obtained by performing combining processing on each of the first picture data of the first level device received by the interactive panel;

[0170] The second display module 52 is configured to display the picture data of the virtual display device corresponding to the identification information based on the identification information in the sixth picture data.

[0171] Optionally, referring to Figure 19 Fig. 5 shows a structural schematic diagram of a picture processing apparatus according to an example embodiment of the present specification. The apparatus 5 further includes a fifth receiving module 53 configured to receive the seventh picture data sent by the interactive panel; the seventh picture data is obtained by performing combining processing on the third picture data sent by the second level device by the interactive panel; the third picture data is obtained by performing combining processing on the second picture data of the second level device and the first picture data by each of the second level device; and display the picture data of the virtual display device corresponding to the identification information based on the seventh picture data.

[0172] It should be noted that the screen processing apparatus provided in the above embodiments is only used for illustrating the division of the above functional modules when the screen processing method is performed, and in actual applications, the above functions can be completed by different functional modules according to the needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the screen processing apparatus and the screen processing method provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be described here.

[0173] The above embodiment numbers of the present specification are only for description, and do not represent the advantages and disadvantages of the embodiments. In some cases, the actions or steps recited in the claims can be executed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.

[0174] The present specification also provides a computer storage medium, which stores a computer program. The computer program is executed by a processor to implement the screen processing method of the above Figures 1-10 embodiments. The specific implementation process can refer to the specific description of the above Figures 1-10 embodiments, which will not be described here.

[0175] Please refer to Figure 20 , which shows a structural schematic diagram of an electronic device provided by an exemplary embodiment of the present specification. The electronic device in the present specification can include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140 and a bus 150. The processor 110, the memory 120, the input device 130 and the output device 140 can be connected through the bus 150.

[0176] The processor 110 can include one or more processing cores. The processor 110 connects various parts within the entire electronic device with various interfaces and lines, performs various functions of the terminal 100 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Alternatively, the processor 110 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 110 can integrate a combination of one or several of a central processing unit (CPU), a graphics processor (GPU), and a modem, etc. Among them, the CPU mainly processes an operating system, a user page, and an application program, etc.; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 110, but can be implemented by a separate communication chip.

[0177] The memory 120 can include a random access memory (RAM) and can also include a read-only memory (ROM). Alternatively, the memory 120 includes a non-transitory computer-readable storage medium. The memory 120 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc., and the operating system can be an Android system, an IOS system developed by Apple Inc., a system based on the deep development of the Android system, a system based on the deep development of the IOS system, or other systems.

[0178] The memory 120 can be divided into an operating system space and a user space, the operating system runs in the operating system space, and native and third-party application programs run in the user space. In order to ensure that different third-party application programs can achieve good running effect, the operating system allocates corresponding system resources for different third-party application programs. However, there are also differences in the demand for system resources in different application scenarios in the same third-party application program. For example, in the local resource loading scenario, the third-party application program has a higher requirement for the disk reading speed; in the animation rendering scenario, the third-party application program has a higher requirement for the GPU performance. However, the operating system and the third-party application program are independent of each other, and the operating system often cannot timely perceive the current application scenario of the third-party application program, so that the operating system cannot perform targeted system resource adaptation according to the specific application scenario of the third-party application program.

[0179] In order to enable the operating system to distinguish the specific application scenario of the third-party application program, it is necessary to open up the data communication between the third-party application program and the operating system, so that the operating system can obtain the current scenario information of the third-party application program at any time, and then perform targeted system resource adaptation based on the current scenario.

[0180] The input device 130 is configured to receive input instructions or data, and the input device 130 includes but is not limited to a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 140 is configured to output instructions or data, and the output device 140 includes but is not limited to a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are a touch display screen.

[0181] The touch display screen can be designed as a full screen, a curved screen, or a special-shaped screen. The touch display screen can also be designed as a combination of a full screen and a curved screen, a combination of a special-shaped screen and a curved screen, which is not limited in the embodiments of the present application.

[0182] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above-mentioned drawings does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the drawings, or combine certain components, or different component arrangements. For example, the electronic device also includes radio frequency circuit, input unit, sensor, audio circuit, WiFi module, power supply, Bluetooth module and other components, which are not described here.

[0183] In Figure 20 In the electronic device shown in the figure, the processor 110 can be configured to invoke the computer application program stored in the memory 120, and specifically perform the following operations:

[0184] Confirm the hierarchical type of the virtual display device based on the device state;

[0185] send the first picture data of the first hierarchical device to a second hierarchical device when the hierarchical type indicates that the virtual display device is the first hierarchical device;

[0186] merge the second picture data of the second hierarchical device with the first picture data to obtain third picture data when the hierarchical type indicates that the virtual display device is the second hierarchical device, and send the third picture data to a receiving device for the receiving device to display picture data of any of the virtual display devices based on the third picture data.

[0187] In one embodiment, the processor 110, when performing the operation of sending the first picture data of the first hierarchical device to a second hierarchical device when the hierarchical type indicates that the virtual display device is the first hierarchical device, specifically performs the following operation:

[0188] send the first picture data of the first hierarchical device to a third hierarchical device when the hierarchical type indicates that the virtual display device is the first hierarchical device; the third hierarchical device is configured to merge the first picture data with fourth picture data of the third hierarchical device, and send fifth picture data obtained by the merging to the second hierarchical device.

[0189] In one embodiment, the processor 110, when performing the operation of merging the second picture data of the second hierarchical device with the first picture data to obtain third picture data when the hierarchical type indicates that the virtual display device is the second hierarchical device, and sending the third picture data to a receiving device, specifically performs the following operation:

[0190] merge the second picture data of the second hierarchical device with the fifth picture data to obtain third picture data when the hierarchical type indicates that the virtual display device is the second hierarchical device, and send the third picture data to a receiving device.

[0191] In one embodiment, the processor 110 can further perform the following operation:

[0192] merge the fourth picture data of the third hierarchical device with the first picture data to obtain fifth picture data when the hierarchical type indicates that the virtual display device is the third hierarchical device, and send the fifth picture data to the second hierarchical device.

[0193] In one embodiment, the processor 110, when performing the operation of confirming the hierarchical type of the virtual display device based on the device state, specifically performs the following operation:

[0194] acquire a device state, judge the device state based on a predefined communication protocol to obtain a judgment result;

[0195] determine a hierarchical type of the virtual display device according to the judgment result.

[0196] In one embodiment, the processor 110, when determining the hierarchical type of the virtual display device based on the device state, specifically performs the following operations:

[0197] determine a device idleness of the virtual display device based on the device state;

[0198] determine the hierarchical type of the virtual display device based on the device idleness.

[0199] In one embodiment, the processor 110, when determining the hierarchical type of the virtual display device based on the device idleness, specifically performs the following operations:

[0200] if the device idleness is less than or equal to an idleness threshold, determine the hierarchical type of the virtual display device as a first hierarchical type;

[0201] if the device idleness is greater than the idleness threshold, determine the hierarchical type of the virtual display device as a second hierarchical type.

[0202] In one embodiment, the processor 110, when performing the merging processing of the second picture data of the second hierarchical device and the first picture data to obtain the third picture data when the hierarchical type indicates that the virtual display device is a second hierarchical device, specifically performs the following operations:

[0203] when the hierarchical type indicates that the virtual display device is a second hierarchical device, perform merging processing of second picture data of the second hierarchical device and the first picture data to generate a merged data stream;

[0204] obtain first identification information corresponding to a setting of the first picture data and second identification information corresponding to a setting of the second picture data;

[0205] write the first identification information and the second identification information into the merged data stream to obtain third picture data.

[0206] In one embodiment, the processor 110 can also be configured to invoke a computer application stored in the memory 120 and specifically perform the following operations:

[0207] receive third picture data sent by the second level device; the third picture data is obtained by the second level device after performing merging processing on second picture data of the second level device and first picture data sent by the first level device to the second level device;

[0208] based on the identification information in the third picture data, display picture data of the virtual display device corresponding to the identification information.

[0209] In one embodiment, the processor 110 can also be configured to invoke a computer application stored in the memory 120, and specifically perform the following operations:

[0210] confirm the level type of the virtual display device based on the device state;

[0211] when the level type indicates that the virtual display device is a first level device, send first picture data of the first level device to the interactive panel, so that the interactive panel performs merging processing on each of the first picture data to obtain sixth picture data, and send the sixth picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the sixth picture data; the first level device is at least two.

[0212] In one embodiment, the processor 110 can also perform the following operations:

[0213] when the level type indicates that the virtual display device is a first level device, send first picture data of the first level device to a second level device, the second level device is configured to perform merging processing on the first picture data and second picture data of the second level device, send third picture data obtained by the merging processing to the interactive panel, so that the interactive panel performs merging processing on each of the third picture data to obtain seventh picture data, and send the seventh picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data.

[0214] In one embodiment, the processor 110 can also perform the following operations:

[0215] when the level type indicates that the virtual display device is a second level device, perform merging processing on second picture data of the second level device and the first picture data to obtain third picture data, send the third picture data to the interactive panel, so that the interactive panel performs merging processing on each of the third picture data to obtain seventh picture data, and send the seventh picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data.

[0216] In one embodiment, the processor 110 can further be configured to invoke a computer application stored in the memory 120, and specifically perform the following operations:

[0217] When receiving the first picture data sent by the first level device, the first picture data is merged to generate a merged data stream;

[0218] Confirm the identification information corresponding to each of the third picture data, write the identification information into the merged data stream, generate the seventh picture data, and send the seventh picture data to the receiving device, so that the receiving device displays the picture data of any virtual display device based on the seventh picture data.

[0219] In one embodiment, the processor 110 can further perform the following operations:

[0220] When receiving the third picture data sent by the second level device, the third picture data is merged to generate a merged data stream;

[0221] Confirm the identification information corresponding to each of the third picture data, write the identification information into the merged data stream, generate the seventh picture data, and send the seventh picture data to the receiving device, so that the receiving device displays the picture data of any virtual display device based on the seventh picture data.

[0222] In one embodiment, the processor 110 can further be configured to invoke a computer application stored in the memory 120, and specifically perform the following operations:

[0223] Receive the sixth picture data sent by the interactive panel; the sixth picture data is obtained by merging the first picture data of each of the first level devices received by the interactive panel;

[0224] Based on the identification information in the sixth picture data, display the picture data of the virtual display device corresponding to the identification information.

[0225] In one embodiment, the processor 110 can further perform the following operations:

[0226] Receive the seventh picture data sent by the interactive panel; the seventh picture data is obtained by merging the third picture data sent by the second level device by the interactive panel; the third picture data is obtained by merging the second picture data of the second level device and the first picture data by each of the second level devices;

[0227] based on the seventh picture data, display picture data of a virtual display device corresponding to the identification information.

[0228] In the embodiments of the present application, the level type of the virtual display device is determined based on the device state, when the level type indicates that the virtual display device is a first level device, the first picture data of the first level device is sent to a second level device, when the level type indicates that the virtual display device is a second level device, the second picture data of the second level device is combined with the first picture data to obtain third picture data, and the third picture data is sent to a receiving device for displaying the picture data of any virtual display device based on the third picture data. By combining the picture data in the virtual display device, the receiving device only receives one stream transmitted by the second level device in the virtual display device, which reduces the network load pressure in the multi-virtual display device scenario and reduces the decoding load pressure of the receiving device. The level type of the virtual display device is determined based on the device state and the judgment result, so that different level type devices are determined in the virtual display device, different device functions are realized, and three level types can be divided, when the level type indicates that the virtual display device is a first level type, the first picture data of the first level device is sent to a third level device, the third level device combines the first picture data with the fourth picture data of the third level device, the fifth picture data obtained by combining is sent to the second level device, and the second level device combines the fifth picture data with the second picture data of the second level device to obtain the third picture data. By setting the third level device as a relay between the first level device and the second level device, the video processing burden of the second level device can be reduced, and the picture processing efficiency can be improved. The level type of the virtual display device is determined based on the device state, when the level type indicates that the virtual display device is a first level device, the first picture data of the first level device is sent to an interactive panel, the interactive panel combines each first picture data to obtain sixth picture data, and the sixth picture data is sent to a receiving device for displaying the picture data of any virtual display device based on the sixth picture data. The interactive panel is directly communicated with the receiving device, so that the receiving device only needs to receive the sixth picture data obtained by combining the first picture data of each first level device in the interactive panel, and the picture data of each virtual display device can be viewed, which reduces the load pressure caused by decoding and rendering of the receiving device and increases the endurance. According to the device type indicated by the level type, the first level device sends the first picture data to the second level device, the second level device combines the first picture data and the second picture data to obtain the third picture data, the interactive panel combines the third picture data to obtain the seventh picture data, and the seventh picture data is sent to the receiving device for displaying the picture data of any virtual display device based on the seventh picture data.The processing pressure of the interactive panel is reduced by setting the second level device, and the efficiency is improved. Meanwhile, the receiving device receives the seventh picture data sent by the interactive panel to display the picture data of each virtual display device.

[0229] In addition, the embodiments of the present specification provide a computer program product, which comprises a computer program. When the computer program is executed by a processor of an electronic device, the processor can at least implement the picture processing method provided in the foregoing Figures 1 to 10 embodiments.

[0230] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments of the method can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0231] The above only describes the preferred embodiments of the present specification, and of course cannot limit the scope of the rights of the present specification. Therefore, equivalent changes made according to the claims of the present specification still fall within the scope of the present specification.

Claims

1. A picture processing method, characterized by, The method is applied to a virtual display device, and comprises the following steps: confirming a hierarchical type of the virtual display device based on a device state; when the hierarchical type indicates that the virtual display device is a first hierarchical device, sending first picture data of the first hierarchical device to a second hierarchical device; when the hierarchical type indicates that the virtual display device is a second hierarchical device, performing a merging process on second picture data of the second hierarchical device and the first picture data to obtain third picture data, and sending the third picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the third picture data; the step of confirming the hierarchical type of the virtual display device based on the device state comprises the following steps: confirming a device idle degree of the virtual display device based on the device state; if the device idle degree is less than or equal to an idle degree threshold, confirming that the hierarchical type of the virtual display device is a first hierarchical type; if the device idle degree is greater than the idle degree threshold, confirming that the hierarchical type of the virtual display device is a second hierarchical type.

2. The method of claim 1, wherein, the step of, when the hierarchical type indicates that the virtual display device is a first hierarchical device, sending first picture data of the first hierarchical device to a second hierarchical device comprises the following steps: when the hierarchical type indicates that the virtual display device is a first hierarchical device, sending first picture data of the first hierarchical device to a third hierarchical device; the third hierarchical device is used for performing a merging process on the first picture data and fourth picture data of the third hierarchical device, and sending fifth picture data obtained by the merging process to the second hierarchical device; the step of, when the hierarchical type indicates that the virtual display device is a second hierarchical device, performing a merging process on second picture data of the second hierarchical device and the first picture data to obtain third picture data, and sending the third picture data to a receiving device comprises the following steps: when the hierarchical type indicates that the virtual display device is a second hierarchical device, performing a merging process on second picture data of the second hierarchical device and the fifth picture data to obtain third picture data, and sending the third picture data to a receiving device.

3. The method of claim 2, wherein, the method further comprises the following steps: when the hierarchical type indicates that the virtual display device is a third hierarchical device, performing a merging process on fourth picture data of the third hierarchical device and the first picture data to obtain fifth picture data, and sending the fifth picture data to the second hierarchical device.

4. The method of claim 1, wherein, the step of, when the hierarchical type indicates that the virtual display device is a second hierarchical device, performing a merging process on second picture data of the second hierarchical device and the first picture data to obtain third picture data comprises the following steps: when the hierarchical type indicates that the virtual display device is a second hierarchical device, performing a merging process on second picture data of the second hierarchical device and the first picture data to obtain third picture data, and sending the third picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the third picture data; obtaining first identification information corresponding to the first picture data and second identification information corresponding to the second picture data; Write the first identification information and the second identification information into the merged data stream to obtain third picture data.

5. A picture processing method characterized by comprising: The method applied to a receiving device, the method comprising: Receiving third picture data sent by a second level device; the third picture data is obtained by the second level device performing merging processing on second picture data of the second level device and first picture data sent by a first level device to the second level device; the first level device is a virtual display device of a first level type, the first level type is determined when a device idle degree of the virtual display device is less than or equal to an idle degree threshold; the second level device is a virtual display device of a second level type, the second level type is determined when the device idle degree of the virtual display device is greater than the idle degree threshold; Displaying picture data of any of the virtual display devices based on the third picture data.

6. A picture processing method characterized by comprising: The method applied to a virtual display device, the method comprising: Confirming a level type of the virtual display device based on a device state; When the level type indicates that the virtual display device is a first level device, sending first picture data of the first level device to a second level device, the second level device is used to perform merging processing on the first picture data and second picture data of the second level device, sending third picture data obtained by the merging to an interactive panel, so that the interactive panel performs merging processing on each of the third picture data to obtain seventh picture data, and sending the seventh picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data; The confirming of the level type of the virtual display device based on the device state comprises: Confirming a device idle degree of the virtual display device based on the device state; If the device idle degree is less than or equal to an idle degree threshold, confirming that the level type of the virtual display device is a first level type; If the device idle degree is greater than the idle degree threshold, confirming that the level type of the virtual display device is a second level type.

7. The method of claim 6, wherein, The method further comprises: When the level type indicates that the virtual display device is a second level device, performing merging processing on second picture data of the second level device and the first picture data to obtain third picture data, sending the third picture data to an interactive panel, so that the interactive panel performs merging processing on each of the third picture data to obtain seventh picture data, and sending the seventh picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data.

8. A picture processing method characterized by comprising: The method applied to an interactive panel, the method comprising: When receiving the third picture data sent by the second level device, the third picture data is processed to generate a merged data stream; the third picture data is obtained by processing the second picture data of the second level device and the first picture data of the first level device; the first level device is a virtual display device of a first level type, and the first level type is determined when the device idle degree of the virtual display device is less than or equal to an idle degree threshold; the second level device is a virtual display device of a second level type, and the second level type is determined when the device idle degree of the virtual display device is greater than the idle degree threshold; The identification information corresponding to each third picture data is confirmed, the identification information is written into the merged data stream to generate seventh picture data, and the seventh picture data is sent to a receiving device to display the picture data of any virtual display device based on the seventh picture data; the identification information is used to identify the source of each picture data in the third picture data.

9. A picture processing method, characterized by, The method applied to a receiving device comprises: Receiving the seventh picture data sent by the interactive panel; the seventh picture data is obtained by processing the third picture data sent by the second level device; the third picture data is obtained by processing the second picture data of the second level device and the first picture data of the first level device; the first level device is a virtual display device of a first level type, and the first level type is determined when the device idle degree of the virtual display device is less than or equal to an idle degree threshold; the second level device is a virtual display device of a second level type, and the second level type is determined when the device idle degree of the virtual display device is greater than the idle degree threshold; Displaying the picture data of any virtual display device based on the seventh picture data.

10. An image processing apparatus characterized by comprising: The device comprises: A first confirmation module is configured to confirm the level type of a virtual display device based on a device state; the confirmation of the level type of the virtual display device based on the device state comprises: confirming the device idle degree of the virtual display device based on the device state; if the device idle degree is less than or equal to an idle degree threshold, confirming that the level type of the virtual display device is a first level type; if the device idle degree is greater than the idle degree threshold, confirming that the level type of the virtual display device is a second level type; A sending module is configured to send the first picture data of the first level device to a second level device when the level type indicates that the virtual display device is a first level device; A first merging module is configured to process the second picture data of the second level device and the first picture data to obtain third picture data when the level type indicates that the virtual display device is a second level device, and send the third picture data to a receiving device to display the picture data of any virtual display device based on the third picture data.

11. An image processing apparatus characterized by comprising: The device comprises: The first receiving module is configured to receive third picture data sent by a second level device; the third picture data is obtained by performing merging processing on second picture data of the second level device and first picture data sent by a first level device to the second level device; the first level device is a virtual display device of a first level type, and the first level type is determined when a device idle degree of the virtual display device is less than or equal to an idle degree threshold; the second level device is a virtual display device of a second level type, and the second level type is determined when the device idle degree of the virtual display device is greater than the idle degree threshold; The first display module is configured to display picture data of any virtual display device based on the third picture data.

12. An image processing apparatus characterized by comprising: The apparatus comprises: The second confirming module is configured to confirm a level type of a virtual display device based on a device state; the confirming the level type of the virtual display device based on the device state comprises: confirming a device idle degree of the virtual display device based on the device state; if the device idle degree is less than or equal to an idle degree threshold, confirming that the level type of the virtual display device is a first level type; if the device idle degree is greater than the idle degree threshold, confirming that the level type of the virtual display device is a second level type; The second sending module is configured to, when the level type indicates that the virtual display device is a first level device, send first picture data of the first level device to a second level device; the second level device is configured to perform merging processing on the first picture data and second picture data of the second level device, send third picture data obtained by the merging to an interactive panel, so that the interactive panel performs merging processing on each third picture data to obtain seventh picture data, and send the seventh picture data to a receiving device, so that the receiving device displays picture data of any virtual display device based on the seventh picture data.

13. An image processing apparatus characterized by comprising: The apparatus comprises: The fourth receiving module is configured to, when receiving third picture data sent by the second-level device, perform merging processing on each of the third picture data to generate a merged data stream; the third picture data is obtained by performing merging processing on second picture data of the second-level device and first picture data of a first-level device by each of the second-level devices; the first-level device is a first-level type virtual display device, and the first-level type is determined when a device idle degree of the virtual display device is less than or equal to an idle degree threshold; the second-level device is a second-level type virtual display device, and the second-level type is determined when the device idle degree of the virtual display device is greater than the idle degree threshold; identifying identification information corresponding to each of the third picture data, writing the identification information into the merged data stream to generate seventh picture data, and sending the seventh picture data to a receiving device, so that the receiving device displays picture data of any of the virtual display devices based on the seventh picture data; and the identification information is used to identify a source of each of the picture data in the third picture data.

14. An image processing apparatus characterized by comprising: The apparatus includes: The fifth receiving module is configured to receive seventh picture data sent by an interactive panel; the seventh picture data is obtained by performing merging processing on third picture data sent by the second-level device by the interactive panel; the third picture data is obtained by performing merging processing on second picture data of the second-level device and first picture data of a first-level device by each of the second-level devices; the first-level device is a first-level type virtual display device, and the first-level type is determined when a device idle degree of the virtual display device is less than or equal to an idle degree threshold; the second-level device is a second-level type virtual display device, and the second-level type is determined when the device idle degree of the virtual display device is greater than the idle degree threshold; and displaying picture data of any of the virtual display devices based on the seventh picture data.

15. An electronic device, comprising: The apparatus includes: A processor and a memory; The memory stores a computer program, and the computer program is adapted to be loaded and executed by the processor to perform steps of the method in any one of claims 1 to 9.

16. A computer storage medium, comprising, The computer storage medium stores a computer program, and the computer program is executed by the processor to perform steps of the method in any one of claims 1 to 9.

Images