Data processing method, storage medium and electronic equipment

Abstract

The invention discloses a data processing method, a storage medium and electronic equipment, and relates to the technical field of multimedia, the data processing method comprises the following steps: merging screen images of a plurality of target terminals to obtain a fused screen image; in response to identification of first touch data for a target terminal, adjusting the fused screen image according to the first touch data; and segmenting the adjusted fused screen image, and displaying sub-images obtained by segmentation on each target terminal so as to improve the operation efficiency of screen touch.

Description

Technical Field

[0001] The present invention relates to the field of multimedia technologies, and in particular, to a data processing method, a storage medium, and an electronic device. Background Art

[0002] With the continuous deepening of the education informatization reform, the wide application of multimedia touch large-screen integrated machines in school education not only greatly enriches teaching means, but also promotes the transformation of the traditional teaching mode to a highly interactive multimedia teaching mode. In particular, it is worth noting that the classroom configuration has been upgraded from a single device to a pattern where multiple touch devices are used together, which not only requires the multimediaization of teaching content, but also poses new challenges to the supporting technologies, especially how to efficiently use these devices for collaborative teaching.

[0003] Currently, multimedia touch large-screen products driven by the Android system on the market generally have limitations in the control mechanism, which is mainly reflected in that their control units only support single-screen operations. Then, if it is necessary to control multiple device screens, each set of devices needs to be equipped with its own control system. It is not only necessary to start the same application program or multimedia content on each device respectively, but also due to the lack of an effective synchronization and linkage mechanism between devices, the displayed content cannot be uniformly managed and coordinated for display, seriously affecting the information consistency and interaction experience in a multi-screen environment. Therefore, this control method of single-screen operation has low efficiency. Summary of the Invention

[0004] The present invention aims to at least solve one of the technical problems in the related technologies to some extent. For this purpose, an object of the present invention is to provide a data processing method, a storage medium, and an electronic device to improve the operation efficiency of screen touch.

[0005] According to a first aspect of an embodiment of the present invention, a data processing method is provided, and the method includes:

[0006] Merge the screen images of multiple target terminals to obtain a fused screen image;

[0007] In response to recognizing first touch data for a target terminal, adjust the fused screen image according to the first touch data;

[0008] Segment the adjusted fused screen image and display the segmented sub-images on each target terminal.

[0009] Optionally, the step of in response to recognizing first touch data for a target terminal and adjusting the fused screen image according to the first touch data includes:

[0010] Convert the first touch data into second touch data in the image coordinate system of the fused screen image;

[0011] Adjust the fused screen image based on the adjustment instruction represented by the second touch data.

[0012] Optionally, the adjustment instruction includes any one of the following: a rotation instruction, a sliding instruction, and a size adjustment instruction.

[0013] Optionally, the merging of the screen images of multiple target terminals to obtain a fused screen image includes:

[0014] Generate buffer pixel data based on the pixel data of the screen images of each target terminal;

[0015] Generate layer data based on the buffer pixel data;

[0016] Synthesize the generated layer data into a fused screen image.

[0017] Optionally, the generating of buffer pixel data based on the pixel data of the screen images of each target terminal includes:

[0018] If there is a copy instruction for the first target terminal, then copy the pixel data of the screen image of the first target terminal based on the number of target terminals to obtain copied pixel data, and merge the generated copied pixel data and the pixel data of the screen image of the first target terminal into a target buffer to obtain buffer pixel data.

[0019] Optionally, the generating of buffer pixel data based on the pixel data of the screen images of each target terminal includes:

[0020] If there is a swap instruction for the second target terminal, after merging the pixel data of each target terminal into a target buffer, swap the position of the pixel data of the second target terminal in the target buffer with the position of the pixel data of a third target terminal indicated by the swap instruction in the target buffer.

[0021] Optionally, the buffer pixel data is generated based on GraphicBuffer, and the generating of layer data based on the buffer pixel data includes:

[0022] Pass the buffer pixel data to SurfaceFlinger through BufferQueue to obtain the layer data generated by SurfaceFlinger.

[0023] Optionally, the splitting of the adjusted fused screen image includes:

[0024] Store the adjusted fused screen image in the framebuffer, and segment the framebuffer based on the Hardware Composer.

[0025] According to a second aspect of an embodiment of the present invention, there is provided a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above data processing method.

[0026] According to a third aspect of an embodiment of the present invention, there is provided an electronic device including: a memory and a processor; the memory stores a computer program, which, when executed by the processor, implements the above data processing method.

[0027] In the solution provided by the embodiment of the present invention, by merging the screen images of multiple target terminals to obtain a fused screen image and implementing screen operations on the fused screen image based on the first touch data, it is possible to affect the screen changes of each target terminal through the fused screen image only by operating on one target terminal. Furthermore, when the fused screen image is segmented and displayed on each target terminal, the display result of each target terminal is controlled by the first touch data, thereby achieving synchronization and consistency in control, and the operation process only needs to be implemented on any one terminal instead of separately controlling multiple target terminals, improving the operation efficiency of screen touch and the cross-screen interaction experience.

[0028] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present invention. Description of the Drawings

[0029] Figure 1 is a flowchart of the first data processing method provided by an embodiment of the present invention;

[0030] Figure 2 is a flowchart of the second data processing method provided by an embodiment of the present invention;

[0031] Figure 3 is a flowchart of the third data processing method provided by an embodiment of the present invention;

[0032] Figure 4 is a structural diagram of an electronic device provided by an embodiment of the present invention. Detailed Embodiments

[0033] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0034] A data processing method, a storage medium, and an electronic device according to embodiments of the present invention will be described below with reference to the accompanying drawings.

[0035] In one embodiment of the present invention, refer to Figure 1 , a data processing method is provided, including the following steps S101 - S103.

[0036] S101: Merge the screen images of multiple target terminals to obtain a fused screen image;

[0037] S102: In response to recognizing first touch data for a target terminal, adjust the fused screen image according to the first touch data;

[0038] S103: Segment the adjusted fused screen image and display the segmented sub - images on each target terminal.

[0039] The target terminal can be any type of touch device, such as a mobile phone, a tablet computer, etc.

[0040] The touch data can be obtained based on gesture actions such as clicking and dragging by the user on the screen. For any target terminal device, a corresponding relationship between different gesture actions and different types of adjustment instructions can be pre - stored, so that the touch data containing gesture actions can be converted into corresponding instructions. For example, in the preset corresponding relationship, when the user double - clicks on the screen, the corresponding adjustment instruction is to enlarge the screen, etc.

[0041] In the case of having multiple first touch data, the adjustment instructions represented by the first touch data can be executed in order according to the generation time of the first touch data, or overwritten in chronological order, with the later adjustment instruction overwriting the earlier one.

[0042] The adjustment instructions can specifically include: rotation instruction, sliding instruction, size adjustment instruction.

[0043] Among them, the rotation instruction can be to flip the screen, such as rotating the screen clockwise by 90 degrees, etc., or to perform a mirror flip on the screen image.

[0044] The size adjustment instruction includes instructions indicating enlargement or scaling.

[0045] In addition, in the absence of the first touch data, adjustment may not be performed, and the content of multiple screen images may be directly driven for independent output.

[0046] In one embodiment of the present invention, the fused screen image may be adjusted in the following manner: converting the first touch data into second touch data in the image coordinate system of the fused screen image; and adjusting the fused screen image based on the adjustment instruction represented by the second touch data.

[0047] For the first touch data, the terminal device of the screen where it is located may preset a coordinate system to record the coordinate position where the first touch data is generated. The method of obtaining the fused screen image is as Figure 2 shown. After the Android main control chip connects three target terminals through HDMI (High-Definition Multimedia Interface), by splicing the screen images of each terminal device, Android can splice the screen images of Screen 1, Screen 2, and Screen 3 with a resolution of 3840*2160, that is, realize screen fusion through Android Hardware, and the resolution of the obtained fused screen image is 11520*2160.

[0048] In some other embodiments, the number of target terminals may be more than three, and the sizes of the screen images of the target terminals may also be different.

[0049] In this case, the second touch data in the image coordinate system of the fused screen image may be obtained based on the splicing direction, splicing order, and the coordinate position generated by the original first touch data.

[0050] For example, for Figure 2 shown, since it is a vertical splicing, the ordinate of the second touch data remains unchanged, and the abscissa is the abscissa of the original coordinate position plus the widths of several screen images. Specifically, taking the origin of the image coordinate system of the fused screen image being at the lower left corner as an example, the coordinates of the second touch data of Screen 1 remain unchanged, the abscissa of the second touch data of Screen 2 is the original abscissa + 3840, and the abscissa of the second touch data of Screen 3 is the original abscissa + 3840*2.

[0051] In this way, the click and flip operations on the original target terminal can be accurately mapped to the specified area in the fused screen image.

[0052] Correspondingly, during the splitting process, splitting is performed according to the splicing direction and the ratio of the widths of the screen images of each target terminal, and sub-images can be obtained. In this way, Screen 1, Screen 2, and Screen 3 can all obtain the adjusted sub-images, realizing three-screen differential display, that is, the screens of each different target terminal display independent content respectively.

[0053] In another embodiment, a single screen image, such as Screen 1, can also be stretched to 11520*2160 and split for display, so that the adapted media files or applications can achieve triple-screen display, and triple-screen linkage control can be realized by operating Screen 1.

[0054] Alternatively, through the above mirroring operation, it is possible to specify mirroring to a dual screen or a triple screen to ensure instant sharing and synchronization of information. And in this mode, since the first touch data can be sent from any target terminal, the interaction is not limited to the original screen, and the mirrored screen can also be operated.

[0055] In the solution provided by the embodiment of the present invention, by merging the screen images of multiple target terminals, a fused screen image is obtained, and screen operations are implemented on the fused screen image based on the first touch data, so that by operating on one target terminal, the screen changes of each target terminal can be affected through the fused screen image. Furthermore, when the fused screen image is split and displayed on each target terminal, the display result of each target terminal is controlled by the first touch data, thereby achieving synchronization and consistency in control, and the operation process only needs to be implemented on any one terminal, rather than separately controlling on multiple target terminals, improving the operation efficiency of screen touch and the cross-screen interaction experience.

[0056] As described above, by means of a single Android operating system, unified control of multiple touch display screens can be achieved, fundamentally avoiding the capital redundancy of repeated purchase of complete machines and significantly optimizing cost-effectiveness; moreover, it supports the function of magnifying content to triple-screen display, and this design optimization significantly improves the visual experience of remote students in large classrooms. Through the screen mirroring function, flexible and instant scheduling and management of screen content can be realized, greatly enhancing the multitasking flexibility in teaching scenarios. Compared with the limitation in traditional large-screen systems that the course process must be frequently interrupted to switch display content, this solution significantly reduces the number of interruptions in the teaching process and improves the coherence and teaching efficiency of the classroom.

[0057] In one embodiment of the present invention, the splicing of the screen is achieved in the following manner, including:

[0058] Generating buffer pixel data based on the pixel data of the screen images of each target terminal;

[0059] Generating layer data based on the buffer pixel data;

[0060] Synthesizing the generated layer data into a fused screen image.

[0061] Specifically, the buffer pixel data is generated based on GraphicBuffer. The buffer pixel data is passed to SurfaceFlinger through BufferQueue, and the layer data generated by SurfaceFlinger can be obtained.

[0062] In this way, after the pixel data is stored in order, SurfaceFlinger will render each abstract layer of each image in sequence according to geometric attributes and hierarchical order, complete the splicing of the screen image, and complete the rendering of the fused screen image through the GPU.

[0063] In one embodiment, if there is a copy instruction for the first target terminal, the pixel data of the screen image of the first target terminal is copied based on the number of target terminals to obtain copied pixel data, and the generated copied pixel data and the pixel data of the screen image of the first target terminal are merged into the target buffer to obtain buffer pixel data.

[0064] Take Figure 2 as an example. If the first target terminal is screen 1, in the case of three target terminals, the pixel data of the screen image of screen 1 is copied twice. The copied pixel data is the copied pixel data, and it is merged into the target buffer GraphicBuffer in order, so that 3 sub-images with the same content can be included when forming the fused screen image.

[0065] In one embodiment, generating buffer pixel data based on the pixel data of the screen images of each target terminal includes:

[0066] If there is a swap instruction for the second target terminal, after the pixel data of each target terminal is merged into the target buffer, the position of the pixel data of the second target terminal in the target buffer is swapped with the position of the pixel data of the third target terminal indicated by the swap instruction in the target buffer.

[0067] For example, to make screen 1 display the content of screen 3 and screen 3 display the content of screen 1, just swap the positions in the target buffer, which will change the positions of the sub-images when rendering in the original order.

[0068] Specifically, segmenting the adjusted fused screen image includes:

[0069] Store the adjusted fused screen image in the framebuffer and segment the framebuffer based on the Hardware Composer.

[0070] Figure 3 is the overall process of screen image merging and segmentation, including the following steps:

[0071] The generated application image data is saved to the GraphicBuffer, where the graphic data is the screen images of each target terminal;

[0072] The data is passed to the SurfaceFlinger through the BufferQueue;

[0073] The SurfaceFlinger synthesizes the final display frame through the GPU, that is, fuses the screen images;

[0074] The fused screen image is submitted to the Hardware Composer to display the framebuffer;

[0075] The framebuffer is divided and displayed on the corresponding screens, so that sub-images after division are displayed on screen 1, screen 2, and screen 3.

[0076] In one embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method of any embodiment is implemented.

[0077] In one embodiment of the present invention, an electronic device is provided, including: a memory and a processor; a computer program is stored on the memory. When the computer program is executed by the processor, the data processing method of any embodiment is implemented.

[0078] Figure 4 It is a structural block diagram of the electronic device according to the embodiment of the present invention.

[0079] As Figure 4 shown, the electronic device 400 includes: a processor 401 and a memory 403. Among them, the processor 401 and the memory 403 are connected, such as connected through a bus 402. Optionally, the electronic device 400 may further include a transceiver 404. It should be noted that in practical applications, the transceiver 404 is not limited to one, and the structure of the electronic device 400 does not constitute a limitation to the embodiment of the present invention.

[0080] The processor 401 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of the present invention. The processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0081] The bus 402 may include a path for transmitting information between the above components. The bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0082] The memory 403 is used to store a computer program corresponding to the data processing method of the above embodiment of the present invention, and this computer program is controlled and executed by the processor 401. The processor 401 is used to execute the computer program stored in the memory 403 to implement the content shown in the foregoing method embodiments.

[0083] Among them, the electronic device 400 includes but is not limited to: mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The illustrated electronic device 400 is merely an example and should not impose any limitation on the functions and scope of use of the embodiments of the present invention.

[0084] Note that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection part with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0085] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0086] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0087] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present invention.

[0088] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise specifically and clearly defined.

[0089] In the present invention, unless otherwise clearly specified and defined, the terms "mounted", "connected", "coupled", "fixed", etc. shall be construed in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal communication of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0090] In the present invention, unless otherwise clearly specified and defined, the first feature being "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on top of" the second feature may be that the first feature is directly above or obliquely above the second feature, or merely indicates that the first feature has a higher horizontal height than the second feature. The first feature being "under", "beneath" and "underneath" the second feature may be that the first feature is directly below or obliquely below the second feature, or merely indicates that the first feature has a lower horizontal height than the second feature.

[0091] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations to the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A data processing method, characterized in that, The method includes: Combining the screen images of multiple target terminals to obtain a fused screen image; In response to recognizing first touch data for a target terminal, adjusting the fused screen image according to the first touch data; Segmenting the adjusted fused screen image and displaying the segmented sub-images on each target terminal.

2. The method according to claim 1, wherein The step of, in response to recognizing first touch data for a target terminal, adjusting the fused screen image according to the first touch data includes: Converting the first touch data into second touch data in the image coordinate system of the fused screen image; Adjusting the fused screen image based on the adjustment instruction represented by the second touch data.

3. The method according to claim 2, wherein The adjustment instruction includes any one of the following: a rotation instruction, a sliding instruction, and a size adjustment instruction.

4. The method according to claim 1, characterized in that, The step of combining the screen images of multiple target terminals to obtain a fused screen image includes: Generating buffer pixel data based on the pixel data of the screen images of each target terminal; Generating layer data based on the buffer pixel data; Combining the generated layer data into a fused screen image.

5. The method according to claim 4, wherein The step of generating buffer pixel data based on the pixel data of the screen images of each target terminal includes: If there is a copy instruction for a first target terminal, copying the pixel data of the screen image of the first target terminal based on the number of target terminals to obtain copied pixel data, and merging the generated copied pixel data and the pixel data of the screen image of the first target terminal into a target buffer to obtain buffer pixel data.

6. The method according to claim 4, characterized in that, The step of generating buffer pixel data based on the pixel data of the screen images of each target terminal includes: If there is a swap instruction for a second target terminal, after merging the pixel data of each target terminal into a target buffer, swapping the position of the pixel data of the second target terminal in the target buffer with the position of the pixel data of a third target terminal indicated by the swap instruction in the target buffer.

7. The method according to claim 4, characterized in that The buffer pixel data is generated based on GraphicBuffer. The step of generating layer data based on the buffer pixel data includes: Passing the buffer pixel data to SurfaceFlinger through BufferQueue to obtain the layer data generated by SurfaceFlinger.

8. The method according to claim 1, wherein The step of segmenting the adjusted fused screen image includes: Storing the adjusted fused screen image in framebuffer and segmenting framebuffer based on Hardware Composer.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1-8.

10. An electronic device, characterized in that, It includes: A memory and a processor; A computer program is stored on the memory. When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1-8.

Images