Split control method, device and equipment based on mobile phone oled screen and medium

Abstract

The present disclosure relates to a split type control method and device based on a mobile phone oled screen, equipment and medium, belonging to the technical field of screen control, the method comprising: obtaining target modality data output by a user through a sensing device and a first screen of a mobile phone; generating control intent information according to the target modality data; according to the preference information characteristics reflected by the control intent information, extracting a target container layer matched with the preference information characteristics in the interface level tree currently associated with the first screen; generating a cooperative control instruction on the second screen according to the container element associated with the target container layer, the screen type of the first screen and the current main and auxiliary screen configuration; in the case of first interaction operation on the first screen, controlling the second screen to work cooperatively according to the cooperative control instruction.

Description

Technical Field

[0001] This disclosure relates to the technical field of screen control, and more specifically, to a split-type control method, apparatus, device, and medium based on a mobile phone OLED screen. Background Technology

[0002] With OLED screens becoming mainstream in smartphones, dual-screen hardware forms such as foldable screens and split-screen phones have emerged. Both screens have independent touch, independent display, and self-illuminating dimming capabilities, and manufacturers generally provide basic dual-screen mirroring and simple split-screen functions.

[0003] Currently, existing smartphones either directly copy the entire page from the first screen to the second screen when splitting the screen, or display the entire page on both the first and second screens simultaneously. This makes it difficult to analyze the page in layers and distinguish between different containers such as main text, interactive controls, and pop-ups, resulting in a poor user experience for visually impaired users. Summary of the Invention

[0004] One objective of this disclosure is to provide a new technical solution for split-type control based on a mobile phone OLED screen.

[0005] According to a first aspect of this disclosure, a split-type control method based on a mobile phone OLED screen is provided, the method comprising: The target modal data output by the user is acquired through the sensing devices and the first screen of the mobile phone; Based on the target modal data, control intent information is generated; Based on the preference information features reflected by the control intent information, the target container layer that matches the preference information features is extracted from the interface hierarchy tree currently associated with the first screen. Based on the container element associated with the target container layer, the screen type of the first screen, and the current main and secondary screen configuration, generate collaborative control instructions for the second screen. When the first interactive operation is performed on the first screen, the second screen is controlled to work in coordination according to the collaborative control command.

[0006] Optionally, the target modal data includes text data output from the first screen, touch data, and audio data output from the mobile phone microphone; generating control intent information based on the target modal data includes: Based on the text data and the contextual association information of the associated session window, the basic semantic parameters are obtained; Based on the touch data, the user's gesture features on the target display area of ​​the first screen are extracted to obtain touch mapping parameters; Based on the audio data, extract the user's acoustic expression parameters; The basic semantic parameters, the touch mapping parameters, and the acoustic expression parameters are linearly fused to obtain control intent information.

[0007] Optionally, extracting a target container layer matching the preference information features in the interface hierarchy tree currently associated with the first screen based on the preference information features reflected by the control intent information includes: Using the preference information features reflected by the control intent information as matching conditions, in the interface hierarchy tree currently associated with the first screen, the upper container layer in the interface hierarchy tree is matched to obtain the first upper container layer that matches the preference information features; the lower container layer of the second upper container layer in the interface hierarchy tree other than the first upper container layer is matched to obtain the target lower container layer that matches the preference information features, and finally the target container layers corresponding to different levels are obtained.

[0008] Optionally, the matching conditions further include screen size information of the first screen; the preference information feature includes a preference intent identifier; the step of matching the upper container layer in the interface hierarchy tree currently associated with the first screen to obtain a first upper container layer that matches the preference information feature includes: In the interface hierarchy tree currently associated with the first screen, filter out candidate upper container layers that contain the screen size information and preference intent identifier; Select candidate upper container layers whose number of upper container layers is greater than or equal to a set threshold, and use them as the first upper container layer; wherein, the level of the upper container layer corresponds to the next level of the candidate upper container layer; The step of matching the lower container layer of the second upper container layer (excluding the first upper container layer) in the interface hierarchy tree to obtain the target lower container layer that matches the preference information features includes: For the lower container layers of the second upper container layer in the interface hierarchy tree (excluding the first upper container layer), filter the candidate lower container layers that contain the screen size information and preference intent identifier; Select candidate lower container layers whose number of lower container layers is greater than or equal to a set threshold, and use them as target lower container layers; wherein, the level of the lower container layer corresponds to the next level of the candidate lower container layer.

[0009] Optionally, generating collaborative control instructions for the second screen based on the container element associated with the target container layer, the screen type of the first screen, and the current primary and secondary screen configuration includes: Determine the element type of the container element associated with the target container layer; When the screen type of the first screen is the main screen and the current main and secondary screens are configured for differentiated display, a display control instruction is generated to display a container element of the element type as plain text on the first screen, and a collaborative control instruction is generated to display a container element of the element type as a control on the second screen. or, When the screen type of the first screen is a secondary screen and the current primary and secondary screens are configured for mirrored display, a collaborative control instruction is generated to display the container element on the second screen and set the container element of the element type to touch state.

[0010] Optionally, the mobile phone further includes a speaker; the method further includes: Based on the user's set accessibility mode, the sensing device detects the target screen in the first screen or the second screen held by the user, and controls the speaker to play the first prompt information of the container element presented on the target screen.

[0011] Optionally, the sensing device further includes a camera; before controlling the second screen to cooperate according to the collaborative control command when the first interactive operation is performed on the first screen, the method further includes: Enter the preview mode where the first and second screens work together; In the preview mode, the user's facial expression tags are determined using the camera; When the facial expression tag is a specific tag, control instructions are executed to adjust the collaborative operation of the first screen and the second screen based on the user's set usage mode; wherein, the usage mode includes an accessibility mode and a normal usage mode; the control instructions include a first control instruction corresponding to the accessibility mode and a second control instruction corresponding to the normal usage mode, the first control instruction plays a second prompt message containing multiple adjustment items through a speaker, and the first control instruction presents an adjustment window containing multiple adjustment items on the first screen; In response to the user's selection of a target adjustment item among the plurality of adjustment items, an adjustment instruction corresponding to the target adjustment item is executed on the first screen and / or the second screen.

[0012] According to a second aspect of this disclosure, a split-type control device based on a mobile phone OLED screen is also provided, the device comprising: The acquisition module is used to acquire target modal data output by the user through the sensing devices and the first screen of the mobile phone; The first generation module is used to generate control intent information based on the target modal data; The extraction module is used to extract a target container layer that matches the preference information features in the interface hierarchy tree currently associated with the first screen, based on the preference information features reflected by the control intent information. The second generation module is used to generate collaborative control instructions for the second screen based on the container element associated with the target container layer, the screen type of the first screen, and the current main and secondary screen configuration. The control module is used to control the second screen to work in coordination according to the collaborative control instructions when the first interactive operation is performed on the first screen.

[0013] According to a third aspect of this disclosure, a computer system is also provided, the computer system including a processor, which implements the split control method based on a mobile phone OLED screen in the first aspect when the processor executes program instructions or code.

[0014] For example, the computer system also includes a memory for storing program instructions or code.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the above-described split control method based on a mobile phone OLED screen at runtime.

[0016] According to a fifth aspect of this disclosure, a computer program product is also provided, comprising a computer program that, when executed, causes a computer to perform the steps of the above-described split-type control method based on a mobile phone OLED screen.

[0017] According to a sixth aspect of this disclosure, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the aforementioned split-type control method based on a mobile phone OLED screen through the computer program.

[0018] One beneficial effect of this disclosure is that the split control method based on a mobile phone OLED screen provided by the present invention can acquire target modal data output by the user through the sensing device of the mobile phone and the first screen. Based on the target modal data, control intent information is generated. Then, based on the control intent information, a target container layer matching the preference information features is extracted from the interface hierarchy tree currently associated with the first screen. Based on the container element associated with the target container layer, the screen type of the first screen, and the current main and secondary screen configuration, a collaborative control instruction for the second screen is generated. When the first interactive operation is performed on the first screen, the second screen is controlled to work in coordination according to the collaborative control instruction, so that the content displayed on the interface of the first screen can be distinguished and displayed separately as controls, text, images, or videos on the second screen, so that users with visual impairments can view the content displayed separately on the second screen, effectively improving the user experience.

[0019] Other features and advantages of the embodiments of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the embodiments of the present disclosure.

[0021] Figure 1 A flowchart illustrating a split-type control method based on a mobile phone OLED screen according to some embodiments is shown; Figure 2 A schematic diagram of the structure of a mobile phone according to some embodiments is shown; Figure 3 A schematic diagram showing the generation of control instructions according to some embodiments is shown; Figure 4 A schematic diagram of a split control device based on a mobile phone OLED screen is shown according to some embodiments; Figure 5 A schematic diagram of the hardware structure of an electronic device according to some embodiments is shown. Detailed Implementation

[0022] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0023] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0024] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0025] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0026] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0027] <Method Implementation> Figure 1 This is a flowchart illustrating a split-type control method based on a mobile phone OLED screen according to one embodiment. The implementing entity is the mobile phone controller.

[0028] like Figure 1 As shown, the split control method based on the mobile phone OLED screen in this embodiment may include the following steps S110 to S150: Step S110: Obtain the target modal data output by the user through the sensing device of the mobile phone and the first screen.

[0029] In this embodiment, the sensing device can be a collection of multi-sensor hardware built into the mobile phone, including a microphone, touch sensor, front / rear camera, grip sensor, and posture sensor, and is not limited thereto.

[0030] In this embodiment, the first screen can be the OLED screen that the user is currently primarily operating on, and can be a main screen / secondary screen. For example... Figure 1 As shown, the first screen can be configured with an independent speaker and microphone, the second screen can be configured with an independent speaker and microphone, and the phone's controller, the first screen, and other sensing devices can be integrated into one unit.

[0031] In this embodiment, target modal data of text, video, and / or audio modes can be obtained synchronously through the sensing device and the first screen. This target modal data may include text data input by the user, touch data generated by touching the first screen, audio data input through the microphone, image data input through the front / rear camera, etc.

[0032] Step S120: Generate control intent information based on the target modal data.

[0033] In some embodiments, step S120 may include the following steps S210 to S240: Step S210: Obtain basic semantic parameters based on the contextual association information of the text data and the associated session window.

[0034] In this embodiment, the basic semantic parameters include and contextual semantic intent parameters By using a Large Language Model (LLM), the basic intent and contextual intent of the text data are extracted to obtain... and contextual semantic intent parameters .

[0035] Step S220: Based on the touch data, extract the user's gesture features on the target display area of ​​the first screen to obtain touch mapping parameters.

[0036] In this embodiment, based on touch data, the user's pressing action and pressing duration, and other operation gesture features on the target display area of ​​the first screen are extracted to obtain touch mapping parameters that characterize the user's touch action intention on the target display area of ​​the first screen. For example, such as Figure 2 As shown, the intention of the touch action in the target display area of ​​the first screen can be: the user performs a zoom gesture on the U1 window in the interface of the first screen, or the user performs a drag gesture on the U2 window in the interface of the first screen.

[0037] Step S230: Extract the user's acoustic expression parameters based on the audio data.

[0038] In this embodiment, RMS energy tracing is used to extract audio energy from audio data to determine whether there is an audio curve exceeding a set volume value at a macroscopic volume. Based on the audio curve exceeding the set volume value and the aforementioned basic semantic parameters and / or touch mapping parameters, the user's emotional change characteristics are determined to obtain the user's acoustic expression parameters. For example, if touch mapping parameters represent a user's gesture of deleting text displayed on the interface, and the audio curve represents brief high-frequency features, then the user's acoustic expression parameters can be represented as the emotional marker of "dislike".

[0039] Step S240: Linearly fuse the basic semantic parameters, touch mapping parameters, and acoustic expression parameters to obtain control intent information.

[0040] In this embodiment, control intent information This can be represented as: control intent information = +Contextual semantic intent parameters +Touch mapping parameters +Acoustic expression parameters .

[0041] Step S130: Based on the preference information features reflected by the control intent information, extract the target container layer that matches the preference information features from the interface hierarchy tree currently associated with the first screen.

[0042] In some embodiments, step S130 may include the following step S310: Step S310: Using the preference information features reflected by the control intent information as the matching condition, in the interface hierarchy tree currently associated with the first screen, the upper container layer in the interface hierarchy tree is matched to obtain the first upper container layer that matches the preference information features; the lower container layer of the second upper container layer in the interface hierarchy tree other than the first upper container layer is matched to obtain the target lower container layer that matches the preference information features, and finally the target container layer corresponding to different levels is obtained.

[0043] In this embodiment, the interface hierarchy tree is the page layering architecture of the user's screen when running the app. It has multiple container layers, and these container layers can have corresponding hierarchical relationships. For example, container layer A1 is the root layer, container layer A11 is a child node of container layer A1, and container layer A111 is a child node of container layer A11. Each container layer carries different UI modules, and the tree structure stores the page layout and element attributes.

[0044] In some embodiments, the matching criteria further include screen size information of the first screen; the preference information features include a preference intent identifier; step S310 may include the following steps S410 to S440: Step S410: In the interface hierarchy tree currently associated with the first screen, filter out candidate upper container layers that contain screen size information and preference intent identifiers.

[0045] Step S420: Select candidate upper container layers whose number of upper container layers is greater than or equal to a set threshold, and use them as the first upper container layer; wherein, the level of the upper container layer corresponds to the next level of the candidate upper container layer.

[0046] Step S430: For the lower container layers of the second upper container layer (excluding the first upper container layer) in the interface hierarchy tree, filter the candidate lower container layers that contain screen size information and preference intent identifiers.

[0047] Step S440: Select candidate lower container layers whose number of lower container layers is greater than or equal to a set threshold, and use them as target lower container layers; wherein, the level of the lower container layer corresponds to the next level of the candidate lower container layer.

[0048] In this embodiment, the threshold can be greater than or equal to 2. By selecting candidate upper container layers with multiple upper container layers and candidate lower container layers with multiple lower container layers, non-content container layers with only a single lower container layer can be effectively filtered out.

[0049] In this embodiment, a preset identifier mapping table can be used to determine the preference intent identifier corresponding to different control intent information. This identifier mapping table can be based on different control intent information. Different pre-defined offset intent identifiers. In the interface hierarchy tree currently associated with the first screen, firstly filter the container layer containing screen size information and preference intent identifiers at the root level, then filter the container layer below the root level containing screen size information and preference intent identifiers, and so on, until all container layers have been filtered. That is, first filter the first upper-level container layer among the candidate upper-level container layers, then filter the target lower-level container layer among the second upper-level container layers, so as to achieve the filtering of each level of the interface hierarchy tree currently associated with the first screen. Then, traverse the interface hierarchy tree from top to bottom, first separate the main screen content container, and then extract the secondary screen control container to adapt to the dual-screen division of labor mode where the main screen displays the screen / text and the secondary screen carries the operation controls.

[0050] In some instances, prior to step S410, the method further includes: determining the page structure associated with each candidate upper-level container layer; when the page structure of the candidate upper-level container layer is a WebView embedded page structure, extracting the Bounds information of the candidate upper-level container layer to determine whether the candidate upper-level container layer is associated with screen size information. The associated screen size information may include whether the Bounds information of the candidate upper-level container layer contains the size information of the first screen or contains a size information smaller than the first screen. When the page structure of the candidate upper-level container is a Native original page structure, the associated screen size information is directly extracted.

[0051] Step S140: Generate collaborative control instructions for the second screen based on the container element associated with the target container layer, the screen type of the first screen, and the current main and secondary screen configuration.

[0052] In some embodiments, step S140 may include the following steps S510 to S530: Step S510: Determine the element type of the container element associated with the target container layer.

[0053] In this embodiment, the element type of the container element can be determined by the bound information associated with the container element. The element type can include plain text type and control type.

[0054] Step S520: When the screen type of the first screen is the main screen and the current main and secondary screens are configured for differentiated display, generate a display control instruction for displaying a container element of plain text type on the first screen, and generate a collaborative control instruction for displaying a container element of control type on the second screen.

[0055] In this embodiment, as Figure 2 As shown, the text in the first screen display window U2 is displayed, and under the action of the collaborative control command, the controls K3, K4 and K5 in the second screen display interface U2 are displayed.

[0056] Step S530: When the screen type of the first screen is a secondary screen and the current main and secondary screens are configured as mirrored displays, generate a collaborative control instruction for the container element displayed on the second screen and the container element of the element type being a control, and set to touch state.

[0057] In this embodiment, the first screen is a secondary screen and does not have touch functionality. Under the action of the collaborative control command, the second screen displays the same content as the first screen, and the controls in the interface can be directly touched.

[0058] In this embodiment, the controller reads the bound attribute data of all container elements within each target container layer and automatically distinguishes element categories by parsing layer identifiers and touch response fields: elements that only contain text and have no interactive capabilities are determined to be plain text; buttons, sliders, input boxes, etc., with click, swipe, and input interaction functions are determined to be control types. This method relies solely on local UI rendering attributes for judgment, eliminating the need for image recognition and resulting in fast calculation speed.

[0059] In step S150, when performing the first interactive operation on the first screen, the second screen is controlled to work in coordination according to the collaborative control instruction.

[0060] In some examples, the first interaction could be that the user browses the interface of the first screen. When browsing to a window that the user is interested in, the user can double-click the window. At this time, the current main and secondary screens are configured to be displayed separately, and the corresponding controls in the window can be displayed on the second screen, making it easier for the user to view the controls in the window on the second screen.

[0061] In some embodiments, the mobile phone further includes a speaker; prior to step S150, the method further includes the following step S610: Step S610: Based on the user's set accessibility mode, the user senses the target screen in the first or second screen being held by the user through the sensing device, and controls the speaker to play the first prompt information of the container element presented on the target screen.

[0062] In this embodiment, the sensing device can be a camera, a thermal sensor, or a pressure sensor. Users can choose to set their phones to an accessibility mode. In this mode, the corresponding sensing device can detect hand tremors or sustained gripping movements, determining whether the user is holding a first or second screen, and controlling the phone to display the first prompt information of the container element displayed on the target screen. For example, such as... Figure 2 As shown, the target screen held by the user is the second screen. Therefore, the phone's speaker can broadcast prompts related to controls K3, K4, and K5.

[0063] In this embodiment, dual-screen interaction is adapted for people with disabilities. The system automatically identifies when the user is holding the screen and reads the content aloud, solving the problem that visually impaired users cannot distinguish the content on the two OLED screens. The integrated sensing device eliminates the need for manual switching of the reading target, achieving a high degree of automation. Furthermore, it only reads the content on the screen the user is holding, avoiding repetitive reading of all information from both screens, reducing redundant voice and improving the user experience.

[0064] In some embodiments, the sensing device further includes a camera; prior to step S150, the method further includes the following steps S710 to S740: Step S710: Enter the preview mode where the first and second screens work together.

[0065] In this embodiment, before executing the collaborative control command, a preview window can pop up on the first screen. After the user selects the confirmation control in the preview window, the user enters the preview mode where the first screen and the second screen work together.

[0066] Step S720: In preview mode, determine the user's facial expression tags using the camera.

[0067] Step S730: When the facial expression tag is a specific tag, based on the user's set usage mode, execute the control command to adjust the collaborative operation of the first screen and the second screen; wherein, the usage mode includes an accessibility mode and a normal usage mode; the control command includes a first control command corresponding to the accessibility mode and a second control command corresponding to the normal usage mode, the first control command is to play a second prompt message with multiple adjustment items through the speaker, and the first control command is to present an adjustment window with multiple adjustment items on the first screen.

[0068] In this embodiment, the specific label can be a label that automatically identifies facial expressions representing discomfort, such as squinting, frowning, staring for a long time, or shaking one's head.

[0069] Step S740: In response to the user's selection of a target adjustment item among multiple adjustment items, execute the adjustment instruction for the corresponding target adjustment item on the first screen and / or the second screen.

[0070] In some examples, in preview mode, the user's facial expression label is a specific label to determine the user's set usage mode. When the usage mode is accessibility mode, a corresponding voice prompt is generated and output to the user based on the screen types of the first and second screens and the current screen configuration. For example, if the first screen is the main screen, the second screen is the secondary screen, and the current screen is configured for differentiated display, then the voice prompt would be: "The main screen displays the description content of the product details page in the interface; the secondary screen displays the evaluation, purchase, and favorite options of the product details page. Do you need to adjust them? Adjustment methods include switching the screen playback order, playback speed, and setting a pause prompt." This allows the user to select the target adjustment item from multiple adjustment options by speaking, executing adjustment commands for the corresponding target adjustment item on the first and / or second screens.

[0071] In other examples, in preview mode, the user's facial expression tags are specific labels to determine the user's chosen usage mode. If the usage mode is normal, corresponding container elements are generated and displayed on the first and / or second screens based on the screen types and current screen configurations of the first and second screens. Users can directly adjust the layout of these container elements on the first and / or second screens via touch, causing the first and / or second screens to adjust accordingly.

[0072] In this embodiment, the user completes the layout adjustment of the first and second screens in the preview mode. After the official linkage, there is no need to repeatedly refresh the interface to modify the dual-screen content, which reduces the number of times the screen pixels switch on and off frequently, reduces the probability of OLED screen leaving afterimages and burn-in marks after long-term use, and extends the lifespan of the OLED screen.

[0073] <Equipment Example 1> Figure 4 This is a schematic diagram of a split-type control device based on a mobile phone OLED screen according to one embodiment. Figure 4 As shown, the split-type control device 400 based on a mobile phone OLED screen may include: The acquisition module 410 is used to acquire target modal data output by the user through the sensing device of the mobile phone and the first screen; The first generation module 420 is used to generate control intent information based on the target modal data; The extraction module 430 is used to extract the target container layer that matches the preference information features in the interface hierarchy tree currently associated with the first screen, based on the preference information features reflected by the control intent information. The second generation module 440 is used to generate collaborative control instructions for the second screen based on the container element associated with the target container layer, the screen type of the first screen, and the current main and secondary screen configuration. The control module 450 is used to control the second screen to work in coordination according to the collaborative control instructions when the first interactive operation is performed on the first screen.

[0074] Optionally, the first generation module 420 is further configured to obtain basic semantic parameters based on text data and contextual information of the conversation window associated with the text data; extract the user's gesture features on the target display area of ​​the first screen based on touch data to obtain touch mapping parameters; extract the user's acoustic expression parameters based on audio data; and linearly fuse the basic semantic parameters, touch mapping parameters and acoustic expression parameters to obtain control intent information.

[0075] Optionally, the extraction module 430 is further configured to use the preference information features reflected by the control intent information as matching conditions to match the upper container layer in the interface hierarchy tree currently associated with the first screen, to obtain the first upper container layer that matches the preference information features; and to match the lower container layer of the second upper container layer in the interface hierarchy tree other than the first upper container layer, to obtain the target lower container layer that matches the preference information features, and finally obtain the target container layer corresponding to different levels.

[0076] Optionally, the extraction module 430 is further configured to: filter candidate upper-level container layers containing screen size information and preference intent identifiers in the interface hierarchy tree currently associated with the first screen; filter candidate upper-level container layers whose number of upper-level container layers is greater than or equal to a set threshold, and use them as first upper-level container layers; wherein the level of the upper-level container layer corresponds to the next level of the candidate upper-level container layer; for the lower-level container layers of the second upper-level container layer in the interface hierarchy tree other than the first upper-level container layer, filter candidate lower-level container layers containing screen size information and preference intent identifiers; filter candidate lower-level container layers whose number of lower-level container layers is greater than or equal to a set threshold, and use them as target lower-level container layers; wherein the level of the lower-level container layer corresponds to the next level of the candidate lower-level container layer.

[0077] Optionally, the second generation module 440 is further configured to determine the element type of the container element associated with the target container layer; when the screen type of the first screen is the main screen and the current main and secondary screens are configured for differentiated display, it generates a display control instruction for displaying a container element of plain text type on the first screen, and generates a collaborative control instruction for displaying a container element of control type on the second screen; or, when the screen type of the first screen is the secondary screen and the current main and secondary screens are configured for mirrored display, it generates a collaborative control instruction for setting a container element of control type to touch state on the second screen.

[0078] Optionally, the device also includes an information control module, which, based on the user-defined accessibility mode, senses the target screen in the first or second screen held by the user through a sensing device, and controls the speaker to play the first prompt information of the container element presented on the target screen.

[0079] Optionally, the device further includes a preview module for entering a preview mode of the first and second screens working together; in the preview mode, the user's facial expression label is determined via a camera; if the facial expression label is a specific label, control instructions for adjusting the collaborative operation of the first and second screens are executed based on the user's set usage mode; wherein, the usage mode includes an accessibility mode and a normal usage mode; the control instructions include a first control instruction corresponding to the accessibility mode and a second control instruction corresponding to the normal usage mode, the first control instruction playing a second prompt message containing multiple adjustment items through a speaker, and the first control instruction presenting an adjustment window containing multiple adjustment items on the first screen; in response to the user's selection of a target adjustment item among the multiple adjustment items, adjustment instructions for the corresponding target adjustment item are executed on the first screen and / or the second screen.

[0080] <Equipment Example 2> Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to another embodiment.

[0081] like Figure 5 As shown, the electronic device 500 includes a processor 510 and a memory 520, the memory 520 being used to store an executable computer program, and the processor 510 being used to execute methods as described in any of the above method embodiments under the control of the computer program.

[0082] Each module of the split control device 400 based on the mobile phone OLED screen can be implemented by the processor 510 in this embodiment executing the computer program stored in the memory 520, or it can be implemented by other structures, which are not limited here.

[0083] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0084] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0085] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0086] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0087] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0089] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0091] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A split-type control method based on a mobile phone OLED screen, characterized in that, The method includes: The target modal data output by the user is acquired through the sensing devices and the first screen of the mobile phone; Based on the target modal data, control intent information is generated; Based on the preference information features reflected by the control intent information, the target container layer that matches the preference information features is extracted from the interface hierarchy tree currently associated with the first screen. Based on the container element associated with the target container layer, the screen type of the first screen, and the current main and secondary screen configuration, generate collaborative control instructions for the second screen. When the first interactive operation is performed on the first screen, the second screen is controlled to work in coordination according to the collaborative control command.

2. The method according to claim 1, characterized in that, The target modal data includes text data and touch data output from the first screen, and audio data output from the mobile phone microphone; the step of generating control intent information based on the target modal data includes: Based on the text data and the contextual association information of the associated session window, the basic semantic parameters are obtained; Based on the touch data, the user's gesture features on the target display area of ​​the first screen are extracted to obtain touch mapping parameters; Based on the audio data, extract the user's acoustic expression parameters; The basic semantic parameters, the touch mapping parameters, and the acoustic expression parameters are linearly fused to obtain control intent information.

3. The method according to claim 1, characterized in that, The step of extracting a target container layer matching the preference information features in the interface hierarchy tree currently associated with the first screen, based on the preference information features reflected by the control intent information, includes: Using the preference information features reflected by the control intent information as matching conditions, in the interface hierarchy tree currently associated with the first screen, the upper container layer in the interface hierarchy tree is matched to obtain the first upper container layer that matches the preference information features; the lower container layer of the second upper container layer in the interface hierarchy tree other than the first upper container layer is matched to obtain the target lower container layer that matches the preference information features, and finally the target container layers corresponding to different levels are obtained.

4. The method according to claim 3, characterized in that, The matching conditions also include the screen size information of the first screen; the preference information features include a preference intent identifier; the step of matching the upper container layer in the interface hierarchy tree currently associated with the first screen to obtain the first upper container layer that matches the preference information features includes: In the interface hierarchy tree currently associated with the first screen, filter out candidate upper container layers that contain the screen size information and preference intent identifier; Select candidate upper container layers whose number of upper container layers is greater than or equal to a set threshold, and use them as the first upper container layer; wherein, the level of the upper container layer corresponds to the next level of the candidate upper container layer; The step of matching the lower container layer of the second upper container layer (excluding the first upper container layer) in the interface hierarchy tree to obtain the target lower container layer that matches the preference information features includes: For the lower container layers of the second upper container layer in the interface hierarchy tree (excluding the first upper container layer), filter the candidate lower container layers that contain the screen size information and preference intent identifier; Select candidate lower container layers whose number of lower container layers is greater than or equal to a set threshold, and use them as target lower container layers; wherein, the level of the lower container layer corresponds to the next level of the candidate lower container layer.

5. The method according to claim 3, characterized in that, The step of generating collaborative control instructions for the second screen based on the container element associated with the target container layer, the screen type of the first screen, and the current primary and secondary screen configuration includes: Determine the element type of the container element associated with the target container layer; When the screen type of the first screen is the main screen and the current main and secondary screens are configured for differentiated display, a display control instruction is generated to display a container element of the element type as plain text on the first screen, and a collaborative control instruction is generated to display a container element of the element type as a control on the second screen. or, When the screen type of the first screen is a secondary screen and the current primary and secondary screens are configured for mirrored display, a collaborative control instruction is generated to display the container element on the second screen and set the container element of the element type to touch state.

6. The method according to claim 1, characterized in that, The mobile phone also includes a speaker; the method further includes: Based on the user's set accessibility mode, the sensing device detects the target screen in the first screen or the second screen held by the user, and controls the speaker to play the first prompt information of the container element presented on the target screen.

7. The method according to claim 1, characterized in that, The sensing device further includes a camera; before controlling the second screen to cooperate in operation according to the collaborative control command when the first interactive operation is performed on the first screen, the method further includes: Enter the preview mode where the first and second screens work together; In the preview mode, the user's facial expression tags are determined using the camera; When the facial expression tag is a specific tag, control instructions are executed to adjust the collaborative operation of the first screen and the second screen based on the user's set usage mode; wherein, the usage mode includes an accessibility mode and a normal usage mode; the control instructions include a first control instruction corresponding to the accessibility mode and a second control instruction corresponding to the normal usage mode, the first control instruction plays a second prompt message containing multiple adjustment items through a speaker, and the first control instruction presents an adjustment window containing multiple adjustment items on the first screen; In response to the user's selection of a target adjustment item among the plurality of adjustment items, an adjustment instruction corresponding to the target adjustment item is executed on the first screen and / or the second screen.

8. A split-type control device based on a mobile phone OLED screen, characterized in that, The device includes: The acquisition module is used to acquire target modal data output by the user through the sensing devices and the first screen of the mobile phone; The first generation module is used to generate control intent information based on the target modal data; The extraction module is used to extract a target container layer that matches the preference information features in the interface hierarchy tree currently associated with the first screen, based on the preference information features reflected by the control intent information. The second generation module is used to generate collaborative control instructions for the second screen based on the container element associated with the target container layer, the screen type of the first screen, and the current main and secondary screen configuration. The control module is used to control the second screen to work in coordination according to the collaborative control instructions when the first interactive operation is performed on the first screen.

9. An electronic device, characterized in that, The system includes a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images