Layout display method and related device

Abstract

The present application provides a layout display method, comprising: receiving a gesture input, and displaying a first layout, the first layout being written content of the gesture input, and the written content comprising a plurality of writing elements; and receiving a first layout switching instruction for the first layout, and, on the basis of the first layout switching instruction, displaying a second layout instead of the first layout, the second layout comprising a portion of the writing elements among the plurality of writing elements, wherein the portion of the writing elements corresponds to a same user intent, and, compared to the first layout, in the second layout, a writing element not comprised thereby and the portion of the writing elements do not correspond to a same user intent, and a gesture input time of the writing element not comprised thereby is before a gesture input time of at least one writing element among the portion of the writing elements. A layout reproduced in this way can present clearer and more complete written content, so that the user experience is improved.

Description

A layout display method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on January 31, 2024, with application number 202410144601.3 and application name “A layout display method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal software, and in particular to a layout display method and related equipment. Background Art

[0003] With the rapid development of the internet and communications technologies, office informatization has become an increasingly important component of the entire IT service system. Among these, interactive touch-screen display devices (such as interactive whiteboards), which are most user-friendly, have become extremely popular and have become standard accessories for digital conferences. Whiteboards are generally suitable for multi-person discussions, serving a similar purpose to blackboards.

[0004] Smart whiteboard devices with handwriting recognition often have low recognition rates or even no recognition in complex writing scenarios. This is mostly due to users adding a large number of meaningless strokes when writing on the blackboard. When users write on smart electronic whiteboards, they often repeatedly change and edit the content of their writing as their thoughts change, which can lead to the loss of important information. When multiple users use electronic whiteboards, they write and overlap on the same page, making it difficult to accurately recognize handwriting and causing the loss of important information. When using whiteboards for meetings, many meaningless strokes such as emphasis lines, underlines, and circles are added to different writing points, making the key points unclear. At this point, if users want to save clear and concise blackboard writing, they can only erase and rewrite, or frequently save the layout manually or automatically while writing, which greatly affects input efficiency and reduces the user experience.

[0005] Summary of the Invention

[0006] A first aspect of an embodiment of the present application provides a layout display method, the method comprising: receiving input and displaying a first layout, wherein the first layout is the input written content, and the written content includes multiple writing elements; receiving a first layout switching instruction for the first layout, and replacing the first layout with a second layout based on the first layout switching instruction, wherein the second layout includes some writing elements among the multiple writing elements; the some writing elements correspond to the same user intention, and the second layout does not correspond to the same user intention as the writing elements not included in the first layout and the some writing elements, and the input time of the not included writing elements is before the input time of at least one writing element among the some writing elements.

[0007] When key content cannot be identified or the layout is chaotic due to interference information (such as redundant written content, written content input by different users), the embodiment of the present application can identify redundant information in the publication page and eliminate the redundant information to reproduce the layout containing key information, wherein redundant information can be considered as information that is different from the main semantics of the layout. Users often have one or more input intentions when inputting, that is, part of the user's input can correspond to the same user intention, and another part of the input can correspond to other user intentions, or the written content input by different users corresponds to different user intentions. The so-called partial input can correspond to the same user intention here can be understood as: the written content of non-redundant content input by the user to express the same subject content can correspond to the same user intention. Usually, the same subject content corresponds to the same semantics.

[0008] Redundant content can be understood as: when the content is missing, the semantic integrity of the layout is not affected.

[0009] The input may specifically be gesture input.

[0010] The idea of ​​the embodiment of the present application is to record the user's input information. When the user needs to reproduce the historical layout (or it can be described as the user requesting to switch the layout), the redundant information in the layout can be eliminated, thereby reproducing the written content with the same user intention. The layout reproduced in this way can present clearer and more complete written content, thereby improving the user experience.

[0011] In an embodiment of the present application, the second layout may be a layout obtained by reconstructing the first layout. Compared to the first layout, the second layout eliminates elements that do not belong to the same user intention. Since the withdrawal operation is not performed according to a time similar to the user's input, the input time of the eliminated elements will be earlier than the input time of the retained elements (the written elements that are later than the eliminated elements are retained because they and other retained elements belong to the same user intention). The written elements included in the second layout are determined based on whether the corresponding user intentions are consistent. Compared to the existing technology that only considers the order of input time to perform layout reproduction, the embodiment of the present application reproduces the layout based on the input user intention.

[0012] In an embodiment of the present application, the first layout may be the layout obtained after receiving the user's input. Since the user often has multiple topics or some content that has a visual impact on important content (that is, redundant content) when inputting, in this case, the user may have a need to reproduce one or more specified layouts in history. Each layout may correspond to a user intention, so that the user can easily view the clear and well-defined layout content. Among them, the written content with the same user intention may be: having the same semantics and no redundant information.

[0013] In a possible implementation, the writing element is one or more complete strokes.

[0014] In one possible implementation, the same user intention includes: a writing element input by the same user; or, if missing, it will affect the semantic integrity of the written content included in the second layout.

[0015] In one possible implementation, the receiving of the first layout switching instruction for the first layout includes: receiving a selection instruction for part of the written content in the first layout, and a layout switching instruction for the part of the handwritten content; the part of the writing element belongs to the part of the written content; or, receiving a selection instruction for all of the written content in the first layout, and a layout switching instruction for all of the written content.

[0016] When the user wishes to reproduce one or more designated layouts in history, he or she may input a first layout switching instruction for the first layout. The first layout switching instruction may be used to instruct to reconstruct the layout based on part or all of the written content in the first layout.

[0017] In one possible implementation, it is also possible to specify that some elements in the first layout will not be eliminated when the layout is reconstructed. For example, a fix instruction for a target element among the multiple writing elements can be received, and the fix instruction is used to indicate that when the first layout is replaced with the second layout (the reconstructed layout), the display of the target element is maintained. Exemplarily, the circled content can be identified as the smallest unit, and the user can select a unit individually and choose to fix or eliminate it. The same unit can also be identified and the same unit can be operated as a whole. After a unit is selected to be fixed or eliminated, the unit is not affected by the sliding of the slider bar.

[0018] By means of the above method, a space for optional operation is provided to the user, so that the user can choose to fix or delete important unit information.

[0019] In a possible implementation, the first layout switching instruction is generated for a sliding operation of a slider.

[0020] In a possible implementation, replacing the first layout with the second layout includes: sequentially eliminating some writing elements in the first layout until the second layout is displayed.

[0021] In a possible implementation, the step of sequentially eliminating some of the writing elements in the first layout includes:

[0022] Some of the writing elements in the first layout are eliminated in sequence according to the relevance between the writing elements and the user intention.

[0023] In one possible implementation, the first layout switching instruction is generated for an operation on multiple candidate layouts, each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0024] In a possible implementation, the method further includes: caching the input data corresponding to the first layout; and determining display content of the second layout based on the data.

[0025] After recording the input data, the input data can be cached, and subsequent redundant information identification and layout generation operations are performed only when the user requests to switch the layout, thereby reducing the overhead of computing resources.

[0026] In a possible implementation, determining the display content of the second layout based on the data is performed after receiving the first layout switching instruction for the first layout.

[0027] In one possible implementation, determining the display content of the second layout based on the data includes: determining the display content of multiple candidate layouts including the second layout through a language model based on the data, each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0028] In one possible implementation, the data includes at least one of the following: static features of the writing elements, the relationship between the static features of the writing elements adjacent in space, the relationship between the static features of the writing elements adjacent in time, and the motion features of the writing elements.

[0029] The embodiment of the present application constructs features based on the time, space, and semantic information of the strokes, and the recognition results are more accurate.

[0030] In a second aspect, the present application provides a layout display device, the device comprising: a processing module and a display module;

[0031] The processing module is used to receive input and control the display module to display a first layout, where the first layout is the input writing content, and the writing content includes multiple writing elements; receive a first layout switching instruction for the first layout, and control the display module based on the first layout switching instruction to replace the first layout with a second layout, where the second layout includes some writing elements among the multiple writing elements; the some writing elements correspond to the same user intention, and the second layout does not correspond to the same user intention as the writing elements not included in the first layout and the some writing elements, and the input time of the not included writing elements is before the input time of at least one writing element among the some writing elements.

[0032] The input may specifically be gesture input.

[0033] In a possible implementation, the writing elements included in the second layout are determined based on whether the corresponding user intentions are consistent.

[0034] In a possible implementation, the writing element is one or more complete strokes.

[0035] In a possible implementation, the same user intention includes:

[0036] Input is via the same writing element as the user input; or,

[0037] If it is missing, it will affect the semantic integrity of the written content included in the second layout.

[0038] In a possible implementation, the processing module is specifically configured to:

[0039] receiving a selection instruction for a portion of the handwritten content in the first layout, and a layout switching instruction for the portion of the handwritten content; the portion of the handwritten elements belongs to the portion of the handwritten content; or

[0040] A selection instruction for all the written contents in the first layout and a layout switching instruction for all the written contents are received.

[0041] In a possible implementation, before replacing the first layout with the second layout, the processing module is further configured to:

[0042] A fixed instruction for a target element among the plurality of writing elements is received, wherein the fixed instruction is used to instruct that the display of the target element be maintained when the first layout is replaced with the second layout.

[0043] In a possible implementation, the first layout switching instruction is generated for a sliding operation of a slider.

[0044] In a possible implementation, the processing module is specifically configured to control the display module to sequentially make some writing elements in the first layout disappear until the second layout is displayed.

[0045] In a possible implementation, the step of sequentially eliminating some of the writing elements in the first layout includes:

[0046] Some of the writing elements in the first layout are eliminated in sequence according to the relevance between the writing elements and the user intention.

[0047] In one possible implementation, the first layout switching instruction is generated for an operation on multiple candidate layouts, each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0048] In a possible implementation, the processing module is further configured to:

[0049] caching the input data corresponding to the first layout;

[0050] The display content of the second page is determined according to the data.

[0051] In a possible implementation, determining the display content of the second layout based on the data is performed after receiving the first layout switching instruction for the first layout.

[0052] In a possible implementation, the processing module is specifically configured to:

[0053] Based on the data, the display content of multiple candidate layouts including the second layout is determined through a language model, each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0054] In one possible implementation, the data includes at least one of the following:

[0055] The static features of the writing elements, the relationship between the static features of the writing elements adjacent in space, the relationship between the static features of the writing elements adjacent in time, and the action features of the writing elements.

[0056] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, a touch screen, and a bus, wherein: the processor, the memory, and the touch screen are connected via the bus;

[0057] The memory is used to store computer programs or instructions;

[0058] The processor is used to call or execute the program or instruction stored in the memory, and is also used to control the touch screen to implement the steps described in the first aspect and any possible implementation method of the first aspect.

[0059] In a fourth aspect, the present application provides a computer storage medium comprising computer instructions, which, when executed on an electronic device or a server, execute the steps described in the first aspect and any possible implementation of the first aspect.

[0060] In a fifth aspect, the present application provides a computer program product, which, when running on an electronic device or a server, executes the steps described in the first aspect and any possible implementation of the first aspect.

[0061] In a sixth aspect, the present application provides a chip system comprising a processor for supporting an execution device or a training device in implementing the functions described in the aforementioned aspects, such as transmitting or processing data or information described in the aforementioned methods. In one possible design, the chip system further comprises a memory for storing program instructions and data necessary for the execution device or the training device. The chip system may consist of a single chip or may include a chip and other discrete components. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] FIG1 is a schematic diagram of the product structure provided in an embodiment of the present application;

[0063] FIG2 is a structural block diagram of an electronic device according to an embodiment of the present application;

[0064] FIG3 is a schematic diagram of an embodiment of a layout display method provided in an embodiment of the present application;

[0065] Figures 4, 5B, 6, 9B, 10, and 13 are schematic diagrams of the layouts in the embodiments of the present application;

[0066] Figures 5A, 7 to 9A, 11 and 12 are diagrams illustrating a gesture in an embodiment of the present application;

[0067] FIG14 is a schematic diagram of the structure of a layout display device provided in an embodiment of the present application;

[0068] FIG15 is a schematic structural diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0070] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0071] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0072] First, an exemplary operating environment of the present application is introduced.

[0073] The embodiments of the present application may be applied in a system 100 including a touch screen 103 .

[0074] 1 , which shows a system 100 to which an embodiment of the present application is applied, wherein the system 100 may include an electronic device 101 and a pen 102 associated with the electronic device 101 .

[0075] The electronic device 101 may be the electronic whiteboard (or electronic interactive smart board) shown in FIG1 , and the electronic device 101 includes a touch screen 103. It should be understood that the electronic device 101 may also be a portable mobile device including a touch screen, such as, but not limited to, a mobile or portable computing device (such as a smart phone), a personal computer, a server computer, a handheld device (such as a tablet) or a laptop device, a multi-processor system, a game console or controller, a microprocessor-based system, a set-top box, a programmable consumer electronic product, a mobile phone, a mobile computing and / or communication device with a wearable or accessory form factor (such as a watch, glasses, a headset or earbuds), a network PC, a minicomputer, a mainframe computer, a distributed computing environment including any of the above systems or devices, and the like.

[0076] The touch screen 103 is capable of recognizing a user's contact gesture.

[0077] In one implementation, the touch screen 103 can be an infrared touch screen. The infrared touch screen is composed of infrared emitting and receiving sensing elements mounted on the touch screen frame. On the screen surface, an infrared detection network is formed. Any touching object can change the infrared light at the contact point to realize touch screen operation. The implementation principle of the infrared touch screen is similar to that of surface acoustic wave touch, which uses infrared emitting and receiving sensing elements. These elements form an infrared detection network on the screen surface. The object of the touch operation (such as the contact between the side of the hand and the touch screen 103 in the embodiment of the present application) can change the infrared light at the contact point, which is then converted into the coordinate position of the touch, thereby realizing the recognition of the touch gesture.

[0078] In one implementation, the touch screen 103 can be a capacitive touch screen, which works by utilizing the current induction of the human body. For example, the capacitive touch screen can be a four-layer composite glass screen, with the inner surface and the interlayer of the glass screen each coated with a layer of ITO (nano indium tin oxide). The outermost layer is a thin layer of silica glass protective layer. The interlayer ITO coating serves as the working surface, and four electrodes are drawn out from the four corners. The inner layer ITO serves as a shielding layer to ensure a good working environment. When the user's hand touches the metal layer (for example, the contact between the side of the hand and the touch screen 103 in the embodiment of the present application), due to the human body's electric field, the user and the touch screen surface form a coupling capacitor. For high-frequency current, the capacitor is a direct conductor, so the finger draws a very small current from the contact point. This current flows out from the electrodes on the four corners of the touch screen, and the current flowing through these four electrodes is proportional to the distance from the finger to the four corners. The controller accurately calculates the ratio of these four currents to derive the position of the touch point, thereby realizing the recognition of the touch gesture.

[0079] It should be understood that the touch screen 103 can also be other types of touch screens with contact gesture recognition capabilities, or can be replaced with a touch screen that only has a display function but can cooperate with other external devices (such as sensors) to realize contact gesture recognition, which is not limited here.

[0080] In one example, the pen 102 can also provide input to the electronic device 101 by contacting or otherwise interacting with the touch screen 103 .

[0081] The touch screen 103 can display patterns and texts, and can also provide a writing and drawing interface for users to write and draw freely, such as the whiteboard interface provided by an electronic whiteboard, the writing and drawing board provided by an application (APP), etc.

[0082] In the embodiment of the present application, the user can touch the touch screen 103 with the side of the hand, and the electronic device 101 can trigger the display of the layout after detecting the user's contact gesture.

[0083] The above introduces the exemplary operating environment of the present application. Next, the internal structure of the electronic device 101 is described with reference to an example.

[0084] Refer to FIG. 2 , which is a schematic structural diagram of an electronic device 101 provided in an embodiment of the present application.

[0085] It is understood that the structure shown in FIG2 does not constitute a specific limitation on the electronic device 101. In other embodiments of the present application, the electronic device 101 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0086] The electronic device 101 may include an input / output controller 218 that can output information to one or more output devices 222 (e.g., a touch screen or a speaker) that are separate from or integrated with the electronic device 101. The input / output controller 218 can also be used to receive input from one or more input devices 220 (e.g., a keyboard, a microphone, or a touch screen). In one embodiment, the output device 222 can also serve as the input device 220. An example of such a device can be a touch screen. In some embodiments, a user can provide input to the input device 220 and / or receive output from the output device 222.

[0087] In an embodiment of the present application, the input device 220 may be a touch screen. The user provides gesture input to the input / output controller 218 by contacting the touch screen with the side of the hand. The input / output controller 218 may transmit the gesture input to the processor for processing by the processor 204.

[0088] The electronic device 101 may include one or more processors 204, which may include one or more processing units. For example, the processor 204 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0089] The controller of the processor 204 can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and execution.

[0090] Processor 204 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 204 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 204. If processor 204 needs to use the instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 204 latency, and thus improves system efficiency.

[0091] In some embodiments, the processor 204 may include one or more interfaces, such as, but not limited to, a mobile industry processor interface (MIPI), an external memory interface, and / or a universal serial bus (USB) interface.

[0092] The MIPI interface can be used to connect the processor 204 to peripheral devices such as a touch screen. The MIPI interface can include a display serial interface (DSI), etc. The processor 204 and the touch screen communicate via the DSI interface to implement the display function of the touch screen.

[0093] It is understood that the interface connection relationship between the modules illustrated in this embodiment is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 101. In other embodiments of the present application, the electronic device 101 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0094] The electronic device 101 can realize the display function through GPU, touch screen, and application processor etc. GPU is a microprocessor for image processing, connected to touch screen and application processor. GPU is used for performing mathematical and geometric calculations, for graphics rendering. Processor 204 may include one or more GPUs, which execute program instructions to generate or change display information. Specifically, one or more GPUs in processor 204 can realize the rendering task of image (such as the rendering task related to drawing layout, distance value, length value, etc. in this application, and the rendering result is passed to application processor or other display driver, which is triggered by application processor or other display driver to display layout, distance value, length value, etc.).

[0095] A touch screen may include a display screen and associated sensors (eg, pressure sensors and touch sensors).

[0096] Display screens are used to display images, videos, and more. They include display panels. Display panels can be liquid crystal displays (LCDs), organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniLEDs, microLEDs, micro-oLEDs, and quantum dot light-emitting diodes (QLEDs).

[0097] The pressure sensor is used to sense pressure signals and can convert pressure signals into electrical signals. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device comprising at least two parallel plates having a conductive material. When a force acts on the pressure sensor, the capacitance between the electrodes changes. The electronic device 101 can determine the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the touch screen, the electronic device 101 can detect the intensity of the touch operation based on the pressure sensor. The electronic device 101 can also calculate the position of the touch based on the detection signal of the pressure sensor. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions.

[0098] A touch sensor, also known as a "touch control device," can be provided on a touch screen, which is composed of a touch sensor and a display screen. The touch sensor is used to detect touch operations applied on or near the touch sensor. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided through the display screen. In other embodiments, the touch sensor can also be provided on the surface of the electronic device 101, in a location different from that of the display screen.

[0099] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable applications such as intelligent cognition in electronic device 101, such as touch gesture recognition.

[0100] The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 101. The external memory card communicates with the processor 204 via the external memory interface to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0101] The memory 214 can be used to store computer executable program code, which includes instructions. The memory 214 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system 206, application software 208 required for at least one function (such as an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 101 (such as image data, etc.), etc. In addition, the memory 214 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 204 executes various functional applications and data processing of the electronic device 101 by running instructions stored in the memory 214 and / or instructions stored in a memory provided in the processor.

[0102] Among them, the method described in the embodiment of the present application can be a code stored in the memory 214 (or an external memory), and the processor 110 can obtain the code in the memory to implement the method provided by the embodiment of the present application.

[0103] The electronic device 101 can also interact with other electronic devices through the communication device 216 .

[0104] It should be understood that, alternatively or additionally, the methods described herein may be at least partially performed by one or more hardware logic components. For example, and not limitation, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), and programmable logic devices (PLDs). For example, functions such as contact gesture detection and gesture category determination may be implemented based on hardware logic components.

[0105] In addition, at least part of the functionality of the various elements in the figures may be distributedly performed by other elements in the figures or entities not shown in the figures (e.g., processors, web services, servers, applications, computing devices, etc.), without limitation here.

[0106] Smart whiteboard devices with handwriting recognition often have low recognition rates or even no recognition in complex writing scenarios. This is mostly due to users adding a large number of meaningless strokes when writing on the blackboard. When users write on smart electronic whiteboards, they often repeatedly change and edit the content of their writing as their thoughts change, which can lead to the loss of important information. When multiple users use electronic whiteboards, they write and overlap on the same page, making it difficult to accurately recognize handwriting and causing the loss of important information. When using whiteboards for meetings, many meaningless strokes such as emphasis lines, underlines, and circles are added to different writing points, making the key points unclear. At this point, if users want to save clear and concise blackboard writing, they can only erase and rewrite, or frequently save the layout manually or automatically while writing, which greatly affects input efficiency and reduces the user experience.

[0107] For ease of understanding, the layout display method provided in the embodiment of the present application is described in detail with reference to the accompanying drawings and application scenarios.

[0108] 3 , which is a flow chart of a layout display method according to an embodiment of the present application, as shown in FIG3 , the layout display method according to an embodiment of the present application includes:

[0109] 301. Receive input and display a first page, where the first page is the input written content, and the written content includes multiple writing elements.

[0110] The input may be a gesture input or an input performed through an external device, such as a writing or drawing input performed through a mouse. For ease of description, the embodiments of the present application are mainly described using gesture input as an example.

[0111] In one possible implementation, a user may input handwritten content through gesture input, thereby displaying a corresponding layout on an electronic device. The input handwritten content may include multiple writing elements, wherein the writing element may be one or more complete strokes.

[0112] In a possible implementation, data of the user's gesture input may be obtained.

[0113] In the embodiment of the present application, the gesture input may be an input of a contact gesture, and the electronic device may detect the contact gesture with the touch screen.

[0114] The touch screen may include a touch-sensitive surface, sensor or sensor group that receives input from the user based on tactile contact, and "detecting a contact gesture with the touch screen" may be understood as detecting a contact gesture with the touch-sensitive surface on the touch screen.

[0115] First, devices and modules related to detecting contact gestures on the touch screen are described.

[0116] In one possible implementation, the touch screen can cooperate with a detection module to detect touch gestures on the touch screen. The detection module can be a program, data structure, or a subset thereof stored in memory and related to touch gesture detection, or a portion thereof stored in memory in the form of a program, data structure, or a subset thereof, and a portion thereof in the form of a hardware logic module. The touch screen can capture touch data, and the detection module can perform various actions related to touch gesture detection based on the touch data.

[0117] In one possible implementation, the touch screen can capture the contact data between the user and the touch-sensitive surface in real time (for example, an electrical signal), and transmit the contact data to the detection module. The detection module can perform various actions related to contact gesture detection based on the contact data.

[0118] Next, the detection module is described to implement the detection of contact gestures on the touch screen.

[0119] In one implementation, the detection module can determine the intensity and / or its changes when the user contacts the touch screen, and the size and / or its changes of the contact area based on the contact data, and then determine the gesture type of the contact gesture based on the above information.

[0120] Here, "intensity" can be understood as the force or pressure (force per unit area) of a contact (e.g., contact by the side of a hand) on the touch-sensitive surface of the touch screen. The intensity of the contact can be determined using a variety of methods and a variety of sensors or a combination of sensors. For example, below the touch-sensitive surface, one or more sensors adjacent to the touch-sensitive surface are used to measure the force at different points on the touch-sensitive surface of the touch screen. In some specific implementations, the force measurements from multiple sensors can be combined (e.g., weighted average) to determine the intensity of the contact. In addition, the size of the contact area detected on the touch-sensitive surface and / or its changes, the capacitance of the touch-sensitive surface near the contact and / or its changes, and / or the resistance of the touch-sensitive surface near the contact and / or its changes can also be used to indirectly represent the force or pressure of the contact on the touch-sensitive surface.

[0121] Here, the "contact area" can be understood as the area that can be detected and in contact with the touch-sensitive surface. The area can be a continuous area or a discrete area composed of dense touch points (for example, the contact area can be a long strip area composed of a group of densely distributed contact points).

[0122] The detection module can perform various actions related to contact gesture detection based on the contact data, and further determine the gesture type of the contact gesture, the movement of the contact gesture, the stop status of the contact gesture, and the like.

[0123] For example, the detection module can determine whether contact has occurred and the type of contact gesture (e.g., detecting a finger press event, or a contact event on the side of the hand), determine whether there is movement of the contact and track the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events, or a drag event on the side of the hand), and determine whether the contact has stopped (e.g., detecting a finger lift event, or detecting a lift event on the side of the hand).

[0124] Among them, the above-mentioned "movement of the contact gesture" can also be referred to as the movement of the contact area of ​​the contact gesture, and the data related to the movement of the contact area may include but is not limited to the rate (value), speed (value and direction) and / or acceleration (change in value and / or direction) of the movement of the contact area.

[0125] In one possible implementation, the gesture input data may include at least one of the following: static features of the writing element, the relationship between the static features of the writing elements adjacent in space, the relationship between the static features of the writing elements adjacent in time, and the action features of the writing element.

[0126] On electronic devices with electronic whiteboards, writing on the board typically generates writing information, including stroke trajectory information. Writing on the board creates a series of stroke trajectories, which can be represented by a sequence of coordinate points. Devices with pressure sensors also generate stroke pressure information. The electronic whiteboard records the pressure applied during writing, which can be used to determine the thickness and intensity of the writing. Stroke speed information can also be recorded, which can be used to determine the fluency and consistency of the writing. Stroke direction information can also be recorded, which can be used to determine the direction and tilt of the writing. Stroke color information: Electronic whiteboards typically support multiple colors, and the color used can also be recorded. Stroke timestamp information: Electronic whiteboards typically record the timestamp of the user's writing, which can be used to determine the speed and rhythm of the writing. Action information can also be included, such as when the user erases, circles, or moves a part.

[0127] For example, referring to Figure 5A , the input gesture data in Figure 5A may include single-stroke features, features of adjacent strokes within a spatial space K, features of adjacent strokes within a temporal space K, and motion features. Single-stroke features include stroke projection length, curvature, and center of gravity, stroke bounding box length and width, and stroke convex hull geometric features; features of adjacent strokes within a spatial space K include the distance between adjacent stroke start and end points and the center of gravity of adjacent strokes; features of adjacent strokes within a temporal space K include the mean and variance of the number of points, and the distance between the centers of gravity of adjacent strokes; and motion features include motion speed, motion type, number of points in the motion trajectory, and motion contact area.

[0128] The embodiment of the present application constructs features based on the time, space, and semantic information of the strokes, and the recognition results are more accurate.

[0129] After recording the gesture input data, the gesture input data can be cached, and subsequent redundant information recognition and layout generation operations are performed only when the user requests to switch the layout, thereby reducing the overhead of computing resources.

[0130] Redundant information recognition can include gesture input type recognition, meaning that various types of information generated by user writing can be cached. If the classification engine is resident, it will occupy a large amount of the device's computing and memory resources, potentially causing device lag and affecting the normal operation of other functions. To reduce resource usage, this technology does not maintain a resident stroke classification engine in the background. Instead, a resident cache engine is used, caching user writing and operation information in memory for subsequent classification and calculation operations.

[0131] 302. Receive a first layout switching instruction for the first layout, and replace the first layout with a second layout based on the first layout switching instruction, wherein the second layout includes some of the multiple writing elements; the some of the writing elements correspond to the same user intention, and the second layout does not correspond to the same user intention as the writing elements not included in the first layout and the some of the writing elements, and the input time of the not included writing elements is before the input time of at least one of the some of the writing elements.

[0132] When key content cannot be identified or the layout is chaotic due to interference information (such as redundant written content, written content input by different users), the embodiment of the present application can identify redundant information in the publication page and eliminate the redundant information to reproduce the layout containing key information, wherein redundant information can be considered as information that is different from the main semantics of the layout. Users often have one or more input intentions when inputting, that is, part of the user's gesture input can correspond to the same user intention, and another part of the gesture input can correspond to other user intentions, or the written content input by different users corresponds to different user intentions. The so-called partial gesture input can correspond to the same user intention here can be understood as: the written content of non-redundant content input by the user to express the same subject content can correspond to the same user intention. Usually, the same subject content corresponds to the same semantics.

[0133] Redundant content can be understood as: when the content is missing, the semantic integrity of the layout is not affected. For example, referring to Figure 4, the multiple circles and double underlines in the written content of area labeled 2 in Figure 1 can be considered as redundant content. When this part of the content is missing, the semantic integrity of the written content in area 2 will not be affected. For another example, referring to Figure 4, the single circle and single underline in the written content of area labeled 3 in Figure 1 can be considered as redundant content. When this part of the content is missing, the semantic integrity of the written content in area 3 will not be affected.

[0134] The idea of ​​the embodiment of the present application is to record the user's gesture input information. When the user needs to reproduce the historical layout (or it can be described as the user requesting to switch the layout), the redundant information in the layout can be eliminated, thereby reproducing the written content with the same user intention. The layout reproduced in this way can present clearer and more complete written content, thereby improving the user experience.

[0135] In an embodiment of the present application, the second layout may be a layout obtained by reconstructing the first layout. Compared with the first layout, the second layout eliminates elements that do not belong to the same user intention. Since the withdrawal operation is not performed according to a gesture input time similar to the user's, the input time of the eliminated elements will be earlier than the input time of the retained elements (the writing elements that are later than the eliminated elements are retained because they and other retained elements belong to the same user intention).

[0136] In an embodiment of the present application, the first layout may be the layout obtained after receiving the user's gesture input. Since there are often multiple topics or some content that has a visual impact on important content (that is, redundant content) when the user inputs, in this case, the user may have a need to reproduce one or more specified layouts in history. Each layout may correspond to a user intention, so that the user can easily view the clear and well-defined layout content. Among them, the written content with the same user intention may be: having the same semantics and no redundant information.

[0137] When the user wishes to reproduce one or more designated layouts in history, he or she may input a first layout switching instruction for the first layout. The first layout switching instruction may be used to instruct to reconstruct the layout based on part or all of the written content in the first layout.

[0138] For example, in one possible implementation, a selection instruction for a portion of the written content in the first layout and a layout switching instruction for the portion of the handwritten content may be received.

[0139] For example, in one possible implementation, a selection instruction for all the written contents in the first layout and a layout switching instruction for all the written contents may be received.

[0140] Exemplarily, the selection instruction may be circle selection or select all. For example, the user may click a circle selection button to circle part of the content manually or with a writing pen, or double-click the circle selection button to circle all pages.

[0141] In one possible implementation, it is also possible to specify that some elements in the first layout will not be eliminated when the layout is reconstructed. For example, a fix instruction for a target element among the multiple writing elements can be received, and the fix instruction is used to indicate that when the first layout is replaced with the second layout (the reconstructed layout), the display of the target element is maintained. Exemplarily, the circled content can be identified as the smallest unit, and the user can select a unit individually and choose to fix or eliminate it. The same unit can also be identified and the same unit can be operated as a whole. After a unit is selected to be fixed or eliminated, the unit is not affected by the sliding of the slider bar.

[0142] By means of the above method, a space for optional operation is provided to the user, so that the user can choose to fix or delete important unit information.

[0143] For example, referring to 5B , the user can select the written content to be operated (eg, the written content within the dotted circle), and the user can perform a fix operation or an erase operation on the circled content by interacting with the control.

[0144] In a possible implementation, when a user requests to switch a layout, subsequent redundant information identification and layout generation operations may be performed.

[0145] In one possible implementation, the detection module can perform various actions related to gesture detection based on the data of gesture input to determine the gesture type, where the gesture type is, for example but not limited to, click, double-click, long press, drag, static contact between the side of the hand and the touch screen, dragging the side of the hand, rotating the side of the hand, double-clicking the side of the hand, etc.

[0146] It should be understood that classification may also mean not identifying specific categories, but simply distinguishing different categories, that is, clustering.

[0147] In a possible implementation, the detection module may be a pre-trained neural network model, which has the ability to recognize the gesture category indicated by the contact gesture based on input gesture data.

[0148] For example, referring to Figure 5A , a stroke relationship graph can be generated based on static features in gesture input data (e.g., single stroke features, features of adjacent strokes within K spatially, and features of adjacent strokes within K temporally). A motion relationship graph can be generated based on motion data, and then a feature vector for each stroke can be obtained. Coarse and fine classification can then be performed to obtain a classification result.

[0149] For example, refer to Figure 6, which is a schematic diagram of the classification results, where different shades of color correspond to different categories. This information can be used as a reference factor for subsequent layout determination. Refer to Figure 7, which is a schematic diagram of the stroke classification process.

[0150] Regarding the rule-matching-based coarse classification module in Figure 5A: In order to ensure that the data obtained by the stroke classifier is clean enough, a rule-matching-based coarse classification module is constructed here to remove non-human information or some noise in the data, such as meaningless points, aimless strokes, or strokes with a writing speed lower than a certain threshold, and to eliminate such information from the data.

[0151] Regarding the stroke classifier based on a deep neural network in Figure 5A. This module trains a neural network based on attention to train the stroke classifier neural network. Through this module, the feature data input in step e is classified into text, graphics, flowcharts, tables, or other categories of data. The calculation method involved in the self-attention is as follows:

[0152] Where Q, K, and V represent Query, Key, Value, and d k is the feature dimension. Q, K, V are: Q = W Q X; K=W K X; V=W V X;

[0153] As you can see, the query, index, and content of self-attention are all for itself. The core goal of self-attention is to select the information that is more critical to the current task goal from a lot of information and focus on it.

[0154] In addition, the position information of the writing element can be determined based on the data of the input gesture, and this information can be used as a reference factor for subsequent layout determination.

[0155] In one possible implementation, the aforementioned types of data can be converted into structured data, which can be used as a reference for subsequent layout determination. Specifically, referring to FIG8 , the structured data can be input into a language model so that the language model can generate the layout (or what can be called layout reconstruction).

[0156] In one possible implementation, the intention of the input gesture can also be understood based on the data of the input gesture. For example, the data of the input gesture can be processed by a neural network to obtain the intention classification of each input gesture, that is, to identify which input gestures correspond to the same user category. This information can be used as a reference factor for subsequent layout determination.

[0157] In the recognition scenario, after the content input by the user passes through the classification engine, different categories of data will be generated, such as text data, graphic data, and table data, but the classification engine will not specifically identify the content of this data.

[0158] After acquiring this data, it is fed into various models using existing methods. For text data, various text recognition technologies, including but not limited to text recognition, word segmentation, part-of-speech tagging, and named entity recognition, are used to process and provide feedback to the user. For graphic data, the graphics can be segmented, recognized, and classified, identifying their individual nodes and connections to restore a complete printed flow chart.

[0159] The present invention is also based on this process design. This module includes, but is not limited to, a text recognition engine, a multi-language recognition engine, a graphic recognition engine, and a table recognition engine. This module may have multiple engines depending on the number of classification engines. Therefore, a multi-engine scheduling module is additionally provided to schedule the appropriate engines for multiple recognition tasks, as well as to coordinate models. For a detailed solution, please refer to Figure 8.

[0160] For example, referring to Figure 9A, in Figure 9A, the intention of the input gesture can be recognized (such as intention 1 and intention 2 shown in the figure), and then the structured data and the intention category of the input gesture are input into the language model so that the language model can generate the layout (or it can be called layout reconstruction).

[0161] Next, we will introduce the reconstructed layout information in the embodiment of this application.

[0162] Taking the reconstruction of the first layout to obtain the second layout as an example, the second layout can include part of the written content in the first layout, and part of the content corresponds to the same user intention, that is, written content with the same semantics and no redundant information. Through the above method, on the one hand, the written content can be structurally reconstructed during the writing process, which greatly improves the user's input efficiency and experience. On the other hand, since the reconstructed layout can correspond to a user intention, the user can easily view the clear and main content of the layout.

[0163] In a possible implementation, some writing elements in the first layout may disappear in sequence until the second layout is displayed.

[0164] In one possible implementation, some written elements in the first layout may be eliminated sequentially based on the relevance of the written elements to the user's intent. For example, redundant information typically has a low relevance to the user's intent (relevance identification can be implemented based on a neural network, and the specific relevance can be learned by the neural network).

[0165] In one possible implementation, the writing elements in the first layout can be gradually eliminated (for example, based on the size of the correlation between the writing elements and the user's intention, the stroke elements with less correlation can be eliminated first). The user can choose how many strokes to eliminate. For example, when eliminating to the second layout, the user thinks that the second layout is the desired layout, and thus the second layout is displayed.

[0166] The above-mentioned process of gradually eliminating the written strokes can be achieved by the user through a sliding operation. That is, in a possible implementation, the first layout switching instruction is generated for a sliding operation of the slider.

[0167] For example, the sliding of the slider may have two-directional operations, and different directions correspond to different input time directions. Moving to the left may gradually eliminate, and moving to the right may gradually restore.

[0168] For example, when a user selects a layout to be reconstructed, a slider bar can be activated. If the selected content is partially circled, the slider bar activates below the circled content; if the entire content is selected, the slider bar activates below the screen. The user-based slider operation can be illustrated in Figure 9B. The user can select how many redundant strokes to eliminate by sliding the slider bar, preventing the loss of important information during the process.

[0169] In addition to operating the slider, multiple candidate layouts may be presented, each corresponding to the same user intent, with different candidate layouts being different, and the user selecting the layout to be displayed from the multiple candidate layouts. That is, in one possible implementation, the first layout switching instruction is generated for an operation on multiple candidate layouts, each candidate layout including some of the multiple writing elements, and each candidate layout corresponding to the same user intent, with different candidate layouts being different.

[0170] For example, in a possible implementation, the system cache module will always cache the original handwriting and the operation actions during the handwriting process, and the stroke classification engine will sleep at this time; then, when the user circles a part of the whiteboard, the stroke classification engine will start, and the stored strokes and operation actions will be pre-processed to extract data features and send them to the stroke classification engine. The stroke classifier will recognize text, graphics, tables, redundant strokes, etc., and then send them to the corresponding recognition engine; secondly, when the data is sent to the recognition engine, the engine scheduling system will schedule a recognition engine to perform targeted recognition on the data. The corresponding recognition engine will send the recognition results to the final reconstruction module; finally, the reconstruction module will perform layout analysis and element layout on the recognized results, and give multiple versions of results. The reconstructed layout will be ranked by importance according to the results given by the recognition engine, and then sent to the display interface. The user can slide the slider to choose to eliminate or retain the layout results.

[0171] In one possible implementation, when reconstructing the layout, the display content of multiple candidate layouts including the second layout can be determined based on the input gesture data through a language model. Each candidate layout corresponds to the same user intention, and different candidate layouts are different.

[0172] In one possible implementation, the recognition results based on the stroke classifier can be sent to the recognition engine. In the scenario where the user has finished writing and circled the writing module, the content of the circled part has been feature extracted, denoised or removed of meaningless strokes, and the mural classifier has been used to label each feature data with various types of labels. At this time, it will be sent to different recognition engines according to the labels of the feature data. If the number of labels is large and the various feature data are not completely independent, engine scheduling and collaborative recognition are required. In one possible implementation, the recognition engine can be scheduled collaboratively. This module designs multi-threaded scheduling to solve the engine scheduling problem. In actual use, it can be designed to be first-come-first-served, shortest job first, priority scheduling, time slice rotation or other scheduling algorithms.

[0173] In one possible implementation, data can be recognized by various engines. Because handwritten stroke data usually carries time sequence information, the device often retains timestamp information when the user writes. Therefore, optionally, the recognition engine of the embodiment of the present application can be based on, but not limited to, LSTM as a recognition engine. For example, in a handwritten English recognition engine, multi-threaded scheduling invokes the English recognition engine, and the feature data is fed into the LSTM network by the classification engine for sequence modeling to obtain the output of each sequence step, and finally classification is performed to obtain the final recognition result.

[0174] When the user pauses and circles a line, the circle option will have the option to circle part or all of the layout. If the circle all option is selected, the stroke and layout reconstruction engine will be activated to analyze and reconstruct the layout. The function of this module is mainly to obtain the different overall information generated during the writing process and display it in a structured layout. For example, when users write according to their thoughts, they often edit repeatedly, which may cause important information to be lost in the middle of the process. At this time, the module will understand the user's intention, retain K layouts through the language model, reconstruct them, and then let the user slide to select. The specific solution can be shown in Figure 9A:

[0175] In addition, the recognition results based on the recognition engine can also be sent to the stroke reconstruction module. After the recognition engine has recognized various results and structured them, the data is sent to the stroke reconstruction module. The overall stroke reconstruction technology based on semantics and intent can be shown in Figure 10: the structured data contains element index, data type, data content, bounding box and other data. The structured data is sent to the language model, and the language model gives the confidence score of the text, and finally sent to the reconstruction module. In the early feature extraction module, the feature extracted data is sent to the intent understanding module. After judgment by the intent understanding module, the strokes are divided into different parts according to different intentions and labeled, and then this data is sent to the reconstruction module.

[0176] At this point, the layout reconstruction module has received the final content results after language model processing, as well as the stroke intention labels corresponding to the content. The reconstruction module will reconstruct the stroke layout based on this information, and divide it into different layouts and send them to the display layer.

[0177] The functional performance is divided into two parts: circled partial recognition and circled full recognition. Circled partial recognition removes the slider displayed by the system, changing redundant strokes. Circled full recognition removes the slider, changing the layout and redundant strokes. The circled content can also be selected, cleared, or fixed, and is not affected by the slider.

[0178] Next, an application example of this application is introduced with reference to the accompanying drawings:

[0179] Figure 11 is a schematic diagram of the layout reconstruction process, wherein, as shown in Figure 11, the user can input on the electronic whiteboard, and the generated information includes stroke information, action information and timestamp information. The action information includes but is not limited to erasing, gesture erasing, undoing, deleting, clicking, circling, overwriting and other information.

[0180] To reduce resource usage, the cache module stores all information written by users in memory for subsequent computing operations.

[0181] When the user circles a certain part, the feature extraction module is first called up, and the data of the module cached in the memory is extracted according to the feature design, and then sent to the stroke classification engine and stroke reconstruction module.

[0182] The stroke classifier classifies features into different data, such as text, graphics, charts, process lines, etc., and labels the feature data with categories and sends them to the next module.

[0183] At this time, the stroke recognition engine uses the data passed in from the upper layer to mobilize various recognition engine modules in a multi-threaded manner to recognize the data, that is, to recognize the specific content of various types of data, such as what characters are in the text data, what graphics are in the graphic data, etc.

[0184] The data results recognized by the recognition engine are integrated, where each data includes data index, data type, data content, and bounding box information. The data is then sent to the language model module in the stroke reconstruction module.

[0185] The language model will assign confidence scores to the recognition results to obtain the most reasonable results and send them to the layout reconstruction module for final integration.

[0186] In addition, the text recognition engine can be used to classify the written content according to the strokes, and classify it into units of the smallest understandable granularity for the user to select. When the user clicks on the unit, the fix, eliminate and multiple selection buttons will appear. The multiple selection function is for selecting similar units. After selecting fix or eliminate, the unit will not participate in the layout reconstruction and analysis.

[0187] For example, only the circle selection module can be analyzed, that is, there is no intention analysis and multi-layout results in this module. This module refers to removing redundant strokes for the current single layout and providing a sliding bar at the display level. Sliding the sliding bar can select how many redundant strokes to eliminate.

[0188] Referring to Figure 12, which is a diagram illustrating an embodiment of the present application in a handwriting recognition scenario, the layout reconstruction module can match the final results with the intent labels one by one, place the recognition results of the same intent on the same layout according to the final layout format, and provide a multi-layout display.

[0189] Refer to Figure 13, which is a schematic diagram of the application of an embodiment of the present application in reconstructing the content of multiple people overlapping lost information on the same page.

[0190] As shown in Figure 13 (a), User 1 writes their own plan based on their ideas. User 2 continues to modify and add to it. In Figure 13 (b), User 2 may rewrite or edit the layout of User 1, resulting in the loss of some of User 1's information.

[0191] 14 , which is a schematic diagram of the structure of a layout display device provided in an embodiment of the present application, and applied to an electronic device, the device 1400 includes:

[0192] Processing module 1401 and display module 1402;

[0193] The processing module 1401 is used to receive input and control the display module 1402 to display a first layout, where the first layout is the input writing content, and the writing content includes multiple writing elements; receive a first layout switching instruction for the first layout, and control the display module based on the first layout switching instruction to replace the first layout with a second layout, where the second layout includes some writing elements among the multiple writing elements; the some writing elements correspond to the same user intention, and the second layout does not correspond to the same user intention as the writing elements not included in the first layout and the some writing elements, and the input time of the not included writing elements is before the input time of at least one writing element among the some writing elements.

[0194] For a detailed description of the processing module 1401 , please refer to the description of steps 301 and 302 , which will not be repeated here.

[0195] In a possible implementation, the writing elements included in the second layout are determined based on whether the corresponding user intentions are consistent.

[0196] In a possible implementation, the writing element is one or more complete strokes.

[0197] The input may specifically be gesture input.

[0198] In a possible implementation, the same user intention includes:

[0199] Input is via the same writing element as the user input; or,

[0200] If it is missing, it will affect the semantic integrity of the written content included in the second layout.

[0201] In a possible implementation, the processing module 1401 is specifically configured to:

[0202] receiving a selection instruction for a portion of the handwritten content in the first layout, and a layout switching instruction for the portion of the handwritten content; the portion of the handwritten elements belongs to the portion of the handwritten content; or

[0203] A selection instruction for all the written contents in the first layout and a layout switching instruction for all the written contents are received.

[0204] In a possible implementation, before replacing the first layout with the second layout, the processing module 1401 is further configured to:

[0205] A fixed instruction for a target element among the plurality of writing elements is received, wherein the fixed instruction is used to instruct to keep displaying the target element when the first layout is replaced with the second layout.

[0206] In a possible implementation, the first layout switching instruction is generated for a sliding operation of a slider.

[0207] In a possible implementation, the processing module 1401 is specifically configured to control the display module 1402 to sequentially make some writing elements in the first layout disappear until the second layout is displayed.

[0208] In a possible implementation, the step of sequentially eliminating some of the writing elements in the first layout includes:

[0209] Some of the writing elements in the first layout are eliminated in sequence according to the relevance between the writing elements and the user intention.

[0210] In one possible implementation, the first layout switching instruction is generated for an operation on multiple candidate layouts, each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0211] In a possible implementation, the processing module 1401 is further configured to:

[0212] caching the input data corresponding to the first layout;

[0213] The display content of the second page is determined according to the data.

[0214] In a possible implementation, determining the display content of the second layout based on the data is performed after receiving the first layout switching instruction for the first layout.

[0215] In a possible implementation, the processing module 1401 is specifically configured to:

[0216] Based on the data, the display content of multiple candidate layouts including the second layout is determined through a language model, each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0217] In one possible implementation, the data includes at least one of the following:

[0218] The static features of the writing elements, the relationship between the static features of the writing elements adjacent in space, the relationship between the static features of the writing elements adjacent in time, and the action features of the writing elements.

[0219] Next, a terminal device provided in an embodiment of the present application is introduced. The terminal device may be the device described in Figure 14. Please refer to Figure 15. Figure 15 is a structural schematic diagram of a terminal device provided in an embodiment of the present application. The terminal device 1500 may be specifically an electronic whiteboard, a virtual reality VR device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which are not limited here. Specifically, the terminal device 1500 includes: a receiver 1501, a transmitter 1502, a processor 1503 and a memory 1504 (wherein the number of processors 1503 in the terminal device 1500 may be one or more, and Figure 15 takes one processor as an example), wherein the processor 1503 may include an application processor 15031 and a communication processor 15032. In some embodiments of the present application, the receiver 1501, the transmitter 1502, the processor 1503 and the memory 1504 may be connected via a bus or other means.

[0220] Memory 1504 may include read-only memory and random access memory, and provides instructions and data to processor 1503. A portion of memory 1504 may also include non-volatile random access memory (NVRAM). Memory 1504 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0221] Processor 1503 controls the operation of the terminal device. In specific applications, the various components of the terminal device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0222] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1503. Processor 1503 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1503. The above processor 1503 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components. The processor 1503 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or register. The storage medium is located in memory 1504. Processor 1503 reads the information in memory 1504 and, in conjunction with its hardware, completes the steps of the above method. Specifically, processor 1503 can read the information in memory 1504 and, in conjunction with its hardware, complete the steps related to steps 301 and 302 in the above embodiment.

[0223] Receiver 1501 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the terminal device. Transmitter 1502 can be used to output digital or character information through the first interface. Transmitter 1502 can also be used to send instructions to the disk pack through the first interface to modify the data in the disk pack. Transmitter 1502 can also include a display device such as a touch screen.

[0224] An embodiment of the present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the steps of the method described in the embodiment corresponding to FIG. 3 in the above embodiment.

[0225] An embodiment of the present application also provides a computer-readable storage medium, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps of the image processing method in the method described in the above embodiment.

[0226] The image display device provided in the embodiment of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip in the execution device to execute the data processing method described in the above embodiment, or to enable the chip in the training device to execute the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0227] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0228] Through the description of the above embodiments, it is clear to those skilled in the art that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course it can also be implemented by means of dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0229] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0230] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive (SSD)).

Claims

1. A layout display method, characterized in that: The method comprises: Receive input and display a first page, where the first page is the input written content, and the written content includes multiple writing elements; A first layout switching instruction for the first layout is received, and the first layout is replaced with a second layout based on the first layout switching instruction, wherein the second layout includes some of the multiple writing elements; the some of the writing elements correspond to the same user intention, and the second layout does not correspond to the same user intention as the writing elements not included in the first layout and the some of the writing elements, and the input time of the not included writing elements is before the input time of at least one of the some of the writing elements.

2. The method according to claim 1, characterized in that The writing elements included in the second layout are determined based on whether the corresponding user intentions are consistent.

3. The method according to claim 1 or 2, characterized in that The input is specifically gesture input.

4. The method according to any one of claims 1 to 3, characterized in that: The same user intent includes: Through the same user input writing element; or, If it is missing, it will affect the semantic integrity of the written content included in the second layout.

5. The method according to any one of claims 1 to 4, characterized in that: The receiving of the first layout switching instruction for the first layout includes: receiving a selection instruction for a portion of the handwritten content in the first layout, and a layout switching instruction for the portion of the handwritten content; the portion of the handwritten elements belongs to the portion of the handwritten content; or A selection instruction for all the written contents in the first layout and a layout switching instruction for all the written contents are received.

6. The method according to any one of claims 1 to 5, characterized in that: Before replacing the first layout with the second layout, the method further includes: A fixed instruction for a target element among the plurality of writing elements is received, wherein the fixed instruction is used to instruct that the display of the target element be maintained when the first layout is replaced with the second layout.

7. The method according to any one of claims 1 to 6, characterized in that: The first layout switching instruction is generated in response to a sliding operation of the slider.

8. The method according to any one of claims 1 to 7, characterized in that: The step of replacing the first layout with the second layout includes: Some writing elements in the first layout disappear in sequence until the second layout is displayed.

9. The method according to claim 8, characterized in that The step of sequentially eliminating some of the writing elements in the first layout includes: Some of the writing elements in the first layout are eliminated in sequence according to the relevance between the writing elements and the user intention.

10. The method according to any one of claims 1 to 9, characterized in that: The first layout switching instruction is generated for operations on multiple candidate layouts, each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

11. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: caching the input data corresponding to the first layout; The display content of the second page is determined according to the data.

12. The method according to claim 11, characterized in that The determining of the display content of the second layout based on the data is performed after the receiving of the first layout switching instruction for the first layout.

13. The method according to claim 11 or 12, characterized in that Determining the display content of the second page according to the data includes: Based on the data, the display content of multiple candidate layouts including the second layout is determined through a language model, each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

14. The method according to any one of claims 11 to 13, characterized in that: The data includes at least one of the following: The static features of the writing elements, the relationship between the static features of the writing elements adjacent in space, the relationship between the static features of the writing elements adjacent in time, and the action features of the writing elements.

15. The method according to any one of claims 1 to 14, characterized in that: The writing element is one or more complete strokes.

16. A layout display device, characterized in that: The device includes: a processing module and a display module; The processing module is used to receive input and control the display module to display a first layout, where the first layout is the input writing content, and the writing content includes multiple writing elements; receive a first layout switching instruction for the first layout, and control the display module based on the first layout switching instruction to replace the first layout with a second layout, where the second layout includes some writing elements among the multiple writing elements; the some writing elements correspond to the same user intention, and the second layout does not correspond to the same user intention as the writing elements not included in the first layout and the some writing elements, and the input time of the not included writing elements is before the input time of at least one writing element among the some writing elements.

17. The device according to claim 16, characterized in that The writing elements included in the second layout are determined based on whether the corresponding user intentions are consistent.

18. The device according to claim 16 or 17, characterized in that The input is specifically gesture input.

19. The device according to any one of claims 16 to 18, characterized in that The same user intent includes: written elements input by the same user's gesture; or, If it is missing, it will affect the semantic integrity of the written content included in the second layout.

20. The device according to any one of claims 16 to 19, characterized in that The processing module is specifically used to: receiving a selection instruction for a portion of the handwritten content in the first layout, and a layout switching instruction for the portion of the handwritten content; the portion of the handwritten elements belongs to the portion of the handwritten content; or, A selection instruction for all the written contents in the first layout and a layout switching instruction for all the written contents are received.

21. The device according to any one of claims 16 to 20, characterized in that Before replacing the first layout with the second layout, the processing module is further configured to: A fixed instruction for a target element among the plurality of writing elements is received, wherein the fixed instruction is used to instruct that the display of the target element be maintained when the first layout is replaced with the second layout.

22. The device according to any one of claims 16 to 21, characterized in that The first layout switching instruction is generated in response to a sliding operation of the slider.

23. The device according to any one of claims 16 to 22, characterized in that The processing module is specifically used to control the display module to make some writing elements in the first layout disappear in sequence until the second layout is displayed.

24. The device according to claim 23, characterized in that The step of sequentially eliminating some of the writing elements in the first layout includes: Some of the writing elements in the first layout are eliminated in sequence according to the relevance between the writing elements and the user intention.

25. The device according to any one of claims 16 to 24, characterized in that The first layout switching instruction is generated for operations on multiple candidate layouts, each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

26. The device according to any one of claims 16 to 25, characterized in that The processing module is further configured to: caching the data of the gesture input corresponding to the first layout; The display content of the second page is determined according to the data.

27. The device according to claim 26, characterized in that The determining of the display content of the second layout based on the data is performed after the receiving of the first layout switching instruction for the first layout.

28. The device according to claim 26 or 27, characterized in that The processing module is specifically used to: Based on the data, the display content of multiple candidate layouts including the second layout is determined through a language model, each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

29. The device according to any one of claims 26 to 28, characterized in that The data includes at least one of the following: The static features of the writing elements, the relationship between the static features of the writing elements adjacent in space, the relationship between the static features of the writing elements adjacent in time, and the action features of the writing elements.

30. The device according to any one of claims 16 to 29, characterized in that The writing element is one or more complete strokes.

31. An electronic device, characterized in that: The device includes a processor, a memory, a display screen, and a bus, wherein: The processor, the memory and the display screen are connected via the bus; The memory is used to store computer programs or instructions; The processor is used to call or execute the program or instruction stored in the memory, and is also used to control the touch screen to implement the method steps described in any one of claims 1-15.

32. A computer-readable storage medium comprising a program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 15.

33. A computer program product comprising instructions, characterized in that When the computer program product is run on a terminal, the terminal is enabled to execute the method according to any one of claims 1 to 15.

Images