Layout display method and related equipment

Abstract

The invention provides a layout display method which comprises the steps that gesture input is received, a first layout is displayed, the first layout is writing content input by the gesture, and the writing content comprises a plurality of writing elements; a first layout switching instruction for the first layout is received, the first layout is replaced and displayed to be a second layout based on the first layout switching instruction, and the second layout comprises part of the writing elements in the multiple writing elements; the partial writing elements correspond to the same user intention, and the writing elements not included in the second layout and the partial writing elements do not correspond to the same user intention compared with the writing elements not included in the first layout. The gesture input time of the non-included writing elements is before the gesture input time of at least one writing element in the partial writing elements. The layout reproduced in this way can present more clear and complete writing content, so that the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of terminal software, and particularly to a layout display method and related devices. Background Art

[0002] With the rapid development of Internet and communication technologies, office informatization has increasingly become an important part of the entire IT service system. Among them, the most user-friendly interactive touch display devices (such as interactive whiteboards) are popular and have become standard accessories for digital conferences. An electronic whiteboard is generally suitable for multi-person discussion scenarios and serves a role similar to that of a blackboard.

[0003] In a smart whiteboard device with handwriting recognition, there are usually problems of low recognition rate and inability to recognize in complex writing scenarios. Most of the reasons are that users add a large number of meaningless strokes when writing on the blackboard. When users write on a smart electronic whiteboard, they usually change and edit the writing content repeatedly according to their writing ideas, which may lead to the loss of important information. When multiple users use an electronic whiteboard, they will write and overlay on the same layout at the same time, resulting in the inability to accurately recognize the handwriting and the loss of important information. When using a whiteboard in a meeting, for different writing points, users will write more meaningless strokes such as emphasis lines, underlines, and circles, resulting in the lack of prominence of key content. At this time, if users want to save a clear and understandable writing, they can only erase and rewrite it, or manually or automatically save the layout frequently during writing, which will greatly affect the input efficiency and reduce the user experience. Summary of the Invention

[0004] In a first aspect of an embodiment of this application, a layout display method is provided. The method includes: receiving an input and displaying a first layout, where the first layout is the writing content of the input, and the writing content includes multiple writing elements; receiving a first layout switching instruction for the first layout, and based on the first layout switching instruction, replacing and displaying the first layout with a second layout, where the second layout includes some of the multiple writing elements; the some writing elements correspond to the same user intention, and the writing elements not included in the second layout compared with the first layout and the some writing elements do not correspond to the same user intention, and the input time of the writing elements not included is before the input time of at least one of the some writing elements.

[0005] When the key content cannot be recognized or the layout is chaotic due to interference information (such as redundant writing content, writing content input by different users), the embodiments of the present application can identify the redundant information in the layout and remove the redundant information to reproduce the layout containing the key information. Among them, the redundant information can be considered as information that is different from the main semantics of the layout. When the user inputs, there is often one or more input intentions, that is, some of the user's inputs can correspond to the same user intention, and some other inputs can correspond to other user intentions, or the writing content input by different users corresponds to different user intentions. The so-called part of the input corresponding to the same user intention can be understood as: the writing content of non-redundant content input by the user to express the same theme content can correspond to the same user intention. Usually, the same theme content corresponds to the same semantics.

[0006] The redundant content can be understood as: when this content is missing, the semantic integrity of the layout is not affected.

[0007] Among them, the input can specifically be gesture input.

[0008] The idea of the embodiments of the present application is that the input information of the user can be recorded. When the user needs to reproduce the historical layout (or it can be described as the user requests to switch the layout), the redundant information in the layout can be removed, so as to reproduce the writing content with the same user intention. The layout reproduced in this way can present a writing content with clearer and more complete content, thereby improving the user experience.

[0009] In the embodiments of the present application, the second layout can be the layout obtained after reconstructing the first layout. Compared with the first layout, the second layout removes the elements that do not belong to the same user intention. Since it is not a withdrawal operation similar to the user's input time, the input time of the removed elements will be earlier than the input time of some of the remaining elements (the writing elements later than the removed elements are retained because they belong to the same user intention as other retained elements). The writing elements included in the second layout are determined based on whether the corresponding user intentions are the same. Compared with the prior art that only considers the sequence of input time for layout reproduction, the embodiments of the present application perform layout reproduction based on the input user intention.

[0010] In the embodiments of the present application, the first layout can be the layout obtained after receiving the user's input. Since there are often multiple topics or some content that visually affects the important content (i.e., redundant content) when the user is inputting, in this case, the user may have a need to reproduce one or more specified layouts in history. Each layout can correspond to a user intention, so that the user can easily view the layout content clearly and with a clear main body. Among them, the writing content with the same user intention can be: having the same semantics and no redundant information.

[0011] In a possible implementation, the writing element is one or more complete strokes.

[0012] In a possible implementation, the same user intention includes: the writing elements input by the same user; or, when missing, it will affect the semantic integrity of the writing content included in the second layout.

[0013] In a possible implementation, the receiving of the first layout switching instruction for the first layout includes: receiving a selection instruction for a part of the writing content in the first layout, and a layout switching instruction for the part of the handwritten content; the part of the writing elements belongs to the part of the writing content; or, receiving a selection instruction for all of the writing content in the first layout, and a layout switching instruction for all of the writing content.

[0014] When the user hopes to reproduce one or more specified layouts in history, the user can input a first layout switching instruction for the first layout. The first layout switching instruction can be used to indicate the reconstruction of the layout based on part or all of the writing content in the first layout.

[0015] In a possible implementation, some elements in the first layout can also be specified not to be eliminated during the layout reconstruction. For example, a fixed instruction for a target element among the multiple writing elements can be received. The fixed instruction is used to indicate that when the first layout is replaced and displayed as the second layout (the reconstructed layout), the display of the target element is maintained. Exemplarily, the selected content can be recognized as the smallest unit. The user can single-click to select a certain unit and can choose to fix or eliminate it. And the same units can be recognized and the same units can be operated on as a whole. After a unit is selected to be fixed or eliminated, the unit is not affected by the sliding of the scroll bar.

[0016] By the above method, an operable space is provided for the user, so that the user can independently select to fix or eliminate important unit information.

[0017] In a possible implementation, the first layout switching instruction is generated for the sliding operation of the slider.

[0018] In a possible implementation, the replacing the display of the first layout with the second layout includes: making some writing elements in the first layout disappear in sequence until the second layout is displayed.

[0019] In a possible implementation, the making some writing elements in the first layout disappear in sequence includes:

[0020] Making some writing elements in the first layout disappear in sequence according to the relevance between the writing elements and the user intention.

[0021] In a possible implementation, the first layout switching instruction is generated for the operation on multiple candidate layouts. Each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0022] In a possible implementation, the method further includes: caching the input data corresponding to the first layout; determining the display content of the second layout according to the data.

[0023] After recording the input data, the input data can be cached, and only when the user requests to switch the layout, subsequent redundant information recognition and layout generation operations are performed, so as to reduce the overhead of computing resources.

[0024] In a possible implementation, the determining the display content of the second layout according to the data is performed after receiving the first layout switching instruction for the first layout.

[0025] In a possible implementation, the determining the display content of the second layout according to the data includes: determining the display content of multiple candidate layouts including the second layout according to the data through a language model. Each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0026] In a possible implementation, the data includes at least one of the following: static features of the writing elements, relationships between static features of adjacent writing elements in space, relationships between static features of adjacent writing elements in time, and action features of the writing elements.

[0027] The embodiments of the present application construct features for the time, space, and semantic information of strokes, and the recognition result is more accurate.

[0028] Second aspect, the present application provides a layout display device, the device comprising: a processing module and a display module;

[0029] The processing module is configured to receive an input, control the display module to display a first layout, the first layout being the written content of the input, the written content comprising a plurality of writing elements; receive a first layout switching instruction for the first layout, and based on the first layout switching instruction, control the display module to replace the display of the first layout with a second layout, the second layout comprising some of the plurality of writing elements; the some writing elements correspond to the same user intention, and the writing elements not included in the second layout compared to the first layout and the some writing elements do not correspond to the same user intention, and the input time of the writing elements not included is before the input time of at least one of the some writing elements.

[0030] Wherein, the input may specifically be a gesture input.

[0031] In a possible implementation, the writing elements included in the second layout are determined based on whether the corresponding user intentions are the same.

[0032] In a possible implementation, the writing element is one or more complete strokes.

[0033] In a possible implementation, the same user intention includes:

[0034] The writing elements input by the same user; or,

[0035] When missing, it will affect the semantic integrity of the written content included in the second layout.

[0036] In a possible implementation, the processing module is specifically configured to:

[0037] Receive a selection instruction for some of the written content in the first layout, and a layout switching instruction for the some handwritten content; the some writing elements belong to the some written content; or,

[0038] Receive a selection instruction for all of the written content in the first layout, and a layout switching instruction for all of the written content.

[0039] In a possible implementation, before replacing the display of the first layout with the second layout, the processing module is further configured to:

[0040] Receive a fixation instruction for a target element among the multiple writing elements, where the fixation instruction is used to indicate that the display of the target element is maintained when the first layout is replaced and displayed as the second layout.

[0041] In a possible implementation, the first layout switching instruction is generated for a sliding operation of a slider.

[0042] In a possible implementation, the processing module is specifically configured to control the display module to sequentially disappear some writing elements in the first layout until the second layout is displayed.

[0043] In a possible implementation, the sequentially disappearing some writing elements in the first layout includes:

[0044] Sequentially disappearing some writing elements in the first layout according to the magnitude of the relevance between the writing elements and the user intention.

[0045] In a possible implementation, the first layout switching instruction is generated for an operation on multiple candidate layouts, each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0046] In a possible implementation, the processing module is further configured to:

[0047] Cache the input data corresponding to the first layout;

[0048] Determine the display content of the second layout according to the data.

[0049] In a possible implementation, the determining the display content of the second layout according to the data is performed after receiving the first layout switching instruction for the first layout.

[0050] In a possible implementation, the processing module is specifically configured to:

[0051] Determine the display content of multiple candidate layouts including the second layout according to the data through a language model, each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0052] In a possible implementation, the data includes at least one of the following:

[0053] The static features of the writing elements, the relationship between the static features of adjacent writing elements in space, the relationship between the static features of adjacent writing elements in time, the action features of the writing elements.

[0054] In a third aspect, the present application provides an electronic device, including: a processor, a memory, a touch screen, and a bus, where: the processor, the memory, and the touch screen are connected through the bus;

[0055] The memory is used to store computer programs or instructions;

[0056] The processor is used to call or execute the programs or instructions stored on the memory, and is further used to control the touch screen to implement the steps described in the first aspect and any possible implementation manner in the first aspect.

[0057] In a fourth aspect, the present application provides a computer storage medium, including computer instructions, which, when running on an electronic device or a server, execute the steps described in the first aspect and any possible implementation manner in the first aspect.

[0058] In a fifth aspect, the present application provides a computer program product, which, when running on an electronic device or a server, executes the steps described in the first aspect and any possible implementation manner in the first aspect.

[0059] In a sixth aspect, the present application provides a chip system, which includes a processor for supporting an execution device or a training device to implement the functions involved in the above aspects. For example, sending or processing the data or information involved in the above method. In a possible design, the chip system further includes a memory for storing the necessary program instructions and data for the execution device or the training device. The chip system can be composed of chips or include chips and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the product structure provided by an embodiment of the present application;

[0061] Figure 2 It is a block diagram of the structure of the electronic device according to an embodiment of the present application;

[0062] Figure 3 It is a schematic diagram of an embodiment of a layout display method provided by an embodiment of the present application;

[0063] Figure 4 and Figure 5B and Figure 6 and Figure 9B and Figure 10 and Figure 13 It is a layout schematic in an embodiment of the present application;

[0064] Figure 5A and Figures 7 to 9A and Figure 11 andFigure 12 A gesture indication in an embodiment of the present application;

[0065] Figure 14 A structural schematic of a layout display device provided in an embodiment of the present application;

[0066] Figure 15 A structural schematic diagram of a terminal device provided in an embodiment of the present application. Specific embodiments

[0067] The embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0068] The embodiments of the present application will be described below with reference to the accompanying drawings. Those skilled in the art will know that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are equally applicable to similar technical problems.

[0069] The terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing the embodiments of the present application. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device comprising a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these processes, methods, products or devices.

[0070] First, an exemplary operating environment of the present application will be introduced.

[0071] The embodiments of the present application can be applied in a system 100 including a touch screen 103.

[0072] Refer to Figure 1 , Figure 1 shows the system 100 to which the embodiments of the present application are applied. Among them, the system 100 may include an electronic device 101 and a pen 102 associated with the electronic device 101.

[0073] The electronic device 101 may be Figure 1The shown electronic whiteboard (or electronic interactive smart board), the electronic device 101 includes a touch screen 103. It should be understood that the electronic device 101 can also be a portable mobile device including a touch screen, such as but not limited to a mobile or portable computing device (such as a smart phone), a personal computer, a server computer, a handheld device (such as a tablet) or a laptop device, a multiprocessor system, a game console or controller, a microprocessor-based system, a set-top box, a programmable consumer electronics product, a mobile phone, a mobile computing and / or communication device with a wearable or accessory form factor (such as a watch, glasses, a headset or earbuds), a network PC, a minicomputer, a mainframe computer, a distributed computing environment including any one of the above systems or devices, and so on.

[0074] The touch screen 103 can recognize the contact gestures of the user.

[0075] In one implementation, the touch screen 103 can be an infrared touch screen. The infrared touch screen is composed of infrared emission and reception sensing elements installed on the outer frame of the touch screen. On the screen surface, an infrared detection network is formed. Any touching object can change the infrared rays at the contact point to achieve touch screen operation. The implementation principle of the infrared touch screen is similar to that of the surface acoustic wave touch screen. It uses infrared emission and reception sensing elements. These elements form an infrared detection network on the screen surface. The touching object of the touch operation (such as the contact between the side of the hand and the touch screen 103 in the embodiment of the present application) can change the infrared rays at the contact point, and then be converted into the coordinate position of the touch, and then the recognition of the touch gesture is realized.

[0076] In one implementation, the touch screen 103 can be a capacitive touch screen. The capacitive touch screen works by using the current induction of the human body. Exemplarily, the capacitive touch screen can be a four-layer composite glass screen. The inner surface and the interlayer of the glass screen are each coated with a layer of ITO (nanometer indium tin metal oxide). The outermost layer is a thin layer of silica glass protective layer. The interlayer ITO coating serves as the working surface, and four electrodes are led out at the four corners. The inner layer ITO is the shielding layer, which can ensure a good working environment. When the user's hand touches the metal layer (such as the contact between the side of the hand and the touch screen 103 in the embodiment of the present application), due to the human body electric field, a coupling capacitor is formed between the user and the touch screen surface. For high-frequency current, the capacitor is a direct conductor. So the finger draws a very small current from the contact point. This current flows out from the electrodes at the four corners of the touch screen respectively, and the current flowing through these four electrodes is proportional to the distance from the finger to the four corners. The controller calculates the position of the touch point through the precise calculation of the ratio of these four currents, and then the recognition of the touch gesture can be realized.

[0077] It should be understood that the touch screen 103 can also be other types of touch screens with touch gesture recognition capabilities, or be replaced with a touch screen that only has a display function but can cooperate with other external devices (such as sensors) to achieve touch gesture recognition, which is not limited here.

[0078] In one example, the pen 102 can also interact with the touch screen 103 by contacting it or in other ways to provide input to the electronic device 101.

[0079] The touch screen 103 can display patterns and text, and can also provide a writing and drawing interface for users to freely write and draw, such as the whiteboard interface provided by an electronic whiteboard, the writing board provided by an application (APP), etc.

[0080] In the embodiments of the present application, the user can contact the touch screen 103 with the side of the hand, and after the electronic device 101 detects the user's touch gesture, it can trigger the display of the layout.

[0081] The above introduced the exemplary operating environment of the present application. Next, the internal structure of the electronic device 101 will be described in combination with an example.

[0082] See Figure 2 , Figure 2 which is a schematic structural diagram of the electronic device 101 provided by the embodiments of the present application.

[0083] It can be understood that Figure 2 the schematic structure does not constitute a specific limitation on the electronic device 101. In other embodiments of the present application, the electronic device 101 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0084] The electronic device 101 may include an input / output controller 218. The input / output controller 218 can output information to one or more output devices 222 (such as a touch screen or a speaker) that are separate from or integrated with the electronic device 101. The input / output controller 218 can also be used to receive input from one or more input devices 220 (such as a keyboard, a microphone, or a touch screen). In one embodiment, the output device 222 can also act as the input device 220. An example of such a device can be a touch screen. In some embodiments, the user can provide input to the input device 220 and / or receive output from the output device 222.

[0085] In the embodiments of the present application, the input device 220 may be a touch screen. The user provides gesture input to the input / output controller 218 by contacting the touch screen with the side of the hand, and the input / output controller 218 may transfer the gesture input to the processor for processing by the processor 204.

[0086] The electronic device 101 may include one or more processors 204, and these processors may include one or more processing units. For example, the processor 204 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0087] The controller of the processor 204 may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0088] A memory may also be provided in the processor 204 for storing instructions and data. In some embodiments, the memory in the processor 204 is a cache memory. This memory may save the instructions or data that the processor 204 has just used or recycled. If the processor 204 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 204, and thus improves the efficiency of the system.

[0089] In some embodiments, the processor 204 may include one or more interfaces. For example, the interface may include, but is not limited to, a mobile industry processor interface (MIPI), an external memory interface, and / or a universal serial bus (USB) interface, etc.

[0090] The MIPI interface may be used to connect the processor 204 to peripheral devices such as a touch screen. The MIPI interface may include a display serial interface (DSI), etc. The processor 204 and the touch screen communicate through the DSI interface to implement the display function of the touch screen.

[0091] It can be understood that the interface connection relationships between the modules illustrated in this embodiment are only illustrative descriptions and do not constitute a structural limitation on the electronic device 101. In other embodiments of the present application, the electronic device 101 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0092] The electronic device 101 can implement the display function through a GPU, a touch screen, an application processor, etc. The GPU is a microprocessor for image processing, and is connected to the touch screen and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics rendering. The processor 204 may include one or more GPUs, which execute program instructions to generate or change display information. Specifically, one or more GPUs in the processor 204 can implement the rendering tasks of images (such as the rendering tasks related to drawing layouts, distance values, length values, etc. in this application, and transmit the rendering results to the application processor or other display drivers, and the application processor or other display drivers trigger the display screen to display layouts, distance values, length values, etc.).

[0093] The touch screen may include a display screen and related sensors (such as a pressure sensor and a touch sensor).

[0094] The display screen is used to display images, videos, etc. The display screen includes a display panel. The display panel may adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0095] A pressure sensor is used to sense pressure signals and can convert pressure signals into electrical signals. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can include at least two parallel plates with conductive materials. When a force acts on the pressure sensor, the capacitance between the electrodes changes. The electronic device 101 can determine the intensity of the pressure according to the change in capacitance. When a touch operation acts on the touch screen, the electronic device 101 can detect the intensity of the touch operation according to the pressure sensor. The electronic device 101 can also calculate the position of the touch according to the detection signal of the pressure sensor. In some embodiments, touch operations with the same touch position but different touch operation intensities can correspond to different operation instructions.

[0096] A touch sensor, also known as a "touch control device". The touch sensor can be disposed on the touch screen, and the touch screen is composed of the touch sensor and the display screen. The touch sensor is used to detect touch operations acting on or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen. In some other embodiments, the touch sensor can also be disposed on the surface of the electronic device 101, at a different position from the display screen.

[0097] The NPU is a neural-network (NN) computing processor. By learning from the structure of biological neural networks, such as learning from the transmission mode between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 101 can be realized. For example, the recognition task of touch gesture categories can be realized based on the NPU.

[0098] The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 101. The external memory card communicates with the processor 204 through the external memory interface to achieve the data storage function. For example, files such as music and videos can be saved in the external memory card.

[0099] The memory 214 can be used to store computer-executable program code, and the executable program code includes instructions. The memory 214 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system 206, application software 208 required for at least one function (such as an image playback function, etc.). The data storage area can store data created during the use of the electronic device 101 (such as image data, etc.). In addition, the memory 214 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 204 executes various functional applications and data processing of the electronic device 101 by running the instructions stored in the memory 214, and / or the instructions stored in the memory provided in the processor.

[0100] Among them, the method described in the embodiments of the present application can be code stored in the memory 214 (or an external memory), and the processor 110 can obtain the code in the memory to implement the method provided by the embodiments of the present application.

[0101] The electronic device 101 can also interact with other electronic devices through the communication device 216.

[0102] It should be understood that alternatively or additionally, the methods described herein can be performed at least in part by one or more hardware logic components. By way of example and not limitation, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chips (SOCs), and programmable logic devices (PLDs), etc. For example, functions such as the detection of touch gestures and the determination of gesture categories can be implemented based on hardware logic components.

[0103] In addition, at least a part of the functionality of various elements in the drawings can be distributedly executed by other elements in the drawings or entities not shown in the drawings (such as processors, web services, servers, applications, computing devices, etc.), which is not limited here.

[0104] In complex writing scenarios, intelligent whiteboard devices with handwriting recognition usually have problems such as low recognition rates and inability to recognize. Most of the reasons are that users add a large number of meaningless strokes when writing on the blackboard. When users write on an intelligent electronic whiteboard, they usually repeatedly change and edit the content of the blackboard according to their writing ideas, which may lead to the loss of important information. When multiple users use the electronic whiteboard, they will write and overlap on the same page at the same time, resulting in the inability to accurately recognize the handwriting and the loss of important information. When using the whiteboard in a meeting, for different writing points, more meaningless strokes such as emphasis lines, underlines, and circles will be written, resulting in the lack of prominence of key content. At this time, if the user wants to save a clear and concise blackboard writing, they can only erase and rewrite it, or manually or automatically save the page frequently during writing, which will greatly affect the input efficiency and reduce the user experience.

[0105] For the sake of easy understanding, in combination with the accompanying drawings and application scenarios, the page display method provided by the embodiments of the present application will be specifically described.

[0106] Refer to Figure 3 , Figure 3 which is a flowchart of a page display method provided by an embodiment of the present application. As Figure 3 shown, the page display method provided by the embodiments of the present application includes:

[0107] 301. Receive an input and display a first page, where the first page is the writing content of the input, and the writing content includes multiple writing elements.

[0108] Among them, the input can specifically be a gesture input, or an input through an external device, such as a writing and drawing input through a mouse. For the sake of convenient description, the embodiments of the present application mainly take the gesture input as an example for description.

[0109] In a possible implementation, the user can input handwritten content through gesture input, so as to display a corresponding page on the electronic device. The input handwritten content can include multiple writing elements, where the writing elements can be one or more complete strokes.

[0110] In a possible implementation, the data of the user's gesture input can be obtained.

[0111] In the embodiments of the present application, the gesture input can be an input of a contact gesture, and the electronic device can detect the contact gesture with the touch screen.

[0112] Among them, the touch screen can include a touch-sensitive surface, a sensor or a sensor group that receives input from the user based on tactile contact. "Detecting the contact gesture with the touch screen" can be understood as detecting the contact gesture with the touch-sensitive surface on the touch screen.

[0113] First, the devices and modules related to detecting contact gestures on the touch screen are described.

[0114] In a possible implementation, the touch screen can cooperate with a detection module to detect contact gestures on the touch screen. Among them, the detection module can be a program, data structure, or a subset thereof stored in the memory related to contact gesture detection, or a part exists in the memory in the form of a program, data structure, or a subset thereof, and a part is in the form of a hardware logic module. The touch screen can capture contact data, and the detection module can perform various actions related to contact gesture detection based on the contact data.

[0115] In a possible implementation, the touch screen can capture contact data of the user's contact with the touch-sensitive surface (for example, it can be an electrical signal) in real time, and transmit the contact data to the detection module. The detection module can perform various actions related to contact gesture detection based on the contact data.

[0116] Next, the detection module's implementation of detecting contact gestures on the touch screen is described.

[0117] In one implementation, the detection module can determine the magnitude and / or change of the intensity when the user contacts the touch screen based on the contact data, and the size and / or change of the contact area, and then determine the gesture type of the contact gesture based on the above information.

[0118] Among them, "intensity" can be understood as the force or pressure (force per unit area) of the contact (for example, the side of the hand contacts) on the touch-sensitive surface of the touch screen. The intensity of the contact can be determined using a variety of methods and a variety of sensors or combinations of sensors. For example, below the touch-sensitive surface, one or more sensors adjacent to the touch-sensitive surface can be used to measure the force at different points on the touch-sensitive surface of the touch screen. In some specific implementations, the force measurement values from multiple sensors can be combined (for example, weighted average) to determine the intensity of the contact. In addition, the size and / or change of the contact area detected on the touch-sensitive surface, the capacitance and / or change of the touch-sensitive surface near the contact, and / or the resistance and / or change of the touch-sensitive surface near the contact can also be used to indirectly represent the force or pressure of the contact on the touch-sensitive surface.

[0119] Among them, "contact area" can be understood as the area that can be detected in contact with the touch-sensitive surface, and this area can be a continuous area or a discrete area composed of dense touch points (for example, the contact area can be a long strip area composed of a group of densely distributed contact points).

[0120] The detection module can perform various actions related to contact gesture detection based on the contact data, and then determine the gesture type of the contact gesture, the movement situation of the contact gesture, the stop situation of the contact gesture, and so on.

[0121] For example, the detection module can determine whether contact has occurred and the type of contact gesture (e.g., detecting a finger press event or a side-of-hand contact event), determine whether there is movement of the contact, and track the movement on the touch-sensitive surface (e.g., detecting one or more finger drag events or a side-of-hand drag event), and determine whether the contact has stopped (e.g., detecting a finger lift event or a side-of-hand lift event).

[0122] Wherein, the above-mentioned "movement of the contact gesture" can also be referred to as the movement of the contact area of the contact gesture, and the data related to the movement of the contact area can include, but are not limited to, the rate (magnitude) of the movement of the contact area, the speed (magnitude and direction), and / or the acceleration (change in magnitude and / or direction).

[0123] In a possible implementation, the data of the gesture input can include at least one of the following: the static features of the writing elements, the relationship of the static features between the adjacent writing elements in space, the relationship of the static features between the adjacent writing elements in time, and the action features of the writing elements.

[0124] In an electronic device equipped with an electronic whiteboard, when a user writes on the electronic whiteboard, writing information is usually generated, which involves stroke trajectory information. When the user writes on the electronic whiteboard, a series of stroke trajectories are generated, and these trajectories can be represented by a sequence of coordinate points; in a device equipped with a pressure sensor, there will also be stroke pressure information. At this time, the electronic whiteboard will record the stroke pressure when the user writes, which can be used to judge the thickness and darkness of the user's writing; stroke speed information, the stroke speed when the user writes can also be recorded, which can be used to judge the smoothness and coherence of the user's writing; stroke direction information, the stroke direction when the user writes can also be recorded, which can be used to judge the direction and inclination of the user's writing; stroke color information, the electronic whiteboard usually supports strokes of multiple colors, and the color used by the user when writing can also be recorded; stroke timestamp information, the electronic whiteboard usually records the timestamp when the user writes, which can be used to judge the speed and rhythm of the user's writing. There will also be action information, such as the user performing erasing, selecting, moving a certain part, etc.

[0125] For example, referring to Figure 5A , Figure 5AThe data of the input gesture can include single-stroke features, adjacent stroke features within space K, adjacent stroke features within time K, and action features. The single-stroke features include stroke projection stroke length, curvature, center of gravity, stroke Bbox length and width, and geometric features of the stroke convex hull; the adjacent stroke features within space K include the distance between the starting and ending points of adjacent strokes and the center of gravity of adjacent strokes; the adjacent stroke features within time K include the mean and variance of the number of dot traces and the center of gravity distance between adjacent strokes; the action features include action speed, action type, the number of action trajectory points, and action contact area.

[0126] The embodiments of this application construct features based on the time, space, and semantic information of strokes, and the recognition result is more accurate.

[0127] After recording the data of the gesture input, the data of the gesture input can be cached, and subsequent redundant information recognition and layout generation operations are only performed when the user requests a layout switch, so as to reduce the overhead of computing resources.

[0128] Redundant information recognition can include the type recognition of gesture input, that is, various types of information generated by the user's writing can be cached. If the classification engine resides permanently, it will occupy a large amount of computing resources and memory resources of the device, which may cause the device to freeze and affect the normal operation of other functions. To reduce resource occupancy, this technology does not permanently reside the stroke classification engine in the background, but permanently resides the cache engine, and caches the information of the user's writing and operations in the memory for subsequent classification calculation operations.

[0129] 302. Receive a first layout switch instruction for the first layout, and based on the first layout switch instruction, replace and display the first layout with a second layout. The second layout includes some of the multiple writing elements; the some writing elements correspond to the same user intention, and the writing elements not included in the second layout compared with the first layout and the some writing elements do not correspond to the same user intention, and the input time of the writing elements not included is before the input time of at least one of the some writing elements.

[0130] When key content cannot be recognized or the layout is chaotic due to interference information (such as redundant writing content, writing content input by different users), the embodiments of the present application can identify the redundant information in the layout and remove the redundant information to reproduce the layout containing the key information. Among them, the redundant information can be considered as information that is different from the main semantics of the layout. When the user inputs, there is often one or more input intentions, that is, some of the user's gesture inputs can correspond to the same user intention, and some other gesture inputs can correspond to other user intentions, or the writing content input by different users corresponds to different user intentions. The so-called some gesture inputs can correspond to the same user intention can be understood as: the writing content of non-redundant content input by the user to express the same theme content can correspond to the same user intention. Usually, the same theme content corresponds to the same semantics.

[0131] The redundant content can be understood as: when this content is missing, the semantic integrity of the layout is not affected. For example, referring to Figure 4 , Figure 1 the multiple circles and double underlines in the writing content of the label 2 area in Figure 4 , Figure 1 can be considered as redundant content. When this part of the content is missing, the semantic integrity of the writing content in the 2 area will not be affected. Another example, referring to

[0132] the single circle and single underline in the writing content of the label 3 area in

[0133] can be considered as redundant content. When this part of the content is missing, the semantic integrity of the writing content in the 3 area will not be affected.

[0134] In the embodiments of the present application, the first layout can be the layout obtained after receiving the gesture input of the user. Since there are often multiple themes or some content that visually affects important content (i.e., redundant content) when the user inputs, in this case, the user may have a need to reproduce one or more specified layouts in history. Each layout can correspond to a user intention, so that the user can easily view the clear and main-layout content. Among them, the writing content with the same user intention can be: having the same semantics and no redundant information.

[0135] When the user hopes to reproduce one or more specified layouts in history, a first layout switching instruction for the first layout can be input. The first layout switching instruction can be used to indicate the reconstruction of the layout based on part or all of the writing content in the first layout.

[0136] For example, in a possible implementation, a selection instruction for part of the writing content in the writing content of the first layout and a layout switching instruction for the part of the handwritten content can be received.

[0137] For example, in a possible implementation, a selection instruction for all of the writing content in the writing content of the first layout and a layout switching instruction for all of the writing content can be received.

[0138] Exemplarily, the selection instruction can be a marquee selection or a full selection. For example, the marquee selection button can be clicked to manually or use a writing pen to marquee select part of the content, or double-click the marquee selection button to marquee select the entire page.

[0139] In a possible implementation, some elements in the first layout can also be specified not to be eliminated during the layout reconstruction. For example, a fixation instruction for a target element among the multiple writing elements can be received. The fixation instruction is used to indicate that when the first layout is replaced and displayed as the second layout (the reconstructed layout), the display of the target element is maintained. Exemplarily, the content selected by the marquee can be recognized as the smallest unit. The user can single-click to select a certain unit and can choose to fix or eliminate it. And the same units can be recognized and the same units can be operated on as a whole. After the unit is selected to be fixed or eliminated, the unit is not affected by the sliding of the scroll bar.

[0140] By the above method, an operable space for selection is provided to the user, so that the user can independently select to fix or eliminate important unit information.

[0141] For example, referring to 5B, the user can select the writing content to be operated on (for example, the writing content within the dashed circle), and the user can perform a fixation operation or an elimination operation on the content selected by the marquee through interaction with the control.

[0142] In a possible implementation, when a user requests a layout switch, subsequent redundant information recognition and layout generation operations can be performed.

[0143] In a possible implementation, the detection module can perform various actions related to gesture detection based on the data of the gesture input, so as to determine the gesture type. Among them, the gesture type is, for example but not limited to, click, double-click, long press, drag, static contact of the side of the hand with the touch screen, side drag of the hand, side rotation of the hand, side double-click of the hand, and so on.

[0144] It should be understood that classification can also be not to identify specific categories, but only to distinguish different categories, that is, clustering.

[0145] In a possible implementation, the detection module can be a pre-trained neural network model, and this neural network model has the ability to identify the gesture category indicated by the contact gesture based on the data of the input gesture.

[0146] For example, referring to Figure 5A , a stroke relationship graph can be obtained based on the static features (such as single-stroke features, adjacent stroke features within space K, adjacent stroke features within time K) in the data of the gesture input, an action relationship graph can be obtained based on the action data, and then the feature vector of each stroke can be obtained. Then, rough classification and fine classification are carried out, so as to obtain the classification result.

[0147] Exemplarily, referring to Figure 6 , Figure 6 is a schematic of the classification result. Among them, different shades of color correspond to different categories. This information can be used as a reference factor for subsequent layout determination. Referring to Figure 7 , Figure 7 is a schematic process of stroke classification.

[0148] Regarding Figure 5A the rough classification module based on rule matching in : To ensure that the data obtained by the stroke classifier is clean enough, a rough classification module based on rule matching is constructed here. The purpose is to remove non-artificial information or part of the noise in the data, such as meaningless points, aimless strokes, or strokes with a writing speed less than a certain threshold, etc., and remove such information from the data.

[0149] Regarding Figure 5A the stroke classifier based on the deep neural network in . This module trains the neural network based on attention to train the stroke classifier neural network. Through this module, the feature data sent in step e is classified, and the strokes are classified into text, graphics, flowcharts, tables or other category data. The calculation method of self-attention involved is as follows:

[0150]

[0151] Among them, Q, K, and V respectively represent Query, Key, and Value, and d k is the feature dimension. Q, K, and V are respectively:

[0152] Q = W Q X;

[0153] K = W K X;

[0154] V = W V X;

[0155] It can be seen that the query, index, and content of self-attention are all itself. The core goal of self-attention is to select the information that is more critical to the current task goal from numerous information and focus the attention on it.

[0156] In addition, the position information of writing elements can be determined based on the input gesture data, and this information can be used as a reference factor for subsequent layout determination.

[0157] In a possible implementation, the above several types of data can be converted into structured data, and the structured data can be used as a reference factor for subsequent layout determination. Specifically, referring to Figure 8 , the structured data can be input into the language model so that the language model can generate the layout (or it can be called layout reconstruction).

[0158] In a possible implementation, the intention of the input gesture can also be understood based on the input gesture data. For example, the data of the input gesture can be processed by a neural network to obtain the intention classification of each input gesture, that is, to identify which input gestures correspond to the same user category, and this information can be used as a reference factor for subsequent layout determination.

[0159] In the recognition scenario, after the content input by the user passes through the classification engine, different types of data will be generated, such as text data, graphic data, and table data, but the classification engine will not specifically identify the content of these data.

[0160] After obtaining these data, they are sent to various models according to the existing methods. For text data, various text recognition technologies are used, including but not limited to text recognition, word segmentation, part-of-speech tagging, named entity recognition and other processing technologies, and the results are fed back to the user. For graphic data, the graphic can be segmented, recognized and classified to identify each node and connection line therein, so as to restore the complete printed flowchart.

[0161] In fact, the present invention is also designed based on this process. This module includes, but is not limited to, a text recognition engine, a multilingual recognition engine, a graphic recognition engine, a table recognition engine, etc. Depending on the number of classification engines, there may be multiple engines in this module. Therefore, there is an additional multi-engine scheduling module here, which schedules the corresponding engines for multiple recognition tasks respectively, as well as model collaboration. The specific solution can be referred to Figure 8 for illustration.

[0162] For example, referring to Figure 9A , Figure 9A the intention of the input gesture can be recognized (such as the intentions shown in the figure Figure 1 and Figure 2 ), and then the structured data and the intention category of the input gesture are input into the language model so that the language model can generate the layout (or it can be called layout reconstruction).

[0163] Next, the reconstructed layout information in the embodiments of the present application will be introduced.

[0164] Taking the layout reconstruction of the first layout to obtain the second layout as an example, the second layout may include some of the writing contents in the first layout, and some contents correspond to the same user intention, that is, the writing contents with the same semantics and no redundant information. In this way, on the one hand, the writing content can be structurally reconstructed during the writing process, greatly improving the user's input efficiency and experience. On the other hand, since the reconstructed layout can correspond to a user intention, the user can easily view the layout content clearly and with a clear main body.

[0165] In a possible implementation, some writing elements in the first layout can be gradually disappeared until the second layout is displayed.

[0166] In a possible implementation, some writing elements in the first layout can be disappeared in sequence according to the relevance between the writing elements and the user intention. For example, redundant information usually has a relatively small relevance to the user intention (the recognition of relevance can be realized based on a neural network, and the specific amount can be learned by the neural network).

[0167] In a possible implementation, the writing elements in the first layout can be gradually eliminated (for example, the stroke elements with smaller relevance are eliminated first based on the relevance between the writing elements and the user intention), and the user can choose how many strokes to eliminate. For example, when eliminating to the second layout, the user thinks that the second layout is the desired layout, so the second layout is displayed.

[0168] The above process of gradually eliminating the writing strokes can be realized by the user's sliding operation. That is to say, in a possible implementation, the first layout switching instruction is generated for the sliding operation of the slider.

[0169] For example, the sliding of the slider may have two-directional operations, and different directions correspond to different input time directions. Moving to the left may gradually eliminate, and moving to the right may gradually restore.

[0170] For example, when the user selects a layout that needs to be reconstructed, the slider bar can be activated. If it is part of the selected content, the slider bar will be activated at the bottom of the selected content. If it is all selected, the slider bar will be activated at the bottom of the screen. The user-based slider bar operation can be exemplified as follows: Figure 9B As shown, the user can slide the slider to choose how many redundant strokes to eliminate to prevent the loss of important information in the intermediate process.

[0171] In addition to operating the slider, multiple candidate layouts may be presented, each corresponding to the same user intent, with different candidate layouts being different, and the user selecting the layout to be displayed from the multiple candidate layouts. That is, in one possible implementation, the first layout switching instruction is generated for an operation on multiple candidate layouts, each candidate layout including some of the multiple writing elements, and each candidate layout corresponding to the same user intent, with different candidate layouts being different.

[0172] For example, in a possible implementation, the system cache module will always cache the original handwriting and the operation actions during the handwriting process, and the stroke classification engine will sleep at this time; then, when the user circles a part of the whiteboard, the stroke classification engine will start, and the stored strokes and operation actions will be pre-processed to extract data features and send them to the stroke classification engine. The stroke classifier will recognize text, graphics, tables, redundant strokes, etc., and then send them to the corresponding recognition engine; secondly, when the data is sent to the recognition engine, the engine scheduling system will schedule a recognition engine to perform targeted recognition on the data. The corresponding recognition engine will send the recognition results to the final reconstruction module; finally, the reconstruction module will perform layout analysis and element layout on the recognized results, and give multiple versions of results. The reconstructed layout will be ranked by importance according to the results given by the recognition engine, and then sent to the display interface. The user can slide the slider to choose to eliminate or retain the layout results.

[0173] In one possible implementation, when reconstructing the layout, the display content of multiple candidate layouts including the second layout can be determined based on the input gesture data through a language model. Each candidate layout corresponds to the same user intention, and different candidate layouts are different.

[0174] In a possible implementation, the recognition results based on the stroke classifier can be sent to the recognition engine. In the scenario where the user finishes writing and circles the writing module, the content of the circled part has undergone feature extraction, denoising or removal of meaningless strokes, and each feature data has been labeled with various labels by the mural classifier. At this time, the feature data will be sent to different recognition engines according to the labels. If the number of labels is large and various feature data are not completely independent, engine scheduling and collaborative recognition are required. In a possible implementation, the recognition engine can perform scheduling and collaboration. This module is designed with multi-threaded scheduling to solve the engine scheduling problem. In actual use, it can be specifically designed as first-come-first-served, shortest job first, priority scheduling, time slice rotation or other scheduling algorithms.

[0175] In a possible implementation, data can be recognized by various engines. Since the data of handwritten strokes usually carry timing information, and the device often retains timestamp information when the user is writing, therefore, optionally, the recognition engine in the embodiments of the present application can be based on but not limited to using LSTM as the recognition engine. For example, for a handwritten English recognition engine, at this time, the multi-threaded scheduling invokes the English recognition engine, and the feature data is sent from the classification engine into the LSTM network for sequence modeling to obtain the output of each sequence step, and finally classification is performed to obtain the final recognition result.

[0176] When the user pauses writing and circles, the circle selection option will have options to circle a part or the entire layout. If the option to circle the entire layout is selected, the stroke and layout reconstruction engine will be started to analyze and reconstruct the layout. The main function of this module is to obtain different overall information generated during the writing process and display it in a structured layout. For example, when the user is writing along with their thoughts, they often edit repeatedly, which may cause important information in the intermediate process to be lost. At this time, according to the user's intention understanding, this module retains K layouts through the language model, reconstructs them, and allows the user to slide and select. The specific solution can be as Figure 9A shown:

[0177] In addition, the recognition results based on the recognition engine can also be sent to the stroke reconstruction module. After the recognition engine recognizes various results and structures them, this data is sent to the stroke reconstruction module. The overall stroke reconstruction technology based on semantics and intention can be as Figure 10 shown: The structured data includes element index, data type, data content, bounding box, etc. The structured data is sent into the language model, and the language model gives the confidence score of the text, and finally it is sent into the reconstruction module. In the early feature extraction module, a part of the feature extraction data is sent into the intention understanding module. After being judged by the intention understanding module, the strokes are divided into different parts according to different intentions and labeled, and then this data is sent into the reconstruction module.

[0178] At this time, the layout reconstruction module has received the final content result processed by the language model and the stroke intention tags corresponding to the content one by one. The reconstruction module will reconstruct the stroke layout according to this information and send it to the display layer in different layouts.

[0179] The functional performance level is divided into two parts: partial selection recognition and full selection recognition. For partial selection recognition, finally slide the scroll bar displayed by the system, and the redundant strokes change; for full selection recognition, slide the scroll bar, and the layout and redundant strokes change. Among them, the content selected can also be selected, eliminated or fixed, and is not affected by the scroll bar.

[0180] Next, an application example of the present application will be introduced in conjunction with the accompanying drawings:

[0181] Figure 11 It is a schematic diagram of a process for layout reconstruction. Among them, as Figure 11 shown, the user can input on the electronic whiteboard, and the generated information includes stroke information, action information and timestamp information. The action information includes, but is not limited to, information such as erasure, gesture erasure, undo, delete, point selection, selection, rewriting, etc.

[0182] To reduce resource occupation, the cache module stores all the information generated by the user's writing in the memory for subsequent calculation operations.

[0183] When the user makes a selection on a certain part, the feature extraction module is first invoked to extract the features of the data of this module cached in the memory according to the feature design and send it to the stroke classification engine and the stroke reconstruction module.

[0184] The stroke classifier classifies the features into different types of data, such as text, graphics, charts, process lines, etc., and attaches category labels to the feature data and sends them to the next module together.

[0185] At this time, the stroke recognition engine mobilizes various recognition engine modules in a multi-threaded manner to recognize the data according to the data passed in from the upper layer, that is, to recognize the specific content of various types of data, such as what characters are in the text data and what graphics are in the graphic data.

[0186] Integrate the data results recognized by the recognition engine. Each data includes data index, data type, data content, and bounding box information. Then the data is sent to the language model module in the stroke reconstruction module.

[0187] The language model will score the confidence of the recognition result, finally obtain the most reasonable result and send it to the layout reconstruction module for final integration.

[0188] In addition, the text recognition engine can be classified according to strokes to recognize and classify the written content into units of the smallest understandable granularity for the user to select. When the user clicks on such a unit, fixed, elimination, and multi-selection buttons will appear. The multi-selection function is for selecting similar units. After selecting fixed or elimination, the unit will not participate in layout reconstruction and analysis.

[0189] For example, analysis can be performed only on the selected module, that is, there is no intent analysis and multi-layout results for this module. This module refers to removing redundant strokes from the current single layout and providing a scroll bar at the display level. Sliding the scroll bar allows selection of how many redundant strokes to eliminate.

[0190] Refer to Figure 12 , Figure 12 which is an application diagram of the embodiment of this application in the handwritten recognition scenario. Among them, the layout reconstruction module can correspond the final result and the intent label one by one, and place the recognition results of the same intent in the same layout according to the final layout format, and provide multi-layout display.

[0191] Refer to Figure 13 , Figure 13 which is an application schematic diagram of the embodiment of this application in reconstructing the lost information content of multiple people's overwriting on the same layout.

[0192] As Figure 13 shown in Figure a in, User 1 writes his own solution according to his idea. User 2 then continues to modify and add. In Figure 13 shown in Figure b in, User 2 may rewrite or edit on the layout of User 1, resulting in partial loss of the information written by User 1.

[0193] Refer to Figure 14 , Figure 14 which is a structural schematic diagram of a layout display device provided by the embodiment of this application, applied to an electronic device. The device 1400 includes:

[0194] A processing module 1401 and a display module 1402;

[0195] The processing module 1401 is configured to receive an input, control the display module 1402 to display a first layout, where the first layout is the written content of the input, and the written content includes a plurality of writing elements; receive a first layout switching instruction for the first layout, and based on the first layout switching instruction, control the display module to replace the display of the first layout with a second layout, where the second layout includes some of the plurality of writing elements; the some writing elements correspond to the same user intention, and the writing elements not included in the second layout compared to the first layout and the some writing elements do not correspond to the same user intention, and the input time of the writing elements not included is before the input time of at least one of the some writing elements.

[0196] For the specific description of the processing module 1401, reference can be made to the descriptions in steps 301 and 302, which will not be elaborated here.

[0197] In a possible implementation, the writing elements included in the second layout are determined based on whether the corresponding user intentions are the same.

[0198] In a possible implementation, the writing element is one or more complete strokes.

[0199] Among them, the input can specifically be gesture input.

[0200] In a possible implementation, the same user intention includes:

[0201] Writing elements input by the same user; or,

[0202] When missing, it will affect the semantic integrity of the written content included in the second layout.

[0203] In a possible implementation, the processing module 1401 is specifically configured to:

[0204] Receive a selection instruction for some of the written content in the first layout, and a layout switching instruction for the some handwritten content; the some writing elements belong to the some written content; or,

[0205] Receive a selection instruction for all of the written content in the first layout, and a layout switching instruction for all of the written content.

[0206] In a possible implementation, before replacing the display of the first layout with the second layout, the processing module 1401 is further configured to:

[0207] Receive a fixation instruction for a target element among the multiple writing elements, where the fixation instruction is used to indicate that when replacing the display of the first layout with the second layout, the display of the target element is maintained.

[0208] In a possible implementation, the first layout switching instruction is generated for a sliding operation of a slider.

[0209] In a possible implementation, the processing module 1401 is specifically configured to control the display module 1402 to sequentially disappear some writing elements in the first layout until the second layout is displayed.

[0210] In a possible implementation, the sequentially disappearing of some writing elements in the first layout includes:

[0211] Sequentially disappearing some writing elements in the first layout according to the magnitude of the relevance between the writing elements and the user intention.

[0212] In a possible implementation, the first layout switching instruction is generated for an operation on multiple candidate layouts, each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0213] In a possible implementation, the processing module 1401 is further configured to:

[0214] Cache the input data corresponding to the first layout;

[0215] Determine the display content of the second layout according to the data.

[0216] In a possible implementation, the determining of the display content of the second layout according to the data is performed after receiving the first layout switching instruction for the first layout.

[0217] In a possible implementation, the processing module 1401 is specifically configured to:

[0218] Determine the display content of multiple candidate layouts including the second layout according to the data through a language model, each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

[0219] In a possible implementation, the data includes at least one of the following:

[0220] The static features of the writing elements, the relationship between the static features of adjacent writing elements in space, the relationship between the static features of adjacent writing elements in time, the action features of the writing elements.

[0221] Next, a terminal device provided by an embodiment of the present application will be introduced. The terminal device may be the device described in Figure 14 . Please refer to Figure 15 . Figure 15 FIG. is a schematic structural diagram of a terminal device provided by an embodiment of the present application. The terminal device 1500 may specifically be an electronic whiteboard, a virtual reality (VR) device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited herein. Specifically, the terminal device 1500 includes: a receiver 1501, a transmitter 1502, a processor 1503, and a memory 1504 (where the number of processors 1503 in the terminal device 1500 may be one or more, Figure 15 taking one processor as an example in ). Among them, the processor 1503 may include an application processor 15031 and a communication processor 15032. In some embodiments of the present application, the receiver 1501, the transmitter 1502, the processor 1503, and the memory 1504 may be connected through a bus or other means.

[0222] The memory 1504 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1503. A part of the memory 1504 may further include a non-volatile random access memory (NVRAM). The memory 1504 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof. Among them, the operation instructions may include various operation instructions for implementing various operations.

[0223] The processor 1503 controls the operation of the terminal device. In a specific application, each component of the terminal device is coupled together through a bus system. Among them, the bus system may include a power bus, a control bus, a status signal bus, etc. in addition to a data bus. However, for the sake of clarity, all kinds of buses are referred to as a bus system in the figure.

[0224] The method disclosed in the embodiments of the present application can be applied to or implemented by the processor 1503. The processor 1503 can be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor 1503 or instructions in software form. The above-mentioned processor 1503 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 1503 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 1504, and the processor 1503 reads the information in the memory 1504 and combines its hardware to complete the steps of the above method. Specifically, the processor 1503 can read the information in the memory 1504 and combine its hardware to complete the relevant steps in steps 301 to 302 in the above embodiments.

[0225] The receiver 1501 can be used to receive input digital or character information, and generate signal inputs related to the relevant settings and function controls of the terminal device. The transmitter 1502 can be used to output digital or character information through the first interface; the transmitter 1502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1502 can also include a display device such as a touch screen.

[0226] An embodiment of the present application also provides a computer program product. When it runs on a computer, it causes the computer to execute the Figure 3 steps of the method described in the corresponding embodiment in the above embodiment.

[0227] An embodiment of the present application also provides a computer-readable storage medium. A program for signal processing is stored in the computer-readable storage medium. When it runs on a computer, it causes the computer to execute the steps of the image processing method in the method described in the foregoing embodiment.

[0228] The image display device provided by the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit, etc. The processing unit may execute the computer-executable instructions stored in the storage unit to cause the chip in the execution device to execute the data processing method described in the above embodiments, or to cause the chip in the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the radio access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0229] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in the present application, the connection relationships between the modules indicate that there is a communication connection between them, and specifically, it may be implemented as one or more communication buses or signal lines.

[0230] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that makes a contribution to the prior art, can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, such as a floppy disk, a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disc of a computer, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0231] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0232] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be stored by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A layout display method, characterized in that, The method includes: Receiving an input and displaying a first layout, where the first layout is the written content of the input, and the written content includes multiple writing elements; Receiving a first layout switching instruction for the first layout, and based on the first layout switching instruction, replacing and displaying the first layout with a second layout, where the second layout includes some of the multiple writing elements; the some writing elements correspond to the same user intention, and the writing elements not included in the second layout compared to the first layout and the some writing elements do not correspond to the same user intention, and the input time of the writing elements not included is before the input time of at least one of the some writing elements.

2. The method according to claim 1, wherein The writing elements included in the second layout are determined based on whether the corresponding user intentions are the same.

3. The method according to claim 1 or 2, characterized in that, The input is specifically a gesture input.

4. The method according to any one of claims 1 to 3, characterized in that The same user intention includes: Writing elements input by the same user; or, When missing, it will affect the semantic integrity of the written content included in the second layout.

5. The method according to any one of claims 1 to 4, characterized in that, The receiving of the first layout switching instruction for the first layout includes: Receiving a selection instruction for some of the written content in the first layout and a layout switching instruction for the some handwritten content; the some writing elements belong to the some written content; or, Receiving a selection instruction for all of the written content in the first layout and a layout switching instruction for all of the written content.

6. The method according to any one of claims 1 to 5, characterized in that, Before replacing and displaying the first layout with the second layout, the method further includes: Receiving a fixation instruction for a target element among the multiple writing elements, where the fixation instruction is used to indicate that when replacing and displaying the first layout with the second layout, the display of the target element is maintained.

7. According to the method described in any one of claims 1 to 6, characterized in that, The first layout switching instruction is generated by a sliding operation on a slider.

8. The method according to any one of claims 1 to 7, characterized in that, The replacing and displaying the first layout with the second layout includes: Sequentially making some writing elements in the first layout disappear until the second layout is displayed.

9. The method according to claim 8, characterized in that, The sequentially making some writing elements in the first layout disappear includes: Sequentially making some writing elements in the first layout disappear according to the magnitude of the relevance between the writing elements and the user intention.

10. The method according to any one of claims 1 to 9, characterized in that, The first layout switching instruction is generated by an operation on multiple candidate layouts, each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Caching the data of the input corresponding to the first layout; Determining the display content of the second layout according to the data.

12. The method according to claim 11, wherein The determining of the display content of the second layout according to the data is performed after receiving the first layout switching instruction for the first layout.

13. The method according to claim 11 or 12, characterized in that, The determining of the display content of the second layout according to the data includes: Based on the data, through a language model, determine the display content of multiple candidate layouts including the second layout, where each candidate layout corresponds to the same user intention, and different candidate layouts are different.

14. The method according to any one of claims 11 to 13, characterized in that The data includes at least one of the following: The static features of the writing elements, the relationship between the static features of adjacent writing elements in space, the relationship between the static features of adjacent writing elements in time, and the motion features of the writing elements.

15. The method according to any one of claims 1 to 14, characterized in that, The writing element is one or more complete strokes.

16. A layout display device, characterized in that, The device includes: a processing module and a display module; The processing module is configured to receive an input, control the display module to display a first layout, where the first layout is the input writing content, and the writing content includes multiple writing elements; receive a first layout switching instruction for the first layout, and based on the first layout switching instruction, control the display module to replace the display of the first layout with a second layout, where the second layout includes some of the multiple writing elements; the some writing elements correspond to the same user intention, and the writing elements not included in the second layout compared to the first layout and the some writing elements do not correspond to the same user intention, and the input time of the writing elements not included is before the input time of at least one of the some writing elements.

17. The device according to claim 16, characterized in that, The writing elements included in the second layout are determined based on whether the corresponding user intentions are the same.

18. The device according to claim 16 or 17, characterized in that, The input is specifically a gesture input.

19. The device according to any one of claims 16 to 18, characterized in that, The same user intention includes: Writing elements input by the same user's gesture; or, Those that will affect the semantic integrity of the writing content included in the second layout when missing.

20. The device according to any one of claims 16 to 19, characterized in that, The processing module is specifically configured to: Receive a selection instruction for some of the writing content in the first layout and a layout switching instruction for the some handwritten content; the some writing elements belong to the some writing content; Or, Receive a selection instruction for all of the writing content in the first layout and a layout switching instruction for all of the writing content.

21. The device according to any one of claims 16 to 20, characterized in that, Before replacing the display of the first layout with the second layout, the processing module is further configured to: Receive a fixation instruction for a target element among the multiple writing elements, where the fixation instruction is used to indicate that when replacing the display of the first layout with the second layout, keep the display of the target element.

22. The device according to any one of claims 16 to 21, characterized in that, The first layout switching instruction is generated for a sliding operation of a slider.

23. The device according to any one of claims 16 to 22, characterized in that, The processing module is specifically configured to control the display module to gradually disappear some of the writing elements in the first layout until the second layout is displayed.

24. The device according to claim 23, characterized in that, The gradually disappearing of some of the writing elements in the first layout includes: Gradually disappearing some of the writing elements in the first layout in order of the magnitude of the correlation between the writing elements and the user intention.

25. The device according to any one of claims 16 to 24, characterized in that, The first layout switching instruction is generated for operations on multiple candidate layouts. Each of the candidate layouts includes some of the multiple writing elements, and each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

26. The device according to any one of claims 16 to 25, characterized in that, The processing module is further configured to: Cache the data of the gesture input corresponding to the first layout. Determine the display content of the second layout according to the data.

27. The device according to claim 26, characterized in that, The determining of the display content of the second layout according to the data is performed after receiving the first layout switching instruction for the first layout.

28. The device according to claim 26 or 27, characterized in that, Specifically, the processing module is configured to: Determine, according to the data and through a language model, the display content of multiple candidate layouts including the second layout. Each of the candidate layouts corresponds to the same user intention, and different candidate layouts are different.

29. The device according to any one of claims 26 to 28, characterized in that The data includes at least one of the following: The static features of the writing elements, the relationships between the static features of the writing elements adjacent in space, the relationships between the static features of the writing elements adjacent in time, and the motion features of the writing elements.

30. The device according to any one of claims 16 to 29, characterized in that The writing element is one or more complete strokes.

31. An electronic device, characterized in that, The device includes a processor, a memory, a display screen, and a bus, where: The processor, the memory, and the display screen are connected through the bus; The memory is used to store computer programs or instructions; The processor is used to call or execute the programs or instructions stored on the memory, and is further used to control the touch screen to implement the method steps described in any one of claims 1-15.

32. A computer-readable storage medium includes a program, which when running on a computer, causes the computer to execute the method described in any one of claims 1 to 15.

33. A computer program product comprising instructions, characterized in that, When the computer program product runs on the terminal, the terminal is caused to execute the method described in any one of claims 1-15.

Images