Methods, apparatus and devices for generating online help statements
By analyzing and identifying the semantic text and sequence of interface elements in the screen recording, online help statements are automatically generated, solving the problem of low development efficiency of online help documentation and enabling the simultaneous development of screen recording guidance and GUI online help documentation.
Patent Information
- Application Number
- CN202310024941.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-09
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-01-09
AI Technical Summary
In existing technologies, the development efficiency of online help documentation is low. Screen recording tutorials and GUI online help documentation require dual human input and have inconsistent content.
By analyzing screen recordings, the semantic text and order of appearance of interface elements are identified, and online help statements are automatically generated. Combined with deep learning image recognition methods, screen recording instructions and GUI online help documentation can be developed together.
It improves the development efficiency of online help documentation and ensures consistency between screen recording instructions and GUI online help documentation.
Smart Images

Figure CN116009864B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of communication technology, and in particular to a method, apparatus and device for generating online help statements. Background Technology
[0002] To facilitate user operation of software or system applications, products typically need to provide GUI (Graphical User Interface) online help to guide users through GUI operations.
[0003] Currently, online help documentation needs to be written by developers, but the development efficiency of online help documentation is relatively low.
[0004] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention
[0005] This application provides a method, apparatus, and device for generating online help statements. By analyzing screen recordings, the semantic text of interface elements (such as page actions, user operation actions, and user-operated page controls) and the order in which these elements appear are identified. Based on the order of appearance, the semantic text is combined to automatically generate online help statements, thereby improving the development efficiency of online help documents.
[0006] In a first aspect, embodiments of this application provide a method for generating online help statements, including:
[0007] The system acquires multiple screen recording frames, which are obtained by recording the user's interactions with the graphical user interface. It analyzes these frames to determine the semantic text of each interface element and the order in which these elements appear. The semantic text represents the name of the corresponding interface element. The interface elements include user actions, page controls manipulated by the user, and page actions of the graphical user interface. Based on the order in which these interface elements appear, the semantic text is semantically combined to obtain online help statements.
[0008] This solution can capture screen recordings during the recording process, analyze the screen recordings to identify the semantic text of interface elements (such as page actions, user actions, and user-operated page controls), as well as the order in which these elements appear. Based on this order, the semantic text is combined to automatically generate online help statements.
[0009] In one possible implementation, analyzing the multi-frame screen recording to determine the semantic text of each of the multiple interface elements includes:
[0010] Extract the element images corresponding to each of the multiple interface elements from the multi-frame screen recording; perform semantic recognition on the element images corresponding to each of the multiple interface elements to obtain the semantic text corresponding to each of the multiple interface elements.
[0011] In this solution, by identifying the element frames representing interface elements in multiple screen recording frames and then performing semantic recognition on these element frames, the accuracy of the obtained semantic text can be ensured.
[0012] In one example, extracting the element images corresponding to each of the multiple interface elements from the multi-frame screen recording includes:
[0013] The multi-frame screen recordings are input into the interface element detection model; based on the output of the interface element detection model, the element images corresponding to each of the multiple interface elements are obtained.
[0014] In this embodiment of the application, the element screen representing the interface element is identified by the interface element detection model.
[0015] Optionally, analyzing the multi-frame screen recording to determine the order of appearance of the multiple interface elements includes:
[0016] Based on the output of the interface element detection model, the order in which the element images corresponding to each of the multiple interface elements appear is obtained.
[0017] In this solution, the interface element detection model simultaneously identifies the element images and their order of appearance of multiple interface elements.
[0018] In one example, the element screen has a first category, which indicates a user action, a user-operated page control, or a page action.
[0019] Optionally, the step of performing semantic recognition on the element screens corresponding to each of the plurality of interface elements to obtain the semantic text corresponding to each of the plurality of interface elements includes:
[0020] For each of the multiple interface elements, the element screen is input into a semantic analysis model under the first category of the element screen. Based on the output of the semantic analysis model, the semantic text of the element screen is obtained.
[0021] In this solution, the accuracy of semantic recognition results for element images can be ensured by using a semantic analysis model corresponding to the type of element image.
[0022] In one possible implementation, acquiring multiple screen recording frames includes:
[0023] Recording is performed when and / or after the first change occurs in the graphical user interface to obtain multiple screen recording frames; wherein, the first change indicates a change in mouse position or a change in page content.
[0024] In this solution, recording occurs when or after changes occur in the graphical user interface, which reduces the number of screen recordings while ensuring the reference value of the screen recordings.
[0025] In one possible implementation, the page action includes at least one of the following: page navigation, opening a page, closing a page, and changing page content.
[0026] In one possible implementation, the page controls include at least one of the following: page labels, buttons, and text boxes.
[0027] In one possible implementation, the user operation includes at least one of the following: mouse click, mouse right-click, mouse long press and slide, and keyboard input.
[0028] Secondly, embodiments of the present invention provide an online help statement generation device. This device includes several modules, each module executing a step in the display method provided in the first aspect of the present invention. The division of modules is not limited here. For the specific functions performed by each module of this online help statement generation device and the beneficial effects achieved, please refer to the functions of each step in the display method provided in the first aspect of the present invention; further details will not be repeated here.
[0029] For example, the online help statement generation device includes:
[0030] The screen recording module is used to acquire multiple screen recording frames; wherein, the multiple screen recording frames are obtained by recording the process of the user operating the graphical user interface, and the graphical user interface includes at least one page control.
[0031] The analysis module is used to analyze the multi-frame screen recording to determine the semantic text of each of the multiple interface elements and the order in which the multiple interface elements appear; wherein, the semantic text is used to represent the name of the corresponding interface element; the multiple interface elements include user operation actions during the operation, user-operated page controls, and page actions of the graphical user interface.
[0032] The combination module is used to semantically combine the multiple semantic texts based on the order in which the multiple interface elements appear, to obtain online help statements.
[0033] In one possible implementation, the analysis module is used to extract the element images corresponding to each of the multiple interface elements from the multi-frame screen recording; and to perform semantic recognition on the element images corresponding to each of the multiple interface elements to obtain the semantic text corresponding to each of the multiple interface elements.
[0034] In one example, the analysis module includes:
[0035] The detection unit is used to input the multi-frame screen recording into the interface element detection model; based on the output of the interface element detection model, the element images corresponding to each of the multiple interface elements are obtained.
[0036] Optionally, the detection unit is further configured to obtain the appearance order of the element screens corresponding to each of the plurality of interface elements based on the output of the interface element detection model.
[0037] In one example, the element screen has a first category, which indicates a user action, a user-operated page control, or a page action.
[0038] Optionally, the analysis module includes:
[0039] The semantic analysis unit is used to input the element screen corresponding to each of the plurality of interface elements into a semantic analysis model under a first category of the element screen, and obtain the semantic text of the element screen based on the output of the semantic analysis model.
[0040] In one possible implementation, a screen recording module is used to record multiple screen recording frames when and / or after a first change occurs in the graphical user interface; wherein the first change indicates a change in mouse position or a change in page content.
[0041] In one possible implementation, the page action includes at least one of the following: page navigation, opening a page, closing a page, and changing page content.
[0042] In one possible implementation, the page controls include at least one of the following: page labels, buttons, and text boxes.
[0043] In one possible implementation, the user operation includes at least one of the following: mouse click, mouse right-click, mouse long press and slide, and keyboard input.
[0044] Thirdly, embodiments of this application provide an online help statement generation apparatus, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is used to execute the method provided in the first aspect.
[0045] Fourthly, embodiments of this application provide an online help statement generation apparatus, characterized in that the apparatus executes computer program instructions to perform the method provided in the first aspect. Exemplarily, the apparatus may be a chip or a processor.
[0046] In one example, the device may include a processor that can be coupled to memory, read instructions from the memory, and execute the methods provided in the first aspect according to those instructions. The memory may be integrated into the chip or processor, or it may be independent of the chip or processor.
[0047] Fifthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect.
[0048] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect. Attached Figure Description
[0049] Figure 1 This is a structural architecture diagram of an electronic device provided in an embodiment of this application;
[0050] Figure 2 This is a flowchart illustrating an online help statement generation method provided in an embodiment of this application;
[0051] Figure 3 This is a schematic diagram of a graphical user interface provided in an embodiment of this application;
[0052] Figure 4a Yes Figure 3 Flowchart of the operation of the graphical user interface Figure 1 ;
[0053] Figure 4b Yes Figure 4a A flowchart illustrating the operation of the graphical user interface;
[0054] Figure 5 Yes Figure 3 Flowchart of the operation of the graphical user interface Figure 2 ;
[0055] Figure 6a Yes Figure 3 Flowchart of the operation of the graphical user interface Figure 3 ;
[0056] Figure 6b Yes Figure 6aA flowchart illustrating the operation of the graphical user interface;
[0057] Figure 7 yes Figure 2 A flowchart illustrating step 202 in the middle section;
[0058] Figure 8 This is a flowchart illustrating another method for generating online help statements provided in an embodiment of this application;
[0059] Figure 9 This is a schematic diagram of the structure of an online help statement generation device provided in an embodiment of this application. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.
[0061] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.
[0062] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.
[0063] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.
[0064] The following explanations cover some of the terms used in this embodiment. It should be noted that these explanations are for the convenience of those skilled in the art and are not intended to limit the scope of protection claimed by this invention.
[0065] GUI: Graphical User Interface, refers to a computer graphical user interface that uses graphics to display information, allowing users to operate the computer system.
[0066] Screen recording: The act of recording desktop operations using screen recording software.
[0067] Online help: Online help refers to help documents that can be viewed directly on a computer. They are usually help files stored in a specific file format, such as .HLP and .CHM files.
[0068] Page controls: These are visual graphic "elements" that can be placed on a window, such as buttons and input boxes. Most of them have the function of performing actions or triggering code execution and completing responses through "events".
[0069] Deep learning models provide a function that outputs a specific result for a given input. Deep learning models enable deep learning. By learning the inherent patterns and hierarchical representations of sample data, deep learning acquires information such as text, images, and sounds, which can explain data patterns. The ultimate goal of deep learning is to enable machines to possess analytical and learning capabilities like humans, and to recognize data such as text, images, and sounds.
[0070] To facilitate user experience with software or system applications, products typically need to provide online GUI help to guide users through GUI operations. With the increasing prevalence of video tutorials, users often request video instructions alongside online GUI help. Among different types of video tutorials, screen recordings account for a large proportion due to their low cost and short production cycle.
[0071] Current screen recording software can only record the screen; it cannot recognize the screen content or generate documentation. Current online GUI help editing platforms only support text-based document editing. This results in screen recording tutorials and online GUI help documentation being developed in parallel, requiring two separate teams of personnel. Furthermore, inconsistencies in understanding of the product between the GUI help or screen recording developers can lead to discrepancies between the tutorial documentation and video tutorials for the same operation.
[0072] Currently, in the field of visual image recognition, there are applications for image-to-text conversion. The industry can use deep learning-based AI recognizers to identify text or object names in images or video frames and convert them into text.
[0073] However, the above solution only achieves direct image-to-text recognition and simple image-to-text mapping conversion. It cannot guide the generation of online help statements in the field of online help, nor can it enable the joint development of screen recording guidance and GUI online help documentation, thus requiring a separate investment of manpower.
[0074] Based on this, embodiments of the present invention propose a method for generating online help statements.
[0075] This invention employs a deep learning-based image recognition method to identify the semantic text and appearance order of multiple interface elements during GUI screen recording. Based on this order, the semantic text is then combined into online help statements for the GUI. This solution automatically generates online help statements during screen recording by converting the identified interface elements into text and combining them into online help statements. This allows for the simultaneous development of screen recording instructions and GUI online help documentation, improving the efficiency of online help text development. This is merely a brief description of the method; detailed information is provided below.
[0076] First, the electronic device to which the online help statement generation method provided in the embodiments of this application is applied will be described.
[0077] The electronic device in this embodiment can be at least capable of displaying a page, such as a mobile phone, tablet, or computer.
[0078] Figure 1 This is a schematic diagram illustrating the structure of an embodiment of the electronic device provided by the present invention. (See diagram below.) Figure 5 As shown, the electronic device 100 in this embodiment includes: a processor 111, a memory 112, a non-volatile memory 1121, a random access memory 1122, an antenna, a communication module 113, an audio module 114, a microphone 1141, a speaker 1142, a display screen 115, an input device 116, a mouse 1161, and a keyboard 1162.
[0079] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of the present invention, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0080] The following is combined Figure 1 A detailed description of some of the components of the electronic device 100 is provided.
[0081] Processor 111 may include one or more processors, such as an application processor (AP), modem, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processors may be independent devices or integrated into one or more processors. For example, processor 111 may process content to be displayed on the display window of an application on electronic device 100. For example, controller may generate operation control signals based on instruction opcodes and timing signals to control the execution of instructions.
[0082] In one example, processor 111 may also include memory for storing instructions and data. In some examples, the memory in processor 111 is a cache memory. This memory can store instructions or data that processor 111 has just used or is recurring. If processor 111 needs to use the instruction or data again, it can directly retrieve it from the memory to avoid repeated access, reduce processor 111's waiting time, and improve system efficiency.
[0083] The memory 112 may include internal memory for storing computer-executable program code, which includes instructions. The processor 111 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), client, etc. The data storage area may store data created during the use of the electronic device 100 (such as audio signals, phone books, etc.), etc. In addition, the internal memory may include non-volatile memory 1121, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc., and random access memory 1122, which is generally referred to as RAM. In this embodiment of the invention, the non-volatile memory 1121 may store screen recordings as described below; during actual operation, image files, character sets, dot matrix data or vector data in the font library, the operating system, and the client may be loaded into the random access memory 1122.
[0084] In one example, memory 112 also includes an external memory card, such as a Micro SD card, connected via an external memory interface to expand the storage capacity of electronic device 100. The external memory card communicates with processor 111 via the external memory interface to perform data storage functions. For example, music, video, and other files can be stored on the external memory card.
[0085] The wireless communication function of electronic device 100 can be realized through an antenna, communication module 113, modem, and baseband processor.
[0086] Antennas are used to transmit and receive electromagnetic wave signals. The antennas in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, an antenna can be reused as a diversity antenna for a wireless local area network. In other examples, antennas can be used in conjunction with tuning switches.
[0087] The communication module 113 can receive electromagnetic waves through at least two antennas, filter and amplify the received electromagnetic waves, and transmit them to a modem for demodulation. It can also amplify the signal modulated by the modem and radiate it as electromagnetic waves through the antennas. In some examples, at least some functional modules of the communication module 113 can be housed in the processor 111. In some examples, at least some functional modules of the communication module 113 and at least some modules of the processor 111 can be housed in the same device.
[0088] In addition, the communication module 113 can receive electromagnetic waves through at least two antennas, and perform filtering, frequency modulation, and other processing on the received electromagnetic waves before sending the processed signal to the processor 111. It can also receive signals to be transmitted from the processor 111, perform frequency modulation and amplification on them, and then convert them into electromagnetic waves for radiation via the antenna.
[0089] In some examples, electronic device 100 can communicate with networks and other devices via wireless communication technologies. These wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), 5G, New Radio (NR), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite-Based Augmentation Systems (SBAS).
[0090] Electronic device 100 can implement audio functions such as music playback and recording through audio module 114, speaker 1141, microphone 1142, and application processor.
[0091] The audio module 114 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 114 can also be used for encoding and decoding audio signals. In some examples, the audio module 114 may be located within the processor 111, or some functional modules of the audio module 114 may be located within the processor 111.
[0092] Microphone 1141, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 1141, inputting the sound signal into microphone 1141. Electronic device 100 may have at least one microphone 1141. In some other examples, electronic device 100 may have two microphones 1141, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 1141, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.
[0093] The speaker 1142, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 114A.
[0094] In some possible cases, the audio module 114 can also be connected to a headphone jack, which can be a USB interface 130, or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.
[0095] Electronic device 100 implements display functions through a GPU, display screen 115, memory 112, digital-to-analog converter, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 115 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. Processor 111 may include one or more GPUs, which execute program instructions to generate or modify display information. Memory 112 includes video memory, used to store data processed by the GPU or the application processor, representing information for each pixel that can be output to the display screen 115; this video memory can be independent and does not occupy random access memory 1122; it can also share a portion of random access memory 1122 as video memory; or it can be both independent video memory and a portion of shared random access memory 1122. The digital-to-analog converter is used to convert the video memory data into analog signals, and the display screen 115 uses this analog information for display. It is worth noting that...
[0096] Specifically, the display screen 115 is used to display images, videos, etc. The display screen 115 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some examples, the electronic device 100 may include one or more display screens 115. In one example, the display screen 115 can be used to display an application interface, an application display window, etc.
[0097] Electronic device 100 operates display screen 115 through mouse 1161 and keyboard 1162 in input device 116. Electronic device 100 can receive operations from mouse 1161 and key input from keyboard 1162, generating signal inputs related to user settings and function control of electronic device 100.
[0098] Additionally, the input device 116 may also include buttons. These buttons include a power button, volume buttons, and an input keypad. The buttons can be mechanical buttons or touch-sensitive buttons.
[0099] Furthermore, the electronic device 100 may also include a universal serial bus (USB) interface, a charging management module, a power management module, a battery, buttons, etc. The charging function of the electronic device 100 can be implemented through the charging management module, the power management module, and the battery.
[0100] Of course, for the sake of simplicity, Figure 1 Only some of the components of the electronic device 100 relevant to this application are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 100 may include any other suitable components depending on the specific application. Furthermore, the electronic device may be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Those skilled in the art will understand that... Figure 1 This is merely an example of electronic device 100 and does not constitute a limitation on electronic device. It may include more or fewer components than shown, or combine certain components, or use different components.
[0101] The online help statement generation method provided in the embodiments of this application will be described in detail below.
[0102] Figure 2 This is a flowchart illustrating the online help statement generation method provided in this application embodiment. This embodiment can be applied to electronic devices, specifically servers or general computers, such as the aforementioned electronic device 100. Figure 2 As shown, the online help statement generation method provided in this application embodiment includes at least the following steps:
[0103] Step 201: Acquire multiple screen recording frames; wherein, the multiple screen recording frames are obtained by recording the process of the user operating the graphical user interface.
[0104] According to one feasible implementation, the display screen 115 of the electronic device 100 displays a graphical user interface (GUI) and buttons for starting and ending screen recording. When the user clicks the start screen recording button using the mouse 1161, the electronic device 100 begins recording the GUI. At the same time, the user operates the GUI using the mouse 1161 and keyboard 1162. During the recording process, the electronic device 100 captures the screen recording interface on the display screen 115 and obtains multiple screen recording frames of the GUI.
[0105] For example, suppose as Figure 3 As shown, the graphical user interface (GUI) includes 11 tabs, each of which can represent a function.
[0106] In one example, a user can integrate the various functions that a graphical user interface (GUI) can provide, and then operate them one by one.
[0107] Optionally, users can click "Start Screen Recording" before operating a function and click "End Screen Recording" after operating a function; then they can start screen recording again when operating the next function.
[0108] For example, the user clicks the "Start Screen Recording" button, such as... Figure 4a As shown, when the mouse moves over tab 6, tab 6 changes color and transforms. Clicking on tab 6 will bring up the configured subpage 1 in the user interface, as shown below. Figure 4b As shown, when the mouse moves over the close button, the close button changes color. Clicking the close button closes configuration subpage 1; afterwards, the user clicks the end screen recording button.
[0109] For example, the user clicks the "Start Screen Recording" button, such as... Figure 5 As shown, when the mouse moves to tab 7, tab 7 changes shape. Clicking tab 7 changes the content of the user interface. Afterward, the user clicks the button to end the screen recording.
[0110] For example, the user clicks the "Start Screen Recording" button, such as... Figure 6a As shown, when the mouse moves to tab 2, tab 2 changes color. Clicking tab 2 redirects the user interface to product introduction page 1, opening product introduction page 1. Figure 6b As shown, when the mouse moves to the sliding component and is long-pressed and slid, the content displayed on page 1 of the product introduction changes; then, the user clicks the button to end the screen recording.
[0111] Optionally, users can click "Start Screen Recording" before operating a function, continue to the next function after operating one function, and click "End Screen Recording" after operating multiple functions.
[0112] For example, when a user clicks the "Start Screen Recording" button, they follow the instructions... Figure 4a and Figure 4b Perform the operation as shown; then follow... Figure 5 Perform the operation as shown; then follow... Figure 6a and Figure 6b Follow the instructions shown; then, the user clicks the button to end screen recording.
[0113] In one possible scenario, if the recording time from start to finish is short, such as recording one function-related operation at a time, then multi-frame screen recording can include all the screen footage recorded from the start to the end of the recording process.
[0114] In one possible scenario, if the recording time from start to finish is relatively long, for example, if multiple function-related operations are recorded sequentially.
[0115] In one example, multi-frame screen recording can capture all the frames from the start of recording to the current moment.
[0116] In another example, the multiple screen recording frames can be a preset number of frames captured in the most recent screen recording. The preset number of frames can be determined based on actual needs, and this application embodiment does not impose a specific limitation on it. For example, the preset number can be 4 frames.
[0117] In another example, the multi-frame screen recording can be multiple frames captured from the current moment and a preset time period preceding the current moment. The preset time period can be determined based on actual needs, and this embodiment does not impose a specific limitation on it; for example, it can be 4 seconds.
[0118] It is worth noting that, in this embodiment of the application, recording can be performed when the graphical user interface changes, such as mouse movement or changes in the page content of the graphical user interface.
[0119] For example, in Figure 4a and Figure 4bThe screen recording shows the user's online help process, including mouse movement, tab color changes, pop-up subpage 1, and button color changes. Frames 1 to 3 record the mouse moving to tab 6, frame 4 records clicking tab 6 and the page after tab 6 changes color and transforms, frame 5 records the pop-up subpage 1, frames 6 to 10 record the mouse moving to the close button, frame 11 records the close button changing color, and frame 12 records the page after subpage 1 is closed, resulting in 12 frames of screen recording.
[0120] For example, in Figure 5 During the user's online help operation, the screen recording included mouse movement, tab deformation, and page content changes. Frames 1 to 3 recorded the mouse movement to tab 7, frame 4 recorded clicking tab 7 and the page after tab 7 deformed, and frame 5 recorded the page after the content changed, resulting in 5 frames of screen recording.
[0121] For example, in Figure 6a and Figure 6b The screen recording captures various actions during the user's online help process, including mouse movement, tab color changes, page navigation, button color changes, and slider movement. Frames 1 to 3 record the mouse moving to tab 2; frame 4 records clicking tab 2 and the page changing color; frame 5 records the page after navigation; frames 6 to 8 record the mouse moving to the slider; and frames 9 to 11 record the mouse being held down and moved, resulting in 11 frames of screen recording.
[0122] Step 202: Analyze the multi-frame screen recording to determine the semantic text of each interface element and the order in which the interface elements appear. The semantic text is used to represent the name of the corresponding interface element. The multiple interface elements include user actions during the operation, page controls operated by the user, and page actions of the graphical user interface.
[0123] Here, multiple interface elements can include several user actions, several user-operated page controls, and several page actions.
[0124] For example, interface elements can be page actions. A page action can be understood as a significant change in the graphical user interface caused by user interaction. Correspondingly, the semantic text of a page action can be page navigation, closing the page, popping up a subpage, closing the subpage, etc.
[0125] For example, interface elements can be user actions. For example, the semantic text of user actions can be mouse click, mouse right click, mouse long press and slide, mouse long press, keyboard input.
[0126] For example, interface elements can be page controls. Page controls are components that interact with the user. For example, the semantic text of a page control can be a page label, subpage, text box, page, button, etc.
[0127] Step 203: Based on the order in which multiple interface elements appear, semantically combine multiple semantic texts to obtain online help statements.
[0128] For example, in Figures 4a to 4b During the user's online help process, several interface elements and their corresponding semantic text are shown below:
[0129] Interface element IE11: Page control - Tab 6, semantic text is Tab 6.
[0130] Interface element IE12: User action - the action of clicking on tab 6, with semantic text "user click".
[0131] Interface element IE13: Page Action - The pop-up subpage 1 shown in the screen recording of frames 4 and 5, with semantic text "pop-up configuration subpage".
[0132] Interface element IE14: Page Controls - Close button in frame 11, semantic text is Close button.
[0133] Interface element IE15: Page Action - Action of clicking the close button in frame 11, semantic text is "user click".
[0134] Interface element IE16: Page Action - The screen recordings of frames 11 and 12 show the action of closing subpage 1, with the semantic text "subpage closed".
[0135] The order in which the 6 interface elements appear in IE11 to IE16 is: tab 6, user click, pop-up configuration subpage, close button, user click, close subpage. The online help statements that can be obtained include: S11: User click on tab 6 will pop up the configuration subpage; user click on the close button in the configuration subpage will close the subpage.
[0136] For example, in Figure 5 During the user's online help process, several interface elements and their corresponding semantic text are shown below:
[0137] Interface element IE21: Page control - Tab 7, semantic text is Tab 7.
[0138] Interface element IE22: User action - the action of clicking on tab 7, semantic text includes user click.
[0139] Interface elements ie23: Page actions - Changes in page content shown in the screen recording from frame 4 to frame 5, semantic text including page content changes.
[0140] The order in which the three interface elements (IE21 to IE23) appear is: tab 7, user click, and page content change. The online help statements that can be obtained include: S21: User click on tab 7 changes the page content.
[0141] For example, in Figures 6a to 6b During the user's online help process, several interface elements and their corresponding semantic text are shown below:
[0142] Interface element IE31: Page control - Tab2, semantic text is Tab2.
[0143] Interface element IE32: User action - the action of clicking on tab 2, semantic text includes user click.
[0144] Interface element ie33: Page Action - The page jump action shown in the screen recording of frames 4 and 5, with semantic text indicating a jump to the product introduction page.
[0145] Interface element IE34: Page controls - the sliding component in frame 9, semantic text is sliding component.
[0146] Interface elements IE35: User actions - actions of long press and slide on sliding components, semantic text includes long press and slide.
[0147] Interface element ie36: Page Action - The page content scrolling action shown in the screen recordings of frames 9 and 11, with the semantic text being "page content scrolling".
[0148] The order in which the 7 interface elements appear in IE31 to IE37 is: tab 2, user click, jump to product introduction page, sliding component, long press and slide action, page content scrolling. The online help statements that can be obtained include: S31: Clicking tab 2 will jump to the product introduction page; long press and slide action on the sliding component of the product introduction page will scroll the page content.
[0149] This solution analyzes screen recordings to identify the semantic text of interface elements (such as page actions, user actions, and user-operated page controls) and the order in which these elements appear. Based on this order, the semantic text is combined to automatically generate online help statements, improving the development efficiency of online help documentation. Furthermore, screen recordings can be developed as video recordings along with online help statements, enabling the joint development of screen recording instructions and GUI online help documentation, further enhancing the overall development efficiency of online help documentation.
[0150] Figure 7 As shown Figure 2 The flowchart of step 202 in the illustrated embodiment is shown.
[0151] like Figure 7 As shown above, in the above Figure 2 Based on the illustrated embodiment, in this embodiment of the application, step 202 may specifically include the following steps:
[0152] Step 2021: Extract the element images corresponding to each of the multiple interface elements from the multi-frame screen recording.
[0153] It should be noted that the element screen refers to the screen footage from multiple frames where interface elements and their semantic text can be identified. It should be understood that although the element screen is obtained, its specific meaning is unknown at this point; only through semantic analysis can the semantic text of the element screen be determined.
[0154] For example, the interface element can be a page action. Correspondingly, the element screen can be multiple frames in a multi-frame screen recording; for example, the element screen can be composed of two screen recording frames before and after the page action.
[0155] For example, interface elements can be user actions. Correspondingly, the element screen can be two frames from a multi-frame screen recording, before and after the page action. Furthermore, it can also include an area related to the mouse position, comprehensively determining the user action. For instance, changes in tabs and the screen after the page action can determine if a mouse click occurred; for example, such as... Figure 4a As shown, in frame 4, tab 6 changes shape and color, and a subpage is added in frame 5. Therefore, the user action is a mouse click. For example, whether text is added can indicate keyboard input; for example, the position change of a sliding component and the before-and-after screen changes can indicate a long mouse press and slide. For instance, as... Figure 6b As shown, frames 9 to 11 can reflect changes in mouse position, changes in the position of sliding components, and changes in page content. The user's operation is a long press and slide of the mouse. For example, the presence of a right mouse button action can be determined by whether a new option list has been added to the page. For example, when using Word, right-clicking while copying text will bring up a list of options such as cut, copy, paste, font, and paragraph.
[0156] For example, the interface element can be a page control. A page control is a component that interacts with the user. Correspondingly, the element screen can be a screen recording of the user operating the page control from multiple screen recording frames. In some possible implementations, if the same page control exists in the graphical user interface, the element screen can also include the title screen of the area where the page control is located. For example, if the page control is a "View More" button and the title of the area is "Success Cases," then the element screen also includes the area where the success cases are located. In some possible implementations, if the page control is a subpage, page, etc., to facilitate the description of the subpage or page name, the element screen can further include the screen in the page control that represents the page name.
[0157] For example, in Figures 4a to 4b The following are the recognizable elements and corresponding semantic text during the user's online help process:
[0158] Element screen IE11: Page controls, Figure 4a The semantic text of the page area containing page 6 in frame 4 is page 6.
[0159] Element screen IE12: User operation actions, Figure 4a The screen recording frames 4 and 5 contain semantic text including user clicks.
[0160] Element screen IE13: Page actions, Figure 4a The 4th and 5th frames of the screen recording contain the semantic text "pop-up configuration subpage".
[0161] Element screen IE14: Page controls, Figure 4b The page area where the close button is located in frame 11 has the semantic text "close button".
[0162] Element screen IE15: Page actions, Figure 4b The semantic text in frames 6 to 12 of the screen recording is mouse clicks.
[0163] Element screen IE16: Page actions, Figure 4b The screen recording frames 11 to 12 show the subpage closed.
[0164] For example, in Figure 5 The following are the recognizable elements and corresponding semantic text during the user's online help process:
[0165] Element in IE21: Page control, the page area containing page 7 in frame 4, with semantic text "page 7".
[0166] Element screen (IE22): User actions, screen recordings of frames 4 and 5, semantic text including user clicks.
[0167] Element screen (IE23): Page actions, screen recording from frame 4 to frame 5, semantic text including changes in page content.
[0168] For example, in Figures 6a to 6b The following are the recognizable elements and corresponding semantic text during the user's online help process:
[0169] Element screen IE31: page controls, Figure 6a The semantic text of the page area containing tab 2 in frame 4 is tab 2.
[0170] Element screen IE32: page controls, Figure 6b The page area where the sliding component is located in frame 9 has the semantic text "sliding component".
[0171] IE33 element screen: user operation actions, Figure 6a The screen recording frames 4 and 5 contain semantic text including user clicks.
[0172] Element screen IE34: User operation actions, Figure 6b The screen recording frames 9 to 11 contain semantic text including long press and swipe.
[0173] Element screen IE35: Page actions, Figure 6b The 4th and 5th frames of the screen recording contain the semantic text indicating a redirect to the product introduction page.
[0174] Element screen IE36: Page actions, Figure 6b The area containing the sliding component and page content in the screen recording from frames 9 to 11 is labeled with the semantic text "Page content scrolling".
[0175] In one possible scenario, multiple screen recording frames are input into an interface element detection model. This model outputs multiple first categories and corresponding bounding boxes within each category within the multi-frame recording. These bounding boxes, representing the screen's position, are typically expressed using coordinates, such as the coordinates of two points on the diagonal of the bounding box. Then, for each first category, based on the corresponding bounding boxes, a single element frame is extracted from the multi-frame recording. For example, the first category can be page actions, page controls, or user actions. In some optional cases, page controls may require text for comprehension; therefore, the element frame corresponding to a page control may include a text-only frame to indicate the control's name, facilitating subsequent semantic analysis. Similarly, page actions may require text for comprehension; therefore, the element frame corresponding to a page action may include a text-only frame to represent the name of a subpage or page, also facilitating semantic analysis. It is noteworthy that multiple element frames can share the same first category.
[0176] Correspondingly, the electronic device stores an information mapping table A1, which is used to represent the correspondence between interface elements and interface element identifiers in the first interface element. For example, if the interface element is a page action, the interface element identifier is 1; if the first interface element is a page control, the interface element identifier is 2; and if the interface element is page text, the interface element identifier is 3.
[0177] It should be noted that the interface element detection model is equivalent to classifying page controls, page actions, and user operations to obtain element images. At this point, it can be determined whether the element image is a page control, page action, or user operation. Then, semantic analysis of the element image can reveal the specific name of the page control, page action, or user operation.
[0178] In practical applications, a dataset can be constructed, comprising multiple samples, each consisting of an image sequence. This image sequence represents the user's interaction with the online help of the sample graphical user interface. Here, the sample graphical user interface can be a reference graphical user interface, generally one other than those described in this application. Additionally, the image sequence has multiple labels representing the page controls operated by the user, the page actions, and the area where the user's actions occurred. Then, a pre-defined interface element detection model is trained based on the multiple samples in the dataset, resulting in a trained interface element detection model.
[0179] Step 2022: Determine the order in which multiple interface elements appear, based on the element screens corresponding to each of the multiple interface elements.
[0180] When performing semantic combination, it is necessary to first determine the order in which multiple element screens appear, and then combine the semantic text corresponding to each of the multiple element screens based on the order of appearance to obtain the online help statement.
[0181] In practical applications, the appearance time of multiple interface elements can be estimated, and the order of appearance of multiple interface elements can be determined based on the order of their appearance.
[0182] For example, when the element screen is a page control, the recording time of the screen recording screen containing the element screen can be directly used as the occurrence time. Specifically, the occurrence time indicates the most recent recording time of the operation on the page control, which can also be understood as the time when the user operates on the page control. For example, element screen IE11: page control, Figure 4a If the page area containing tab 6 is in frame 4, then the recording time of frame 4 is taken as the appearance time of tab 6. For example, in IE31, the element screen... Figure 6b If the page area where the sliding component is located is in frame 9, then the screen recording time of frame 9 is taken as the appearance time of the sliding component.
[0183] For example, when an element screen represents a user action, the recording time of the previous frame of the screen recording that indicates a change in page content can be used as the occurrence time. Specifically, the occurrence time indicates when the user action occurs. For instance, in element screen ie12: user action, screen recording frames 4 and 5, semantic text includes user click, and the page content changes in the screen recording frame 5, then the recording time of the screen recording frame 4 is used as the occurrence time of the user click.
[0184] For example, when an element's screen represents a page action (i.e., a page change), the screen recording moment representing the page after the change can be used as the appearance moment. Specifically, the appearance moment indicates the screen recording moment closest to the moment of the page change, which can also be understood as the display moment of the page action. For instance, in IE13, the element screen shows a page action, with screen recordings of frames 4 and 5, and the semantic text indicating a pop-up subpage. Frame 5 represents the page after the change, so the screen recording moment of frame 5 is used as the appearance moment.
[0185] Optionally, in practical applications, the interface element detection model can output multiple screen recording screen identifiers for each of the first categories. The screen recording screen time indicated by the screen recording screen identifier is the occurrence or display time of the interface element. Correspondingly, the electronic device stores an information mapping table A2, representing the screen recording screen identifiers and their corresponding screen recording screens. Correspondingly, in the dataset used by the interface element detection model, the sample labels should also include the identifiers of the screen recording screens indicating the occurrence of the interface element. In this way, the interface element detection model can output multiple screen recording screen identifiers for each of the first categories. Subsequently, based on the screen recording screen identifiers for each of the multiple first categories, the occurrence time of the element screens corresponding to each of the multiple first categories is obtained, and further, the occurrence order of the multiple interface elements is obtained.
[0186] Step 2023: Perform semantic recognition on the element screens corresponding to each of the multiple interface elements to obtain the semantic text corresponding to each of the multiple interface elements.
[0187] Each element in the image has a first category. Since different first categories differ significantly, different first categories require different semantic analysis models.
[0188] Specifically, for each of the multiple element images, the element image is input into the semantic analysis model under the first category of the element image, and the semantic text of the element image is obtained based on the output of the semantic analysis model.
[0189] For example, if the first category is page actions, the semantic analysis model can be called a page action recognition model. If the first category is page controls, the semantic analysis model can be called a page control recognition model; if the first category is user operation actions, the semantic analysis model can be called a user operation action recognition model. In some optional cases, the element screens corresponding to page controls and page actions may contain page text screens. Deep learning models can be used to identify the page text screens within these element screens and assign page text identifiers (to indicate that the screen contains text).
[0190] In one optional implementation, the electronic device 100 stores a model library, which includes pre-trained page control recognition models, page action recognition models, and user operation action recognition models; furthermore, it may also include page text recognition models; the electronic device 100 calls the models in the model library to realize the text conversion of multiple element screens.
[0191] Here, the page control recognition model outputs page control identifiers. Correspondingly, the electronic device 100 stores an information mapping table B, which is used to represent the correspondence between page control names and page control identifiers. For example, if the page control name is an input box, the page control identifier is 1; if the page control name is an OK button, the page control identifier is 2.
[0192] For each element screen representing a page control, if there is no page text screen in the element screen, the element screen is input into the page control recognition model to obtain the page control identifier output by the page control recognition model. Based on information mapping table B, the page control name corresponding to the page control identifier is obtained as semantic text.
[0193] If the element screen contains page text, then the text outside the page text within the element screen is input into the page control recognition model, resulting in the output page control identifier; the page text is input into the page text recognition model, which outputs the recognized page text; the recognized page text and the page control name corresponding to the page control identifier are used as semantic text. For example, for Figure 4a In the example, there is a tab 6 with the name "Configuration". Assuming the element screen is the area containing tab 6 and the text screen of the "Configuration" page, then the page control is identified as a tab, the page text is "Configuration", and the corresponding semantic text is "Configuration tab".
[0194] Here, the page action recognition model outputs a page action identifier. Correspondingly, the electronic device 100 stores an information mapping table C, which is used to represent the correspondence between page action names and page action identifiers.
[0195] For each element screen representing a page action, the element screen is input into the page action recognition model to obtain the page action identifier output by the page action recognition model. Based on the information mapping table C, the page action name corresponding to the page action identifier is obtained as semantic text.
[0196] If the element screen contains page text, then the elements other than the page text are input into the page action recognition model to obtain the page control identifier output by the page action recognition model; the page text is input into the page text recognition model to output the recognized page text; the recognized page text and the page action name corresponding to the page action identifier are combined into semantic text. For example, for Figure 4a The configuration of subpage 1 in the middle, assuming the element screen is Figure 4a The screen recordings of frames 4 and 5, as well as the text on the "Configuration" page in subpage 1, indicate that the page action is "Pop-up subpage," the page text is "Configuration," and the corresponding semantic text is "Pop-up configuration subpage."
[0197] Here, the user operation action recognition model outputs the user operation action identifier. Correspondingly, the electronic device 100 stores an information mapping table D, which is used to represent the correspondence between the user operation action name and the user operation action identifier.
[0198] For each element screen representing a user action, the element screen is input into the user action recognition model to obtain the user action identifier output by the user action recognition model. Based on the information mapping table D, the user action is obtained.
[0199] The name of the corresponding user action is used as semantic text.
[0200] In addition, the training methods for the page control recognition model, page action recognition model, and user operation action recognition model can be found in the training methods for the interface element detection model. The only differences are the dataset and the structure of the model.
[0201] This solution uses a deep learning model to automatically identify the semantic text of interface elements (such as page actions, user actions, and user-operated page controls) and the order in which these elements appear. This allows for the automatic generation of online help messages.
[0202] This sentence improves the efficiency of developing online help documentation.
[0203] Furthermore, such as Figure 8 As shown, in Figure 8 Based on steps 201 to 203 shown, this embodiment of the application further includes at least the following step: Step 204: Displaying online help statements.
[0204] like Figure 3 Therefore, in addition to displaying the graphical user interface (GUI) and screen recording buttons on the display screen 115 of the electronic device 100, it also displays generated online help statements. For example... Figure 3 As shown, the display screen 115 of the electronic device 100 is divided into three areas: one area displays the graphical user interface (GUI), one area displays online help statements, and one area displays buttons for starting and ending screen recording.
[0205] Step 205: Based on the user's modification of the online help statement, obtain the modified online help statement.
[0206] In some possible implementations, users can display an online help statement area on the electronic device 100, select the online help statement and modify it directly, making the modified contact help statement more aesthetically pleasing, fluent, and in line with reading habits.
[0207] For example, suppose the online help statement is: S11: When the user clicks on tab 6, a configuration subpage pops up; when the user clicks the close button in the configuration subpage, the subpage is closed.
[0208] The revised online help statement is as follows:
[0209] 1: Click tab 6 to bring up the configuration subpage.
[0210] 2: Click the close button in the configuration subpage to close the subpage.
[0211] In addition, such as Figure 3 Therefore, an "Export Text" button is displayed on the screen 115 of the electronic device 100, and the user clicks it to export.
[0212] The text-based button allows you to export and save the online help text in a desired format, such as .doc. This allows for subsequent modification of the online help statements in the saved file, significantly improving the efficiency of online help document development. For example... Figure 3 As shown, the display screen 115 of the electronic device 100 displays an area for online help statements and a button for exporting text.
[0213] In addition, such as Figure 3 As shown in the embodiment of this application, the screen recording file can also be stored. The screen recording file indicates the video formed by the screen recording screen between the start and end of the recording. The user can save the screen recording file by clicking the "Export Screen Recording File" button.
[0214] Save it in the desired format. For example... Figure 3 As shown, the display screen 115 of the electronic device 100 displays the areas for starting and ending screen recording, a thumbnail of the screen recording file, and a button for exporting the screen recording file.
[0215] In summary, this solution allows for the modification of existing online help statements; subsequent modifications are merely text polishing, thus improving the development efficiency of online help statements. Furthermore, screen recordings can be exported as video recordings and developed alongside the online help statements, enabling the joint development of screen recording guidance and GUI online help documentation, further enhancing the development efficiency of online help documentation.
[0216] Based on the same concept as the embodiments of the method of the present invention, the embodiments of the present invention also provide an online help statement generation device. The online help statement generation device includes several modules, each module being used to execute various steps in the online help statement generation method provided in the embodiments of the present invention. The division of modules is not limited here. Those skilled in the art will clearly understand that in practical applications, the various steps in the online help statement generation method provided in the embodiments of the present invention can be assigned to different modules as needed, that is, the internal structure of the device can be divided into different modules to complete all or part of the functions described above. The modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the modules are only for easy differentiation and are not intended to limit the scope of protection of the present invention. The specific working process of the modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0217] For example, the online help statement generation device is used to execute the online help statement generation method provided in the embodiments of the present invention. Figure 9 This is a schematic diagram of the online help statement generation device provided in an embodiment of this application. Figure 9 As shown, the online help statement generation device provided in this application embodiment includes:
[0218] The screen recording module 901 is used to acquire multiple screen recording frames; wherein, the multiple screen recording frames are obtained by recording the process of the user operating the graphical user interface, and the graphical user interface includes at least one page control.
[0219] Analysis module 902 is used to analyze the multi-frame screen recording to determine the semantic text of each of the multiple interface elements and the order in which the multiple interface elements appear; wherein, the semantic text is used to represent the name of the corresponding interface element; the multiple interface elements include user operation actions during the operation, user-operated page controls, and page actions of the graphical user interface.
[0220] The combination module 903 is used to semantically combine the multiple semantic texts based on the order in which the multiple interface elements appear, to obtain online help statements.
[0221] In one possible implementation, the analysis module 902 is used to extract the element images corresponding to each of the multiple interface elements from the multi-frame screen recording; and to perform semantic recognition on the element images corresponding to each of the multiple interface elements to obtain the semantic text corresponding to each of the multiple interface elements.
[0222] In one example, analysis module 902 includes:
[0223] The detection unit is used to input the multi-frame screen recording into the interface element detection model; based on the output of the interface element detection model, the element images corresponding to each of the multiple interface elements are obtained.
[0224] Optionally, the detection unit is further configured to obtain the appearance order of the element screens corresponding to each of the plurality of interface elements based on the output of the interface element detection model.
[0225] In one example, the element screen has a first category, which indicates a user action, a user-operated page control, or a page action.
[0226] Optionally, the analysis module 902 includes:
[0227] The semantic analysis unit is used to input the element screen corresponding to each of the plurality of interface elements into a semantic analysis model under a first category of the element screen, and obtain the semantic text of the element screen based on the output of the semantic analysis model.
[0228] In one possible implementation, a screen recording module is used to record when and / or after a first change occurs in the graphical user interface, the first change indicating a change in mouse position or a change in page content.
[0229] In one possible implementation, the page action includes at least one of the following: page navigation, opening a page, closing a page, and changing page content.
[0230] In one possible implementation, the page controls include at least one of the following: page labels, buttons, and text boxes.
[0231] In one possible implementation, the user operation includes at least one of the following: mouse click, mouse right-click, mouse long press and slide, and keyboard input.
[0232] Among some feasible implementation methods, Figure 1 The memory 112 of the electronic device 100 can store computer programs. When the processor 111 executes the computer program, it implements the steps in the above-described method embodiment for generating online help statements, for example... Figure 2 Steps 201 to 203 are shown. Figure 8Steps 201 to 205 are shown. Alternatively, when the processor 111 executes the computer program, it implements the functions of each module in the above-described device embodiment. For example, the computer program can be divided into one or more modules / units, which can be a series of computer program instruction segments capable of performing specific functions. One or more modules / units are stored in the memory 112 and executed by the processor 111 to complete the present invention. For example, the computer program can be divided into a screen recording module 901, an analysis module 902, and a combination module 903; the specific functions of each module are described above.
[0233] In addition to the methods, apparatus, and electronic devices described above, embodiments of this application may also provide a computer program product comprising computer program instructions. When executed by a processor, these computer program instructions cause the processor to perform the steps of the online help statement generation methods described in the "Methods" section of this specification for various embodiments of this application. The computer program product may be written in any combination of one or more programming languages to perform the operations of embodiments of this application. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The computer program code may be in source code form, object code form, executable file, or some intermediate form. The computer program code may be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0234] Furthermore, embodiments of this application may also provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the online help statement generation method according to various embodiments of this disclosure as described in the "Method" section above. The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0235] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0236] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0237] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of the various embodiments of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.
[0238] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0239] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.
[0240] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
[0241] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.
Claims
1. A method for generating statements, characterized in that, include: Acquire multiple frames of screen recording; wherein, the multiple frames of screen recording are obtained by recording the process of the user operating the graphical user interface; The multi-frame screen recordings are analyzed to determine the semantic text of each interface element and the order in which the interface elements appear. The semantic text is used to represent the name of the corresponding interface element. The multiple interface elements include user actions during the operation, user-operated page controls, and page actions of the graphical user interface. Based on the order in which the multiple interface elements appear, multiple semantic texts are semantically combined to obtain online help statements; The analysis of the multi-frame screen recording to determine the semantic text of each interface element includes: Extract the element images corresponding to each of the multiple interface elements from the multi-frame screen recording; Semantic recognition is performed on the element screens corresponding to each of the multiple interface elements to obtain the semantic text corresponding to each of the multiple interface elements. Extracting the element images corresponding to each of the multiple interface elements from the multi-frame screen recording includes: The multi-frame screen recording is input into the interface element detection model; the interface element detection model is trained based on multiple samples in the dataset to train a preset interface element detection model; the dataset includes multiple samples, each sample consists of an image sequence, and the image sequence is used to represent the user's operation process of the online help of the sample graphical user interface; Based on the output of the interface element detection model, the element screen corresponding to each of the multiple interface elements is obtained; the element screen includes the page control, the page action, or the user operation action.
2. The method according to claim 1, characterized in that, The step of analyzing the multi-frame screen recording to determine the order of appearance of the multiple interface elements includes: Based on the output of the interface element detection model, the order in which the element images corresponding to each of the multiple interface elements appear is obtained.
3. The method according to claim 1, characterized in that, The element screen has a first category, which indicates user operation actions, user-operated page controls, or page actions.
4. The method according to claim 3, characterized in that, The step of performing semantic recognition on the element screens corresponding to each of the plurality of interface elements to obtain the semantic text corresponding to each of the plurality of interface elements includes: For each of the multiple interface elements, the element screen is input into a semantic analysis model under the first category of the element screen. Based on the output of the semantic analysis model, the semantic text of the element screen is obtained.
5. The method according to claim 1, characterized in that, The acquisition of multiple screen recording frames includes: Recording is performed when and / or after the first change occurs in the graphical user interface to obtain multiple screen recording frames; wherein, the first change indicates a change in mouse position or a change in page content.
6. The method according to claim 1, characterized in that, The page actions include at least one of the following; Page redirection, page opening, page closing, page content modification: and / or The page controls include at least one of the following: Page labels, buttons, text boxes; and / or, The user operation includes at least one of the following: Mouse click, right mouse button, long press and slide mouse, keyboard input.
7. An online help statement generation device, characterized in that, Includes processor and memory; among which, The memory is used to store programs; The processor shown is configured to execute a program stored in the memory, and when the program stored in the memory is executed, to perform the method as described in any one of claims 1 to 6.
Citation Information
Patent Citations
RPA process video key frame extraction and element positioning method
CN115033740A
Automatic text generation
CN1973256A
