Method and apparatus for automatically scrolling and capturing dialog window executed on android-based portable terminal

Abstract

The present invention relates to a technique of automatically scrolling and capturing a dialog window executed on an Android-based portable terminal. The automatically scrolling and capturing method according to one embodiment may comprise the steps of: receiving an input of a screen capture command; acquiring, on the basis of the screen capture command that was input, XML data for the current screen of the portable terminal by using a screen dump function of Android; acquiring at least one piece of XML data while scrolling the screen by a designated pixel amount from the position where the XML data for the current screen was acquired; and generating a screen capture image by integrating the acquired pieces of XML data.

Description

Method and device for automatically scrolling and capturing a dialog window running on an Android-based portable terminal

[0001] The present invention relates to a technology for automatically scrolling and capturing a dialogue window running on an Android-based portable terminal.

[0002] Screen capture technology provides the ability to capture and save the device's current screen as an image.

[0003] The screen of a monitor or display is stored in a framebuffer. The framebuffer maintains information about each pixel on the screen in memory.

[0004] Screen capture primarily involves reading data from this framebuffer and saving it as an image.

[0005]

[0006] Operating systems provide APIs and commands that allow you to capture the screen. For example, on Android, you can capture the screen using the "adb shell screencap" command.

[0007] Some operating systems may also provide an API that developers can use to take screenshots programmatically.

[0008] Graphics and image processing libraries are used to capture and store images. These libraries provide a variety of functions and tools needed to create and manipulate images.

[0009] For example, in Python, libraries such as PIL (Pillow) and OpenCV are widely used.

[0010] When capturing a screen, it's important to consider the device's screen resolution and aspect ratio. Failure to do so may result in the captured image being distorted or appearing at a different aspect ratio than the original.

[0011] The process of capturing a screen can impact device performance and resource usage. Therefore, it's important to perform the capture appropriately to optimize performance.

[0012] Some platforms may restrict capturing other apps' screens due to security and permission issues. This may require user consent or specific permissions.

[0013] Screen capture technology is used in a variety of environments, and these underlying technologies must be applied appropriately, taking into account the characteristics of the device and the operating system.

[0014] The purpose of the present invention is to automatically scroll and capture a dialog window running on an Android-based portable terminal.

[0015] The purpose of the present invention is to accurately and quickly process the capture of a conversation window of a portable terminal.

[0016] An automatic scroll capture method according to one embodiment may include a step of receiving a screen capture command, a step of obtaining XML data for a current screen of the mobile terminal by utilizing a screen dump function of Android based on the input screen capture command, a step of obtaining at least one XML data while scrolling the screen by a selected number of pixels at a location where the XML data for the current screen is obtained, and a step of generating a screen capture image by integrating the obtained XML data.

[0017] In one embodiment, the step of obtaining at least one XML data while scrolling the screen by a selected number of pixels from a position where XML data for the current screen is obtained may include the step of obtaining at least one XML data while changing the position from the nth position where XML data for the current screen is obtained to a specific position while scrolling by a selected number of pixels.

[0018] In one embodiment, the step of obtaining at least one XML data while scrolling the screen by a selected number of pixels at a location where XML data for the current screen is obtained may include the step of obtaining at least one XML data while changing the location until the end of the dialog box being executed is reached.

[0019] In one embodiment, the step of obtaining at least one XML data while scrolling the screen by a selected number of pixels at a location where XML data for the current screen is obtained may include the step of identifying an end position included in a screen capture command, and the step of obtaining at least one XML data while scrolling the screen until the identified end position is reached.

[0020] The step of generating a screen capture image by integrating the acquired XML data according to one embodiment may include the steps of identifying common data between consecutive XML data, combining a capture image for a current screen and a capture image for a next screen, but combining the capture image after the portion corresponding to the common data in the capture image for the next screen, and generating a screen capture image by combining capture images for all remaining acquired XML data.

[0021] A device for automatically scrolling and capturing a dialog window running on an Android-based portable terminal according to an embodiment may include a screen capture command input unit for receiving a screen capture command, an XML data acquisition unit for obtaining XML data for a current screen of the portable terminal by utilizing a screen dump function of Android based on the input screen capture command, and for obtaining at least one XML data while scrolling the screen by a selected number of pixels at a position where the XML data for the current screen is obtained, and a screen capture image generation unit for generating a screen capture image by integrating the obtained XML data.

[0022] According to one embodiment, the XML data acquisition unit can acquire at least one XML data by changing the position from the nth position where XML data for the current screen is acquired to a specific position while scrolling by a selected number of pixels.

[0023] According to one embodiment, the XML data acquisition unit can acquire at least one XML data by changing the position until it reaches the end of the executed dialog window.

[0024] According to one embodiment, the XML data acquisition unit can identify an end position included in a screen capture command and acquire at least one XML data while scrolling the screen until the identified end position is reached.

[0025] The screen capture image generation unit according to one embodiment can generate a screen capture image by identifying common data between consecutive XML data, combining a capture image for a current screen and a capture image for a next screen, combining a capture image after a portion corresponding to the common data in the capture image for the next screen, and combining capture images for all remaining XML data acquired.

[0026] According to one embodiment, a dialog window running on an Android-based portable terminal can be automatically scrolled and captured.

[0027] According to one embodiment, the capture of a conversation window of a portable terminal can be processed accurately and quickly.

[0028] FIG. 1 is a drawing illustrating a method for automatically scrolling and capturing a conversation window in an Android-based portable terminal according to one embodiment.

[0029] Figure 2 is a diagram illustrating a dialog box running on an Android-based portable terminal.

[0030] Figure 3 is a diagram illustrating XML data for a running dialog box.

[0031] Figure 4 is a diagram showing coordinates within XML data.

[0032] FIG. 5 is a drawing illustrating a device for automatically scrolling and capturing a conversation window in an Android-based portable terminal according to one embodiment.

[0033] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0034] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes modifications, equivalents, or alternatives that fall within the spirit and technical scope of the present invention.

[0035] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are intended solely to distinguish one component from another. For example, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component," without departing from the scope of the invention.

[0036] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions that describe relationships between components, such as "between," "immediately between," or "directly adjacent to," should be interpreted similarly.

[0037] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0038] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0039]

[0040] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals in each drawing represent the same components.

[0041] FIG. 1 is a drawing illustrating a method for automatically scrolling and capturing a conversation window in an Android-based portable terminal according to one embodiment.

[0042] An automatic scroll capture method according to one embodiment receives a screen capture command from a user or a running program (step 101).

[0043] Next, an automatic scroll capture method according to an embodiment obtains XML data for the current screen of the mobile terminal by utilizing the screen dump function of Android based on an input screen capture command (step 102), and at a location where XML data for the current screen is obtained, at least one XML data can be obtained by scrolling the screen by a selected number of pixels (step 103).

[0044] When a screen capture command is input from a user or a running program, it can act as a trigger to start an automatic scroll and capture operation.

[0045] To obtain XML data, you can use the Android screen dump feature to obtain XML data for the current screen based on the input screen capture command. Android's screen dump feature provides the UI structure of a dialog box and corresponding information in XML format.

[0046] A series of steps can be repeated to obtain automatic scrolling and XML data. Specifically, the acquired XML data can be analyzed to automatically scroll the chat window, taking into account factors such as the chat window height and scroll units, and additional XML data can be obtained. This process can be repeated until a predefined condition is met or the chat window ends.

[0047] That is, the automatic scroll capture method according to one embodiment determines whether a set condition is met (step 104), and if the set condition is not met, XML data can be obtained by scrolling the screen by a selected number of pixels (step 105).

[0048] If the judgment result of step 104 satisfies the set conditions, the automatic scroll capture method according to one embodiment can generate a screen capture image by integrating the acquired XML data (step 106).

[0049] The set condition may be a condition that determines whether the current location has changed to a specific location through scrolling.

[0050] For example, in order to obtain at least one XML data while scrolling the screen by a selected number of pixels, at least one XML data can be obtained by changing the position from the nth position where XML data for the current screen is obtained to a specific position while scrolling by a selected number of pixels.

[0051] In this process, at least one XML data can be acquired by changing the position until the end of the running dialog box is reached.

[0052] Additionally, in order to obtain at least one XML data while scrolling the screen by a selected number of pixels, an end position included in the screen capture command can be identified, and at least one XML data can be obtained while scrolling the screen until the identified end position is reached.

[0053] In addition, in order to generate a screen capture image by integrating the acquired XML data, common data is identified between consecutive XML data, and the capture image for the current screen and the capture image of the next screen are combined, but the capture image after the part corresponding to the common data in the capture image of the next screen can be combined.

[0054] Additionally, you can create a screen capture image by combining the captured images for all the remaining XML data you have acquired.

[0055] This process synthesizes all the collected information and automatically scrolls the remaining parts of the screen according to each XML data to generate a capture image.

[0056] By considering information such as the location, size, and text of each UI element included in XML data, capture images of various parts can be combined, and the screen part represented by each XML data can be accurately placed in the corresponding location, so that a screen capture image of the entire dialog box can be created.

[0057] The images generated in this way can be presented visually to users or used for various purposes such as automated testing, analysis, and documentation.

[0058] FIG. 2 is a drawing (200) showing a dialog box running on an Android-based portable terminal.

[0059] Figure 2 is a visual representation of a dialog box of an application running in an Android environment.

[0060] Figure 2 depicts a chat window displayed on the screen of a portable terminal. The chat window contains various UI elements, including buttons, text, and images, which can be visually identified. Furthermore, a portion of the chat window contains XML data, as shown in Figure 3, providing structural information about the corresponding chat window.

[0061]

[0062] Figure 2 can be utilized to visually understand and analyze the screen of an application running on an Android terminal for screen capture.

[0063] There are various methods and technologies used to capture chat windows, mainly screen capture and image processing technologies.

[0064] You can use screen capture technology to capture the chat window.

[0065] The present invention is directed to an Android-based portable terminal, for example, an Android smartphone.

[0066] In Android, you can capture the screen of the current portable device using commands such as adb shell screencap.

[0067] Image processing technology can be utilized to extract a specific area from a captured image and analyze necessary information, but in the present invention, XML data for the current screen can be utilized.

[0068] Certain UI elements, such as screen coordinates and objects within the chat window, can be expressed as XML data.

[0069] Additionally, Android provides XML data about the current screen.

[0070] In the present invention, automation tools such as UI Automator and Appium can be utilized to automate capture and analysis tasks using acquired XML data.

[0071] FIG. 3 is a drawing (300) showing XML data for a running dialog box.

[0072] Figure 3 visually represents the UI structure of the dialog box and details for each UI element.

[0073] Figure 3 shows a hierarchical arrangement of various UI elements within the dialog box. Each UI element displays attributes such as position, size, and text, allowing for easy identification of its characteristics. Furthermore, the location of the XML data displayed allows for an overview of how each element is arranged within the dialog box.

[0074] Figure 3 visually represents the information collected by the XML data acquisition unit, and can be used to easily understand and analyze the structure of the dialog box.

[0075] By displaying XML data, you can drill down into specific parts of a dialog box or provide a basis for automated tasks.

[0076] Figure 4 is a drawing (400) showing the appearance of coordinates in XML data.

[0077] When capturing while scrolling the screen, only the coordinate values ​​of the same speech bubble will change, and this change value can be used to determine how much the image has scrolled.

[0078] Figure 4 visually represents the coordinates of a specific UI element extracted from XML data.

[0079] Figure 4 shows coordinate values ​​representing the location of a specific UI element contained in XML data. These coordinate values ​​generally represent a relative position relative to the upper left corner of the screen. The values ​​along the X and Y axes can accurately express the location of the UI element on the screen.

[0080] Figure 4 shows the coordinate information collected by the XML data acquisition unit, which can be used to visually confirm the coordinate information and analyze the precise location of UI elements. This coordinate information can then be used in automatic scrolling and capture operations.

[0081] FIG. 5 is a drawing illustrating a device (500) that automatically scrolls and captures a conversation window in an Android-based portable terminal according to one embodiment.

[0082] An automatic scroll capture device (500) according to one embodiment may include a screen capture command input unit (510), an XML data acquisition unit (520), a screen capture image generation unit (530), and a control unit (540).

[0083] First, the screen capture command input unit (510) receives a screen capture command.

[0084] The screen capture command input unit (510) can receive a screen capture command from a user or through a separate application that is being run.

[0085] In addition, commands input into the screen capture command input section (510) are mainly input through a terminal or user interface, and the configuration of the commands may include specific parameters or options related to screen capture.

[0086] The screen capture command input unit (510) can process the input command and induce the system or application to perform appropriate actions. For example, it can perform tasks such as verifying the validity of the command or collecting necessary additional information. Furthermore, the input command can be interpreted to determine the scope, format, or other relevant settings for the screen capture.

[0087] Next, the XML data acquisition unit (520) acquires XML data for the current screen of the mobile terminal by utilizing the screen dump function of Android based on the input screen capture command, and acquires at least one XML data while scrolling the screen by a selected number of pixels at the location where the XML data for the current screen was acquired.

[0088] In the process of acquiring XML data, the XML data acquisition unit (520) can extract detailed information about the UI structure and corresponding elements displayed on the screen in XML format.

[0089] The XML data acquisition unit (520) calls the screen dump function and processes the XML data received as a result to extract the hierarchical structure, attributes, coordinates, and other related information of the UI elements displayed on the screen. The XML data thus acquired can be used as basic information for automatic scrolling and capture operations in subsequent steps.

[0090] An XML data acquisition unit (520) according to one embodiment can acquire at least one XML data by changing the position from the nth position where XML data for the current screen is acquired to a specific position while scrolling by a selected number of pixels.

[0091] The XML data acquisition unit (520) can start from the nth position where XML data for the current screen is acquired, change the position of the screen by scrolling by a selected number of pixels, and move to a specific position. Through this process, at least one piece of XML data can be acquired.

[0092] Scrolling to a specific location can be accomplished by leveraging the scrolling mechanism provided by the Android platform. The XML data acquisition unit can handle the scrolling motion and acquire additional XML data for that location whenever a new screen area appears. Collecting XML data from multiple locations in this manner allows for a variety of information spanning the entire dialog box.

[0093] According to another embodiment, the XML data acquisition unit (520) can acquire at least one XML data by changing the position until it reaches the end of the dialog window being executed.

[0094] The XML data acquisition unit (520) can detect the end of the chat window and continuously collect XML data at that location whenever the chat window is scrolled. When the chat window reaches the end, the acquired XML data constitutes a final result, thereby collecting XML data across the entire contents of the chat window, and this information can be utilized for automatic scrolling and capture operations.

[0095] According to another embodiment, an XML data acquisition unit (520) can identify an end position included in a screen capture command and acquire at least one XML data while scrolling the screen until the identified end position is reached.

[0096] The XML data acquisition unit (520) can continuously collect XML data while scrolling to a specified end point when a user specifies a command. Upon reaching the end point, the acquired XML data is synthesized to form a final result. In this way, the dialog box can be automatically scrolled and captured only to a specific point where capture is desired.

[0097] The screen capture image generation unit (530) can generate a screen capture image by integrating the acquired XML data.

[0098] The screen capture image generation unit (530) can interpret information about each UI element of the screen contained in XML data and combine them to create an image representing the entire screen. The position, size, text, and other attributes of the UI elements can be appropriately positioned based on the XML data and converted into an image. Through this image generation process, an image can be formed that visually represents the screen capture requested by the user.

[0099] For example, the screen capture image generation unit (530) can identify common data between consecutive XML data and combine the captured image for the current screen with the captured image for the next screen. In this process, the captured image after the portion corresponding to the common data in the captured image for the next screen can be combined, and the captured images for all remaining XML data obtained can be combined to generate the screen capture image.

[0100] The control unit (540) can be interpreted as a central processing unit (CPU) and can perform various operations and process data within the system.

[0101] In particular, the control unit (540) can read commands from memory, interpret and execute the commands, and perform arithmetic operations such as addition, subtraction, multiplication, and division.

[0102] In addition, the control unit (540) can handle data storage and retrieval, and can also perform the function of reading data from memory and storing the results of performing operations back into memory.

[0103] In addition, the control unit (540) can manage the execution flow of the program, and in particular, can control the flow of the program using commands such as conditional statements (if-else) or iterative statements (for, while). In addition, the control unit (540) can have a small and fast memory device called a register placed inside, and this register can be used to temporarily store data or perform operations.

[0104] The control unit (540) can process and take appropriate action when an external event or exceptional situation occurs, and can quickly access data and instructions by using cache memory that is faster than the main memory.

[0105] In addition, the control unit (540) may use a system bus to communicate with memory or input / output devices, and may provide various power management functions to minimize power consumption.

[0106] Ultimately, by using the present invention, it is possible to automatically scroll and capture a conversation window running on an Android-based portable terminal, and to accurately and quickly process the capture of the conversation window of the portable terminal.

[0107]

[0108] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0109] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0110] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0111] Although the embodiments described above have been described with limited drawings, those skilled in the art will recognize that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0112] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A method for automatically scrolling and capturing a dialog window running on an Android-based portable terminal, Step for receiving screen capture command; A step of obtaining XML data for the current screen of the mobile terminal by utilizing the screen dump function of Android based on the input screen capture command; A step of obtaining at least one XML data while scrolling the screen by a selected number of pixels at a location where XML data for the current screen is obtained; and Step of generating a screen capture image by integrating the XML data obtained above An automatic scroll capture method including:

2. In paragraph 1, The step of obtaining at least one XML data while scrolling the screen by a selected number of pixels at a location where XML data for the current screen is obtained is as follows: A step of obtaining at least one XML data by changing the position to a specific position while scrolling by a selected number of pixels from the nth position where XML data for the current screen is obtained. An automatic scroll capture method including:

3. In paragraph 2, The step of obtaining at least one XML data while scrolling the screen by a selected number of pixels at a location where XML data for the current screen is obtained is as follows: A step of obtaining at least one XML data by changing the position until reaching the end of the dialog box being executed. An automatic scroll capture method characterized by including:

4. In paragraph 2, The step of obtaining at least one XML data while scrolling the screen by a selected number of pixels at a location where XML data for the current screen is obtained is as follows: A step for identifying the end position included in the screen capture command; and A step of obtaining at least one XML data while scrolling the screen until the identified end position is reached. An automatic scroll capture method characterized by including:

5. In paragraph 1, The step of generating a screen capture image by integrating the XML data obtained above is: A step of identifying common data among consecutive XML data; A step of combining a captured image of the current screen with a captured image of the next screen, and combining the captured image after the portion corresponding to the common data in the captured image of the next screen; and Step to create a screen capture image by combining the captured images for all remaining XML data acquired. An automatic scroll capture method characterized by including:

6. In a device that automatically scrolls and captures a conversation window running on an Android-based portable terminal, Screen capture command input section that receives screen capture commands; An XML data acquisition unit that acquires XML data for the current screen of the mobile terminal by utilizing the screen dump function of Android based on the input screen capture command, and acquires at least one XML data while scrolling the screen by a selected number of pixels at a location where the XML data for the current screen is acquired; and A screen capture image generation unit that generates a screen capture image by integrating the XML data obtained above. An automatic scroll capture device including:

7. In paragraph 6, The above XML data acquisition unit, An automatic scroll capture device characterized in that it acquires at least one XML data by changing the position to a specific position while scrolling by a selected number of pixels from the nth position where XML data for the current screen is acquired.

8. In paragraph 7, The above XML data acquisition unit, An automatic scroll capture device characterized in that it acquires at least one XML data while changing the position until it reaches the end of the dialog window being executed.

9. In paragraph 7, The above XML data acquisition unit, An automatic scroll capture device characterized in that it identifies an end position included in a screen capture command and obtains at least one XML data while scrolling the screen until the identified end position is reached.

10. In paragraph 6, The above screen capture image generation unit, An automatic scroll capture device characterized in that it identifies common data among consecutive XML data, combines a capture image for the current screen with a capture image of the next screen, combines a capture image after a portion corresponding to the common data in the capture image of the next screen, and creates a screen capture image by combining capture images for all remaining XML data obtained.

Images