Method and apparatus for automatically scrolling and capturing chat windows running on Android-based mobile devices

Abstract

The present invention relates to a technique for automatically scrolling and capturing a chat window running on an Android-based mobile device. An automatic scrolling capture method according to one embodiment may include the steps of: inputting a screen capture command; obtaining XML data relating to the current screen of the mobile device by utilizing the Android screen dump function based on the input screen capture command; obtaining at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data relating to the current screen was obtained; and integrating the obtained XML data to generate a screen capture image.

Description

Technical Field

[0001] The present invention relates to a technique for automatically scrolling and capturing a chat window executed on an Android-based mobile terminal.

Background Art

[0002] The technique of capturing a screen provides a function of capturing and storing the current screen of a device as an image.

[0003] The screens of monitors and displays are stored in a frame buffer. The frame buffer maintains information about each pixel of the screen in memory.

[0004] Screen capture mainly includes the process of reading data from this frame buffer and storing it as an image.

[0005] Operating systems provide APIs and instruction words that can capture the screen. For example, in Android, the "adb shell screencap" instruction word can be used to capture the screen.

[0006] Some operating systems may also provide APIs that developers can use to take screenshots via programs.

[0007] To capture and store an image, graphics and image processing libraries are used. Such libraries provide various functions and tools necessary for generating and manipulating images.

[0008] For example, in Python, libraries such as PIL (Pillow) and OpenCV are widely used.

[0009] When capturing a screen, you must consider the device's screen resolution and aspect ratio. Failure to do so may result in a distorted or misrepresented image.

[0010] The screen capture process can affect device performance and resource usage. Therefore, it is important to optimize performance by capturing the screen using an appropriate method.

[0011] Some platforms may have restrictions on capturing the screen of other applications due to security and permission issues. In particular, user consent or specific permissions may be required.

[0012] Screen capture technology is used in a variety of environments, and such background technologies must be applied appropriately, taking into account the characteristics of the device and the features of the operating system. [Overview of the project] [Problems that the invention aims to solve]

[0013] The present invention aims to automatically scroll and capture chat windows running on Android-based mobile devices.

[0014] The present invention aims to accurately and quickly process the capture of chat windows on portable terminal devices. [Means for solving the problem]

[0015] An automatic scroll capture method according to one embodiment may include the steps of: inputting a screen capture command; acquiring XML data relating to the current screen of the mobile device by utilizing the Android screen dump function based on the input screen capture command; acquiring at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data relating to the current screen was acquired; and integrating the acquired XML data to generate a screen capture image.

[0016] The step of acquiring at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data relating to the current screen was acquired according to one embodiment may include the step of acquiring at least one XML data while changing the position to a specific position by scrolling by a selected number of pixels from the nth position where the XML data relating to the current screen was acquired.

[0017] The step of acquiring at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data relating to the current screen was acquired according to one embodiment may include the step of acquiring at least one XML data while changing the position until the end of the chat window being executed is reached.

[0018] The step of acquiring at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data relating to the current screen was acquired according to one embodiment may include the step of identifying an end position included in the screen capture command, and the step of acquiring at least one XML data while scrolling the screen until the identified end position is reached.

[0019] The step of integrating the acquired XML data to generate a screen capture image according to one embodiment may include the steps of: identifying common data among consecutive XML data; combining the capture image of the current screen with the capture image of the next screen, but combining the capture image after the portion corresponding to the common data in the capture image of the next screen; and combining the capture images of all the remaining acquired XML data to generate a screen capture image.

[0020] An embodiment of a device for automatically scrolling and capturing a chat window running on an Android-based mobile terminal may include: a screen capture command input unit into which a screen capture command is input; an XML data acquisition unit that, based on the input screen capture command, utilizes the Android screen dump function to acquire XML data relating to the current screen of the mobile terminal, and acquires at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data relating to the current screen was acquired; and a screen capture image generation unit that integrates the acquired XML data to generate a screen capture image.

[0021] The XML data acquisition unit according to one embodiment can acquire at least one XML data item by changing its position from the nth position from which XML data relating to the current screen was acquired, while scrolling by a selected number of pixels to a specific position.

[0022] The XML data acquisition unit according to one embodiment can acquire at least one XML data while changing its position until it reaches the end of the chat window being executed.

[0023] The XML data acquisition unit according to an embodiment can identify an end position included in a screen capture command word and acquire at least one or more XML data while scrolling the screen until the identified end position is reached.

[0024] The screen capture image generation unit according to an embodiment can identify common data between consecutive XML data and combine a capture image for the current screen and a capture image for the next screen. However, it combines the capture image after the portion corresponding to the common data in the capture image of the next screen and combines the capture images for all the remaining XML data acquired to generate a screen capture image.

Advantages of the Invention

[0025] According to an embodiment, it is possible to automatically scroll and capture a chat window executed on an Android-based mobile terminal.

[0026] According to an embodiment, it is possible to accurately and quickly process the capture of a chat window of a mobile terminal.

Brief Description of the Drawings

[0027] [Figure 1] It is a diagram for explaining a method of automatically scrolling and capturing a chat window on an Android-based mobile terminal according to an embodiment. [Figure 2] It is a diagram showing a chat window executed on an Android-based mobile terminal. [Figure 3] It is a diagram showing XML data related to the executed chat window. [Figure 4] It is a diagram showing a state where coordinates are present in XML data. [Figure 5] It is a diagram for explaining an apparatus for automatically scrolling and capturing a chat window on an Android-based mobile terminal according to an embodiment. [Modes for carrying out the invention]

[0028] The specific structural or functional descriptions of embodiments of the concept of the present invention disclosed herein are merely illustrative for the purpose of illustrating embodiments of the concept of the present invention, and embodiments of the concept of the present invention can be implemented in various forms and are not limited to those described herein.

[0029] Embodiments of the concept of the present invention can be modified in various ways and may take on various forms; such embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit embodiments of the concept of the present invention to any particular disclosure, but rather to include modifications, equivalents, or substitutions that fall within the spirit and technical scope of the present invention.

[0030] Terms such as "first" or "second" can be used to describe various components, but the components should not be limited by such terms. The terms are used solely for the purpose of distinguishing one component from another; for example, without departing from the scope of rights under the concept of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0031] When it is mentioned that one component is "linked" or "connected" to another, it must be understood that this may mean that it is directly linked or connected to the other component, but that there may also be other components between those components. Conversely, when it is mentioned that one component is "directly linked" or "directly connected" to another, it must be understood that there are no other components between those components. Expressions describing the relationship between components, such as "between," "immediately between," or "directly adjacent to," must be interpreted in the same way.

[0032] The terms used herein are used solely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “includes” or “having” are intended to specify the existence of a described feature, figure, stage, operation, component, part, or combination thereof, and should not be understood to preemptively exclude the existence or possibility of adding one or more other features, figures, stages, operations, components, parts, or combinations thereof.

[0033] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as those generally understood by a person of ordinary skill in the art to which this invention pertains. Terms as defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as ideal or overly formal unless explicitly defined herein.

[0034] The following examples will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these examples. The same reference numerals shown in each drawing indicate the same component.

[0035] Figure 1 illustrates a method for automatically scrolling and capturing a chat window on an Android-based mobile device according to one embodiment.

[0036] In one embodiment of the automatic scroll capture method, a screen capture command word is input from the user or the program being executed (step 101).

[0037] Next, the automatic scroll capture method according to one embodiment can acquire XML data relating to the current screen of the mobile terminal device by utilizing the Android screen dump function based on the input screen capture command word (step 102), and acquire at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data relating to the current screen was acquired (step 103).

[0038] When a screen capture command is entered by the user or a running program, it can act as a trigger to start automatic scrolling and capture operations.

[0039] To obtain XML data, the Android screen dump function can be used to retrieve XML data about the current screen based on the input screen capture command. The Android screen dump function provides the UI structure of the chat window and the relevant information in XML format.

[0040] The process of automatically scrolling and acquiring XML data can be repeated. Specifically, the acquired XML data can be analyzed, and additional XML data can be acquired while automatically scrolling the chat window, taking into account factors such as the chat window's height and scroll unit. This process can be repeated until pre-set conditions are met or the end of the chat window is reached.

[0041] In other words, the automatic scroll capture method according to one embodiment determines whether or not the set conditions are met (step 104), and if the set conditions are not met, it can acquire XML data while scrolling the screen by a selected number of pixels (step 105).

[0042] If the results of the determination in step 104 meet the set conditions, the automatic scroll capture method according to one embodiment can integrate the acquired XML data to generate a screen capture image (step 106).

[0043] The set conditions could be used to determine whether the current position has changed to a specific position through scrolling.

[0044] For example, to retrieve at least one XML data while scrolling the screen by a selected number of pixels, one can retrieve at least one XML data from the nth position where XML data related to the current screen was obtained, while scrolling by the selected number of pixels to a specific position.

[0045] In this process, at least one XML data can be obtained while changing the position until the end of the chat window being executed is reached.

[0046] Furthermore, in order to obtain at least one XML data while scrolling the screen by a selected number of pixels, the system can identify the end position included in the screen capture command and obtain at least one XML data while scrolling the screen until it reaches the identified end position.

[0047] Furthermore, in order to integrate the acquired XML data and generate a screen capture image, common data is identified between consecutive XML data, and the capture image of the current screen is combined with the capture image of the next screen. In addition, the capture image of the next screen can be combined after the portion corresponding to the common data.

[0048] Furthermore, it is possible to combine the captured images from all the remaining XML data to generate a screen capture image.

[0049] In this process, all the collected information is integrated, and a capture image can be generated by automatically scrolling the remaining part of the screen based on each XML data.

[0050] By considering information such as the position, size, and text of each UI element contained in the XML data, it is possible to combine capture images of various parts. However, the screen portion represented by each XML data is precisely positioned in its corresponding location, and ultimately, a screen capture image of the entire chat window can be generated.

[0051] The images generated in this way can be presented to users visually or used for a variety of purposes, such as automated testing, analysis, and documentation.

[0052] Figure 200 shows a chat window running on an Android-based mobile device.

[0053] Figure 2 is a visual representation of the chat window of an application running in an Android environment.

[0054] Figure 2 illustrates a chat window displayed on the screen of a mobile device. Various UI elements are arranged within the chat window, including buttons, text, and images, which are visually apparent. Furthermore, as shown in Figure 3, a portion of the chat window contains XML data, allowing for the examination of structural information related to that chat window.

[0055] Figure 2 can be used to visually understand and analyze the screen of an application running on an Android device for screen capture purposes.

[0056] Capturing chat windows involves a variety of methods and techniques, primarily using screen capture and image processing technologies.

[0057] Screen capture technology can be used to capture the chat window.

[0058] This invention relates to Android-based portable devices, such as Android smartphones.

[0059] On Android, you can capture the current screen of your mobile device using commands such as "adb shell screencap".

[0060] Image processing techniques can be used to extract specific areas from captured images and analyze the necessary information, but in this invention, XML data related to the current screen can be utilized.

[0061] Certain UI elements, such as screen coordinates and objects within a chat window, can be represented using XML data.

[0062] Additionally, Android provides XML data related to the current screen.

[0063] In this invention, automation tools such as UI Automator and Appium can be used to automate the capture and analysis process using the acquired XML data.

[0064] Figure 3 is a 300-size image showing XML data related to the chat window being executed.

[0065] Figure 3 visually represents the UI structure of the chat window and provides detailed information about each UI element.

[0066] Figure 3 shows a hierarchical list of various UI elements within the chat window. Each UI element displays attributes such as position, size, and text, making it easy to identify its characteristics. Furthermore, the location where the XML data is displayed allows us to see how each element is positioned within the chat window.

[0067] Figure 3 visually represents the information collected by the XML data acquisition unit, making it easy to understand and analyze the structure of the chat window.

[0068] Visualizing XML data allows for a detailed examination of specific parts of a chat window and provides a basis for automated tasks.

[0069] Figure 4 is Figure 400, which shows how coordinates are contained within XML data.

[0070] When capturing while scrolling the screen, if it's the same speech bubble, only the coordinate values ​​will change, and this change in coordinates indicates how much the image has scrolled.

[0071] Figure 4 visually shows the coordinates of a specific UI element extracted from XML data.

[0072] Figure 4 displays coordinate values ​​indicating the position of a specific UI element included in the XML data. These coordinate values ​​generally represent the relative position with respect to the top-left corner of the screen. The X and Y axis values ​​can accurately represent the position of the UI element on the screen.

[0073] Figure 4 allows for a visual confirmation of the coordinate information collected by the XML data acquisition unit, which can be used to understand and analyze the precise location of UI elements. The coordinate information can then be used in automatic scrolling and capture operations.

[0074] Figure 5 illustrates a device 500 that automatically scrolls and captures a chat window on an Android-based portable terminal according to one embodiment.

[0075] An automatic scroll capture device 500 according to one embodiment may include a screen capture command input unit 510, an XML data acquisition unit 520, a screen capture image generation unit 530, and a control unit 540.

[0076] First, the screen capture command input unit 510 receives the screen capture command.

[0077] The screen capture command input unit 510 can receive screen capture commands from the user or via a separate application that is driven by it.

[0078] Furthermore, the commands input to the screen capture command input unit 510 are mainly entered via a terminal or user interface, and the command structure may include specific parameters and options related to screen capture.

[0079] The screen capture command input unit 510 can process the input command and guide the system or application to perform appropriate actions. For example, it may perform tasks such as checking the validity of the command or collecting necessary additional information. It can also interpret the input command to determine the screen capture range, format, or other related settings.

[0080] Next, the XML data acquisition unit 520, based on the input screen capture command, utilizes the Android screen dump function to acquire XML data relating to the current screen of the mobile terminal, and from the position where the XML data relating to the current screen was acquired, it can acquire at least one XML data while scrolling the screen by a selected number of pixels.

[0081] The XML data acquisition unit 520 can extract the UI structure displayed on the screen and detailed information about those elements in XML format during the process of acquiring XML data.

[0082] The XML data acquisition unit 520 calls the screen dump function and processes the resulting XML data to extract the hierarchical structure, attributes, coordinates, and other related information of the UI elements displayed on the screen. The XML data thus acquired can be used as foundational information for automatic scrolling and capture operations in subsequent steps.

[0083] The XML data acquisition unit 520 according to one embodiment can acquire at least one XML data by changing its position from the nth position from which XML data relating to the current screen was acquired, while scrolling by a selected number of pixels to a specific position.

[0084] The XML data acquisition unit 520 can start from the nth position from which XML data related to the current screen has been acquired, and move to a specific position while changing the screen position by a selected number of pixels. Through this process, it can acquire at least one XML data.

[0085] Scrolling to a specific location can be performed by utilizing the scrolling mechanism provided by the Android platform. The XML data acquisition unit can manipulate the scrolling operation and acquire additional XML data for each new screen area that appears. By collecting XML data at multiple locations in this manner, a variety of information across the entire chat window can be obtained.

[0086] An XML data acquisition unit 520 according to another embodiment can acquire at least one XML data while changing its position until it reaches the end of the chat window being executed.

[0087] The XML data acquisition unit 520 can sense the end of the chat window and continuously collect XML data at that position each time the chat window is scrolled. When the end of the chat window is reached, the acquired XML data constitutes the final result, through which XML data is collected from the entire contents of the chat window, and this information can be used for automatic scrolling and capture operations.

[0088] Furthermore, the XML data acquisition unit 520 according to another embodiment can identify the end position included in the screen capture command and acquire at least one XML data while scrolling the screen until the identified end position is reached.

[0089] The XML data acquisition unit 520 can continuously collect XML data while scrolling to the end position specified by the user in the command. Upon reaching the end position, it integrates the acquired XML data to form a final result, and in this manner, the chat window can be automatically scrolled and captured up to a specific point where capture is desired.

[0090] The screen capture image generation unit 530 can generate a screen capture image by integrating the acquired XML data.

[0091] The screen capture image generation unit 530 can generate an image showing the entire screen by interpreting information about each UI element of the screen contained in the XML data and combining it. Attributes such as the position, size, and text of the UI elements can be appropriately arranged based on the XML data and converted into an image. Through this image generation process, an image that visually represents the screen capture requested by the user can be formed.

[0092] For example, the screen capture image generation unit 530 can identify common data among consecutive XML data and combine the captured image of the current screen with the captured image of the next screen. In this process, the captured image of the next screen can be combined with the captured image after the portion corresponding to the common data, and the captured images of all remaining XML data can be combined to generate a screen capture image.

[0093] The control unit 540 can be interpreted as a central processing unit (CPU) and can perform various calculations and process data within the system.

[0094] In particular, the control unit 540 can read instruction words from memory, interpret and execute those instructions, and can also perform arithmetic operations such as addition, subtraction, multiplication, and division.

[0095] Furthermore, the control unit 540 can process data storage and retrieval, and can also read data from memory, perform calculations, and then store the results back into memory.

[0096] Furthermore, the control unit 540 can manage the program execution flow, and in particular, it can control the program flow using instructions such as conditional statements (if-else) and loop statements (for, while). The control unit 540 may also have small and fast memory devices called registers inside, which may be used to temporarily store data or perform calculations.

[0097] The control unit 540 can process external events and exceptional situations, take appropriate measures, and quickly access data and instructions using cache memory, which is faster than main memory.

[0098] Furthermore, the control unit 540 can use the system bus to communicate with memory and input / output devices, and can provide a variety of power management functions to minimize power consumption.

[0099] In short, the present invention enables automatic scrolling and capture of chat windows running on Android-based mobile devices, and allows for accurate and rapid processing of chat window captures on mobile devices.

[0100] The devices described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, controller, ALU (arithmetic logic unit), digital signal processor, microcomputer, FPA (field programmable array), PLU (programmable logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device can run an operating system (OS) and one or more software applications run on the operating system. The processing device can also access, store, manipulate, process, and generate data in response to software execution. For convenience of understanding, the processing device may have been described as being used as one, but a person with ordinary skill in the art will see that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Furthermore, other processing configurations, such as parallel processors, are also possible.

[0101] Software may include computer programs, code, instructions, or a combination of one or more of these, which can configure a processing unit to operate as desired, or independently or collectively, instruct the processing unit. Software and / or data may be permanently or temporarily embodied in a certain type of machine, component, physical device, virtual device, computer storage medium or device, or transmitted signal wave, in order to be interpreted by a processing unit or to provide instructions or data to a processing unit. Software may be distributed on a networked computer system and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0102] The methods according to the embodiments may be embodied in the form of program instructions executable via various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be specifically designed and configured for the embodiments, or may be publicly known and available to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code executable by a computer using an interpreter or the like. The hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0103] As described above, although the embodiments have been explained with limited drawings, various modifications and variations can be made from the above description by a person with ordinary skill in the art. For example, the described technique may be performed in a different order than described, and / or the components of the described system, structure, apparatus, circuit, etc. may be combined or assembled in a different manner than described, or substituted or replaced by other components or equivalents, and satisfactory results may be achieved.

[0104] Therefore, other embodiments, other embodiments and equivalents of the claims also fall within the scope of the attached claims.

Claims

1. A method for automatically scrolling and capturing a chat window running on an Android-based mobile device, The step in which the screen capture command word is entered, The steps include: obtaining XML data relating to the current screen of the mobile device by utilizing the Android screen dump function based on the input screen capture command; The steps include: obtaining at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data for the current screen was obtained; An automatic scroll capture method, comprising the step of integrating the acquired XML data to generate a screen capture image.

2. The step of obtaining at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data for the current screen was obtained is as follows: The automatic scroll capture method according to claim 1, comprising the step of acquiring at least one XML data while changing the position to a specific position by scrolling by a selected number of pixels, starting from the nth position from which XML data relating to the current screen has been acquired.

3. The step of obtaining at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data for the current screen was obtained is as follows: The automatic scroll capture method according to claim 2, characterized by including the step of acquiring at least one XML data while changing the position until it reaches the end of the chat window being executed.

4. The step of obtaining at least one XML data while scrolling the screen by a selected number of pixels from the position where the XML data for the current screen was obtained is as follows: A step of identifying the end position included in the screen capture command, The automatic scroll capture method according to claim 2, characterized by comprising the step of acquiring at least one XML data while scrolling the screen until it reaches the identified end position.

5. The step of integrating the acquired XML data to generate a screen capture image is: Steps include identifying common data between consecutive XML data, The process involves combining a captured image of the current screen with a captured image of the next screen, wherein the captured image of the next screen includes a step of combining the captured image after the portion corresponding to the common data in the captured image of the next screen. The automatic scroll capture method according to claim 1, characterized by comprising the step of generating a screen capture image by combining the captured images of all remaining XML data that have been acquired.

6. A device that automatically scrolls and captures chat windows running on Android-based mobile devices, A screen capture command input section into which a screen capture command is entered, An XML data acquisition unit that, based on the input screen capture command, uses the Android screen dump function to acquire XML data relating to the current screen of the mobile terminal, and then scrolls the screen by a selected number of pixels from the position where the XML data relating to the current screen was acquired, acquiring at least one XML data; An automatic scroll capture device, including a screen capture image generation unit that integrates the acquired XML data to generate a screen capture image.

7. The XML data acquisition unit is, The automatic scroll capture device according to claim 6, characterized in that, starting from the nth position from which XML data relating to the current screen is obtained, the device changes its position by scrolling by a selected number of pixels to a specific position and obtains at least one XML data.

8. The XML data acquisition unit is, The automatic scroll capture device according to claim 7, characterized in that it acquires at least one XML data while changing the position until it reaches the end of the chat window being executed.

9. The XML data acquisition unit is, The automatic scroll capture device according to claim 7, characterized in that it identifies an end position included in a screen capture command and acquires at least one XML data while scrolling the screen until the identified end position is reached.

10. The aforementioned screen capture image generation unit, The automatic scrolling capture device according to claim 6, characterized in that it identifies common data between consecutive XML data, combines a captured image of the current screen with a captured image of the next screen, combines the captured image after the portion corresponding to the common data in the captured image of the next screen, and combines the captured images of all remaining XML data to generate a screen capture image.

Images