Method and automatic layout module for responsive automatic layout of industrial process graphics

Abstract

The present invention relates to a method for automatically laying out graphic objects and an automatic layout module for a human-machine interface. The method comprises the following steps: receiving (S1) input data, the input data comprising input image data, library data and screen parameter data; and analyzing (S2) the received input data, and creating output image data based on reshaping of the input image data, the reshaping of the input image data being based on the library data and the screen parameter data, the output image data being optimized for a specific screen defined by the screen parameter data.

Description

Technical Field

[0001] The present invention relates to responsive automatic layout of industrial process graphics. In particular, the present invention relates to a method for responsive automatic layout of industrial process graphics. Background Art

[0002] Today, automation engineers create process graphics for specific screen sizes. This results in a poor user experience when the graphics are displayed on a screen with different characteristics (e.g., on a mobile device). Industrial process graphics visualize industrial production processes for human operators and allow interaction with related automation equipment (e.g., sensors and actuators). For example, such process graphics include visualization of tanks, pipelines, valves, sensors, trend graphs, and alarm conditions.

[0003] Today, automation engineers manually create these graphics for specific screen sizes. For example, a process graphic may be created for a large overview screen in an operator control room and may span multiple adjacent screens. In the alternative, a process graphic is optimized for use in a single workstation.

[0004] Today, field personnel use computer devices with a variety of different screen sizes, resolutions, and attributes when interacting with large machines on site. The computer devices include smart watches, smartphones, tablet computers, smart TV devices, and even cross-wall displays. Displaying process graphics designed for different screens on these devices, such as via remote desktop or other means, can lead to cumbersome and undesirable interactions with the graphics.

[0005] Operators may need to use scroll bars extensively, pin and zoom graphics, and may not even be able to activate the graphic elements they want to interact with because the activation area is small. Fonts showing sensor data may be unreadable on smaller screens, and the overview of the plant may be lost on smaller screens. This can be even worse if field personnel are wearing protective gloves or goggles.

[0006] Designing screens of different sizes is expensive, and automation engineers cannot foresee all possible screen variations that will emerge in the consumer electronics market.

[0007] In other fields, responsive web design is a common approach to resize web pages for smaller screen sizes. While these approaches can reduce the size of images, fonts, or tables, or even suppress specific graphic elements, they do not exploit the details of process graphics that allow for reasonable reshaping of the graphics by leveraging domain knowledge. Summary of the invention

[0008] There exists a need to overcome the shortcomings from existing automatic layout implementations of video or image graphics content.

[0009] This need is met by the subject matter of the independent claims. Further exemplary embodiments are evident from the dependent claims and the following description.

[0010] The invention features a so-called "HMI Auto Layouter software" component that can convert existing process graphics to different screen characteristics. The abbreviation "HMI" stands for Human Machine Interface. The Auto Layouter exploits domain knowledge about process graphics and can not only resize graphic and text elements, but also change interaction points. This reduces the cost of providing displays for different screen sizes and helps field personnel to perform operating and maintenance tasks more efficiently.

[0011] A first aspect of the present invention relates to a method for automatically laying out a graphic object, the method comprising the steps of: receiving input data, the input data comprising input image data, library data and screen parameter data. Furthermore, the method for automatically laying out a graphic object comprises the steps of: analyzing the received input data, and creating output image data based on a reshaping of the input image data, the reshaping of the input image data being based on the library data and the screen parameter data, the output image data being optimized for a specific screen defined by the screen parameter data.

[0012] The present invention advantageously uses a software component denoted as an HMI automatic layouter that can convert a process graphic designed for a specific screen into a process graphic optimized for a different screen. The HMI automatic layouter exploits knowledge of graphic shapes used in typical process graphics to reshape and resize the graphics in a reasonable manner.

[0013] Based on the present invention, for input data, the HMI automatic layout receives:

[0014] i) Process graphics designed for a specific size (e.g., desktop, 3840x2160, UHD); and / or

[0015] ii) a library of process graphic shapes and best practices for resizing (e.g. shortening pipeline shapes); and / or

[0016] iii) desired screen characteristics (e.g., resolution such as 2436x1125, display size such as a 6-inch phone or a 10-inch tablet, interaction means such as touch screen or non-touch screen); and / or

[0017] iv) classification of visual shapes into priority categories (essential vs. optional), such as provided via process graphic design tools from automation engineers; and / or

[0018] v) Contextual information about the task to be solved using the graphics, or the current situation of the field operator (e.g., geographic coordinates, service records, or parameters).

[0019] The HMI automatic layout analyzes the process graphics and identifies visual shapes that are candidates for reshaping. If a classification of shape priority categories is not available, a best practice library for reshaping can be referenced. The HMI automatic layout then creates a new process graphic by manipulating the existing process graphic. This creation can be, for example:

[0020] i) Reset font size, eliminate text fields, create subsuming text fields, etc.

[0021] ii) Reshape pipeline visuals, reduce / increase tank, pump, motor, turbine, compressor visual sizes

[0022] iii) Eliminate the trend graph, or introduce a placeholder or button for the trend graph

[0023] iv) Reshape the actuator vision for simpler interaction (e.g. increase size for easier selection)

[0024] v) Compress or expand visuals from multiple process graphic hierarchies

[0025] vi) Select graphical elements according to specific tasks (e.g. root cause analysis in a plant section)

[0026] As output, the HMI automatic layouter creates an optimized process graphic for a specific screen and optionally for a specific task context.

[0027] According to one embodiment of the present invention, inputting image data includes receiving a process graphic designed for a specific screen size, screen resolution, or screen type.

[0028] According to one embodiment of the present invention, the library data includes: library data containing data on process graphic shapes and data on best practices for reshaping input image data.

[0029] According to one embodiment of the present invention, the screen parameter data includes data regarding desired screen characteristics.

[0030] According to one embodiment of the present invention, the data on desired screen characteristics includes data on desired screen resolution, data on desired screen size, data on desired screen type, or data on desired interaction means to be coupled to the screen.

[0031] According to an embodiment of the present invention, the method further comprises the step of generating a classification of visual shapes in at least one priority category for use in reshaping the input image data based on the library data and the screen parameter data.

[0032] According to one embodiment of the present invention, the method further comprises the step of receiving context information about a task to be solved using the displayed graphical object.

[0033] According to one embodiment of the present invention, the method further comprises the following steps: receiving context information about a user mode controlled by a human-machine interface comprising a graphic object.

[0034] According to an embodiment of the invention, the method further comprises the step of identifying graphical objects that are candidates for reshaping.

[0035] According to one embodiment of the present invention, the reshaping of the input image data includes at least one of the following:

[0036] - resizing of fonts of input image data;

[0037] - Elimination of text fields;

[0038] - Create merged text fields;

[0039] -Reshape pipeline visuals;

[0040] - Reduce the size of the object;

[0041] - Increase the size of the object;

[0042] - Eliminate trend graphs;

[0043] -Introducing placeholders;

[0044] - Introducing buttons for placeholders; and

[0045] - Reshape the dialog box.

[0046] A second aspect of the present invention relates to an automatic layout module for a human-machine interface, the automatic layout module comprising a receiver configured to receive input data, the input data comprising input image data, library data and screen parameter data. The automatic layout module comprises a processor configured to: analyze the received input data, and create output image data based on a reshaping of the input image data, the reshaping of the input image data being based on the library data and the screen parameter data, the output image data being optimized for a specific screen defined by the screen parameter data.

[0047] According to one embodiment of the present invention, the receiver is configured to receive input image data, and receiving the input image data includes receiving a process graphic designed for a specific screen size, screen resolution or screen type.

[0048] According to one embodiment of the present invention, the receiver is configured to receive library data, the library data comprising: library data containing data on process graphic shapes and data on best practices for reshaping of input image data.

[0049] According to one embodiment of the invention, the receiver is configured to receive screen parameter data, the screen parameter data comprising data regarding desired screen characteristics.

[0050] According to one embodiment of the present invention, the data on desired screen characteristics includes data on desired screen resolution, data on desired screen size, data on desired screen type, or data on desired interaction means to be coupled to the screen.

[0051] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Hereinafter, embodiments of the present invention are described in more detail with reference to the accompanying drawings.

[0053] Figure 1 A schematic diagram showing a flow chart of a method for automatically laying out graphic objects according to an embodiment of the present invention.

[0054] Figure 2 A schematic diagram of an automatic layout module according to an embodiment of the present invention is shown.

[0055] Figure 3 A data flow chart of a method for automatically laying out graphic objects according to an embodiment of the present invention is shown.

[0056] In principle, the same parts have the same reference numerals in the drawings. The present invention is described in detail below by using examples with reference to the following drawings. The following description will be combined with the drawings in the embodiments of the present invention to provide a clear and complete description of the technical solutions in the embodiments of the present invention.

[0057] Obviously, the described embodiments are only some embodiments of the present invention rather than all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative efforts should fall within the protection scope of the present invention. DETAILED DESCRIPTION

[0058] Figure 1A schematic diagram showing a flow chart of a method for automatically laying out graphic objects according to an embodiment of the present invention.

[0059] exist Figure 1 The method for automatically laying out graphical objects as depicted in the example comprises the following steps:

[0060] As a first step, receiving S1 input data is performed, the input data including input image data, library data and screen parameter data.

[0061] As a second step, analysis S2 of the received input data is performed and output image data is created based on a reshaping of the input image data based on the library data and the screen parameter data, the output image data being optimized for a specific screen defined by the screen parameter data.

[0062] According to an exemplary embodiment of the present invention, the reshaping of the input image data based on the library data and the screen parameter data may, for example, include taking into account the type of each input image data received and processed.

[0063] According to an exemplary embodiment of the present invention, the reshaping of the input image data may include: specific reshaping routines for certain types of input image data.

[0064] According to an exemplary embodiment of the present invention, the pipeline or rectangle is shortened or extended in one dimension (i.e., the width, height or length of the displayed graphical element is scaled), rather than scaling the displayed element in two dimensions. Therefore, with the help of the HMI automatic layouter, the screen parameter data can be used to achieve advantageous adaptation to different aspect ratios of different display types.

[0065] According to an exemplary embodiment of the present invention, with respect to the automatic layout module, the HMI automatic layout can change the layout of the graphics to adapt to the desired screen. For example, as the received input data, the HMI automatic layout receives at least one of the following input data:

[0066] a. Information about the device type (and available input methods) and screen size

[0067] b. Information about user roles, context, usage history and preferences

[0068] c. Definition of the graphic display to be shown, comprising:

[0069] i. Visual Information

[0070] ii. Structural information (hierarchical composition of the display)

[0071] iii. Topology information (connections between components on the display)

[0072] iv. Navigation information (links to other displays)

[0073] v. Contextual information (links to other views and functionality in the system (i.e., non-process graphics))

[0074] d. Prioritize shapes and subcomponents by user role, preferences, and usage history.

[0075] e. Libraries of shapes and subcomponents used.

[0076] According to an exemplary embodiment of the present invention, the HMI automatic layouter uses the information described above to create one or several new graphic displays according to the following manner:

[0077] i) Filtering of visual elements on the display by priority of visual elements based on user role, context / task and preferences

[0078] ii) Additional visual elements can be added based on context, topological information, structural information and priority

[0079] iii) the conforming shapes are adjusted to ensure usability (e.g., ensuring minimum font size) and

[0080] iv) Navigation links to omitted elements or sub-sections of the display may be inserted

[0081] v) If available in the library, shapes and components can be replaced with the variant that best suits the device type

[0082] vi) The topology of the remaining components can be adjusted to optimize the available space (e.g., a graph layout algorithm can be used)

[0083] vii) Interact with the display, and the shapes and elements shown can be adjusted to suit the device type

[0084] According to an exemplary embodiment of the present invention, the HMI automatic layouter sends the created one or more graphic display files to the operator's device (eg, a phone or tablet computer).

[0085] According to an exemplary embodiment of the invention, the layout functionality runs directly on the device (eg, input to the HMI automatic layouter is sent to the client device as described above).

[0086] Figure 2 A schematic diagram of an automatic layout module according to an embodiment of the present invention is shown.

[0087] As in Figure 2The automatic layout module 100 depicted in FIG. 1 can be used for a human-machine interface, and the automatic layout module includes a receiver 10 and a processor 20 .

[0088] The receiver 10 is configured to receive input data, the input data including input image data, library data and screen parameter data.

[0089] The processor 20 is configured to analyze the received input data and create output image data based on a reshaping of the input image data based on the library data and the screen parameter data, the output image data being optimized for a specific screen defined by the screen parameter data.

[0090] According to an exemplary embodiment of the present invention, an automatic layout version of a graph is implemented for multiple applications depending on the device types, users, tasks available and stored in a library database. According to an exemplary embodiment of the present invention, a subset of known device types, users and tasks are pre-created.

[0091] According to an exemplary embodiment of the invention, the priority of shapes and elements for a user and context is adjusted over time to reflect usage history (eg, frequently used elements or frequently followed navigation links are elevated in priority).

[0092] According to an exemplary embodiment of the present invention, the HMI automatic layout may be configured with a set of rules and expressions that further specify its behavior (eg, always replace online trends for tanks with numerical measurement representations).

[0093] Figure 3 A data flow chart of a method for automatically laying out graphic objects according to an embodiment of the present invention is shown.

[0094] It must be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims, whereas other embodiments are described with reference to apparatus type claims.

[0095] However, it will be apparent to those skilled in the art from the above and following descriptions that, unless otherwise specified, any combination of features belonging to one category of subject matter, any combination of features relating to different subject matter is also considered to be disclosed with the present application. However, all features may be combined to provide a synergistic effect that is not merely the simple sum of the features.

[0096] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative or exemplary rather than restrictive; the present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art and those who practice the present invention from a study of the drawings, the present disclosure, and the appended claims.

[0097] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or controller or other unit may fulfill the functions of several items recited in the claims. The fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A method for automatically laying out graphic objects, the method comprising the following steps: - receiving input data, the input data comprising input image data, library data and screen parameter data; as well as - analyzing the received input data and creating output image data based on a reshaping of the input image data, the reshaping of the input image data being based on the library data and the screen parameter data, the output image data being optimized for a screen defined by the screen parameter data; The library data includes: library data including data on process graphic shapes and data on best practices for reshaping the input image data.

2. The method according to claim 1, The input image data includes receiving process graphics, wherein the process graphics are designed for a screen size, a screen resolution or a screen type.

3. The method according to any one of claims 1 to 2, The screen parameter data includes data on desired screen characteristics.

4. The method according to claim 3, The data about the desired screen characteristics include: Data about a desired screen resolution, data about a desired screen size, data about a desired screen type, or data about desired interaction means to be coupled to the screen.

5. The method according to any one of claims 1 to 2, The method further comprises the following steps: A classification of visual shapes in at least one priority category is generated for use in reshaping the input image data based on the library data and the screen parameter data.

6. The method according to any one of claims 1 to 2, The method further comprises at least one of the following steps: receiving contextual information about a task to be solved using the displayed graphical object; and / or Contextual information about a user mode controlled by a human-machine interface including the graphical object is received.

7. The method according to any one of claims 1 to 2, The method further comprises the following steps: Identifies graphics objects that are candidates for reshaping.

8. The method according to claim 7, The reshaping of the input image data comprises at least one of the following: resizing of the font of the input image data; Elimination of text fields; Create a merged text field; Reshape pipelines; Reduce the size of the object; Increase the size of the object; Eliminate trend graphs; Introduce placeholders; introducing a button for said placeholder; and Reshape the dialog box.

9. The method of claim 1 , wherein analyzing the received input data and creating output image data based on a reshaping of the input image data comprises: A classification based on shape priority categories is not available, using library data on best practices for remodeling.

10. An automatic layout module for a human-machine interface, the automatic layout module comprising: - a receiver configured to receive input data, the input data comprising input image data, library data and screen parameter data; as well as - a processor configured to analyze the received input data and create output image data based on a reshaping of the input image data, the reshaping of the input image data being based on the library data and the screen parameter data, the output image data being optimized for a screen defined by the screen parameter data; The library data includes: library data including data on process graphic shapes and data on best practices for reshaping the input image data.

11. The automatic layout module according to claim 10, The receiver is configured to receive the input image data, and receiving the input image data comprises: A process graphic designed for a screen size, screen resolution, or screen type is received.

12. The automatic layout module according to any one of claims 10 to 11, The receiver is configured to receive the screen parameter data, the screen parameter data comprising data regarding desired screen characteristics.

13. The automatic layout module according to claim 12, The data about the desired screen characteristics include: Data about a desired screen resolution, data about a desired screen size, data about a desired screen type, or data about desired interaction means to be coupled to the screen.

Images