Screen creation assistance device and computer-readable recording medium

Abstract

A screen creation assistance device according to the present disclosure comprises: an attribute information determination unit that determines whether a display component can be displayed in a display area on the basis of attribute information indicating the attribute of a display component displayed in the display area and a display condition related to the attribute information set to the display area; a prompt creation unit that, when determined to be not displayable by the attribute information determination unit, creates a prompt for updating the attribute information of the display component on the basis of the attribute information and the display condition; a transmission unit that transmits the prompt to an interactive response device; a reception unit that receives a response including the display component the attribute information of which is updated by the interactive response device; and an output unit that displays the updated display component. The attribute information determination unit determines whether the updated display component can be displayed in the display area each time the reception unit receives a response including the updated display component from the interactive response device.

Description

Screen creation support device and computer-readable recording medium

[0001] The present disclosure relates to a screen creation support device and a computer-readable recording medium.

[0002] When displaying display components that make up a screen on a display device, the size of the display device must be taken into consideration. For example, the size of text and graphics used in the display components must be reduced or enlarged to fit the size of the display device (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2008-250962

[0004] When displaying display components of a size that does not fit the display area of ​​a display device, if the size of the display area is too small compared to the size of the display components, the user may not be able to see the characters and other elements on the display components that have been reduced in size to fit the display area. To create an appropriate screen, it is necessary to modify the display components through trial and error to match the size of the display area, and adjusting their placement and size requires a lot of work. In the field, there is a demand for technology that can appropriately change display components to match the size of the display area.

[0005] The screen creation support device disclosed herein solves the above problem by checking whether display is possible based on attribute information such as the size of the graphics and text size of each display component in relation to the display conditions (size, etc.) of the display area, and if it determines that display is not possible, using a generation AI such as an AI image generation tool to input the display conditions of the display area and the attribute information of the display components (text information, graphic information, number of display components, etc.) to generate simplified display components that match the display conditions of the display area.

[0006] One aspect of the present disclosure is a screen creation support device including: an attribute information determination unit that determines whether a display component can be displayed in a display area based on attribute information indicating attributes of the display component and display conditions related to the attribute information set for the display area; a prompt creation unit that creates a prompt for updating the attribute information of the display component based on the attribute information and the display conditions if the attribute information determination unit determines that the display component cannot be displayed; a transmission unit that transmits the prompt to an interactive response device; a reception unit that receives a response including a display component whose attribute information has been updated by the interactive response device; and an output unit that displays the updated display component, wherein the attribute information determination unit determines whether the updated display component can be displayed in the display area every time the reception unit receives a response including an updated display component from the interactive response device.

[0007] FIG. 1 is a schematic hardware configuration diagram of a screen creation support device according to an embodiment of the present disclosure; FIG. 2 is a block diagram showing schematic functions of a screen creation support device according to a first embodiment; FIG. 3 is a schematic diagram showing an example of reducing a display component; FIG. 4 is a schematic diagram showing examples of a plurality of templates stored in a template storage unit; FIG. 5 is a schematic diagram showing an example of a template; FIG. 6 is a schematic diagram showing an example of a prompt; FIG. 7 is a schematic diagram showing an example of a response of an interactive response device; FIG. 8 is a schematic diagram showing another example of a response of an interactive response device; FIG. 9 is a schematic diagram showing another example of a prompt.

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following description, components having the same or similar functions will be denoted by the same reference numerals. Duplicate descriptions of those components may be omitted.

[0009] In this application, "based on XX" means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0010] 1 is a schematic hardware configuration diagram showing the main parts of a screen creation support device according to a first embodiment of the present disclosure. The screen creation support device 1 according to this embodiment can be implemented on a computer such as a personal computer attached to a control device, a personal computer connected to the control device via a wired / wireless network, a cell computer, a fog computer 6, or a cloud server 7. This embodiment shows an example in which the screen creation support device 1 is implemented on a computer connected via a network to a control device that controls industrial machinery.

[0011] The CPU 11 provided in the screen creation support device 1 according to this embodiment is a processor that controls the entire screen creation support device 1. The CPU 11 reads a system program stored in the ROM 12 via the bus 22 and controls the entire screen creation support device 1 in accordance with the system program. The RAM 13 temporarily stores temporary calculation data, display data, various data acquired from outside, and the like.

[0012] The non-volatile memory 14 is composed of, for example, a battery-backed memory or an SSD (Solid State Drive) (not shown), and retains its stored state even when the screen creation support device 1 is powered off. The non-volatile memory 14 stores programs and data read from an external device 72 via the interface 15, programs and data input via the input device 71, and programs and data acquired from the control device 3 that controls the industrial machine 4 or other devices via the network 5. The programs and data stored in the non-volatile memory 14 may be expanded into the RAM 13 when executed / used. In addition, various system programs such as known analysis programs are written in the ROM 12 in advance.

[0013] The interface 15 is an interface for connecting the CPU 11 of the screen creation support device 1 to an external device 72 such as a USB device. For example, system programs, setting data, etc. are read from the external device 72. In addition, programs, setting data, etc. created or edited within the screen creation support device 1 can be stored in external storage means via the external device 72.

[0014] The interface 20 is an interface for connecting the CPU 11 of the screen creation support device 1 to a wired or wireless network 5. The network 5 may communicate using technologies such as serial communication such as RS-485, Ethernet (registered trademark), optical communication, wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. The network 5 is connected to a control device 3 that controls at least one industrial machine 4 to be verified, an interactive response device 2 that responds to inquiries from users, a fog computer 6, a cloud server 7, etc., and exchanges data with the screen creation support device 1.

[0015] The display device 70 displays data and the like obtained as a result of executing various data and programs loaded into the memory, output via the interface 17. An input device 71, which is composed of at least one input device such as a keyboard, a pointing device, a voice input device, an imaging device, etc., passes commands, data, etc. based on user operations to the CPU 11 via the interface 18.

[0016] The interactive response device 2 is configured as a device that returns data such as response sentences, images, and display components in response to predetermined sentences and data. The interactive response device 2 is equipped with a multi-modal large-scale language model (LLM: Large Language Models) that has learned data such as response sentences, images, and display components in response to predetermined sentences and data. This model can be configured by combining known models such as CLIP (Contrastive Language-Image Pre-training) and GANs (Generative Adversarial Networks). These models can be combined to encode display components provided as input and then decode them into desired display components. In learning, depending on how data such as sentences, images, and display components that respond to specified sentences or data is learned, the interactive response device 2 can be used to convert attribute information in data related to specified sentences, images, and display components, or to convert it into other types of data with a similar concept (for example, a character string into an image).

[0017] The screen creation support device 1 according to the present disclosure is assumed to be connected to an interactive response device 2 via a network 5. This interactive response device 2 has a model that has learned conversion of data such as text containing attribute information of display components that constitute a user interface and display components. If necessary, additional learning (fine tuning) may be performed in advance using data related to display components specialized for a control device associated with an industrial machine 4 as learning data. By narrowing down the information used for learning, it is possible to prepare an interactive response device 2 that returns responses to inquiries with a certain degree of accuracy.

[0018] 2 is a schematic block diagram illustrating functions of the screen creation support device 1 according to the first embodiment of the present disclosure. Each function of the screen creation support device 1 according to this embodiment is realized by the CPU 11 included in the screen creation support device 1 shown in FIG. 1 executing a system program and controlling the operation of each unit of the screen creation support device 1.

[0019] The screen creation support device 1 of this embodiment includes a display area identification unit 100, a display component identification unit 110, an attribute information determination unit 120, a prompt creation unit 130, a transmission unit 140, a reception unit 150, and an output unit 190. The RAM 13 to the non-volatile memory 14 of the screen creation support device 1 also include a template storage unit 200, which is an area in which templates related to prompts are stored in advance.

[0020] The display area identification unit 100 identifies a display area in which display components are to be displayed. The display area identified by the display area identification unit 100 may be, for example, the entire display area of ​​the display device 70 or a partial area within the screen displayed on the display device 70. The display area identification unit 100 may identify the display area based on a user specification. Alternatively, the display area may be identified based on a specification by an application that displays the screen. Display conditions related to attributes such as the type, size, resolution, dot spacing, display color, brightness, contrast ratio, and update cycle of the display area are set for the display area. These display conditions may fix the values ​​of each attribute. The display area identification unit 100 also outputs information related to the identified display area to the attribute information determination unit 120.

[0021] The display element identification unit 110 identifies display elements to be displayed in the display area identified by the display area identification unit 100. The display elements identified by the display element identification unit 110 may be elements such as text, figures, photos, videos, buttons, and backgrounds. Furthermore, the display elements may be composite elements that combine these elements. The display element identification unit 110 may identify display elements based on a user specification. Furthermore, the display element identification unit 110 may identify display elements based on a specification by an application that displays the screen. The display elements include attribute information such as the shape (information related to the image element), size, display color, update cycle, characters, number of characters, and text size of the display element. The display element identification unit 110 outputs information related to the identified display elements to the attribute information determination unit 120.

[0022] The attribute information determination unit 120 determines whether or not the display element can be displayed in the display area based on the attribute information of the display element identified by the display element identification unit 110 and the display conditions for the display area identified by the display area identification unit 100. If it is determined that the display element can be displayed, it outputs a message to that effect to the output unit 190. On the other hand, if it is determined that the display element cannot be displayed, it instructs the prompt creation unit 130 to create a prompt for updating the attribute information so that the display item can be displayed in the display area.

[0023] When the attribute information of a display component is changed in accordance with the display conditions for the display area, the attribute information determination unit 120 determines that the display component can be displayed in the display area if the changed attribute information is within a predetermined tolerance range. Furthermore, if the attribute information deviates from the tolerance range, the attribute information determination unit 120 determines that the display component cannot be displayed in the display area. This determination process by the attribute information determination unit 120 will be described with reference to FIG. 3 . FIG. 3 is a schematic diagram showing an example of reducing a display component. In the example of FIG. 3 , reference numeral 320 denotes a display component that uses images and text to depict three motors lined up vertically. In the display component 320, each motor is represented by an image. Furthermore, text relating to the shaft to which the motor is attached is attached above each motor. The attribute information of this display component 320 is assumed to be 8 cm in height, 6 cm in width, and 2 cm in text size. Consider a case in which this display component 320 is displayed in a display area with display conditions set to "height of 2 cm or less," "width of 6 cm or less," and "text size of 1 cm or more." In this case, the vertical size of the display element 320 must be reduced to 2 cm in order to display it. Therefore, a normal method is used to reduce the size of the display element 320, creating a display element 322 with a vertical size of 2 cm. In this case, the size of the display element 322 is 2 cm in height and 1.5 cm in width. The character size is also 0.5 cm. This character size of 0.5 cm does not satisfy the display condition set for the display area, which is "character size 1 cm or more." In such a case, the attribute information determination unit 120 determines that the display element cannot be displayed in the display area.

[0024] The prompt creation unit 130 creates a prompt that can be input to the interactive response system 2, for updating the attribute information of a display component that the attribute information determination unit 120 has determined cannot be displayed without changing the meaning of the display component. The prompt creation unit 130 creates the prompt based on the attribute information of the display component and the display conditions set in the display area. The prompt creation unit 130 may assign prompt identification information that allows the created prompt to be uniquely identified. The prompt creation unit 130 may create the prompt by applying the attribute information of the display component and the display conditions set in the display area to a prompt template corresponding to, for example, inquiry information. The prompt creation unit 130 outputs the created prompt to the transmission unit 140.

[0025] 4 is a schematic diagram showing an example of multiple templates stored in the template storage unit 200. In the example of FIG. 4, the template storage unit 200 stores multiple templates 1, 2, .... Each template corresponds to a type of display area or a type of display component. The prompt creation unit 130 selects a template stored in the template storage unit 200 according to the type of display area or the type of display component. Then, the selected template is used to create a prompt.

[0026] FIG. 5 is a schematic diagram showing an example of a template. In the example of FIG. 5, the template includes information keys for embedding predetermined information at predetermined positions in the prompt text. The prompt creation unit 130 creates a prompt by embedding corresponding information at each information key position. In the example of FIG. 5, display conditions set in the display area are embedded at the positions of the information keys "DISPLAY_CONDITION1," "DISPLAY_CONDITION2," etc. In addition, display component data is attached at the position of the information key "DISPLAY_PART." Furthermore, attribute information of the display components is embedded at the positions of "DISPLAY_ATTRIBUTE1," "DISPLAY_ATTRIBUTE2," etc.

[0027] The template may include a statement specifying attribute information that must not be changed when updating a display component. For example, if the color of an image component included in the original display component is yellow and you want to maintain the same display color in the updated display component, you can include a statement instructing that the display color of the image component not be changed in the template.

[0028] Figure 6 is a schematic diagram showing an example of a prompt created based on a template. The prompt in Figure 6 requests updating of attribute information of a display component while maintaining its display meaning, in order to display an attached display component with attributes such as a height of 8 cm, a width of 6 cm, and a character size of 2 cm in a display area with display conditions such as a height of 2 cm or less, a width of 6 cm or less, and a character size of 1 cm or more. This prompt is created by embedding the attribute information of the display component, the display conditions set for the display area, and the data of the display component into the template shown in Figure 5.

[0029] The templates are created in advance to match the format of inquiries to the interactive response system 2. For example, if the interactive response system 2 responds to inquiries in the form of a list of items, the templates are preferably for creating prompts that list each of the items. Also, if the interactive response system 2 responds to inquiries in the form of natural language inquiries, the templates are preferably for creating prompts in the form of natural sentences. Each template is preferably created to include information that can be used as a reference when creating responses to inquiries related to expected display components.

[0030] The transmission unit 140 transmits the prompt created by the prompt creation unit 130 to the interactive response apparatus 2. Upon receiving the prompt from the transmission unit 140, the interactive response apparatus 2 generates a response to the received prompt. Then, the generated response is transmitted to the screen creation assistance device 1.

[0031] The receiving unit 150 receives the response transmitted from the interactive response apparatus 2. This response includes data related to the updated display components and attribute information generated by the interactive response apparatus 2. Figures 7 and 8 are schematic diagrams illustrating responses from the interactive response apparatus 2. The receiving unit 150 outputs the received response to the attribute information determining unit 120.

[0032] Here, the attribute information determination unit 120 determines whether the updated display component included in the response received by the receiving unit 150 can be displayed in the display area. This is because the response generated by the generation AI may contain errors or fluctuations, and the display component updated by the interactive response device 2 may not necessarily satisfy the display conditions set for the display area. This determination may be made, as in the initial determination, based on the data and attribute information related to the updated display component and the display conditions related to the display area identified by the display area identification unit 100. If the updated display component can be displayed in the display area, the attribute information determination unit 120 outputs the updated display component to the output unit 190.

[0033] At this time, the attribute information determination unit 120 may further determine the similarity between the original display element and the updated display element. This is because if there are unacceptable differences between the original display element and the updated display element, they cannot be used as the same display element. The similarity between display elements may be calculated based on the similarity calculated for each attribute information of the display element. For example, the overall similarity may be calculated by weighting and accumulating the similarity of each attribute information. For example, the similarity of characters included in a display element can be determined by calculating the Hamming distance or Levenshtein distance of character strings. Furthermore, the similarity of image elements included in a display element may be determined using a known image matching technique. Or, more simply, if the amount of information in the attribute information of the updated display element is increased compared to the attribute information of the original display element, the display elements are not similar and there are unacceptable differences, and the overall similarity may be calculated as 0. If the attribute information determination unit 120 determines that the updated display component can be displayed in the display area and that the similarity between the original display component and the updated display component is within a predetermined tolerance range, the attribute information determination unit 120 outputs the updated display component to the output unit 190.

[0034] For example, in the example of FIG. 7, the character components "X," "Y," and "Z" included in the original display components are also included in the updated display components. On the other hand, when using known shape matching technology for image components, the similarity is low. In such a case, the attribute information determination unit 120 instructs the prompt creation unit 130 to create a prompt to inquire of the interactive response system 2 to update the display components again. FIG. 9 shows an example of a prompt for inquiring of the creation of display components with different shapes.

[0035] In the example of Figure 8, the character components "X," "Y," and "Z" included in the original display components are also included in the updated display components. Furthermore, when a known shape matching technique is used for image components, the degree of similarity is high. In such a case, the attribute information determination unit 120 outputs the updated display components to the output unit 190.

[0036] The output unit 190 outputs the updated display parts determined by the attribute information determination unit 120. The output unit 190 may output the updated display parts to the display device 70. The output unit 190 may also output the updated display parts to be stored in the RAM 13, the non-volatile memory 14, or the external device 72. The output unit 190 may also transmit the updated display parts to other computers such as the control device 3, the fog computer 6, or the cloud server 7 via the network 5.

[0037] The screen creation support device 1 according to this embodiment, which has the above configuration, updates display components in accordance with the display conditions set in the display area. The updated display components are configured to be easily visible in accordance with the display conditions. This is expected to result in display components that are easy for the user to view (read). Since the user no longer needs to manually modify the display components in accordance with the display conditions of the display area, the amount of work required can be reduced.

[0038] Other Embodiments In the above-described embodiment, the screen creation support device 1 is configured to include the template storage unit 200. However, the template storage unit 200 may be provided on another device, such as the fog computer 6 or the cloud server 7. In this case, the screen creation support device 1 references the template storage unit 200 via the network 5. With such a configuration, it becomes possible to collectively manage templates at a site where many screen creation support devices 1 are installed.

[0039] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the invention or the idea and intent of the present disclosure derived from the content described in the claims and their equivalents. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0040] Below are notes related to the embodiments of the present disclosure. (Supplementary Note 1) A screen creation support device (1) according to one aspect of the present disclosure includes: an attribute information determination unit (120) that determines whether a display component can be displayed in a display area, based on attribute information indicating attributes of the display component and display conditions related to the attribute information set in the display area; a prompt creation unit (130) that creates a prompt for updating attribute information of the display component, based on the attribute information and the display conditions, when the attribute information determination unit (120) determines that the display component cannot be displayed; a transmission unit (140) that transmits the prompt to an interactive response device (2); a reception unit (150) that receives a response including a display component whose attribute information has been updated by the interactive response device (2); and an output unit (190) that displays the updated display component; and each time the reception unit (150) receives a response including an updated display component from the interactive response device (2), the attribute information determination unit (120) determines whether the updated display component can be displayed in the display area.

[0041] (Supplementary Note 2) The attribute information determination unit (120) included in a screen creation support device (1) according to another aspect of the present disclosure determines that the updated display element can be displayed if the similarity between the display element before the update and the updated display element is within a predetermined tolerance range. (Supplementary Note 3) The attribute information of the display element handled by the screen creation support device (1) according to another aspect of the present disclosure includes at least one of the shape, size, display color, update cycle, character type, number of characters, and text size of the display element. (Supplementary Note 4) The display conditions set for the display area handled by the screen creation support device (1) according to another aspect of the present disclosure include at least one of the type, size, resolution, dot spacing, display color, brightness, contrast ratio, and update cycle of the display area.

[0042] (Supplementary Note 5) The prompt creation unit (130) included in the screen creation support device (1) according to another aspect of the present disclosure creates a prompt including wording specifying that attribute information of some of the display components should not be updated. (Supplementary Note 6) The attribute information determination unit (120) included in the screen creation support device (1) according to another aspect of the present disclosure calculates the similarity between the display components before the update and the updated display components based on the similarity for each attribute information of the display components.

[0043] (Supplementary Note 7) The attribute information determination unit (120) included in the screen creation support device (1) according to another aspect of the present disclosure determines that there is an unacceptable difference when the amount of information in the attribute information of the display component increases. (Supplementary Note 8) The display components handled by the screen creation support device (1) according to another aspect of the present disclosure include at least one of text, figures, photos, videos, buttons, and backgrounds.

[0044] (Supplementary Note 9) A computer-readable recording medium according to one aspect of the present disclosure operates as an attribute information determination unit (120) that determines whether a display component can be displayed in a display area, based on attribute information indicating attributes of the display component and display conditions related to the attribute information set in the display area; a prompt creation unit (130) that creates a prompt for updating attribute information of the display component, based on the attribute information and the display conditions, when the attribute information determination unit (120) determines that the display component cannot be displayed; a transmission unit (140) that transmits the prompt to an interactive response device (2); a reception unit (150) that receives a response including a display component whose attribute information has been updated by the interactive response device (2); and an output unit (190) that displays the updated display component, and records a program in the attribute information determination unit (120) that causes the attribute information determination unit (120) to determine whether the updated display component can be displayed in the display area, every time the reception unit (150) receives a response including an updated display component from the interactive response device (2).

[0045] REFERENCE SIGNS LIST 1 Screen creation support device 2 Interactive response device 3 Control device 4 Industrial machine 5 Network 6 Fog computer 7 Cloud server 11 CPU 12 ROM 13 RAM 14 Non-volatile memory 15, 17, 18, 20 Interface 22 Bus 70 Display device 71 Input device 72 External device 100 Display area identification unit 110 Display component identification unit 120 Attribute information determination unit 130 Prompt creation unit 140 Transmission unit 150 Reception unit 190 Output unit 200 Template storage unit

Claims

1. A screen creation support device comprising: an attribute information determination unit that determines whether a display component can be displayed in a display area based on attribute information indicating the attributes of the display component and display conditions related to the attribute information set for the display area; a prompt creation unit that creates a prompt to update the attribute information of the display component based on the attribute information and the display conditions if the attribute information determination unit determines that the display component cannot be displayed; a transmission unit that transmits the prompt to an interactive response device; a reception unit that receives a response including a display component whose attribute information has been updated by the interactive response device; and an output unit that displays the updated display component, wherein the attribute information determination unit determines whether the updated display component can be displayed in the display area every time the reception unit receives a response including an updated display component from the interactive response device.

2. The screen creation support device according to claim 1, wherein the attribute information determination unit determines that the updated display component can be displayed if the similarity between the display component before the update and the updated display component is within a predetermined tolerance range.

3. The screen creation support device according to claim 1, wherein the attribute information of the display component includes at least one of the shape, size, display color, update cycle, character type, number of characters, and text size of the display component.

4. The screen creation support device according to claim 1, wherein the display conditions set for the display area include at least one of the type, size, resolution, dot spacing, display color, brightness, contrast ratio, and update cycle of the display area.

5. The screen creation support device according to claim 1, wherein the prompt creation unit creates a prompt including a statement specifying that attribute information of a portion of the display components should not be updated.

6. The screen creation support device according to claim 2, wherein the attribute information determination unit calculates the similarity between the display component before updating and the updated display component based on the similarity for each attribute information of the display component.

7. The screen creation support device according to claim 6, wherein the attribute information determination unit determines that an unacceptable difference exists when the amount of information in the attribute information of the display component increases.

8. The screen creation support device according to claim 1, wherein the display components include at least one of text, figures, photographs, videos, buttons, and backgrounds.

9. A computer-readable recording medium having recorded thereon a program causing a computer to operate as: an attribute information determination unit that determines whether a display component can be displayed in a display area based on attribute information indicating the attributes of the display component to be displayed in the display area and display conditions related to the attribute information set in the display area; a prompt creation unit that creates a prompt to update the attribute information of the display component based on the attribute information and the display conditions if the attribute information determination unit determines that the display component cannot be displayed; a transmission unit that transmits the prompt to an interactive response device; a reception unit that receives a response including a display component whose attribute information has been updated by the interactive response device; and an output unit that displays the updated display component; and causing the attribute information determination unit to determine whether the updated display component can be displayed in the display area every time the reception unit receives a response including an updated display component from the interactive response device.

Images