Graphic locating device, graphic locating method, and program

Abstract

To allow for partially taking care of collaborative work which was conventionally possible only by persons.SOLUTION: A graphic locating device generates a new graphic location image by relocating graphic objects and includes: an image candidate generation unit 12 which takes a first graphic location image as input data and takes pairs of text naming representations and graphic location images which can be generated by relocating individual graphic objects included in the first graphic location image, as a plurality of candidates and takes the plurality of candidates as output data; a selection unit 14 which selects a pair relating to a prescribed candidate out of the plurality of candidates; and a graphic location image generation unit 15 which takes a second graphic location image in the selected pair and first graphic location instruction information for correcting the second graphic location image, as input data and changes locations of individual graphic objects within the second graphic location image according to the first graphic location instruction information for the second graphic location image to generate a third graphic location image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating a new graphic layout image by rearranging graphic objects. [Background technology]

[0002] In recent years, in dialogue systems, humans can interact with computers such as smart speakers to obtain various information and fulfill requests. In addition to dialogue systems that accomplish specific tasks, such as AI dialogue-based customer service systems, there are also dialogue systems such as stuffed animal-type robots that engage in everyday conversations (casual conversations). These systems allow humans to achieve mental stability, satisfy their desire for recognition, and build trusting relationships. Types of dialogue systems are described in Non-Patent Document 1.

[0003] Furthermore, Non-Patent Document 2 states that even with the use of recent large-scale language models, long-term communication is difficult, resulting in a problem of low user satisfaction. As a means for solving this problem, there is a method of providing a dialogue system with information called a common foundation. A "common foundation" refers to information such as knowledge and beliefs shared between dialogue participants in a dialogue, and is sometimes called a mutual belief. In order for a computer to process the contents of a dialogue, the common foundation handled by the computer has been considered to be one of the important concepts, but Non-Patent Document 3 discloses that there has been little research analyzing the process of constructing a common foundation.

[0004] In addition, Non-Patent Document 4 discloses research content in which text chats between two workers completing a task are collected and analyzed as an attempt to build a common foundation. The task of arranging shapes by two people as described in Non-Patent Document 4 is a type of creative collaborative work, and it is desirable for a dialogue system to be able to carry out such creative work together with the user. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] "Natural Language Processing Series 7 Dialogue Systems", edited by Manabu Okumura, co-authored by Mikio Nakano, Kazunori Komatani, Kotaro Funakoshi, and Yukiko Nakano, published 2015.2.13 [Non-Patent Document 2] Sparks of Artificial General Intelligence: Early experiments with GPT-4<https: / / arxiv.org / abs / 2303.12712> [Non-Patent Document 3] "Understanding Processing in Dialogue Systems", Mikio Nakano, SIG-SLUD-B901-01, Language, Speech Understanding and Dialogue Processing, 2019. [Non-Patent Document 4] "Analysis of the process of constructing a common ground for dialogue in a collaborative shape layout task," Wataru Mitsuda, Ryuichiro Higashinaka, Yuhei Ohga, Tetsuya Kinebuchi, Proceedings of the 27th Annual Conference of the Association for Natural Language Processing (NLP2021), March 2021 [Non-Patent Document 5] "Analysis of the usefulness of naming in constructing a common foundation," Yui Saito, Wataru Mitsuda, Ryuichiro Higashinaka, Yasuhiro Minami, Proceedings of the 29th Annual Conference of the Association for Natural Language Processing (NLP2023), March 2023. Summary of the Invention [Problem to be solved by the invention]

[0006] However, since existing dialogue systems cannot properly handle a common base, it is difficult for workers to dialogue or collaborate while properly retaining information that has been determined between them through dialogue. For example, as shown in FIG. 2, human workers name a target geometric layout (a layout that does not yet exist) (e.g., "truck"), or name the geometric layout when communicating it (e.g., "driver's seat of the truck") (see Non-Patent Document 5). In this way, for a task that includes graphical elements, a function is required to set a goal and verbalize it (devise a target geometric layout) and to understand the verbalized task (understand the geometric layout). However, these functions require a common base to align each other's perceptions, so it is difficult to collaborate with a dialogue system for such tasks.

[0007] The present invention has been made in consideration of the above-mentioned points, and has an object to make it possible to partially substitute for collaborative work that has conventionally been difficult to achieve except between humans. [Means for solving the problem]

[0008] In order to solve the above problem, the invention of claim 1 is a geometric arrangement device that generates a new geometric arrangement image by rearranging geometric objects, the geometric arrangement device having: an image candidate generation unit that receives a first geometric arrangement image as input data, rearranges each geometric object included in the first geometric arrangement image, and outputs a set of generable geometric arrangement images and text naming expressions as multiple candidates; a selection unit that selects a set related to a predetermined candidate from the multiple candidates; and a geometric arrangement image generation unit that receives a second geometric arrangement image in the selected set and first geometric arrangement command information for correcting the second geometric arrangement image as input data, and generates a third geometric arrangement image by changing the arrangement of each geometric object in the second geometric arrangement image in accordance with the first geometric arrangement command information for the second geometric arrangement image. Effect of the Invention

[0009] As described above, the present invention has the effect of partially replacing cooperative work that was previously difficult to achieve only between humans. [Brief description of the drawings]

[0010] [Figure 1] 1 is a functional configuration diagram of a graphic layout device according to an embodiment; [Diagram 2] FIG. 13 is a diagram showing an example of a dialogue in which the construction of a common foundation progresses through naming. [Diagram 3] FIG. 4A shows an example of input data to a geometric candidate generating unit, and FIG. 4B shows an example of output data from the geometric candidate generating unit. [Figure 4] 5A shows an example of input data to a geometric layout image generating unit, and FIG. 5B shows an example of output data from the geometric layout image generating unit. [Diagram 5] FIG. 13 is a diagram showing an example of learning data of the figure arrangement image generating unit. [Figure 6] 1 is a diagram illustrating an electrical hardware configuration of a graphic layout device according to an embodiment. [Figure 7] 4 is a flowchart showing the process or operation of the graphic arrangement device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [Overview of the embodiment] In this embodiment, the problem is solved by a collaborative work between two people targeting graphical elements. Specifically, the problem of collaborative graphic layout work described in Non-Patent Documents 3 and 4 will be described, but the method of this embodiment can be applied to any collaborative work in which some goal is achieved by editing a graphical object between two people. The graphic layout device 10 of this embodiment generates a new graphic layout image by rearranging graphic objects as shown below.

[0013] [Functional configuration of the graphic layout device] The functional configuration of a graphic layout device 10 will be described with reference to Fig. 1. Fig. 1 is a functional configuration diagram of a graphic layout device according to an embodiment.

[0014] The graphic layout device 10 has an acquisition unit 11, an image candidate generation unit 12, a display control unit 13, a selection unit 14, a graphic layout image generation unit 15, a graphic layout command generation unit 16, and an output unit 17. Each of these units is a function realized by an instruction from the CPU 101 in FIG. 6 based on a program.

[0015] The acquisition unit 11 acquires data of the graphic layout image A1, graphic layout command information B1, and graphic layout image A4 from outside the graphic layout device 10 in response to a user's operation. The "graphic layout image" is an image generated by arranging each graphic object such as a circle or a square at a predetermined position as shown in Figs. 2 to 4, and is, for example, an image representing a truck. The "graphic layout command information" is information indicating the command content for arranging a predetermined graphic object, and indicates, for example, the content such as "arrange a square as the driver's seat and the loading platform of the truck." The "graphic layout command information" also includes a "naming expression" which is an expression (name) given to the graphic. When the graphic layout information indicates "arrange a square as the driver's seat and the loading platform of the truck," the naming expression is "square," "truck," "driver's seat," "loading platform," etc.

[0016] The image candidate generation unit 12 receives the graphic layout image A1 as input data, rearranges each graphic object included in the graphic layout image A1, and outputs a plurality of candidates for N sets of generable graphic layout images and text naming expressions.

[0017] 3, (a) shows an example of input data to the geometric candidate generating unit, and (b) shows an example of output data from the geometric candidate generating unit. One or more geometric objects are arranged in a geometric arrangement image A1, which is the input data. The output data is a set of a geometric arrangement image and its naming expression, which are generated by changing the arrangement of the input geometric objects.

[0018] The image candidate generation unit 12 is constructed by a deep learning model (an example of a first machine learning model) that uses Transformer (see Reference 1) in machine learning as a base and machine-learns the relationship between images, text, and new images. For example, a machine learning model called StyleCLIP (see Reference 2) has been proposed that takes images and text as inputs and outputs images, and this machine learning model is constructed by learning with data of an initial image, a final image, and a naming expression (one selected from the naming expressions in the utterance content) given to the final image, using the geometric arrangement images collected in Non-Patent Document 4 shown in FIG. 2.

[0019] For example, Non-Patent Document 4 includes approximately 1000 dialogues (utterances), which is a sufficient number to generate input / output pairs. If the model performance is insufficient, a user manually creates a graphic layout image and a naming expression, and uses them for machine learning. When the image candidate generation unit 12 is actually operated, it is necessary to input the text of the naming expression together with the graphic layout image into the model. Therefore, a list of some text (for example, a list of texts used in the data used for learning the model) is prepared, and the text to be input is input by randomly selecting the text to be input from the list, and this text is paired with the output image and output.

[0020] (Reference 1) VASWANI, Ashish, et al. Attention is all you need. Advances in neural information processing systems, 2017, 30. (Reference 2) PATASHNIK, Or, et al. StyleCLIP: Text-driven manipulation of StyleGAN imagery. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021. p. 2085-2094. The display control unit 13 causes the display 106 shown in FIG. 6 to display the multiple pairs (pair candidate list L) generated by the image candidate generating unit 12.

[0021] The selection unit 14 accepts the selection of a predetermined set from among a plurality of sets of a graphic layout image and its naming expression, based on an operation of the input device 108 by a user.

[0022] The graphic layout image generating unit 15 receives as input data the graphic layout image A2 selected by the selecting unit 14 and graphic layout command information B1 newly acquired by the acquiring unit 11 through a user operation, and generates a new graphic layout image A3 as output data by changing the arrangement of each graphic object in the graphic layout image A2 according to the graphic layout command information B1 for the graphic layout image A2. In this case, only the graphic layout image A2 out of the pair of the graphic layout image A2 selected by the selecting unit 14 and its naming expression is used as input data. This allows the graphic layout image A2 to be automatically corrected.

[0023] In FIG. 4, (a) shows an example of input data to the graphic layout image generating unit, and (b) shows an example of output data from the graphic layout image generating unit. The input data is a graphic layout image A2 in which one or more graphic objects are arranged, and graphic layout command information B1 that specifies one or more graphic objects to be moved and indicates the contents thereof. The output data is a graphic layout image A3 generated by changing each of the graphic objects in the input graphic layout image A2 in accordance with the graphic layout command information B1. Note that, like the image candidate generating unit 12, the graphic layout image generating unit 15 can output M pieces of output data (M is an integer) in descending order of certainty, but here it is assumed that graphic layout command information that clearly determines one graphic layout image is input, and only one type of output data is provided. An example of one graphic layout image is one with the highest certainty.

[0024] The graphic layout image generation unit 15 is also constructed by a deep learning model (an example of a second machine learning model) based on Transformer (see Reference 1) in machine learning. For example, like the image candidate generation unit 12, a machine learning model called StyleCLIP (see Reference 2) that takes an image and text as input and outputs an image can be used, and this model can be trained as a machine learning model with the input / output pair shown in FIG.

[0025] Here, a method of creating learning data for machine learning of the graphic layout command generating unit 16 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of learning data for the graphic layout image generating unit.

[0026] Since it is unclear which graphic object the naming expressions of the graphic objects collected by the technology of Non-Patent Document 5 are actually associated with (for example, "rear tires", "above the driver's seat"), the data collected by the technology of Non-Patent Document 5 cannot be used as learning data as it is. Therefore, as shown in FIG. 5, data to which labels indicating which graphic object each naming expression is actually associated with is prepared as learning data. By creating this data for the utterance contents including all naming expressions, it becomes possible to trace how the graphic arrangement image has changed for each utterance content through the naming expressions. By using this data as learning data, it is possible to create one input / output pair shown in FIG. 4 by grouping the graphic arrangement images (two graphic arrangement images) before and after the first utterance of the dialogue and one naming expression during the utterance. Similarly, one input / output pair is created by grouping two graphic arrangement images and one naming expression for the second utterance, the third utterance, .... With the scale of data collected by the technology of Non-Patent Document 4, it becomes possible to create tens of thousands of input / output pairs. The machine learning model of the graphic layout image generating unit 15 can be trained using the training data of the input / output pairs.

[0027] In addition, Figure 5 shows the utterance content of speaker X, "The image of the forehead is a diamond." In this case, as information about the naming expression, "3_diamond" indicates that "diamond" is represented in the sequence number 3 from the top. "0_3_shape / one building" is located between the 0th and 3rd characters of the utterance content, and indicates that a type of naming expression is represented. This indicates that "diamond" is a naming expression that "represents a shape or one of the buildings." In addition, "registered name" indicates that "diamond" is a basic geometric object that has been registered in advance.

[0028] Similarly, in the case of "4_forehead_4_5_shape / one building_name related to the body," in the sequence number 4 from the top, the naming expression "forehead" is located between the fourth and fifth characters of the utterance, indicating that "forehead" is a naming expression that "represents a shape or a building," and further indicating that "forehead" is a "name related to the body."

[0029] In addition, as for the correspondence relationship, it is shown that the serial number 3 "diamond" and the serial number 4 "frame" both correspond to the geometric object at the arrangement position of "a" in the geometric arrangement image.

[0030] The graphic layout command generating unit 16 receives as input data the graphic layout image A3 output from the graphic layout image generating unit 15 and the graphic layout image A4 newly acquired by the acquiring unit 11 through a user operation, generates graphic layout command information B2 for making the graphic layout image A3 closer to the graphic layout image A4, and outputs the generated graphic layout command information B2 as output data. The graphic layout command generating unit 16 is also constructed by a deep learning model (an example of a third machine learning model) based on Transformer (see Reference 1) in machine learning. For example, like the graphic layout image generating unit 15, a machine learning model called StyleCLIP (see Reference 2) that receives an image and text as input and outputs an image can be used, and this model can be trained as a machine learning model with an input / output pair shown in FIG. 4.

[0031] The output unit 17 outputs the graphic layout image A3 from the graphic layout image generation unit 15. The output unit 17 also outputs graphic layout command information B2 from the graphic layout command generation unit 16. Examples of the output method include displaying on the display 107, outputting to a printer and printing, transmitting to another device (such as a PC), etc.

[0032] [Hardware configuration of the graphics layout device] Next, the electrical hardware configuration of the graphic layout device 10 will be described with reference to Fig. 6. Fig. 6 is a diagram showing the electrical hardware configuration of the graphic layout device.

[0033] The graphic layout device 10 is a computer that includes a CPU 101, a ROM 102, a RAM 103, an SSD 104, an external device connection I / F (Interface) 105, a network I / F 106, a display 107, an input device 108, a media I / F 109, and a bus line 110.

[0034] Of these, CPU 101 controls the overall operation of graphic layout device 10. ROM 102 stores programs such as IPL used to drive CPU 101. RAM 103 is used as a work area for CPU 101.

[0035] The SSD 104 reads and writes various data under the control of the CPU 101. Instead of the SSD 104, a HDD (Hard Disk Drive) may be used.

[0036] The external device connection I / F 105 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB memory, and a printer.

[0037] The network I / F 106 is an interface for performing data communication via the communication network 100 .

[0038] The display 107 is a type of display means, such as a liquid crystal display or an organic EL (Electro Luminescence) display, that displays various images.

[0039] The input device 108 is a keyboard, a pointing device, or the like, and is an example of an input means for accepting input operations of characters, numerical values, various instructions, and the like.

[0040] The media I / F 109 controls reading and writing (storing) of data from and to a recording medium 109m such as a flash memory. The recording medium 109m includes DVDs and Blu-ray Discs (registered trademarks).

[0041] The bus line 110 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 101 shown in FIG.

[0042] [Processing or Operation of the Embodiment] Next, the processing or operation of this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the processing or operation of the graphic layout device according to the embodiment.

[0043] S11: The acquisition unit 11 acquires data of the graphic layout image A1 from outside the graphic layout device 10.

[0044] S12: The image candidate generation unit 12 uses the graphic layout image A1 as input data, rearranges each graphic object included in the graphic layout image A1, and generates a plurality of candidates of N pairs of the graphic layout image and the naming expression of the text that can be generated (the graphic layout image A1). N and naming expression b N The output data is each candidate of the pair.

[0045] S13: The display control unit 13 causes the display 106 shown in FIG. 6 to display each of the pairs (pair candidate list L) generated by the image candidate generating unit 12.

[0046] S14: The selection unit 14 accepts the selection of a predetermined set from among a plurality of sets of a graphic layout image and its naming expression, based on the user's operation of the input device 108. Furthermore, the acquisition unit 11 acquires graphic layout command information B1 for correcting the graphic layout image A2 included in the selected set, based on the user's operation of the input device 108.

[0047] S15: The graphic layout image generation unit 15 receives as input data the graphic layout image A2 selected by the selection unit 14 and graphic layout command information B1 newly acquired by the acquisition unit 11 through user operation, and generates a new image layout image A3 as output data by changing the arrangement of each graphic object in the graphic layout image A2 in accordance with the graphic layout command information B1 for the graphic layout image A2.

[0048] S16: The acquisition unit 11 further acquires data of the graphic layout image A4 by the user's operation.

[0049] S17: The graphic layout command generation unit 16 takes as input data the graphic layout image A3 output from the graphic layout image generation unit 15 and the graphic layout image A4 newly acquired by the acquisition unit 11 through user operation, and generates graphic layout command information B2 for moving the graphic layout image A3 closer to the graphic layout image A4, and sets this as output data.

[0050] As a result, the output unit 17 outputs the graphic layout image A3 from the graphic layout image generating unit 15 or the graphic layout command information B2 from the graphic layout command generating unit 16.

[0051] [Main Effects of the Embodiments] As described above, according to the embodiment, collaborative work that was previously difficult to achieve only between humans can now be partially substituted by graphic layout device 10 (dialogue system), and the user and graphic layout device 10 can work together. This allows the user to work while consulting with the dialogue system, improving the quality of the deliverable, and allows the work to be performed efficiently by having the dialogue system perform the work.

[0052] Another advantage is that the two robots can communicate with each other and work together without human intervention.

[0053] 〔supplement〕 The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

[0054] (1) Graphic layout device 10 can be realized by a computer and a program, but this program can also be recorded on a (non-transitory) recording medium or provided via a communication network such as the Internet.

[0055] (2) The CPU 101 may be multiple, not just single. [Explanation of symbols]

[0056] 10. Graphic layout device 11 Acquisition Department 12 Image Candidate Generation Unit 13 Display control section 14 Selection section 15. Graphic layout image generation section 16 Graphics placement command generator 17 Output section

Claims

1. A graphic layout device that generates a new graphic layout image by rearranging graphic objects, comprising: an image candidate generating unit that receives a first geometric layout image as input data, rearranges each geometric object included in the first geometric layout image, and outputs a plurality of candidates of pairs of generable geometric layout images and text naming expressions as output data; A selection unit that selects a set related to a predetermined candidate from among the plurality of candidates; a geometric layout image generating unit which receives as input data a second geometric layout image in the selected set and first geometric layout command information for correcting the second geometric layout image, and changes the layout of each geometric object in the second geometric layout image in accordance with the first geometric layout command information for the second geometric layout image, thereby generating a third geometric layout image; A graphic layout device having the above structure.

2. The graphic arrangement device of claim 1, wherein the image candidate generation unit outputs the plurality of candidates using a first machine learning model that has learned the relationship between images and text and the new image, and the graphic arrangement image generation unit outputs the third graphic arrangement image using a second machine learning model that has learned the relationship between images and text and the new image.

3. 3. A graphic layout device according to claim 1, further comprising: a graphic layout command generating unit that uses the third graphic layout image generated by the graphic layout image generating unit and a newly acquired fourth graphic layout image as input data and generates second graphic layout command information for bringing the third graphic layout image closer to the fourth graphic layout image.

4. 4. The graphic arrangement device according to claim 3, wherein the graphic arrangement command generating unit outputs the second graphic arrangement command information using a third machine learning model that has learned a relationship between images and text and the new image.

5. A geometric arrangement method executed by a geometric arrangement device that generates a new geometric arrangement image by rearranging geometric objects, comprising: The graphic layout device comprises: an image candidate generation process in which a first geometric layout image is used as input data, and each geometric object included in the first geometric layout image is rearranged to generate a plurality of candidate sets of geometric layout images and text naming expressions as output data; A selection process for selecting a set related to a predetermined candidate from among the plurality of candidates; a geometric layout image generating process for generating a third geometric layout image by using a second geometric layout image in the selected set and first geometric layout command information for correcting the second geometric layout image as input data and changing the layout of each geometric object in the second geometric layout image in accordance with the first geometric layout command information for the second geometric layout image; A shape placement method that performs.

6. A program for causing a computer to execute the method according to claim 5.

Images