Drawing method and device based on voice interaction, storage medium and electronic device

Abstract

The present disclosure belongs to the field of voice interaction technology, and relates to a painting method and device, storage medium and electronic device based on voice interaction. The method includes: collecting voice information, and performing voice recognition processing on the voice information to obtain painting elements; inputting the painting elements into a pre-trained generative adversarial network so that the generative adversarial network outputs a painting image. The present disclosure can realize the function of outputting a painting image by inputting the painting elements obtained by voice recognition processing into the generative adversarial network. On the one hand, drawing a painting image through voice information provides users with a new way of painting and enriches the user's expression method; on the other hand, it helps users to complete the expression of the painting image without having to learn the operation method of the drawing tool and master the painting knowledge, thereby improving the accuracy and fit of the user's expression of ideas through painting.

Description

Technical Field

[0001] The present disclosure relates to the field of voice interaction technology, and in particular to a painting method based on voice interaction, a painting device based on voice interaction, a computer-readable storage medium, and an electronic device. Background Art

[0002] Vision has always been the easiest way to express information among the senses. Among them, pictures have a long history of helping humans share ideas. Pictures have a very low threshold for the recipient, but they require higher skills for the transmitter. Therefore, it is often the case that the pictures in memory cannot be intuitively presented through oral description, and the conceived design is difficult to understand only through textual explanation.

[0003] In existing scenarios, users can use at least one of brushes, lines, light and color to draw in drawing tools when depicting images; they can also use software to touch and drag to draw pictures. However, these methods require extremely high painting skills of users, are difficult to implement, and may not necessarily fit the user's ideas and cannot accurately convey ideas.

[0004] In view of this, there is an urgent need in the art to develop a new painting method and device based on voice interaction.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a painting method based on voice interaction, a painting device based on voice interaction, a computer-readable storage medium and an electronic device, thereby at least to a certain extent overcoming the problems of high painting skills requirements and unsatisfactory painting effects caused by the limitations of related technologies.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of an embodiment of the present invention, there is provided a drawing method based on voice interaction, wherein a graphical user interface is presented through a touch terminal, the method comprising: collecting voice information, and performing voice recognition processing on the voice information to obtain drawing elements;

[0009] The painting elements are input into a pre-trained generative adversarial network so that the generative adversarial network outputs a painting image.

[0010] In an exemplary embodiment of the present invention, the method further comprises:

[0011] In response to a touch operation on the graphical user interface, obtaining position information of the touch operation;

[0012] The target position and / or target size of the painting element is adjusted according to the position information to generate a target painting image corresponding to the painting image.

[0013] In an exemplary embodiment of the present invention, the location information includes first location information and second location information;

[0014] The step of obtaining position information of the touch operation in response to the touch operation on the graphical user interface includes:

[0015] In response to a first touch operation performed on the graphical user interface, acquiring the first position information of the first touch operation;

[0016] In response to a second touch operation that is continuous with the first touch operation, the second position information corresponding to the second touch operation is acquired.

[0017] In an exemplary embodiment of the present invention, adjusting the target position and / or target size of the drawing element according to the position information includes:

[0018] determining a target drawing element in the drawing elements according to the first position information, and acquiring element position information of the target drawing element;

[0019] The element position information is adjusted according to the second position information to obtain a target position and / or a target size of the target drawing element.

[0020] In an exemplary embodiment of the present invention, performing speech recognition processing on the speech information to obtain a painting element includes:

[0021] Performing speech recognition processing on the speech information to obtain text information;

[0022] Determine a painting element corresponding to the text information in a preset knowledge graph.

[0023] In an exemplary embodiment of the present invention, determining the painting element corresponding to the text information in a preset knowledge graph includes:

[0024] Determining a painting entity corresponding to the text information and an entity relationship corresponding to the painting entity in a preset knowledge graph;

[0025] The painting entities and the entity relationships are subjected to reference resolution processing to obtain painting elements.

[0026] In an exemplary embodiment of the present invention, the touch control terminal includes: a full-duplex communication touch control terminal.

[0027] According to a second aspect of an embodiment of the present invention, there is provided a painting device based on voice interaction, the device comprising: an information collection module, configured to collect voice information, and perform voice recognition processing on the voice information to obtain painting elements;

[0028] The image generation module is configured to input the painting elements into a pre-trained generative adversarial network so that the generative adversarial network outputs a painting image.

[0029] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor and a memory; wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the drawing method based on voice interaction of any of the above exemplary embodiments is implemented.

[0030] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the drawing method based on voice interaction in any of the above exemplary embodiments is implemented.

[0031] It can be seen from the above technical solutions that the painting method based on voice interaction, the painting device based on voice interaction, the computer storage medium and the electronic device in the exemplary embodiments of the present invention have at least the following advantages and positive effects:

[0032] In the method and device provided by the exemplary embodiments of the present disclosure, by inputting the painting elements obtained by voice recognition processing into the generative adversarial network, the function of outputting a painting image can be realized. On the one hand, drawing a painting image through voice information provides a new way of painting for users and enriches the user's expression; on the other hand, it helps users to complete the expression of painting images without having to learn the operation of drawing tools and master painting knowledge, thereby improving the accuracy and fit of users' expression of ideas through painting.

[0033] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0035] Figure 1 A flowchart schematically illustrates a drawing method based on voice interaction in an exemplary embodiment of the present disclosure;

[0036] Figure 2 A flowchart schematically illustrates a method for performing speech recognition processing on speech information in an exemplary embodiment of the present disclosure;

[0037] Figure 3 A flowchart schematically illustrating a method for determining a painting element in an exemplary embodiment of the present disclosure;

[0038] Figure 4 A flowchart schematically illustrating a method for generating a target painting image in an exemplary embodiment of the present disclosure;

[0039] Figure 5 A flowchart schematically illustrating a method for acquiring location information in an exemplary embodiment of the present disclosure;

[0040] Figure 6 A flowchart schematically illustrating a method for adjusting a target painting element in an exemplary embodiment of the present disclosure;

[0041] Figure 7 A schematic diagram schematically illustrates a flow chart of a drawing method based on voice interaction in an application scenario in an exemplary embodiment of the present disclosure;

[0042] Figure 8 An interface diagram schematically illustrating a method for determining a drawing entity and an entity relationship in an exemplary embodiment of the present disclosure;

[0043] Fig. 9 A schematic diagram schematically illustrates an interface for adjusting a target painting element in an application scenario in an exemplary embodiment of the present disclosure;

[0044] Fig.10 A schematic diagram of an interface for generating a target painting image in an application scenario in an exemplary embodiment of the present disclosure is schematically shown;

[0045] Fig.11 A processing flow chart of a touch terminal implementing full-duplex communication in an application scenario in an exemplary embodiment of the present disclosure is schematically shown;

[0046] Fig.12 A schematic diagram of the structure of a painting device based on voice interaction in an exemplary embodiment of the present disclosure is schematically shown;

[0047] Fig.13 An electronic device for implementing a painting method based on voice interaction in an exemplary embodiment of the present disclosure is schematically shown;

[0048] Fig.14A computer-readable storage medium for implementing a painting method based on voice interaction in an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0049] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0050] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0051] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0052] In view of the problems existing in the related art, the present disclosure proposes a drawing method based on voice interaction, which presents a graphical user interface through a touch terminal.

[0053] Figure 1 A flow chart of a painting method based on voice interaction is shown, as Figure 1 As shown, the painting method based on voice interaction includes at least the following steps:

[0054] Step S110: Collect voice information, and perform voice recognition processing on the voice information to obtain painting elements.

[0055] Step S120: Input the painting elements into a pre-trained generative adversarial network so that the generative adversarial network outputs a painting image.

[0056] In an exemplary embodiment of the present disclosure, by inputting the painting elements obtained by voice recognition processing into the generative adversarial network, the function of outputting a painting image can be realized. On the one hand, drawing a painting image through voice information provides a new way of painting for users and enriches the user's expression; on the other hand, it helps users to complete the expression of painting images without having to learn the operation of drawing tools and master painting knowledge, thereby improving the accuracy and fit of users' expression of ideas through painting.

[0057] The following is a detailed description of each step of the painting method based on voice interaction.

[0058] In step S110, voice information is collected and voice recognition is performed on the voice information to obtain painting elements.

[0059] In the exemplary embodiment of the present disclosure, voice information is information expressed by the user through language and can be recognized and processed by relevant equipment. It can be in Chinese, English, Japanese, Korean, etc., and this exemplary embodiment does not specifically limit this.

[0060] Taking the voice information in Chinese as an example, the voice information can be collected in real time by a touch terminal, and the touch terminal receives the information waiting to be recognized. The touch terminal can be a voice collection device with a recording function such as a smart phone, a smart wearable device, a tablet computer, a laptop computer, etc. Preferably, the touch device is a speaker with a screen.

[0061] Furthermore, the collected voice information can be processed by voice recognition.

[0062] In an alternative embodiment, Figure 2 A flow chart of a method for performing speech recognition processing on speech information is shown, such as Figure 2 As shown, the method at least includes the following steps: In step S210, voice recognition processing is performed on voice information to obtain text information.

[0063] The text information is the speech recognition result obtained after the speech information is subjected to speech recognition processing. Specifically, the speech recognition processing may be implemented by a speech recognition system, or may be implemented according to a speech recognition algorithm, or may be implemented in other ways, which are not specifically limited in this exemplary embodiment.

[0064] For example, the speech recognition algorithm can be a speech recognition algorithm based on a deep convolutional long short-term memory neural network. The speech recognition algorithm is composed of a three-layer network structure of a fully connected convolutional neural network (CNN), a long short-term memory neural network (LSTM) and a deep neural network (DNN).

[0065] Specifically, CNN can be used to reduce frequency domain changes, and the output of CNN can be used to reduce time domain changes through LSTM. The output of the last layer of LSTM is input to DNN to map feature controls to an output layer that is easier to classify.

[0066] After speech recognition processing by a corresponding speech recognition system or speech recognition algorithm, text information corresponding to the speech information can be obtained.

[0067] In step S220, the painting elements corresponding to the text information are determined in the preset knowledge graph.

[0068] In an alternative embodiment, Figure 3 A flow chart of a method for determining a painting element is shown, Figure 3 As shown, the method includes at least the following steps: In step S310, the painting entity corresponding to the text information and the entity relationship corresponding to the painting entity are determined in a preset knowledge graph.

[0069] The preset knowledge graph can be an open domain knowledge graph or a professional domain knowledge graph.

[0070] Among them, the open domain knowledge graph is aimed at general fields and focuses on building common sense knowledge, including structured encyclopedia knowledge. It emphasizes more on the breadth of knowledge and does not make more requirements on the depth of knowledge. The corresponding users are generally ordinary users.

[0071] The professional field knowledge graph is aimed at a specific industry field, such as painting, science and technology, etc. Its data source is the knowledge of a specific industry and is constructed based on the data of the industry. It must have a certain industry depth and be able to solve more professional problems of industry personnel. The corresponding users are generally practitioners in the industry.

[0072] For example, the text information may be "that big minimalist eye in the middle of the dark sky". In the knowledge graph of the painting field, it can be determined that the painting entities corresponding to the text information are [eye, sky], and the relationship of the painting entities can be [(sky, centered, eye), (eye, style is, minimalist), (sky, color is, black)].

[0073] In step S320, the painting entities and entity relationships are subjected to reference resolution processing to obtain painting elements.

[0074] Among them, the coreference resolution technology is a key supporting technology for text summarization and dialogue question-answering systems. Coreference resolution refers to analyzing all the words in a given text passage that represent the same entity, which is usually a person or an object.

[0075] Entities are often scattered in different locations of a text, and the entities involved can usually be expressed in a variety of different ways. In order to extract relevant information from a text more accurately and without omission, the referential relationship must be resolved to obtain a complete description of the corresponding information in the text.

[0076] Generally, the coreference resolution process can be implemented using a pre-trained coreference resolution model, which can be a long short-term memory network model or other network models, and this exemplary embodiment does not specifically limit this.

[0077] Specifically, we manually annotate text samples with reference relationships, and randomly extract several samples from the text samples and input them into the bidirectional long short-term memory network model, which then extracts the feature variables of each word in the text sample in combination with the context. Furthermore, we calculate the probability that the word has a reference relationship with other words based on the feature variables, and infer the entity that the pronoun refers to in the sentence.

[0078] Therefore, the trained coreference resolution model can be used to determine the painting entities referred to by the pronouns in the text information, and the painting entities and entity relationships after the coreference resolution process can be determined as painting elements.

[0079] In this exemplary embodiment, the painting elements corresponding to the voice information can be determined through voice recognition processing and reference resolution processing, which realizes the idea of ​​converting the voice information to the painting elements and improves the accuracy of the painting element determination.

[0080] In step S120, the painting elements are input into a pre-trained generative adversarial network, so that the generative adversarial network outputs a painting image.

[0081] In an exemplary embodiment of the present disclosure, a generative adversarial network (GAN) is a deep learning model and one of the most promising methods for unsupervised learning on complex distributions. The generative adversarial network produces a relatively ideal output through the mutual game learning of at least two models in the framework, namely the generative model and the discriminative model.

[0082] For example, the generative model in the pre-trained generative adversarial network generates a picture of the sky based on the input painting elements, such as the sky; the discriminative model is used to distinguish the distribution of real samples and actual pictures based on the pictures generated by the generative model and the real annotated sky pictures, and learn the distribution of real samples and actual pictures, and finally determine a sky picture output. Therefore, the sky picture is a painting image output by the generative adversarial network.

[0083] It is worth noting that, compared with collecting voice information once to generate a painting image, or adjusting and updating the painting image, it is more friendly to collect voice information while generating the painting image and display the painting image to the user in real time. Therefore, the touch terminal can be limited to meet this demand.

[0084] In an optional embodiment, the touch terminal includes: a full-duplex communication touch terminal.

[0085] Full-duplex communication allows data to be transmitted in both directions at the same time, which is equivalent to the combination of two simplex communications in terms of capability. Full-duplex can realize the bidirectional transmission of signals at the same time and is instantaneous synchronization.

[0086] Furthermore, the touch terminal using full-duplex communication can draw a picture image through sequential continuous dialogue.

[0087] When the touch terminal is a touch terminal capable of achieving full-duplex communication, the user can view the painting image in real time according to the voice information, which is more user-friendly and optimizes the user experience.

[0088] In addition, other generative adversarial networks can also be used to achieve the function of outputting painting images.

[0089] For example, other generative adversarial networks can be object-driven focused generative adversarial networks (Object Generative Adversarial Networks, referred to as Obj-GAN) or story-image sequence generation model (Story Generative Adversarial Networks, referred to as StoryGAN).

[0090] Since the object-driven focused generative adversarial network and story-image sequence generation model incorporate the functions of the previous context decomposition elements, the processing and reference resolution in the knowledge graph after speech recognition can be reduced, but the application scenarios need to be adjusted in actual applications.

[0091] The object-driven focused generative adversarial network suggests removing the ability to adjust the painting image during continuous dialogue and only adapting to one sentence for drawing. The story-image sequence generation model is more suitable for the sequential drawing of continuous images rather than the adjustment of a single painting image, and it is suggested that each sentence in the continuous dialogue is a complete scene.

[0092] After the painting image is generated, the painting image may be further adjusted and optimized by using touch operation.

[0093] In an alternative embodiment, Figure 4 A schematic diagram of a method for generating a target painting image is shown, Figure 4 As shown, the method at least includes the following steps: In step S410, in response to a touch operation on the graphical user interface, position information of the touch operation is obtained.

[0094] Graphical User Interface (GUI) refers to a computer operation user interface that is displayed in graphical form.

[0095] In an optional embodiment, the location information includes first location information and second location information. Figure 5 A flow chart of a method for obtaining location information is shown, such as Figure 5 As shown, the method at least includes the following steps: In step S510, in response to a first touch operation on the graphical user interface, first position information of the first touch operation is obtained.

[0096] The first touch operation may be a click operation or a long press operation, etc., which is not particularly limited in this exemplary embodiment.

[0097] For example, when a target position of a drawing element is to be adjusted, the first touch operation may be a click operation; when a target size of a drawing element is to be adjusted, the first touch operation may be a long press operation.

[0098] Of course, in different adjustment modes, the first touch operation may also be the same, and this exemplary embodiment does not specifically limit this.

[0099] The first position information may be position information of the action point of the first touch operation, or may be other position information representing the first touch operation, which is not particularly limited in this exemplary embodiment.

[0100] In step S520, in response to a second touch operation that is continuous with the first touch operation, second position information corresponding to the second touch operation is acquired.

[0101] The second touch operation is sequential with the first touch operation in terms of timing. The second touch operation may be a sliding operation or other operations, which are not specifically limited in this exemplary embodiment.

[0102] For example, when adjusting the target position of a drawing element, the second touch operation may be a sliding operation performed by one touch medium; when adjusting the target size of the drawing element, the second touch operation may be a sliding operation performed by two touch media.

[0103] Of course, in different adjustment modes, the second touch operation may also be other different operations, and this exemplary embodiment does not specifically limit this.

[0104] The second position information may be position information of the end point of the second touch operation, or other position information representing the second touch operation, which is not particularly limited in this exemplary embodiment.

[0105] In this exemplary embodiment, the first position information and the second position information of the first touch operation and the second touch operation are respectively obtained to facilitate the subsequent determination of the target painting element and the adjustment of the target painting element. The determination method is simple and accurate, and the feasibility is very strong.

[0106] In step S420 , the target position and / or target size of the drawing element is adjusted according to the position information.

[0107] In an alternative embodiment, Figure 6 A flow chart of a method for adjusting a target drawing element is shown. Figure 6 As shown, the method at least includes the following steps: in step S610, a target painting element is determined in the painting elements according to the first position information, and element position information of the target painting element is acquired.

[0108] The previously determined painting element is displayed in the painting image. There may be one or more painting elements, which is not particularly limited in this exemplary embodiment.

[0109] After obtaining the first position information, the painting element that fully or partially overlaps with the first position information may be determined as the target painting element. There may also be other ways or corresponding relationships for determining the target painting element using the first position information, which is not particularly limited in this exemplary embodiment.

[0110] Furthermore, element position information of the target painting element may be obtained. The element position information may be position information of the painting element saved when the graphical user interface generates a painting image, and the position information may be information that can characterize the painting element in the graphical user interface.

[0111] For example, it may be in coordinate form or in other forms, and this exemplary embodiment does not specifically limit this.

[0112] In step S620, the element position information is adjusted according to the second position information to obtain a target position and / or a target size of the target drawing element.

[0113] For example, when the second position information is the position information of a touch medium action termination point, it can be determined to move the target painting element to the target position represented by the second position information; when the second position information is the position information of two touch medium action termination points, it can be determined to enlarge or reduce the size of the target painting element to the target size represented by the second position information.

[0114] It is worth noting that the user can repeatedly adjust the target position and target size of the target painting element through multiple second touch operations to achieve the target painting image that best meets expectations.

[0115] In this exemplary embodiment, the target position and / or target size of the target painting element can be adjusted through the second position information. The adjustment method is simple and can make the display of the target painting element more in line with user expectations.

[0116] After adjusting the target painting elements, a target painting image may be generated according to the adjusted display effect to replace the originally generated painting image as a final painting work.

[0117] The following is a detailed description of the drawing method based on voice interaction in an embodiment of the present disclosure in conjunction with an application scenario.

[0118] Figure 7 A flow chart of a drawing method based on voice interaction in an application scenario is shown, such as Figure 7 As shown, in step S710, voice information is collected and converted into a corresponding voice stream.

[0119] For example, voice information is converted into a voice stream using a preset streaming media protocol.

[0120] In step S711, based on the voice stream, voice recognition processing is performed on the voice information to obtain corresponding voice text, that is, text information.

[0121] For example, the speech recognition processing may be implemented through a speech recognition system, or may be implemented according to a speech recognition algorithm, or may be implemented in other ways, which is not particularly limited in this exemplary embodiment.

[0122] In step S712, the painting entity corresponding to the text information and the entity relationship corresponding to the painting entity are determined in the preset knowledge graph.

[0123] Figure 8 A schematic diagram of an interface for determining a drawing entity and an entity relationship is shown, Figure 8 As shown, the text information is "that big minimalist eye in the center of the dark sky", and in the indication map of the painting field, it can be determined that the painting entities are "eye" and "sky" respectively.

[0124] The entity relationships corresponding to the painting entity are "eye decoration style is minimalist", "eye size is large", "eye element type is human organ", "the current color of the sky is pitch black", and "the element type of the sky is the most natural landscape". In addition, the entity relationship between the two painting entities "eye" and "sky" is "eye is in the center of the sky".

[0125] In step S713, the painting entities and entity relationships are subjected to reference resolution processing to obtain painting elements.

[0126] It is worth noting that when there are no pronouns in the text information, reference resolution processing does not need to be performed.

[0127] Generally, the coreference resolution process can be implemented using a pre-trained coreference resolution model, which can be a long short-term memory network model or other models, and this exemplary embodiment does not specifically limit this.

[0128] In step S714, a first touch operation on the graphical user interface and a second touch operation continuous with the first touch operation are received.

[0129] In step S715, in response to a first touch operation on the graphical user interface and a second touch operation that is continuous with the first touch operation, first position information and second position information of the first touch operation and the second touch operation are acquired.

[0130] For example, the first position information and the second position information may be expressed in time sequence, that is, the first position information and the second position information may be expressed in the form of [(t1, p1), (t2, p2)], wherein t1 is the action time of the first touch operation, p1 is the first position information, and may be expressed using horizontal and vertical coordinates; t2 is the action time of the second touch operation, p2 is the second position information, and may be expressed using horizontal and vertical coordinates.

[0131] In step S716, semantic post-processing is performed on the painting elements.

[0132] The semantic post-processing can be to input the painting elements into a pre-trained generative adversarial network.

[0133] Each painting element can correspond to a neuron of the generative adversarial network. Since the generative adversarial network is trained based on training data, the trained generative adversarial network can reasonably combine painting elements in a scene, that is, insert painting elements into the scene to generate a painting image.

[0134] In addition, the semantic post-processing may further include adjusting the drawing element according to the first position information and the second position information.

[0135] Specifically, the target painting element may be determined in the painting elements according to the first position information, and the element position information of the target painting element may be adjusted according to the second position information to obtain the target painting element displayed at the target position and / or target size.

[0136] Fig. 9 A schematic diagram of an interface for adjusting a target drawing element in an application scenario is shown, such as Fig. 9 As shown, 910 is a target drawing element determined according to the first position information of the first touch operation. At this time, the first position information collected in time sequence is t1, (x1, y1).

[0137] Then, according to the user's demand for the target drawing element to "move a little bit here", a second touch operation can be performed continuously with the first touch operation to move the target drawing element to another position. At this time, the first position information in the time sequence can be collected as t2, (x2, y2). 920 is the target drawing element displayed at the target position after the move.

[0138] It can be seen that moving or resizing the target painting element through touch operation can be translated into the process of eliminating the specified element in the painting image and forcibly inserting a new element. Therefore, adjusting the target painting element can be understood as the process of generating a target painting image to cover the generated painting image.

[0139] In step S717, after the painting image or the target painting image is generated, the picture scene within the effective time can be read and written, that is, the element position information of the painting element and other related information can be read and saved to obtain the picture context.

[0140] Fig.10 A schematic diagram of an interface for generating a target painting image in an application scenario is shown, Fig.10As shown, the voice information collected from the drawing user is "that kind of minimalist style big eyes in the center of the dark sky."

[0141] At this time, a touch terminal capable of implementing full-duplex communication may be used to generate a painting image in real time, as shown in painting image 1010 , which displays a large eye in a minimalist style in the dark sky.

[0142] Fig.11 FIG. 4 shows a processing flow chart of a touch terminal that implements full-duplex communication in an application scenario, such as Fig.11 As shown, the touch terminal can accept the voice information input by the drawing user, and convert the drawing instructions of the voice information into voice stream data, store the voice stream data in the uplink message queue, and draw the drawing image according to the current drawing instructions.

[0143] During the image drawing process, multiple drawing instructions of the user may be stored in the downlink message queue, and the voice stream data in the downlink message queue may be converted into drawing instructions in real time to draw a drawing image and display it on the touch terminal.

[0144] In addition, sound effects can be generated during the process of modifying the painting image, so as to provide the drawing user with a multi-faceted sensory experience through the sound effects, making it easier for the drawing user to perceive the adjustment and update of the painting image.

[0145] Storing the user's drawing instructions in the uplink message queue and the downlink message queue can allow multiple drawing instructions to be superimposed, thereby achieving the effect of continuously identifying the drawing user's intention and continuously modifying the drawing image.

[0146] Therefore, in the obtained Fig.10 After the painting image 1010 is displayed, the user can continue to voice control “make the eyes bigger and red” to obtain the painting image 1020 ; further, the user can voice control again “just add a lightning bolt here” to obtain the painting image 1030 .

[0147] Finally, the painting image is displayed on the touch device.

[0148] In order to use it in actual application scenarios, you can consider reducing the restrictions in the application scenarios. For example, the scene restriction can be "this painting can only combine faces / fruits or other types of elements that are smaller than the threshold", or specifically cut into scenes such as children's picture books and interior design to meet the drawing needs in different application scenarios.

[0149] In the painting method based on voice interaction in the application scenario, the function of outputting a painting image can be realized by inputting the painting elements obtained by voice recognition processing into the generative adversarial network. On the one hand, drawing a painting image through voice information provides users with a new way of painting and enriches the user's expression; on the other hand, it helps users to complete the expression of painting images without having to learn how to operate the drawing tools and master the painting knowledge, which improves the accuracy and fit of users' expression of ideas through painting.

[0150] In addition, in an exemplary embodiment of the present disclosure, a painting device based on voice interaction is also provided. Fig.12 A schematic diagram of the structure of a painting device based on voice interaction is shown, Fig.12 As shown, the painting device 1200 based on voice interaction may include: an information collection module 1210 and an image generation module 1220. Among them:

[0151] The information collection module 1210 is configured to collect voice information and perform voice recognition processing on the voice information to obtain painting elements;

[0152] The image generation module 1220 is configured to input the painting elements into a pre-trained generative adversarial network so that the generative adversarial network outputs a painting image.

[0153] The specific details of the above-mentioned painting device based on voice interaction have been described in detail in the corresponding painting method based on voice interaction, so they will not be repeated here.

[0154] It should be noted that although several modules or units of the painting device 1200 based on voice interaction are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units for embodiment.

[0155] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0156] Refer to the following Fig.13 13 to 14. An electronic device 1300 according to such an embodiment of the present invention is described. Fig.13 The electronic device 1300 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0157] like Fig.13As shown, the electronic device 1300 is in the form of a general computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, a bus 1330 connecting different system components (including the storage unit 1320 and the processing unit 1310), and a display unit 1340.

[0158] The storage unit stores program codes, which can be executed by the processing unit 1310, so that the processing unit 1310 executes the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0159] The storage unit 1320 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1321 and / or a cache memory unit 1322 , and may further include a read-only memory unit (ROM) 1323 .

[0160] The storage unit 1320 may also include a program / utility 1324 having a set (at least one) of program modules 1325, such program modules 1325 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0161] Bus 1330 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0162] The electronic device 1300 may also communicate with one or more external devices 1500 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1300, and / or communicate with any device that enables the electronic device 1300 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1350. Furthermore, the electronic device 1300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1360. As shown, the network adapter 1360 communicates with other modules of the electronic device 1300 via a bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0163] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0164] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0165] refer to Fig.14 As shown, a program product 1400 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0166] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0167] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0168] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0169] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0170] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A drawing method based on voice interaction, presenting a graphical user interface through a touch terminal, characterized in that: The method comprises: Collecting voice information, and performing voice recognition processing on the voice information to obtain painting elements; Inputting the painting elements into a pre-trained generative adversarial network so that the generative adversarial network outputs a painting image; wherein the generative adversarial network is a story image sequence generation model, and the story image sequence generation model has the function of merging the decomposition elements of the previous text; The step of performing speech recognition processing on the speech information to obtain the painting elements includes: Performing speech recognition processing on the speech information to obtain text information; Determining a painting entity corresponding to the text information and an entity relationship corresponding to the painting entity in a preset knowledge graph; Based on the coreference resolution model, coreference resolution processing is performed on the painting entity and the entity relationship to obtain the painting element.

2. The painting method based on voice interaction according to claim 1, characterized in that: The method further comprises: In response to a touch operation on the graphical user interface, obtaining position information of the touch operation; The target position and / or target size of the painting element is adjusted according to the position information to generate a target painting image corresponding to the painting image.

3. The painting method based on voice interaction according to claim 2, characterized in that: The location information includes first location information and second location information; The step of obtaining position information of the touch operation in response to the touch operation on the graphical user interface includes: In response to a first touch operation performed on the graphical user interface, acquiring the first position information of the first touch operation; In response to a second touch operation that is continuous with the first touch operation, the second position information corresponding to the second touch operation is acquired.

4. The painting method based on voice interaction according to claim 3, characterized in that: The step of adjusting the target position and / or target size of the drawing element according to the position information includes: determining a target drawing element in the drawing elements according to the first position information, and acquiring element position information of the target drawing element; The element position information is adjusted according to the second position information to obtain a target position and / or a target size of the target drawing element.

5. The painting method based on voice interaction according to claim 1, characterized in that: The touch control terminal includes: a full-duplex communication touch control terminal.

6. A drawing device based on voice interaction, characterized in that: include: An information collection module is configured to collect voice information and perform voice recognition processing on the voice information to obtain painting elements; An image generation module is configured to input the painting elements into a pre-trained generative adversarial network so that the generative adversarial network outputs a painting image; wherein the generative adversarial network is a story image sequence generation model, and the story image sequence generation model has the function of merging the decomposition elements of the previous text; The information collection module performs voice recognition processing on the voice information to obtain the painting elements, including: Performing speech recognition processing on the speech information to obtain text information; Determining a painting entity corresponding to the text information and an entity relationship corresponding to the painting entity in a preset knowledge graph; Based on the coreference resolution model, coreference resolution processing is performed on the painting entity and the entity relationship to obtain the painting element.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the painting method based on voice interaction described in any one of claims 1 to 5 is implemented.

8. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the voice interaction-based painting method described in any one of claims 1-5 by executing the executable instructions.

Images