Display device, server and picture book interactive processing method

Abstract

The invention relates to a display device, a server and a picture book interactive processing method. The display device includes: a display configured to display a picture book, the picture book containing at least one editable element; the input module is configured to receive an editing instruction input by a user for the picture book; the controller is configured to determine a target element indicated by an editing instruction in response to the editing instruction for at least one part of editable elements in the picture book; on the basis of editing content carried in the editing instruction, obtaining a replacement element matched with the editing content; and controlling the display to display an element-updated picture book, the element-updated picture book being obtained by replacing the target element based on the replacement element. According to the scheme, the picture book interaction result can be flexibly adjusted according to the editing instruction input by the user for the picture book.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device, a server, and a picture book interaction processing method. Background Art

[0002] With the rapid development of technology, more and more additional functions have emerged based on the original functions of display devices. For example, by integrating picture book application software into the display device, users can be provided with functions such as picture book reading and picture book reading aloud.

[0003] In traditional technology, users can flip through picture books or trigger preset animation effects in picture books by clicking on the touch screen of the display device. However, since these interactive contents are pre-set, the interactive results are relatively fixed and it is difficult to meet the interactive needs of different users.

[0004] Therefore, using traditional technology has the problem that the interactive results of picture books cannot be flexibly adjusted. Summary of the invention

[0005] The present application provides a display device, a server and a picture book interaction processing method to solve the problem that the picture book interaction results cannot be flexibly adjusted.

[0006] In a first aspect, some embodiments provide a display device, including:

[0007] A display configured to display a picture book, the picture book comprising at least one editable element;

[0008] An input module is configured to receive an editing instruction input by a user for a picture book;

[0009] The controller is configured as:

[0010] In response to an editing instruction for at least a portion of editable elements in the picture book, determining a target element indicated by the editing instruction;

[0011] Based on the edit content carried in the edit instruction, obtain a replacement element matching the edit content;

[0012] The display is controlled to display an updated picture book of the element, and the updated picture book of the element is obtained by replacing the target element based on the replacement element.

[0013] The above-mentioned display device includes a display, an input module and a controller. Specifically, the input module can be used to receive the editing instructions input by the user for the picture book, and the controller can respond to the editing instructions for at least a part of the editable elements in the picture book, determine the target element indicated by the editing instruction, and thereby obtain the replacement element matching the editing content based on the editing content carried in the editing instruction, and then control the display to display the picture book with updated elements. Among them, the picture book with updated elements is obtained by replacing the target element based on the replacement element. That is, by using the above-mentioned display device, the editing instructions input by the user for the picture book can be received through interaction with the user, so that the elements in the picture book can be edited according to the editing instructions, and the picture book with updated elements can be displayed, that is, the picture book interaction results corresponding to the editing instructions are displayed to the user. If the editing instructions generated by the interaction are different, the above-mentioned display device can be used to display different picture book interaction results to the user, thereby realizing flexible adjustment of the picture book interaction results.

[0014] In a second aspect, some embodiments further provide a server, including:

[0015] A server communication device configured to establish a communication connection with the display device;

[0016] The processor is configured as:

[0017] Receiving an editing instruction sent by the display device through the server communication module; the editing instruction is input for at least a part of the editable elements in the picture book displayed by the display device;

[0018] Based on the editing content carried in the editing instruction, a replacement element matching the editing content is generated; the replacement element is used to replace the target element in the picture book indicated by the editing instruction to obtain a picture book with updated elements.

[0019] The above-mentioned server includes a server communication device and a processor. Specifically, the server communication module can receive the editing instructions sent by the display device, wherein the editing instructions are input for at least a part of the editable elements in the picture book displayed by the display device. Furthermore, the processor can generate replacement elements that match the editing content based on the editing content carried in the editing instructions, wherein the replacement elements are used to replace the target elements in the picture book indicated by the editing instructions, so as to obtain a picture book with updated elements, that is, to obtain the picture book interaction results generated by the editing instructions. Based on this, by using the above-mentioned server, different replacement elements can be generated according to different editing instructions, so as to obtain picture books with different element updates, that is, to obtain different picture book interaction results, thereby realizing flexible adjustment of the picture book interaction results.

[0020] In a third aspect, some embodiments further provide a picture book interaction processing method, which is applied to the display device provided in the first aspect. The picture book interaction processing method includes:

[0021] receiving an editing instruction input by a user for at least a portion of editable elements in a picture book, and displaying the picture book on a display device;

[0022] determining the target element indicated by the editing instruction;

[0023] Based on the edit content carried in the edit instruction, obtain a replacement element matching the edit content;

[0024] The display is controlled to display an updated picture book of the element, and the updated picture book of the element is obtained by replacing the target element based on the replacement element.

[0025] In a fourth aspect, some embodiments further provide a picture book interaction processing device, which is applied to the display device provided in the first aspect. The picture book interaction processing device includes:

[0026] An editing instruction receiving module, used to receive an editing instruction input by a user for at least a part of the editable elements in the picture book, and the picture book is displayed on a display of a display device;

[0027] A target element determination module, used to determine the target element indicated by the editing instruction;

[0028] A replacement element acquisition module, used to acquire a replacement element matching the edited content based on the edited content carried in the edit instruction;

[0029] The picture book display control module is used to control the display to display the updated picture book of elements, and the updated picture book of elements is obtained by replacing the target element based on the replacement element.

[0030] In a fifth aspect, some embodiments further provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the picture book interaction processing method provided in the third aspect is implemented.

[0031] In a sixth aspect, some embodiments further provide a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the picture book interaction processing method provided in the third aspect is implemented.

[0032] In a seventh aspect, some embodiments further provide a computer program product, comprising a computer program, which, when executed by a processor, implements the picture book interaction processing method provided in the third aspect.

[0033] The above-mentioned picture book interaction processing method, device, computer equipment, readable storage medium and program product first receive the editing instructions input by the user for at least a part of the editable elements in the picture book, wherein the picture book is displayed on the display of the display device. Further, the target element indicated by the editing instruction is determined, and then based on the editing content carried in the editing instruction, a replacement element matching the editing content is obtained. Finally, the display is controlled to display the picture book with updated elements, that is, the display is controlled to display the picture book interaction results generated by user interaction, wherein the picture book with updated elements is obtained by replacing the target elements based on the replacement elements. By adopting the above-mentioned technical scheme, different replacement elements can be generated according to different editing instructions, thereby obtaining picture books with different element updates, that is, obtaining different picture book interaction results, and then different picture book interaction results can be displayed to users, thereby realizing flexible adjustment of the picture book interaction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0036] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0037] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0038] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0039] Figure 5 A schematic diagram of a module of a display device provided in some embodiments of the present application;

[0040] Figure 6 A schematic diagram of a flow chart of an element replacement step performed by a controller provided in some embodiments of the present application;

[0041] Figure 7 A schematic diagram of a flow chart of an element replacement step performed with the assistance of a server provided in some embodiments of the present application;

[0042] Figure 8An interactive flow chart of a picture book interactive processing method based on a display device provided in some embodiments of the present application;

[0043] Fig. 9 An interactive flow chart of a picture book interactive processing method based on a display device and a server provided in some embodiments of the present application;

[0044] Fig.10 A flowchart of a picture book interaction processing method provided in some embodiments of the present application. DETAILED DESCRIPTION

[0045] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0046] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0047] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0048] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0049] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0050] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0051] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , the user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0052] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0053] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN) and other networks.

[0054] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function of a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0055] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0056] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0057] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0058] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface. The display device may also include a touch component connected to the display 260, which may be a touch pad, a touch screen, etc.

[0059] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0060] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0061] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0062] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0063] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0064] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0065] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0066] Figure 3 Some embodiments of the present application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0067] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0068] In some embodiments, the control device 100 may be a smart device, for example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0069] In some embodiments, Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0070] The controller 110 includes a processor 112, a RAM 113, a ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between the internal components and the external and internal data processing functions.

[0071] The communication interface 130 implements communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of other near field communication modules such as a WiFi chip 131, a Bluetooth module 132, and an NFC module 133.

[0072] The user input / output interface 140 , wherein the input interface includes at least one of other input interfaces such as a microphone 141 , a touch panel 142 , a sensor 143 , and a button 144 .

[0073] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with a communication interface 130, such as a WiFi, Bluetooth, NFC or other module, and can encode the user input command through the WiFi protocol, Bluetooth protocol, or NFC protocol and send it to the display device 200.

[0074] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0075] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0076] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system may provide a user interface (control the display device), allow the user to interact with the display device 200, and support the running of various application programs.

[0077] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.

[0078] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications layer) (referred to as "application layer"), the application framework layer (Application Framework layer) (referred to as "framework layer"), the system library layer and the kernel layer.

[0079] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0080] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions of applications in the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0081] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager is used to interact with all activities running in the system; a location manager is used to provide system services or applications with access to system location services; a package manager is used to retrieve various information related to the application package currently installed on the device; a notification manager is used to control the display and clearing of notification messages; and a window manager is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0082] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0083] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0084] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 4As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0085] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0086] In some embodiments, Figure 5 As shown, a display device is provided, which includes: a display 510, configured to display a picture book, the picture book includes at least one editable element; an input module 520, configured to receive editing instructions input by a user for the picture book; a controller 530, configured to: determine the target element indicated by the editing instruction in response to the editing instruction for at least a part of the editable elements in the picture book; based on the editing content carried in the editing instruction, obtain a replacement element matching the editing content; control the display to display the picture book with updated elements, the picture book with updated elements is obtained by replacing the target element with the replacement element.

[0087] In this embodiment, the picture book is specifically: an electronic book that is mainly composed of paintings and a small amount of text. The editable elements in the picture book can specifically refer to: editable images in the picture book, and the user can edit the image, form, style, background, color, etc. of the object in the image. The number of target elements can be multiple, so that in this embodiment, multiple target elements in the picture book can be replaced in batches at the same time, for example, all "cloud" images in the picture book are replaced with "star" images.

[0088] The images include but are not limited to: dogs, cats, people, flowers, grass, rivers, stars, eggs, etc. Editing of images includes but is not limited to: changing a dog into a cat, changing an egg into a chicken, changing the moon into a star, etc. Editing of forms includes but is not limited to: changing an upright puppy into a squatting puppy, changing a straight river into a winding river, changing a running person into a running person, changing a person without makeup into a person with makeup, etc. Editing of painting styles includes but is not limited to: changing a cartoon style into an ink painting style, changing a figurative style into an abstract style composed of lines, changing a realistic style into an anime style, etc. Editing of backgrounds includes but is not limited to: changing a night background into a day background, changing a winter background into a spring background, changing an indoor background into an outdoor background, etc. Editing of colors includes but is not limited to: changing the saturation, contrast, and hue of colors, for example, changing a cold color into a warm color.

[0089] Specifically, the user can trigger a start instruction for a picture book application in the display device through the input module, wherein at least one picture book can be displayed in the picture book application. After the picture book application is started, the user can trigger an edit instruction for any picture book displayed on the display through the input module. The triggering method of the start instruction and the triggering method of the edit instruction are not unique.

[0090] Optionally, taking the input module including the touch component as an example, the display of the display device can display the application logo of the picture book application, and the user can select the application logo through the touch component of the display device to trigger the start instruction for the picture book application. If the input module also includes a voice control component, the user can also input a voice interaction instruction through the voice control component to trigger the start instruction for the picture book application.

[0091] Furthermore, when the user inputs a startup command for the picture book application in the display device, the controller of the display device can respond to the startup command and control the display to display the user interface of the picture book application. The user interface (UI) refers to the medium for interaction and information exchange between the system and the user, which can specifically include a graphical user interface (GUI), a command line interface (CLI), a touch interface, a voice user interface (VUI) and other forms.

[0092] Furthermore, after the display shows the user interface of the picture book application, the user can trigger an editing instruction for any picture book displayed on the user interface. Taking the input module including the touch component as an example, the user can select any picture book page in any picture book through the touch component of the display device and input an editing instruction for the selected picture book page. If the input module includes a voice control component, the user can also input an editing instruction for any picture book page in any picture book through the voice control component.

[0093] More specifically, after the input module receives the editing instructions input by the user for the picture book, the input module can forward the editing instructions to the controller, and the controller can determine the target elements indicated by the editing instructions in the picture book in response to the editing instructions for at least a portion of the editable elements in the picture book. Furthermore, the controller can obtain a replacement element that matches the editing content based on the editing content carried in the editing instructions. This replacement element can be generated by a model deployed on the controller based on the editing content, or it can be generated with the help of a model deployed on other devices. Furthermore, the controller can control the display to display the picture book with updated elements, wherein the picture book with updated elements is obtained by replacing the target element with the replacement element.

[0094] In an optional embodiment, when the data processing capability of the controller reaches a preset condition, the controller can execute the step of generating a replacement element matching the edited content based on the edited content carried in the edit instruction, and can also execute the step of replacing the target element with the replacement element to obtain the updated element, and control the display to display the updated element in the area where the target element is located, so as to display the updated element picture book to the user. When the data processing capability of the controller does not reach the preset condition, the controller can forward the edit instruction carrying the edited content to the server, so as to generate the replacement element with the help of the server, and obtain the replacement element from the server, and then, can complete the replacement operation of the target element with the replacement element with the help of the server, and obtain the updated element picture book from the server.

[0095] For example, taking the data processing capability of the controller reaching the preset conditions as an example, an image generation model can be deployed on the controller, so that by inputting the edited content into the image generation model, a replacement element output by the image generation model that matches the edited content is obtained. The image generation model can be a text-based image model trained based on a deep learning framework, which can be used to generate various images that conform to the text description in the edited content. The image generation model used in this embodiment includes but is not limited to: (1) a model trained based on Generative Adversarial Networks (GANs). GANs can achieve data generation through adversarial training between two neural networks, the generator and the discriminator, for example, to achieve text-to-image conversion. (2) a model trained based on Variational Auto-Encoders (VAE). VAE is a form of deep generative model and a generative network structure based on variational Bayesian inference.

[0096] For example, if the generation of replacement elements is achieved with the help of a server or other device, the above-mentioned image generation model may also be deployed on the server or other device to generate matching replacement elements according to the edited content.

[0097] The training process of the above-mentioned image generation model can be specifically as follows: pre-constructing a data set, the data set includes various types of images, and each image carries a label, and the label can specifically include the following content: text content that describes the image, form, style, background, color and other features of the object in the image. Further, using the images with labels in the data set, training a generative adversarial network or a variational autoencoder, so that the generative adversarial network or the variational autoencoder learns how to generate corresponding images according to the text content. When the number of training times reaches a preset number, or the loss function value of the model obtained after training is less than a preset value, the training can be stopped to obtain the desired image generation model.

[0098] The above technical solution has the following advantages or beneficial effects: by using the above display device, the editing instructions input by the user for the picture book can be received through the input module, and the target element indicated by the editing instruction can be determined through the controller in response to the editing instructions for at least a part of the editable elements in the picture book, so as to obtain the replacement element matching the editing content based on the editing content carried in the editing instruction, and then control the display to display the picture book with updated elements. Among them, the picture book with updated elements is obtained by replacing the target element based on the replacement element. That is, by using the above display device, the editing instructions input by the user for the picture book can be received through interaction with the user, so as to edit the elements in the picture book according to the editing instructions, and the picture book with updated elements can be displayed to the user, that is, the picture book interaction result can be displayed to the user. If the editing instructions generated by the interaction are different, by using the above display device, different picture book interaction results can be displayed to the user, so as to realize flexible adjustment of the picture book interaction results.

[0099] In some embodiments, the input module is further configured to: in response to an edit trigger operation for at least a portion of the editable elements in the picture book, receive multiple editing instructions input sequentially; when multiple editing instructions all meet the judgment conditions of the same interaction round, use the last editing instruction input in the same interaction round as the valid editing instruction for at least a portion of the editable elements.

[0100] Among them, the touch component or voice control component in the input module can support the recognition of multiple editing instructions input sequentially in a short period of time, and identify valid editing instructions from multiple editing instructions. The judgment condition that two editing instructions meet the same interaction round can be specifically: the input interval between the two editing instructions does not exceed the preset time length. That is, if two editing instructions are input continuously within the preset time length, the two editing instructions can be regarded as belonging to the same interaction round. If the two editing instructions are not input continuously within the preset time length, the two editing instructions can be regarded as belonging to different interaction rounds and can be executed independently.

[0101] Optionally, the input module may receive a plurality of editing instructions input in sequence in response to an editing trigger operation for at least a portion of the editable elements in the picture book, and determine the input interval duration between any two adjacent editing instructions. When the plurality of editing instructions all meet the judgment condition of the same interaction round, for example, the input interval duration between any two adjacent editing instructions in the plurality of editing instructions does not exceed a preset duration, the input module may determine that the plurality of editing instructions input in sequence belong to the same interaction round, and use the last editing instruction input in the same interaction round as a valid editing instruction for at least a portion of the editable elements.

[0102] Optionally, when multiple editing instructions do not meet the judgment conditions of the same interaction round, that is, when the input interval duration between two editing instructions exceeds a preset duration, the input module may divide the two editing instructions into different interaction rounds, for example, dividing the editing instruction input first into the interaction round executed first, and dividing the editing instruction input later into the interaction round executed later.

[0103] It should be noted that the multiple editing instructions input sequentially may be editing instructions received by different components (including a touch component and a voice control component) or may be editing instructions received by the same component.

[0104] For example, after the picture book application is started, the user can input the first editing instruction "turn the kitten into a puppy" through the touch component for any picture book page of any picture book, then input the second editing instruction "turn the kitten into a white puppy" through the voice control component, and then input the third editing instruction "turn the kitten into a white puppy with a bow" through the voice control component. Based on this, the input module can receive multiple editing instructions input sequentially. If multiple editing instructions meet the judgment conditions of the same interaction round, the input module can only use the third editing instruction "turn the kitten into a white puppy with a bow" as a valid editing instruction, and send the valid editing instruction to the controller, so that the controller can control the display to display the picture book with updated elements. Compared with the original picture book, the "kitten" has been replaced with "white puppy with a bow" in the picture book with updated elements.

[0105] In some embodiments, when multiple editing instructions do not meet the judgment conditions of the same interaction round, if the first editing instruction is "change the kitten into a puppy", the second editing instruction is "change the kitten into a white puppy", and the third editing instruction is "change the kitten into a puppy with a bow tie", and the input interval between the first editing instruction and the second editing instruction exceeds the preset time, and the input interval between the second editing instruction and the third editing instruction does not exceed the preset time, then the input module can use the first editing instruction as a valid editing instruction for the interaction round that needs to be executed first, and send it to the controller first, divide the second editing instruction and the third editing instruction into the interaction round that is executed later, and use the third editing instruction as a valid editing instruction for the interaction round that is executed later. Based on this, the controller can first obtain the replacement element "puppy" according to the editing instruction "change the kitten into a puppy" that needs to be executed first, and control the display to display the picture book with the updated element, in which the "kitten" has been replaced by "puppy". Furthermore, the controller can obtain the replacement element "puppy with a bow" according to the editing instruction "change the kitten into a puppy with a bow" that needs to be executed later, so that "puppy with a bow" can replace "puppy", thereby realizing secondary editing of the elements in the picture book.

[0106] In a possible implementation, when multiple editing instructions satisfy the judgment condition of the same interaction round, if there are non-mutually exclusive text description contents in the multiple editing instructions, the input module can merge the multiple editing instructions to obtain the final effective editing instruction. For example, if the first editing instruction is "change the kitten into a puppy", the second editing instruction is "change the kitten into a white puppy", and the third editing instruction is "change the kitten into a puppy with a bow", the input module can merge them to obtain the effective editing instruction "change the kitten into a white puppy with a bow".

[0107] The above technical solution has the following advantages or beneficial effects: it supports users to continuously input multiple editing instructions (including multiple rounds of voice input and multiple rounds of text input), and can accurately identify valid editing instructions from multiple editing instructions generated by multiple rounds of interactions, so that the controller can accurately control the display to display the updated picture book with the elements required by the user according to the received valid editing instructions, accurately feedback the picture book interaction results to the user, improve the fun and interactivity of the picture book, and enhance user participation.

[0108] In some embodiments, when the input module executes the step of receiving editing instructions input by the user for a picture book, it is further configured to: receive interaction instructions input for the picture book, the interaction instructions containing element keywords; when the picture book contains elements represented by the element keywords, use the interaction instructions as editing instructions for the picture book.

[0109] When the controller executes the step of determining the target element indicated by the editing instruction in response to the editing instruction for at least a portion of the editable elements in the picture book, it is further configured to: determine the element in the picture book corresponding to the element keyword as the target element indicated by the editing instruction.

[0110] The interactive instructions may include text interactive instructions and voice interactive instructions. The element keywords may specifically be key fields carried by the editing content of the elements in the picture book, such as "cat", "dog", "moon", etc., so that the controller can quickly determine the target element to be edited in the picture book according to the element keywords.

[0111] Optionally, the user can input text interaction instructions for any picture book in the picture book application through the touch component, and the user can also input voice interaction instructions for any picture book in the picture book application through the voice control component. Further, the input module can receive text interaction instructions or voice interaction instructions input by the user.

[0112] If the element represented by the element keyword contained in the text interaction instruction or the voice interaction instruction exists in the picture book, the input module can use the interaction instruction as an editing instruction for the picture book and send the editing instruction to the controller. If there are multiple editing instructions, the input module can send the valid editing instruction among the multiple editing instructions to the controller. Furthermore, the controller can determine the element corresponding to the element keyword in the picture book as the target element indicated by the editing instruction.

[0113] When the element represented by the element keyword contained in the text interaction instruction or the voice interaction instruction does not exist in the picture book, the input module can feedback the result of the failed interaction instruction recognition to the controller, and the controller can control the display to display a prompt message to prompt the user to re-enter the interaction instruction, or the controller can control the voice broadcast device in the display device to issue a preset voice prompt to prompt the user to re-enter the interaction instruction.

[0114] In a possible implementation, a hidden element or a visible element may be set in the picture book page of any picture book in the picture book application according to the plot of the picture book. When the user uses the display device to consult the picture book, the user can click the preset hidden element or visible element through the touch component, and the touch component can send the click operation to the controller in the form of an instruction. Further, the controller can control the display to display a text interaction instruction input area based on the click operation, so that the user can enter the text interaction instruction in the text interaction instruction input area through the touch component, or the controller can control the display to display the voice interaction instruction input key, so that the user can enter the voice interaction instruction by continuously pressing the key. Based on this, the input module of the display device (including the touch component and the voice control component) can receive the editing instruction input by the user. Among them, the hidden elements or visible elements set according to the plot of the picture book can enhance the interactivity, exploratory nature, and fun of the picture book, so that the user can edit the elements in the picture book by clicking on the elements in the picture book, thereby increasing the user's participation.

[0115] It should be noted that, instead of clicking on the hidden elements or visible elements set according to the plot, the user can directly input voice interaction instructions through the voice control component to specify the editing of the target elements in the picture book.

[0116] The above technical solution has the following advantages or beneficial effects: it can determine whether there is an executable editing instruction based on the interaction with the user, for example, based on the interaction instruction input by the user, so that the controller can subsequently edit the elements in the picture book based on the editing instruction and display the picture book with updated elements, that is, display the picture book interaction results generated based on the user's picture book. Based on this, the controller can display different picture book interaction results based on different editing instructions input by the user, rather than just feeding back fixed picture book interaction results to the user, thereby realizing flexible adjustment of the picture book interaction results.

[0117] In some embodiments, the picture book includes multiple picture book pages, each picture book page includes at least one picture book element. The controller is further configured to: perform semantic analysis on the editing instruction to determine the element keyword contained in the editing instruction; search for a target picture book page containing a target element corresponding to the element keyword from the multiple picture book pages; control the display to display the target picture book page, and locate and display the target element in the target picture book page.

[0118] The number of target picture book pages may be at least one, which may be the picture book page that the user is viewing, or may be all picture book pages in the picture book that contain the target element.

[0119] Specifically, the controller may pre-store the element description words in each picture book page in the picture book. After receiving the editing instruction sent by the input module, the controller may determine the element keywords included in the editing instruction by semantically parsing the editing instruction, and determine the target picture book page containing the target element corresponding to the element keyword by querying the element description words corresponding to multiple picture book pages in the picture book. Furthermore, after the controller controls the display to display the target picture book page, the controller may first locate the area where the target element is located in the target picture book page, and locate and display the target element to be edited to the user by generating a border for the image in the area, or controlling the image in the area where the target element is located to start flashing, highlighting, etc., to achieve interaction with the user.

[0120] In one possible implementation, when there are multiple picture book pages that all include the target element corresponding to the element keyword, the controller can control the display to display multiple picture book pages including the target element in the form of reduced-size pages, so that the user can select the target picture book page. Alternatively, when the user enters an edit instruction, if the display is displaying a picture book page, and the picture book page also includes the target element corresponding to the element keyword, the controller can use the picture book page being displayed as the target picture book page, and control the display to continue displaying the picture book page.

[0121] Take the example of a user using a picture book application to browse the picture book "XXX". If the semantics of the editing instruction entered by the user is "change the kitten into a puppy", and when the user enters the editing instruction, the display is displaying the Yth picture book page of the picture book "XXX". The controller can control the display to display multiple picture book pages including the target element "kitten" in the form of a reduced version of the page, so that the user can select the final target picture book page, and after the user selects the target picture book page, control the display to position and display the "kittens" in all target picture book pages, and the "kittens" in the selected target picture book page will be replaced with "puppies". Alternatively, because the user is browsing the Yth picture book page of the picture book "XXX", if the Yth picture book page also includes the target element "kitten", the controller can directly use the Yth picture book page as the target picture book page, and control the display to position and display the "kitten" in the Yth picture book page, and after obtaining the replacement element "puppy" later, replace the "kitten" in the Yth picture book page with "puppy".

[0122] The above technical solution has the following advantages or beneficial effects: by semantically parsing the editing instructions, the element keywords contained in the editing instructions can be determined, and then the target elements can be located and displayed in the target picture book page according to the element keywords, which can be used to prompt the user to edit the target element and realize interaction with the user.

[0123] In some embodiments, each picture book page is configured with a semantic area mapping relationship; when the controller executes the step of locating and displaying the target element in the target picture book page, it is further configured to: search for the semantic area mapping relationship configured for the target picture book page based on the element keyword, and determine the area where the target element is located in the target picture book page; locate and display the target element by marking the area where the target element is located.

[0124] The semantic region mapping relationship may specifically be: a mapping relationship between a text description word of an element in a picture book page and the region where the element is located in the picture book page. For example, if there is an element A in region a in a picture book page, and the text description word corresponding to the element A is "flower", then the picture book page may be configured with a semantic region mapping relationship "flower-region a".

[0125] Optionally, a picture book application can store a variety of picture book resources, and in the originally stored picture books, a semantic area mapping relationship can be configured in the picture book page of each picture book. After the user selects a picture book and enters an editing instruction, the controller can search for the semantic area mapping relationship configured for the target picture book page according to the element keywords contained in the editing instruction to determine the area where the target element corresponding to the element keyword is located in the target picture book page, thereby positioning and displaying the target element by marking the area where the target element is located.

[0126] Exemplarily, based on the element keywords contained in the editing instructions, the semantic area mapping relationship configured for the target picture book page is searched, which may specifically include: matching the element keywords with the semantics in the semantic area mapping relationship, and if the match is successful, extracting the area with successful matching semantic association, and determining the area as the area where the target element is located.

[0127] In one embodiment, the marking processing method includes but is not limited to: (1) generating a border at the edge of the area where the target element is located. The border can be a solid line or a dashed line, can be semi-transparent or opaque, and can be configured with a color to locate and display the target element to the user. (2) controlling the image in the area where the target element is located to flash so as to locate and display the target element. (3) while controlling the image in the area where the target element is located to flash, generating a border at the edge of the area where the target element is located.

[0128] The above technical solution has the following advantages or beneficial effects: the preset semantic area mapping relationship can be queried according to the element description words in the editing instruction, so as to locate and display the target element in the target picture book page, prompt the user to edit the target element, and realize interaction with the user.

[0129] In some embodiments, the target element includes a foreground area and a background area. When the controller executes the step of controlling the display to display an updated picture book of the element, it is further configured to: obtain an update element; the update element is obtained by replacing any area in the target element based on the replacement element; and control the display to display the updated element in the area where the target element is located to display the updated picture book of the element.

[0130] It should be noted that after the controller determines the target element according to the editing instruction, if the target element is to be edited, the controller needs to first extract the image corresponding to the target element from the target picture book page, and the extracted image usually carries a part of the foreground and background. Therefore, when replacing the target element later, it is necessary to consider whether to replace both the foreground and the background, or just one of them. For example, if the target element indicated by the editing instruction is "Standing Kitten", the target element extracted by the controller includes not only the foreground image of "Standing Kitten", but also some background images in the area where the "Standing Kitten" is located.

[0131] Optionally, after obtaining the replacement element, the controller may further obtain an update element obtained by replacing any area in the target element based on the replacement element. Specifically, in the case where the replacement element is only used to replace the foreground area in the target element, the controller may obtain an update element consisting of the replacement element (as a new foreground area) and the background area; in the case where the replacement element is only used to replace the background area in the target element, the controller may obtain an update element consisting of the replacement element (as a new background area) and the foreground area; in the case where the replacement element includes two areas, both the foreground area and the background area in the target element are replaced by the replacement element, and the update element is also the replacement element. After obtaining the update element, the controller may control the display to display the update element in the area where the target element is located to display the updated picture book of the element.

[0132] Taking the data processing capability of the controller reaching the preset condition as an example, the controller can independently perform the step of replacing the target element with the replacement element to obtain the updated element without the assistance of other devices, such as Figure 6As shown, a flow chart of the element replacement step performed by the controller is provided. Specifically, the controller can first perform image segmentation on the target element based on an image segmentation model or an image segmentation algorithm to determine the foreground area and the background area in the image corresponding to the target element. Further, the controller replaces the target element with the replacement element to obtain an updated element: if the replacement element obtained is only used to replace the background area, the controller can fuse the segmented foreground area with the replacement element (as a new background area) to obtain an updated element; if the replacement element obtained is only used to replace the foreground area, the controller can fuse the segmented background area with the replacement element (as a new foreground area) to obtain an updated element; if the replacement element is used to replace the entire target element, the entire replacement element is used as the updated element. Finally, the controller can control the display to display the updated element in the area where the target element is located. Among them, when the retained area is fused with the replacement element in the target element, the controller can call a preset image fusion model to perform image edge smoothing and color matching, thereby achieving image fusion.

[0133] Taking the case where the data processing capability of the controller does not meet the preset conditions as an example, a server deployed with an image generation model can assist in generating replacement elements, and the server can assist in element replacement, such as Figure 7 As shown, a flow chart of the element replacement step performed with the assistance of a server is provided. Specifically, the server may also be deployed with an image segmentation model and an image fusion model to perform image segmentation on the target element, determine the foreground area and the background area in the image corresponding to the target element, and then replace the target element with the replacement element to obtain an updated element, and send the updated element to the display device, so that the controller in the display device can control the display to display the updated element in the area where the target element is located.

[0134] The above-mentioned image segmentation models and image fusion models can be obtained based on neural network training. Image segmentation models include but are not limited to: models trained based on FCN (Fully Convolutional Network), which can predict and segment images at the pixel level, providing more refined and accurate segmentation results. Models trained based on U-Net (Convolutional Networks for Biomedical Image Segmentation), which is a neural network designed specifically for image segmentation. It uses a U-shaped network structure that encodes first and then decodes, and can be applied to image segmentation tasks in various fields. Image fusion models include but are not limited to: models trained based on CNN (Convolutional Neural Network), which can fuse the edges of the foreground image with the background image to ensure the consistency of the fused image.

[0135] In one embodiment, during the process of displaying the updated element in the area where the target element is located, the controller can control the display to adjust the brightness, contrast and color balance of the displayed picture book page to ensure that the visual effect of the picture book page finally displayed is consistent.

[0136] For example, in the process of displaying the updated element in the area where the target element is located, the controller can fill, repair, and delete the redundant parts of the edge of the updated element through the pre-configured Inpainting handler according to the existing information of the image in the target picture book page and the image information of the updated element, so as to ensure that the updated element will not destroy the overall display effect of the target picture book page after being merged with the target picture book page. Among them, the core principle of Inpainting technology is based on the similarity and continuity between pixels. By analyzing the pixel information around the missing area, the pixels adjacent to the missing area are found, and based on the color, texture and structure information of these pixels, pixels similar to the surrounding pixels are generated to fill the missing area, or remove unnecessary parts to achieve image repair.

[0137] The above technical solution has the following advantages or beneficial effects: the target element indicated by the editing instruction can be edited according to the editing instruction input by the user, and after the editing is completed, the picture book interaction result (the picture book with updated elements) is automatically displayed to the user, and the display effect of the picture book in the picture book interaction result can be ensured, thereby improving the user's picture book usage experience.

[0138] In an optional embodiment, if Figure 8 As shown, the picture book interaction processing method based on the display device includes the following interaction process:

[0139] Step 1: The user can trigger the picture book application start instruction through the input module (including the touch component and the voice control component) in the display device.

[0140] Step 2: The input module can send a start command to the controller.

[0141] Step 3: The controller controls the picture book application to start according to the start instruction, and controls the display to display the picture book application.

[0142] Step 4: The monitor displays the interface of the picture book application.

[0143] Step 5: After the display shows the interface of the picture book application, the user can trigger an interactive instruction for the picture book displayed on the display through the input module.

[0144] Step 6: When the element represented by the element keyword contained in the interactive instruction exists in the picture book, the input module can use the interactive instruction as an editing instruction for the picture book and send the editing instruction to the controller. It should be noted that when multiple editing instructions input in sequence meet the judgment conditions of the same interactive round, the input module can use the last editing instruction input in the same interactive round as a valid editing instruction and send it to the controller. Otherwise, multiple editing instructions are divided into different interactive rounds, and the valid editing instructions of each interactive round are determined separately and then sent to the controller.

[0145] Step seven: The controller can determine the element in the picture book corresponding to the element keyword contained in the valid editing instruction as the target element indicated by the valid editing instruction based on the received valid editing instruction, and then search for the target picture book page containing the target element corresponding to the element keyword from multiple picture book pages, and determine the area of ​​the target element in the target picture book page, so as to control the display to locate and display the target element while displaying the target picture book page.

[0146] Step 8: The display shows the target picture book page and target element to be edited.

[0147] Step 9: The controller may further obtain a replacement element and an update element obtained by replacing the target element with the replacement element, so as to control the display to display the update element in the area where the target element is located. When the data processing capability of the controller reaches the preset condition, the steps of generating the replacement element and replacing the target element may be performed by the controller. When the data processing capability of the controller does not reach the preset condition, the steps of generating the replacement element and replacing the target element may be performed with the assistance of a server or other device.

[0148] Step 10: The display displays the updated element in the area where the target element is located to display the updated picture book of the element, and displays the picture book interaction result matching the editing instruction to the user.

[0149] In some embodiments, the display device further includes a display device communication device, and the display device communication device can forward the editing instruction to the server so that the server assists the display device in generating the replacement element. A server is provided, which may include:

[0150] A server communication device configured to establish a communication connection with the display device;

[0151] The processor is configured as:

[0152] Receiving an editing instruction sent by the display device through the server communication module; the editing instruction is input for at least a part of the editable elements in the picture book displayed by the display device;

[0153] Based on the editing content carried in the editing instruction, a replacement element matching the editing content is generated; the replacement element is used to replace the target element in the picture book indicated by the editing instruction to obtain a picture book with updated elements.

[0154] Specifically, when the data processing capacity of the controller in the display device does not meet the preset conditions, the server can assist the display device in generating replacement elements. The server can receive the editing instructions sent by the display device through the server communication module, and perform semantic recognition on the editing content carried in the editing instructions to obtain image description words used to describe the replacement elements, thereby generating replacement elements that match the editing content according to the recognized image description words. Furthermore, the server can send the generated replacement elements to the controller so that the controller performs the element replacement step, and the server can also assist in performing the element replacement step based on the generated replacement elements.

[0155] The above technical solution has the following advantages or beneficial effects: by using the above server, the editing instructions sent by the display device can be received through the server communication module, wherein the editing instructions are input for at least a part of the editable elements in the picture book displayed by the display device. Furthermore, a processor can generate a replacement element matching the editing content based on the editing content carried in the editing instruction, wherein the replacement element is used to replace the target element in the picture book indicated by the editing instruction, so as to obtain a picture book with updated elements, that is, to obtain a picture book interaction result generated by the editing instruction. Based on this, by using the above server, different replacement elements can be generated according to different editing instructions, so as to obtain picture books with different updated elements, that is, to obtain different picture book interaction results, thereby realizing flexible adjustment of the picture book interaction results.

[0156] In some embodiments, when the processor executes the step of generating a replacement element matching the editing content based on the editing content carried in the editing instruction, the processor is further configured to: obtain an image generation model; input the editing content carried in the editing instruction into the image generation model, and obtain the replacement element output by the image generation model.

[0157] The image generation model can be a text-based graph model trained based on a deep learning framework, which can be used to generate various images that conform to the text description in the edited content. The image generation model deployed on the server includes but is not limited to: (1) a model trained based on Generative Adversarial Networks (GANs). (2) a model trained based on Variational Auto-Encoders (VAE).

[0158] Specifically, the server may input the image description words identified from the edited content into the image generation model, thereby obtaining the replacement elements output by the image generation model.

[0159] The above technical solution has the following advantages or beneficial effects: the image generation model can be used to generate replacement elements that meet the user's editing needs, so that the replacement elements can be used later to accurately replace the target elements, which is conducive to accurately feedback to the user the picture book interaction results that match the editing instructions.

[0160] In some embodiments, the server is further configured to: identify the foreground area and the background area in the target element; match the target area where object replacement is required from the foreground area and the background area based on the target object described by the replacement element; replace the object in the target area with the target object to obtain an updated element; and update the element for display in the area where the target element is located to display the updated picture book.

[0161] The target object described by the replacement element is the object indicated in the edited content. For example, if the target element includes a foreground area A and a background area B, and the edited content is to change A to C, then the foreground area A is the target object described by the replacement element and is also the target area that needs to be replaced.

[0162] Taking the case where the controller's data processing capability does not meet the preset conditions and the server's assistance in element replacement is required as an example, after receiving the editing instructions sent by the display device, the server can use the image segmentation model to perform image segmentation on the target element, identify the foreground area and background area in the target element, and generate a replacement element. Furthermore, the server can match the target area that needs to be replaced from the foreground area and the background area based on the target object described by the replacement element, and replace the object in the target area with the target object, that is, merge the area in the target area that does not need to be replaced with the replacement element to obtain an updated element. Furthermore, the server can send the updated element to the controller so that the controller controls the display to display the updated element in the area where the target element is located.

[0163] Specifically, if the generated replacement element is only used to replace the background area, the server can merge the segmented foreground area with the replacement element (as a new background area) to obtain an updated element; if the generated replacement element is only used to replace the foreground area, the server can merge the segmented background area with the replacement element (as a new foreground area) to obtain an updated element; if the replacement element is used to replace the entire target element, the server can use the entire replacement element as an updated element.

[0164] The above-mentioned image segmentation model and image fusion model can be obtained based on neural network training. Image segmentation models include but are not limited to: models obtained based on FCN training and models obtained based on U-Net training. Image fusion models include but are not limited to: models obtained based on CNN training, which can fuse the edges of the foreground image with the background image to ensure the consistency of the fused image.

[0165] The above technical solution has the following advantages or beneficial effects: the area that needs to be replaced in the target element can be accurately replaced according to the target object described by the replacement element, thereby achieving accurate editing of the picture book and ensuring that the picture book interaction results that match the editing instructions can be accurately fed back to the user.

[0166] In some embodiments, Fig. 9 As shown, the picture book interaction processing method based on the display device and the server includes the following interaction process:

[0167] Step 1: The user can trigger the picture book application start instruction through the input module (including the touch component and the voice control component) in the display device.

[0168] Step 2: The input module can send a start command to the controller.

[0169] Step 3: The controller controls the picture book application to start according to the start instruction, and controls the display to display the picture book application.

[0170] Step 4: The monitor displays the interface of the picture book application.

[0171] Step 5: After the display shows the interface of the picture book application, the user can trigger an interactive instruction for the picture book displayed on the display through the input module.

[0172] Step 6: When the element represented by the element keyword contained in the interactive instruction exists in the picture book, the input module can use the interactive instruction as an editing instruction for the picture book and send the editing instruction to the controller. It should be noted that when multiple editing instructions input in sequence meet the judgment conditions of the same interactive round, the input module can use the last editing instruction input in the same interactive round as a valid editing instruction and send it to the controller. Otherwise, multiple editing instructions are divided into different interactive rounds, and the valid editing instructions of each interactive round are determined separately and then sent to the controller.

[0173] Step seven: The controller can determine the element in the picture book corresponding to the element keyword contained in the valid editing instruction as the target element indicated by the valid editing instruction based on the received valid editing instruction, and then search for the target picture book page containing the target element corresponding to the element keyword from multiple picture book pages, and determine the area of ​​the target element in the target picture book page, so as to control the display to locate and display the target element while displaying the target picture book page.

[0174] Step 8: The display shows the target picture book page and target element to be edited.

[0175] Step 9: If the server is required to generate a replacement element, the controller may forward the received valid editing instruction to the server. Step 9 and step 7 may be executed simultaneously, and the execution order is not limited to the order shown in this embodiment.

[0176] Step 10: The server generates a replacement element according to the edited content in the received valid edit instruction, and replaces the target element with the replacement element to obtain an updated element, and then returns the updated element to the controller. That is, when the data processing capacity of the controller does not meet the preset conditions, the steps of generating the replacement element and replacing the target element can be performed by the server.

[0177] Step 11: The controller controls the display to display the update element in the area where the target element is located.

[0178] Step 12: The display displays the updated element in the area where the target element is located to display the updated picture book of the element, and displays the picture book interaction result matching the editing instruction to the user.

[0179] Based on the same inventive concept, the embodiment of the present application also provides a picture book interaction processing method applied to the above-mentioned display device. The implementation solution provided by the method to solve the problem is similar to the implementation solution recorded in the above-mentioned display device embodiment, so the specific limitations in one or more picture book interaction processing method embodiments provided below can refer to the above-mentioned limitations on the display device, and will not be repeated here.

[0180] In some embodiments, the present application further provides a picture book interaction processing method, which is applied to the above-mentioned display device. Fig.10 As shown, the picture book interaction processing method includes the following steps:

[0181] Step 1002, receiving an editing instruction input by a user for at least a portion of editable elements in the picture book, and displaying the picture book on a display device;

[0182] Step 1004, determining the target element indicated by the editing instruction;

[0183] Step 1006, based on the edit content carried in the edit instruction, obtaining a replacement element matching the edit content;

[0184] Step 1008, controlling the display to display the updated picture book; the updated picture book is obtained by replacing the target element with the replacement element.

[0185] The above-mentioned picture book interaction processing method first receives the editing instructions input by the user for at least a part of the editable elements in the picture book, wherein the picture book is displayed on the display of the display device. Further, the target element indicated by the editing instruction is determined, and then based on the editing content carried in the editing instruction, a replacement element matching the editing content is obtained. Finally, the display is controlled to display the picture book with updated elements, that is, the display is controlled to display the picture book interaction results generated by user interaction, wherein the picture book with updated elements is obtained by replacing the target elements based on the replacement elements. By adopting the above-mentioned technical scheme, different replacement elements can be generated according to different editing instructions, thereby obtaining picture books with different element updates, that is, obtaining different picture book interaction results, and then different picture book interaction results can be displayed to the user, thereby realizing flexible adjustment of the picture book interaction results.

[0186] In some embodiments, the above-mentioned picture book interaction processing method also includes: in response to an edit trigger operation for at least a portion of the editable elements in the picture book, receiving multiple editing instructions input sequentially; when multiple editing instructions all meet the judgment conditions of the same interaction round, using the last edit instruction input in the same interaction round as a valid editing instruction for at least a portion of the editable elements.

[0187] In some embodiments, the above-mentioned picture book interaction processing method also includes: receiving an interaction instruction input for the picture book, the interaction instruction including an element keyword; when the picture book includes an element represented by the element keyword, using the interaction instruction as an editing instruction for the picture book.

[0188] In some embodiments, the above picture book interaction processing method further includes: determining the element corresponding to the element keyword in the picture book as the target element indicated by the editing instruction.

[0189] In some embodiments, the above-mentioned picture book interaction processing method also includes: performing semantic analysis on the editing instructions to determine the element keywords contained in the editing instructions; searching for a target picture book page containing a target element corresponding to the element keyword from multiple picture book pages; controlling the display to display the target picture book page, and locating and displaying the target element in the target picture book page.

[0190] In some embodiments, the above-mentioned picture book interaction processing method also includes: searching for the semantic area mapping relationship configured for the target picture book page according to the element keyword, and determining the area where the target element is located in the target picture book page; positioning and displaying the target element by marking the area where the target element is located.

[0191] In some embodiments, the above-mentioned picture book interaction processing method also includes: obtaining an update element; the update element is obtained by replacing any area in the target element based on the replacement element; controlling the display to display the update element in the area where the target element is located to display the picture book with the updated element.

[0192] Based on the same inventive concept, the embodiment of the present application also provides a picture book interaction processing device for implementing the picture book interaction processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more picture book interaction processing device embodiments provided below can refer to the limitations of the picture book interaction processing method above, and will not be repeated here.

[0193] In some embodiments, the present application further provides a picture book interaction processing device, which is applied to the above-mentioned display device. In this embodiment, the picture book interaction processing device includes:

[0194] An editing instruction receiving module, used to receive an editing instruction input by a user for at least a part of the editable elements in the picture book, and the picture book is displayed on a display of a display device;

[0195] A target element determination module, used to determine the target element indicated by the editing instruction;

[0196] A replacement element acquisition module, used to acquire a replacement element matching the edited content based on the edited content carried in the edit instruction;

[0197] The picture book display control module is used to control the display to display the updated picture book of elements, and the updated picture book of elements is obtained by replacing the target element based on the replacement element.

[0198] The above-mentioned picture book interaction processing device first receives the editing instructions input by the user for at least a part of the editable elements in the picture book, wherein the picture book is displayed on the display of the display device. Further, the target element indicated by the editing instruction is determined, and then based on the editing content carried in the editing instruction, a replacement element matching the editing content is obtained. Finally, the display is controlled to display the picture book with updated elements, that is, the display is controlled to display the picture book interaction results generated by user interaction, wherein the picture book with updated elements is obtained by replacing the target elements based on the replacement elements. By adopting the above-mentioned technical scheme, different replacement elements can be generated according to different editing instructions, thereby obtaining picture books with different element updates, that is, obtaining different picture book interaction results, and then different picture book interaction results can be displayed to users, thereby realizing flexible adjustment of the picture book interaction results.

[0199] In some embodiments, the editing instruction receiving module is also used to: receive multiple editing instructions input sequentially in response to an editing trigger operation on at least a portion of the editable elements in the picture book; when multiple editing instructions all meet the judgment conditions of the same interaction round, use the last editing instruction input in the same interaction round as the valid editing instruction for at least a portion of the editable elements.

[0200] In some embodiments, the editing instruction receiving module is further used to: receive interaction instructions input for the picture book, the interaction instructions containing element keywords; when the picture book contains elements represented by the element keywords, use the interaction instructions as editing instructions for the picture book.

[0201] In some embodiments, the target element determination module is further used to: determine the element corresponding to the element keyword in the picture book as the target element indicated by the editing instruction.

[0202] In some embodiments, the above-mentioned picture book interaction processing device also includes a positioning and display module, which is used to: perform semantic analysis on the editing instructions to determine the element keywords contained in the editing instructions; search for a target picture book page containing a target element corresponding to the element keyword from multiple picture book pages; control the display to display the target picture book page, and locate and display the target element in the target picture book page.

[0203] In some embodiments, the positioning and display module can also be used to: find the semantic area mapping relationship configured for the target picture book page according to the element keyword, and determine the area where the target element is located in the target picture book page; locate and display the target element by marking the area where the target element is located.

[0204] In some embodiments, the picture book display control module is also used to: obtain an update element; the update element is obtained by replacing any area in the target element based on the replacement element; and control the display to display the update element in the area where the target element is located to display the picture book with the updated element.

[0205] Each module in the above picture book interactive processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0206] In some embodiments, a display device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the above method steps are implemented when the processor executes the computer program.

[0207] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method steps are implemented.

[0208] In some embodiments, a computer program product is provided, comprising a computer program, which implements the above method steps when executed by a processor.

[0209] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0210] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0211] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A display device, characterized in that: The display device comprises: A display configured to display a picture book, wherein the picture book includes at least one editable element; An input module, configured to receive an editing instruction input by a user for the picture book; The controller is configured as: In response to an editing instruction for at least a portion of the editable elements in the picture book, determining a target element indicated by the editing instruction; Based on the editing content carried in the editing instruction, obtaining a replacement element matching the editing content; The display is controlled to display an updated picture book of an element, wherein the updated picture book of an element is obtained by replacing the target element based on the replacement element.

2. The display device according to claim 1, characterized in that The input module is further configured to: In response to an edit triggering operation on at least a portion of the editable elements in the picture book, receiving a plurality of edit instructions input sequentially; When the multiple editing instructions all meet the judgment condition of the same interaction round, the editing instruction input last in the same interaction round is used as the valid editing instruction for the at least part of the editable elements.

3. The display device according to claim 1, characterized in that When the input module executes the step of receiving the editing instruction input by the user for the picture book, the input module is further configured to: receiving an interaction instruction input for the picture book, wherein the interaction instruction includes an element keyword; In the case where the picture book contains the element represented by the element keyword, using the interaction instruction as an editing instruction for the picture book; When the controller executes the step of determining the target element indicated by the editing instruction in response to the editing instruction for at least a part of the editable elements in the picture book, the controller is further configured to: The element in the picture book corresponding to the element keyword is determined as the target element indicated by the editing instruction.

4. The display device according to claim 1, characterized in that The picture book includes a plurality of picture book pages, and each of the picture book pages includes at least one picture book element; The controller is also configured to: Performing semantic analysis on the editing instruction to determine element keywords contained in the editing instruction; Searching, from the plurality of picture book pages, a target picture book page containing a target element corresponding to the element keyword; The display is controlled to display the target picture book page, and the target element is positioned and displayed in the target picture book page.

5. The display device according to claim 4, characterized in that Each of the picture book pages is configured with a semantic area mapping relationship; when the controller executes the step of locating and displaying the target element in the target picture book page, it is further configured to: According to the element keyword, searching for a semantic area mapping relationship configured for the target picture book page, and determining the area where the target element is located in the target picture book page; The target element is positioned and displayed by marking the area where the target element is located.

6. The display device according to claim 1, characterized in that The target element includes a foreground area and a background area. When the controller executes the step of controlling the display to update the picture book of the element, the controller is further configured as follows: Acquire an update element; the update element is obtained by replacing any region in the target element based on the replacement element; The display is controlled to display the updated element in the area where the target element is located, so as to display the updated picture book of the element.

7. A server, characterized in that: The server comprises: A server communication device configured to establish a communication connection with the display device; The processor is configured as: Receiving, through the server communication module, an editing instruction sent by the display device; the editing instruction is input for at least a part of the editable elements in the picture book displayed by the display device; Based on the editing content carried in the editing instruction, a replacement element matching the editing content is generated; the replacement element is used to replace the target element in the picture book indicated by the editing instruction to obtain a picture book with updated elements.

8. The server according to claim 7, characterized in that: When the processor executes the step of generating a replacement element matching the edit content based on the edit content carried in the edit instruction, the processor is further configured to: Get the image generation model; The editing content carried in the editing instruction is input into the image generation model to obtain a replacement element output by the image generation model.

9. The server according to claim 7, characterized in that: The server is also configured to: Identifying a foreground region and a background region in the target element; Based on the target object described by the replacement element, matching a target area requiring object replacement from the foreground area and the background area; Replacing the object in the target area with the target object to obtain an updated element; The update element is used to be displayed in the area where the target element is located to display the updated picture book of the element.

10. A picture book interactive processing method, characterized in that: Applied to the display device according to any one of claims 1 to 6, the method comprises: receiving an editing instruction input by a user for at least a portion of editable elements in a picture book, wherein the picture book is displayed on a display of the display device; Determining a target element indicated by the editing instruction; Based on the editing content carried in the editing instruction, obtaining a replacement element matching the editing content; The display is controlled to display an updated picture book of an element, wherein the updated picture book of an element is obtained by replacing the target element based on the replacement element.

Images