Display method and device of electronic equipment, electronic equipment and storage medium

Abstract

The invention provides a display method and device of electronic equipment, the electronic equipment and a storage medium, the electronic equipment comprises a display screen, and the display method comprises the following steps: obtaining a display image in a display interface of the display screen; performing image recognition on the display image to obtain at least one image element, and setting a label for the image element; obtaining selected target elements in the image elements; inputting the content of the tag corresponding to the target element into a large language model to obtain a text related to the target element; and displaying the content of the text on the display interface. The method comprises the following steps: performing image recognition on a display image to obtain at least one image element, setting a label for the image element, then obtaining a target element in the image elements selected by a user, and then inputting the content of the label corresponding to the target element into a large language model to obtain a text related to the target element. And the content of the text is displayed on the display interface, so that the reading convenience is improved.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to a display method and device for an electronic device, an electronic device, and a storage medium. Background Art

[0002] Electronic ink screen technology uses electrophoresis technology to display images. The display effect of this type of screen is very close to that of traditional paper, so it is also called "electronic paper". Electronic ink is usually made of a thin film composed of a large number of microcapsules. The microcapsules contain pigment particles with different charges. In the initial state, the pigment particles are suspended in the microcapsules. When an electric field in a certain direction is applied, the corresponding pigment particles are pushed to the top, and the microcapsules will display different colors. Microcapsules of different colors form various texts and patterns.

[0003] In recent years, with the development of science and technology, portable electronic devices represented by smart phones, tablets, and e-readers have become more and more popular. More and more readers like to read documents, information or materials on portable electronic devices (especially e-readers, tablets and mobile phones), because electronic ink screens have the advantages of eye protection and low blue light radiation, which are suitable for long-term reading and learning. When readers read documents, information or materials, they will be interested in the characters, buildings, plants, animals and other elements in the documents, information or materials, and have the desire to learn more. At this time, readers may need to actively exit the reading interface and search for related content of the elements, which undoubtedly brings inconvenience to readers and wastes reading time. Summary of the invention

[0004] The present application provides a display method and device for an electronic device, an electronic device, and a storage medium to improve the convenience of reading.

[0005] In a first aspect, the present application provides a display method of an electronic device, wherein the electronic device includes a display screen, and the display method includes: Acquire a display image in a display interface of the display screen; Performing image recognition on the displayed image to obtain at least one image element, and setting a label for the image element; Obtaining a selected target element from the image elements; Inputting the content of the label corresponding to the target element into a large language model to obtain text related to the target element; The content of the text is displayed on the display interface.

[0006] In some embodiments of the present application, the step of obtaining the selected target element in the image element includes: Obtaining position information of a cursor on the display interface; The image element overlapping with the cursor is obtained according to the position information of the cursor and used as the selected target element.

[0007] In some embodiments of the present application, the step of obtaining the image element overlapping with the cursor according to the position information of the cursor and using it as the selected target element includes: Obtaining the image element overlapping the cursor according to the position information of the cursor; Determine whether the number of the image elements that have overlapping parts with the cursor is greater than 1. If so, the image element with the largest overlapping area with the cursor is selected as the target element. Otherwise, the image element that overlaps with the cursor is selected as the target element.

[0008] In some embodiments of the present application, the step of performing image recognition on the displayed image to obtain at least one image element includes: Detecting the displayed image using a target detection model to obtain at least one image element region and position information of the image element region; The image element region is identified by an instance segmentation model to obtain a mask, and a display region corresponding to the mask is the image element.

[0009] In some embodiments of the present application, the display method further includes: The text is associated with the target element and a tag corresponding to the target element and stored in a storage unit.

[0010] In some embodiments of the present application, the label includes a category name, and before the step of inputting the content of the label corresponding to the target element into a large language model to obtain text related to the target element, the display method further includes: Determine whether the category name of the label corresponding to the target element exists in the storage unit. If so, obtain the text corresponding to the category name of the label from the storage unit, and display the content of the text corresponding to the category name of the label on the display interface; if not, execute the next step.

[0011] In some embodiments of the present application, the display method further includes: Obtain the cropped image corresponding to the target element; The step of associating the text with the target element and the label corresponding to the target element and storing them in a storage unit includes: Associating the text with the target element, the cropped image, and a label corresponding to the target element and storing them in a storage unit; The step of determining whether the category name of the tag corresponding to the target element exists in the storage unit, and if so, obtaining text corresponding to the category name of the tag from the storage unit, and displaying the content of the text corresponding to the category name of the tag on the display interface; if not, executing the next step, including: Determine whether the category name of the label corresponding to the target element exists in the storage unit. If so, obtain the text and cropped image corresponding to the category name of the label from the storage unit, and display the content of the text and cropped image corresponding to the category name of the label on the display interface; if not, execute the next step.

[0012] In some embodiments of the present application, the step of displaying the content of the text on the display interface includes: The content of the text is displayed on the display interface in the form of text.

[0013] In some embodiments of the present application, the step of displaying the content of the text on the display interface includes: The content of the text is converted into voice, and the voice is played on the display interface.

[0014] In a second aspect, the present application further provides a display device of an electronic device, wherein the electronic device includes a display screen, and the display device includes: An acquisition module, the acquisition module is used to acquire a display image in a display interface of the display screen; a recognition module, the recognition module being used to perform image recognition on the displayed image to obtain at least one image element, and to set a label for the image element; A target module, the target module is used to obtain a target element selected from the image elements; A model module, the model module is used to input the content of the label corresponding to the target element into a large language model to obtain text related to the target element; A display module is used to display the content of the text on the display interface.

[0015] In a third aspect, the present application further provides an electronic device, including: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the electronic device implements the display method as described in any one of the first aspects.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the display method as described in any one of the first aspects is implemented.

[0017] The present application provides a display method, device, electronic device and storage medium of an electronic device, wherein the electronic device includes a display screen, and the display method includes: obtaining a display image in a display interface of the display screen; performing image recognition on the display image to obtain at least one image element, and setting a label for the image element; obtaining a target element selected from the image elements; inputting the content of the label corresponding to the target element into a large language model to obtain text related to the target element; and displaying the content of the text on the display interface. The present application obtains at least one image element by performing image recognition on the display image, and sets a label for the image element, then obtains the target element from the image element selected by the user, and then inputs the content of the label corresponding to the target element into a large language model to obtain text related to the target element, and displays the content of the text on the display interface, so that the user does not need to exit the reading interface to obtain the relevant content of the target element, thereby improving the convenience of reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of Embodiment 1 of a display method for an electronic device provided in the present application; Figure 2 A flowchart of step S30 of the display method of the electronic device provided in this application; Figure 3 A flowchart of step S32 of the display method of the electronic device provided in this application; Figure 4 A flowchart of step S20 of the display method of the electronic device provided in this application; Figure 5 A flowchart of Embodiment 2 of the display method of the electronic device provided in the present application; Figure 6 This is a flowchart of Embodiment 3 of the display method of the electronic device provided in this application; Figure 7 A flowchart of a fourth embodiment of a display method for an electronic device provided in the present application; Figure 8 A schematic diagram of the structure of a display device of an electronic device provided in this application; Fig. 9 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0019] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only parts related to the embodiments of the present application are shown in the accompanying drawings, rather than all structures.

[0020] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, but not to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more, and it cannot be understood as indicating or implying relative importance. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship. In addition, unless otherwise clearly specified and limited, the terms "set", "install", "connect", "connect", and "connect in series" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be a connection between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood in specific circumstances.

[0021] The display method provided in the embodiments of the present application can be used in electronic devices, especially electronic devices equipped with ink screens. The electronic devices can be electronic devices such as tablet computers, mobile phones, e-readers, personal digital assistants (PDAs) based on ink screens, etc. Specifically, the electronic devices can include an ink screen, an image processing module, a microcontroller, a screen driver module, a timing control circuit, and a power control circuit.

[0022] The following is a detailed description of the various embodiments of the present application.

[0023] Figure 1 This is a flowchart of Example 1 of the display method of the electronic device provided in the present application. The display method of the electronic device provided in the embodiment of the present application can be executed by a display device, which can be implemented by hardware and / or software and integrated in an electronic device equipped with an ink screen.

[0024] The following description is made by taking a display device executing a display method of an electronic device as an example, wherein the electronic device includes a display screen. Figure 1, the display method comprises: Step S10: Acquire a display image in the display interface of the display screen.

[0025] Among them, the display interface is the current interface in the display screen or the interface located at the top layer, that is, the interface that the user is reading. At this time, the display image in the display interface is the relevant content that the user is reading.

[0026] Step S20: Perform image recognition on the displayed image to obtain at least one image element, and set a label for the image element.

[0027] The types of the image elements include: objects, people, animals, plants, scenic spots, buildings, etc., and may also include nouns in text. At least one image element is obtained by performing image recognition on the display image, and a label is set for the image element, and the label includes a text description of the image element and the coordinates of the image element, etc. Specifically, the text description of the image element includes a category name, etc.

[0028] In addition, in order to improve the recognition accuracy of image elements, the name of the displayed image and the context words of the image element can be used as associated data. Specifically, if the displayed image is a page of a book, the title of the book and the context words of the page are used as associated data. This can improve the recognition accuracy of image elements. For example, if the title of the book is a list of wild animals, the recognition result of the image element may include animals such as tigers or lions, but it is unlikely to be a sculpture.

[0029] Step S30: obtaining the selected target element in the image elements.

[0030] After performing image recognition on the display image to obtain image elements and setting labels for the image elements, it is not necessary to display all image elements. Only when the user needs to display a certain image element and selects the image element, the image element is used as the target element, indicating that the user wants to know the relevant content of the target element, and the target element will continue to be processed subsequently.

[0031] Step S40: Input the content of the tag corresponding to the target element into the large language model to obtain text related to the target element.

[0032] The content of the label corresponding to the target element is the relevant content of the target element, and the large language model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. The large language model can handle a variety of natural language tasks, such as text classification, question and answer, dialogue, etc., and is an important way to artificial intelligence. In this application, the text related to the target element is obtained by inputting the content of the label corresponding to the target element into the large language model. For example, if the content of the label corresponding to the target element is Sun Wukong, Sun Wukong is input into the large language model to obtain the text related to the introduction of Sun Wukong.

[0033] Step S50: display the content of the text on the display interface.

[0034] Further, in an embodiment of the present application, the step of displaying the content of the text on the display interface includes: displaying the content of the text on the display interface in the form of text. That is, directly displaying the content of the text in the form of text in a blank position on the display interface.

[0035] In another embodiment of the present application, the step of displaying the content of the text on the display interface includes: converting the content of the text into voice, and playing the voice on the display interface. Displaying the content of the text on the display interface by playing the voice can allow the user to hear the content of the text through the ears, while the eyes can continue to read the content of the displayed image, which can improve the convenience of reading. In addition, the voice can be set in the form of a voice bar, and when the user needs to understand the content of the text, he can click the voice bar to play it.

[0036] To summarize, the present application obtains at least one image element by performing image recognition on the displayed image, sets a label for the image element, then obtains the target element among the image elements selected by the user, and then inputs the content of the label corresponding to the target element into the large language model to obtain text related to the target element, and displays the content of the text on the display interface. Therefore, the user does not need to exit the reading interface to obtain the relevant content of the target element, thereby improving the convenience of reading.

[0037] Please refer to Figure 2 , Figure 2 This is a flow chart of step S30 of the display method of the electronic device provided in the present application. Further, in an embodiment of the present application, the step of obtaining the target element selected from the image elements includes: Step S31, obtaining the position information of the cursor of the display interface; Step S32: obtaining the image element overlapping with the cursor according to the position information of the cursor, and taking it as the selected target element.

[0038] In the present application, the position information of the cursor of the display interface is obtained, and the image element overlapping with the cursor is obtained according to the position information of the cursor. Specifically, in one embodiment of the present application, the position information includes coordinate information, and the coordinate information of the cursor is used to determine whether the cursor overlaps with the image element. When the cursor and the image element have the same coordinate information, the cursor overlaps with the image element. When the cursor overlaps with the image element, it means that the image element is selected by the user and serves as the selected target element.

[0039] Please refer to Figure 3 , Figure 3 This is a flow chart of step S32 of the display method of the electronic device provided in this application. Figure 2 On the basis of the provided display method, further, in an embodiment of the present application, the step of obtaining the image element overlapping with the cursor according to the position information of the cursor and using it as the selected target element includes: Step S321, obtaining the image element overlapping with the cursor according to the position information of the cursor; Step S322, determine whether the number of the image elements that have overlapping parts with the cursor is greater than 1, if so, use the image element with the largest overlapping area with the cursor as the selected target element, otherwise, use the image element that overlaps with the cursor as the selected target element.

[0040] That is, after obtaining the image elements overlapping with the cursor according to the position information of the cursor, it is then determined whether the number of the image elements overlapping with the cursor is greater than 1. When the number of the image elements overlapping with the cursor is greater than 1, it means that the cursor overlaps with at least two of the image elements, and thus the image element with the largest overlapping area with the cursor is obtained and used as the selected target element. When the number of the image elements overlapping with the cursor is not greater than 1, it means that the cursor overlaps with only one of the image elements, and the image element overlapping with the cursor is used as the selected target element.

[0041] Please refer to Figure 4 , Figure 4 This is a flow chart of step S20 of the display method of the electronic device provided in this application. Figure 1On the basis of the provided display method, further, in an embodiment of the present application, the step of performing image recognition on the displayed image to obtain at least one image element includes: Step S21, detecting the displayed image by using a target detection model to obtain at least one image element region and position information of the image element region; Step S22: Identify the image element region through an instance segmentation model to obtain a mask, and the display region corresponding to the mask is the image element.

[0042] Among them, the image element area is an area including the image element, and the image element area can be a regular shape, such as a rectangle, a circle, or an irregular shape. The position information includes coordinate information. The target detection model and the instance segmentation model are both models for image processing. The present application detects the displayed image through a target detection model to obtain at least one image element area and the position information of the image element area, and then identifies the image element area through an instance segmentation model to obtain a mask, and the display area corresponding to the mask is the image element. There are many mature target detection models and instance segmentation models in the prior art, for example: Faster R-CNN model, YOLO model, SSD model, etc. are target detection models; Mask RCNN model, UNet model, etc. are instance segmentation models.

[0043] Please refer to Figure 5 , Figure 5 This is a flow chart of Embodiment 2 of the display method of the electronic device provided in this application. Figure 1 On the basis of the provided display method, further, in an embodiment of the present application, the display method further includes: Step S60: Associating the text with the target element and the label corresponding to the target element and storing them in a storage unit.

[0044] The generated text can be saved in a local storage unit and identified with the target element and the tag corresponding to the target element. When the same document is opened later, it does not need to be recognized again, which can save computing power.

[0045] Please refer to Figure 6 , Figure 6 This is a flow chart of Embodiment 3 of the display method of the electronic device provided in this application. Figure 5 On the basis of the provided display method, further, in an embodiment of the present application, the label includes a category name, and before the step of inputting the content of the label corresponding to the target element into a large language model to obtain text related to the target element, the display method further includes: Step S70, determine whether the category name of the label corresponding to the target element exists in the storage unit. If so, obtain the text corresponding to the category name of the label from the storage unit, and display the content of the text corresponding to the category name of the label on the display interface; if not, execute the next step.

[0046] As can be seen from the above, the text is associated with the target element and the label corresponding to the target element and stored in a storage unit. If the latter target element has the same category name of the label as the former target element, the latter target element is considered to be the same element as the former target element. Therefore, when the category name of the label corresponding to the target element exists in the storage unit, the text corresponding to the category name of the label is obtained from the storage unit, and the content of the text corresponding to the category name of the label is displayed on the display interface, and the next step is no longer executed. When the category name of the label corresponding to the target element does not exist in the storage unit, it is the first time that the target element is encountered, and the content of the label corresponding to the target element needs to be input into the large language model to obtain the text related to the target element. Thereby, efficiency can be improved and duplication of work can be reduced.

[0047] Please refer to Figure 7 , Figure 7 This is a flow chart of Embodiment 4 of the display method of the electronic device provided in this application. Figure 6 Based on the provided display method, further, in an embodiment of the present application, the display method further includes: Step S80, obtaining a cropped image corresponding to the target element; The step of associating the text with the target element and the label corresponding to the target element and storing them in a storage unit includes: Step S61, associating the text with the target element, the cropped image, and a label corresponding to the target element and storing them in a storage unit; The step of determining whether the category name of the tag corresponding to the target element exists in the storage unit, and if so, obtaining text corresponding to the category name of the tag from the storage unit, and displaying the content of the text corresponding to the category name of the tag on the display interface; if not, executing the next step, including: Step S71, determine whether the category name of the label corresponding to the target element exists in the storage unit. If so, obtain the text and cropped image corresponding to the category name of the label from the storage unit, and display the content of the text and cropped image corresponding to the category name of the label on the display interface; if not, execute the next step.

[0048] Among them, obtaining the cropped image corresponding to the target element is to obtain the cropped image corresponding to the target element by cropping or screenshotting the displayed image. Then the text is associated with the target element, the cropped image, and the label corresponding to the target element and stored in a storage unit for subsequent direct call. When the category name of the label corresponding to the target element exists in the storage unit, the text and the cropped image corresponding to the category name of the label are obtained from the storage unit, and the content of the text and the cropped image corresponding to the category name of the label are displayed on the display interface, and the next step is no longer executed. When the category name of the label corresponding to the target element does not exist in the storage unit, it is the first time that the target element is encountered, and the content of the label corresponding to the target element needs to be input into the large language model to obtain the text related to the target element.

[0049] That is, when the latter target element is the same element as the previous target element, the present application obtains the text and cropped image corresponding to the category name of the label from the storage unit, that is, obtains the text and cropped image of the previous target element, and then displays the text and cropped image when the latter target element is selected. This can deepen the user's impression of the image element, help enhance reading memory, and improve reading efficiency.

[0050] Figure 8 This is a schematic diagram of the structure of the display device of the electronic device provided in this application. Figure 8 A display device of an electronic device, the electronic device includes a display screen, and the display device includes: an acquisition module 10, a recognition module 20, a target module 30, a model module 40 and a display module 50.

[0051] The acquisition module 10 is used to acquire the display image in the display interface of the display screen; The recognition module 20 is used to perform image recognition on the display image to obtain at least one image element, and set a label for the image element; The target module 30 is used to obtain the selected target element in the image elements; The model module 40 is used to input the content of the label corresponding to the target element into a large language model to obtain text related to the target element; The display module 50 is used to display the content of the text on the display interface.

[0052] To summarize, the present application obtains at least one image element by performing image recognition on the displayed image, sets a label for the image element, then obtains the target element among the image elements selected by the user, and then inputs the content of the label corresponding to the target element into the large language model to obtain text related to the target element, and displays the content of the text on the display interface. Therefore, the user does not need to exit the reading interface to obtain the relevant content of the target element, thereby improving the convenience of reading.

[0053] The display device of the electronic device provided in the embodiment of the present application is included in the electronic device and can be used to execute any display method provided in the above embodiments, and has corresponding functions and beneficial effects.

[0054] It is worth noting that in the embodiment of the display device of the above-mentioned electronic device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0055] Fig. 9 A schematic diagram of the structure of the electronic device provided in this application, such as Fig. 9 As shown, the electronic device includes one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the electronic device implements the display method of the electronic device as described above, Fig. 9 A processor 201 is taken as an example.

[0056] The memory 202, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules. The processor 201 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 202, that is, realizes the above-mentioned display method.

[0057] The memory 202 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 202 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 202 may further include a memory remotely arranged relative to the processor 201, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0058] The electronic device mentioned above includes a display device, which can be used to execute any of the above display methods and has corresponding functions and beneficial effects.

[0059] The present application also provides a storage medium containing computer executable instructions, which are used to perform the display method when executed by a computer processor. The computer program can also implement the relevant operations in the display method provided in any embodiment of the present application when executed by the processor, and has corresponding functions and beneficial effects.

[0060] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0061] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0062] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0063] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0064] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0065] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A display method for an electronic device, the electronic device comprising a display screen, characterized in that: include: Acquire a display image in a display interface of the display screen; Performing image recognition on the displayed image to obtain at least one image element, and setting a label for the image element; Obtaining a selected target element from the image elements; Inputting the content of the label corresponding to the target element into a large language model to obtain text related to the target element; The content of the text is displayed on the display interface.

2. The display method according to claim 1, characterized in that: The step of obtaining the selected target element in the image element comprises: Obtaining position information of a cursor on the display interface; The image element overlapping with the cursor is obtained according to the position information of the cursor and used as the selected target element.

3. The display method according to claim 2, characterized in that: The step of obtaining the image element overlapping with the cursor according to the position information of the cursor and using it as the selected target element comprises: Obtaining the image element overlapping the cursor according to the position information of the cursor; Determine whether the number of the image elements that have overlapping parts with the cursor is greater than 1. If so, the image element with the largest overlapping area with the cursor is selected as the target element. Otherwise, the image element that overlaps with the cursor is selected as the target element.

4. The display method according to claim 1, characterized in that: The step of performing image recognition on the displayed image to obtain at least one image element comprises: Detecting the displayed image using a target detection model to obtain at least one image element region and position information of the image element region; The image element region is identified by an instance segmentation model to obtain a mask, and a display region corresponding to the mask is the image element.

5. The display method according to claim 1, characterized in that: The display method further includes: The text is associated with the target element and a tag corresponding to the target element and stored in a storage unit.

6. The display method according to claim 5, characterized in that: The label includes a category name. Before the step of inputting the content of the label corresponding to the target element into a large language model to obtain text related to the target element, the display method further includes: Determine whether the category name of the label corresponding to the target element exists in the storage unit. If so, obtain the text corresponding to the category name of the label from the storage unit, and display the content of the text corresponding to the category name of the label on the display interface; if not, execute the next step.

7. The display method according to claim 6, characterized in that: The display method further includes: Obtain the cropped image corresponding to the target element; The step of associating the text with the target element and the label corresponding to the target element and storing them in a storage unit includes: Associating the text with the target element, the cropped image, and a label corresponding to the target element and storing them in a storage unit; The step of determining whether the category name of the tag corresponding to the target element exists in the storage unit, and if so, obtaining text corresponding to the category name of the tag from the storage unit, and displaying the content of the text corresponding to the category name of the tag on the display interface; if not, executing the next step, including: Determine whether the category name of the label corresponding to the target element exists in the storage unit. If so, obtain the text and cropped image corresponding to the category name of the label from the storage unit, and display the content of the text and cropped image corresponding to the category name of the label on the display interface; if not, execute the next step.

8. The display method according to claim 1, characterized in that: The step of displaying the content of the text on the display interface includes: The content of the text is displayed on the display interface in the form of text.

9. The display method according to claim 1, characterized in that: The step of displaying the content of the text on the display interface includes: The content of the text is converted into voice, and the voice is played on the display interface.

10. A display device of an electronic device, the electronic device comprising a display screen, characterized in that: include: An acquisition module, the acquisition module is used to acquire a display image in a display interface of the display screen; a recognition module, the recognition module being used to perform image recognition on the displayed image to obtain at least one image element, and to set a label for the image element; A target module, the target module is used to obtain a target element selected from the image elements; A model module, the model module is used to input the content of the label corresponding to the target element into a large language model to obtain text related to the target element; A display module is used to display the content of the text on the display interface.

11. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the electronic device implements the display method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the display method according to any one of claims 1 to 9 is implemented.

Images