Display device and interface content labeling method

Abstract

The embodiment of the invention provides a display device and an interface content labeling method, and the method comprises the steps: obtaining a currently displayed interface image and meta-information of a user interface in response to a first request labeling instruction, inputting the first request labeling instruction, the interface image and the meta-information into a multi-modal retrieval system, and retrieving associated information, and inputting the associated information, the first annotation request instruction and the interface image into a multi-modal understanding model to generate annotation feedback information, and controlling a display to display the annotation feedback information on a user interface. According to the method, information retrieval is executed on the multi-modal information through the multi-modal retrieval system, the accuracy of information retrieval and understanding can be improved by using the first request labeling instruction, the interface image, the meta-information and the like as the multi-modal information, and then the accuracy of labeling content is improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and an interface content annotation method. Background Art

[0002] A display device refers to a terminal device that can output images. The output images can be image images or video images. Taking video images as an example, the user's needs when watching video images are not limited to viewing. There are also different characters, scenes and other contents in the video images. The user's needs also include understanding the information of different contents of the video images.

[0003] In order to identify the content of the output screen, the text content in the output screen can be extracted and recognized through retrieval enhancement technology. Among them, retrieval enhancement technology obtains annotation information through text retrieval, and can only perform single-mode retrieval, that is, text information, but cannot identify annotation content such as people and scenes in the screen.

[0004] For character recognition, computer vision and machine learning technologies can be used to identify, classify, and annotate the content in the video. For example, if a user needs to understand the information of a person in a video, the user can capture an image, analyze and annotate the video content, and then feed the person information back to the user. However, it can only process information of a single modality and cannot analyze and annotate other content such as scenes, resulting in low accuracy of the annotated content. Summary of the invention

[0005] The present application provides a display device and an interface content annotation method to solve the problem of low accuracy of annotated content.

[0006] In a first aspect, some embodiments of the present application provide a display device, comprising: a display and a controller. The display is configured to display a user interface; the controller is configured to:

[0007] In response to a first request annotation instruction, acquiring the currently displayed interface image and meta information of the user interface, wherein the first request annotation instruction is an instruction generated by the display device according to a preset period and / or an instruction input by a user;

[0008] Inputting the first request annotation instruction, the interface image, and the meta-information into a multimodal retrieval system to retrieve associated information through the multimodal retrieval system, wherein the multimodal retrieval system is used to perform information retrieval according to multimodal information, wherein the multimodal information includes keywords extracted from the first request annotation instruction, the interface image, and the meta-information;

[0009] Inputting the associated information, the first request annotation instruction, and the interface image into a multimodal understanding model to generate annotation feedback information through the multimodal understanding model, wherein the multimodal understanding model is a model generated by training the multimodal information based on a deep learning algorithm;

[0010] The display is controlled to display the annotation feedback information on the user interface.

[0011] In a second aspect, some embodiments of the present application further provide an interface content annotation method, which is applied to the display device described in the first aspect, and the interface content annotation method includes:

[0012] In response to a first request annotation instruction, acquiring the currently displayed interface image and meta information of the user interface, wherein the first request annotation instruction is an instruction generated by the display device according to a preset period and / or an instruction input by a user;

[0013] Inputting the first request annotation instruction, the interface image, and the meta-information into a multimodal retrieval system to retrieve associated information through the multimodal retrieval system, wherein the multimodal retrieval system is used to perform information retrieval according to multimodal information, wherein the multimodal information includes keywords extracted from the first request annotation instruction, the interface image, and the meta-information;

[0014] Inputting the associated information, the first request annotation instruction, and the interface image into a multimodal understanding model to generate annotation feedback information through the multimodal understanding model, wherein the multimodal understanding model is a model generated by training the multimodal information based on a deep learning algorithm;

[0015] The control display displays the annotation feedback information on the user interface.

[0016] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and an interface content annotation method, wherein the method obtains the currently displayed interface image and the meta-information of the user interface by responding to a first request annotation instruction, and then inputs the first request annotation instruction, the interface image, and the meta-information into a multimodal retrieval system to retrieve associated information, and then inputs the associated information, the first request annotation instruction, and the interface image into a multimodal understanding model to generate annotation feedback information, and controls the display to display the annotation feedback information on the user interface. The method performs information retrieval on multimodal information through a multimodal retrieval system, wherein by using the first request annotation instruction, the interface image, and the meta-information as multimodal information, the accuracy of information retrieval and understanding can be improved, thereby improving the accuracy of the annotated content. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0019] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0020] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0021] Figure 4 A schematic diagram of performing recognition on an output screen provided in some embodiments of the present application;

[0022] Figure 5 A schematic diagram of a method for annotating interface content provided in some embodiments of the present application;

[0023] Figure 6 A schematic diagram of setting annotation setting parameters provided in some embodiments of the present application;

[0024] Figure 7 A first schematic diagram of a current interface image provided in some embodiments of the present application;

[0025] Figure 8 A second schematic diagram of a current interface image provided in some embodiments of the present application;

[0026] Fig. 9 A schematic diagram of a process for correcting errors in associated information provided in some embodiments of the present application;

[0027] Fig.10 A schematic diagram of labeling information provided for some embodiments of the present application;

[0028] Fig.11 A schematic diagram of feedback information provided for some embodiments of the present application;

[0029] Fig.12 A schematic diagram of the process of generating supplementary annotation feedback information provided in some embodiments of the present application. DETAILED DESCRIPTION

[0030] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0031] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0032] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0033] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0034] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0035] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0036] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0037] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.

[0038] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .

[0039] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0040] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.

[0041] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0042] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0043] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0044] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.

[0045] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0046] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.

[0047] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0048] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0049] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0050] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.

[0051] In some embodiments, the user input interface 280 may be used to receive instructions from a user.

[0052] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface. For example, the operating system can directly control the display device 200 to provide a user interface, or provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.

[0053] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device 200 .

[0054] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0055] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0056] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0057] like Figure 3 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0058] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.

[0059] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.

[0060] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0061] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device 200 and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0062] Based on the above display device 200, the display device 200 can output a picture, and display the picture through the display 260. To improve the user experience, see Figure 4 The display device 200 can perform recognition and annotation on the output screen, for example, recognize and annotate the character B in the output interface.

[0063] The output screen may be an image screen or a video screen, and the recognition methods of the image screen and the video screen may be different.

[0064] For image screens, i.e. static screens, in some embodiments, image recognition algorithms can be used to recognize image screens, such as convolutional neural networks (CNN), object detection algorithms, etc. CNN can extract image features through structures such as convolutional layers and pooling layers, and classify or recognize them. In the display device 200, CNN can be used to recognize the content of the currently displayed image, such as faces, objects, scenes, etc. Through the pre-trained CNN model, the feature representation of the image can be obtained, and used to recognize the image on the screen in real time, and then the recognized content is annotated.

[0065] Among them, the object detection algorithm can locate and identify multiple objects in the image. The object detection algorithm combines classification and positioning technology to simultaneously identify multiple objects and their locations in the image. In the display device 200, the object detection algorithm can be used to identify multiple targets in the current display screen, such as participants in a meeting, moving objects in a video, etc., and then mark the identified content.

[0066] That is to say, for an image, after acquiring a static image, it is preprocessed, such as denoising, contrast enhancement, etc., and then feature extraction is performed, and the recognition result is output through classification recognition, and the recognition result is annotated. The annotated content of the static image is people, objects, scenes, etc.

[0067] For video images, i.e. dynamic images, in some embodiments, computer vision and machine learning techniques can be used to identify, classify and label the video images. For example, a video file is decoded into a series of video frames, and key frames are extracted from the video or frames are sampled at fixed intervals to reduce the amount of calculation. Features are then extracted from the video frames. The features can be low-level features such as color, texture, shape, or high-level features such as motion and audio. Feature extraction methods include CNN, optical flow method, etc. Among them, the optical flow method estimates the direction and speed of motion by analyzing the changes in pixels or feature points between adjacent frames. In the display device 200, the optical flow method is used to analyze motion information on the screen, for example, detecting motion trajectories, calculating motion speeds, etc.

[0068] The model is trained using the labeled video data set to obtain a pre-trained model. The content to be recognized is then input into the pre-trained model to perform recognition on the video screen using the pre-trained model and output the recognition result.

[0069] That is to say, for video images, we can first capture video frames, then perform static image recognition on each frame, output recognition results through inter-frame information fusion, such as motion estimation, tracking, etc., and annotate the recognition results. The annotation method for video images can perform behavior analysis, video monitoring, motion detection, etc. on dynamic images.

[0070] Big model technology is a deep learning model with a large number of parameters, layers, and units. The deep learning model is built by a deep neural network and has expressive and predictive performance. Big model technology can handle complex tasks and data, and adopts a pre-training and fine-tuning training mode to adapt to downstream tasks. Big model technology is applied to multiple fields. For example, Natural Language Processing (NLP), Computer Vision (CV), multimodal processing, etc. For multimodal processing, the big model can simultaneously process information in multiple modes such as text, images, and audio.

[0071] For example, multimodal models can conduct multimodal interactive question and answer. Through large model technology, an intelligent question and answer system can be built, which can understand the user's questions and provide corresponding answers. However, the intelligent question and answer system based on large models mainly relies on the model's own knowledge reserve, and the accuracy, comprehensiveness and timeliness of the answers are limited. Retrieval enhancement technology (RAG) can also be used. RAG searches for information related to user questions in external knowledge bases through text retrieval to supplement the model's own knowledge reserve. However, RAG is mainly limited to single-modal retrieval, that is, it mainly focuses on text data, and is not suitable for video content analysis and annotation application scenarios.

[0072] Video content contains images, audio, text and other information, which interact with each other and together constitute the overall meaning of the video. However, for the application scenarios of video content analysis and annotation, accurate annotation information cannot be obtained based on text retrieval alone. Therefore, relying solely on text retrieval to obtain supplementary information cannot fully and accurately understand the video content, which limits the application of intelligent question-answering systems in video content analysis and annotation, and thus cannot obtain accurate video content annotation.

[0073] In some embodiments, the video content analysis method of multimodal technology can obtain the corresponding multimodal feature vector by sampling and encoding the multimodal information in the same time period of the video containing a specific time point; using the fusion model, at least two of the image information feature vector, audio information feature vector or text information feature vector in the multimodal feature vector are fused to obtain a fused feature vector; and using the barrage generation model, the fused feature vector is used to generate the barrage of the video at a specific time point. Combined with multimodality, the barrage of the current video time point is automatically generated to enhance the degree of interaction and enrich the video content. However, this method is limited to the single modal information of the video screen, which also leads to low accuracy of video content annotation.

[0074] In order to solve the problem of low accuracy of annotated content, an interface content annotation method is provided in some embodiments of the present application. The display device 200 can perform retrieval through the multimodal information of the multimodal retrieval system through the interface content annotation method, wherein the multimodal information includes instructions input by the user, and / or instructions, interface images and meta-information generated by the display device 200 according to a preset period, and the retrieved related information is understood using a multimodal understanding model to generate accurate and comprehensive annotation content.

[0075] To implement the interface content annotation method, the display device 200 includes at least a display 260 and a controller 250. The display 260 is configured to display a user interface. Figure 5 As shown, the controller 250 is configured to execute program steps corresponding to the interface content annotation method, including the following contents:

[0076] S100: In response to a first request annotation instruction, obtain the currently displayed interface image and meta information of the user interface.

[0077] The first request labeling instruction is used to trigger the display device 200 to display the labeling information. In some embodiments, the first request labeling instruction is an instruction generated by the display device 200 according to a preset period. In some embodiments, the first request labeling instruction is an instruction input by a user. Among them, the first request labeling instruction generated by the display device 200 according to the preset period can be automatically generated by the system or program of the display device 200 according to a preset time period. For example, when the user sets the display device 200 to label the content in the picture with information when playing sports game content. When the video being played is a football game, the display device 200 autonomously initiates a video content labeling request, that is, the first request labeling instruction.

[0078] In some embodiments, the display device 200 may also generate a first request for labeling instruction based on historical interaction data. For example, if the video being played is a contemporary TV series, the display device 200 may infer from the user's historical interaction data that the user may be interested in the specific information of the scene being filmed in the current scene, and may autonomously initiate a first request for labeling instruction.

[0079] Historical interaction data refers to data records generated when the user interacts with the display device 200 before the current viewing. These data records may include user behaviors, preferences, selections, feedback, etc. Historical interaction data may include interaction behaviors, feedback data, setting preferences, etc. Interaction behaviors are interactive actions between users and devices, such as clicks, slides, voice commands, etc. Feedback data is user feedback such as evaluation, rating, comments, and likes on content. Setting preferences is the personalized settings made by users on the device, such as annotation content selection, annotation method, etc.

[0080] The display device 200 can generate a first request annotation instruction at a fixed time interval, for example, 2 minutes after the video is played; for example, when the episode of a TV series is switched; the display device 200 can also generate a first request annotation instruction based on periodic changes triggered by an event, for example, the video screen is switched.

[0081] The instructions generated by the display device 200 according to a preset period can be set as a function switch, and the function switch can also be activated at a time interval set by the user, for example, when a specific playback application or a specific picture is played.

[0082] In addition to setting automatic annotation by time interval, in order to improve annotation accuracy, in some embodiments, by obtaining the first request annotation instruction generated by the user when setting the display device 200 at a fixed time interval, you can also input annotation setting parameters, the annotation setting parameters are the annotation preference setting parameters of the interface image, the annotation setting parameters include the interface image annotation content; then extract the annotation preference information from the annotation setting parameters, the annotation preference information includes at least one annotation type item; according to the annotation preference information, generate a first request annotation instruction according to a preset period.

[0083] like Figure 6 As shown, the annotation setting parameters are defined by the user according to his / her own preferences and needs, and are used to guide how to annotate the interface image. The annotation setting parameters include but are not limited to: annotation content, annotation type, annotation position, and annotation style.

[0084] The annotation content is the interface content of the specified annotation, such as the video title, current play time, remaining time, actor information, and plot summary. The annotation type is used to define the specific form of the annotation, such as text annotation, icon annotation, voice annotation, etc. Among them, text annotation can be displayed directly on the interface image or on the interface for dialogue with the user; icon annotation can be directly represented by icons or symbols; the annotation content can be broadcasted through the voice assistant whether the user is in a dialogue state or not. The annotation position is used to specify the display position of the annotation on the interface to display the content in a non-invasive manner and is easy for users to view. The annotation style includes font size, color, background color, etc. to adapt to different visual needs and interface styles.

[0085] It is understandable that the user can further set the annotation setting parameters by selecting video title, text annotation, non-intrusive, font size, etc.

[0086] The user can determine the annotation type item through the annotation setting parameters, and the annotation type item can include at least one, for example, the annotation setting parameters only include the annotation content, and after responding to the first request annotation instruction, if the user does not make other request annotation instructions, only the annotation content is annotated. For other annotation setting parameters, set them according to the default values, for example, the annotation style and annotation position are set according to the default values.

[0087] In some embodiments, the video content analysis method of multimodal technology can integrate the content through the use of crawler technology, language large models, knowledge graphs and generative artificial intelligence (Artificial Intelligence Generated Content, AIGC), and can automatically generate the latest character special page, which includes character profiles, character-participated programs, character-related programs and special page backgrounds, all of which are processed and synthesized by algorithms, thereby improving the speed and frequency of updating the character special page.

[0088] In some embodiments, the user can send instructions to the display device 200 through the control device 100, the smartphone application, or other interactive methods. For example, by controlling the remote control, the focus is moved to the target annotation object, and a first request annotation instruction is generated. For another example, the target annotation object is clicked through the mobile phone application to generate a first request annotation instruction. For another example, the user pre-sets voice or interactive buttons as a wake-up method to wake up the interactive assistant during video playback. If the video being played is a contemporary TV series, the user wakes up the interactive assistant through the pre-set wake-up method and asks "Where is the current scene, can I go there to play recently?", thereby generating a first request annotation instruction.

[0089] Different from the instructions generated by the display device 200 according to a preset cycle, the instructions input by the user occur instantly. By responding to the user's needs, the interactivity and flexibility can be increased, and the video images can be annotated and analyzed according to the user's needs.

[0090] It is understandable that when the user interface is identified, the user interface is displaying an image interface or a video interface. If the user interface displays an image interface, the current interface image may be a screenshot of the image interface, and after receiving the first request annotation instruction, a screenshot operation is performed. If the user interface displays a video interface, after receiving the first request annotation instruction, a certain video frame of the video screen may be obtained, and this video frame may be the current video frame or another video frame including the current video frame.

[0091] That is, in some embodiments, the currently displayed interface image may be determined based on the first request annotation instruction. If the first request annotation instruction is an instruction generated by the display device 200 according to a preset period, the currently displayed interface image may be the current video frame; if the first request annotation instruction is an instruction input by a user, the currently displayed interface image may be the current video frame, and the currently displayed interface image may also include other video frames of the current video frame.

[0092] For example, the user is watching a video and notices a certain scene. The display device 200 generates instructions according to a preset period to mark or analyze the current video frame. The currently displayed video frame can be captured and processed as a target frame, where the target frame is the current interface image.

[0093] like Figure 7 As shown, the display device 200 is playing a live broadcast of a sports event and can display information about athletes in the live broadcast. However, in a sports event, athletes may perform actions in different postures and these actions may occur in a very short period of time. For example, the current video frame may not be captured. Figure 7The athlete information of the athlete number 5 in the video is obtained. However, after receiving the first request labeling instruction, if the athlete information cannot be obtained, a certain number of video frames can be captured forward or backward starting from the current video frame, such as Figure 8 As shown, 5 frames are captured forward and 5 frames are captured backward. A certain video frame in these video frames can identify the athlete's number. Therefore, the display device 200 uses the video frame that can identify the athlete's number as the current video frame.

[0094] The meta information of the user interface is a series of descriptive information related to the playback content. The meta information of the user interface includes but is not limited to the video title, author, theme, task, release date, tag, language, and comments. Taking the video screen as an example, when the user interface displays the video screen, the video screen is a video file, and the video file itself contains some built-in meta information, such as author, title, creation date, encoding format, resolution, frame rate, duration, etc. When obtaining video files, some third-party databases or websites also collect and organize video files and related information. You can search for specific video files through these databases and obtain meta information.

[0095] The interface types of the interface image include video interface and image interface. The methods for obtaining the current interface image and meta-information are different. In some embodiments, the interface type of the interface image is obtained, and the interface type is a video interface or an image interface; if the interface type is a video interface, the content type of the current interface image is obtained to obtain the image of the current video interface and the meta-information of the video interface according to the content type; if the interface type is an image interface, the image of the image interface and the meta-information of the image interface are obtained.

[0096] If it is detected that the currently displayed interface image contains an identifier of a video stream or a video file, it can be marked as a video interface; if it is detected that a static image file or image metadata is included, it can be marked as an image interface.

[0097] Among them, the content type is an attribute that can describe the video content, such as plot, sports, entertainment, education, etc. Identification can be achieved by analyzing the meta information, title, tag or video content of the video file. After obtaining the content type, the processing process can be optimized according to the content type. For example, a suitable video summary algorithm can be selected according to the content type to generate an image of the current video interface, and the meta information of the video interface can be obtained at the same time. For another example, the meta information of video interfaces in different directions can be obtained according to video interfaces of different content types.

[0098] Taking the video screen as an example, the currently displayed interface image is a frame of the video, the user interface is the video file, and the meta-information of the user interface can be information related to the video file rather than specific information of a frame of the image, such as scene, character, etc.

[0099] The order of obtaining the currently displayed interface image and the meta information of the user interface is not limited. In some embodiments, the information can be obtained in parallel, which can shorten the response time and improve the processing efficiency.

[0100] In some embodiments, the currently displayed interface image may be obtained first, and then the meta information of the user interface may be obtained. If an object in the interface image is the target object requested to be annotated by the user, and the target object can be directly obtained, then the interface image is obtained first, and then the meta information of the user interface is obtained, so that the user can directly view the annotation, and then further analyze or process it. It is understandable that for the person annotation in the directly obtainable interface image, the meta information of the user interface may not be obtained.

[0101] In some embodiments, the meta information of the user interface may be obtained first, and then the currently displayed interface image may be obtained. If the meta information is important for the annotated content, the meta information may be obtained first. In this way, it may be determined whether the interface image needs to be further obtained based on the content of the meta information.

[0102] After obtaining the currently displayed interface image and the meta information of the user interface, a search is performed according to the first request annotation instruction to retrieve the relevant content requested to be annotated.

[0103] S200: Inputting the first request annotation instruction, the interface image, and the meta-information into a multimodal retrieval system to retrieve related information through the multimodal retrieval system.

[0104] When analyzing and annotating video images, it can be based not only on video frames, but also on audio information, time information, etc. Audio information can provide the sound content in the video, such as dialogue, music, ambient sound, etc. Sound information helps to understand the video content more comprehensively. Time information can represent the sequence of events in the video, the duration of actions, etc. When annotating, the timestamp of each label or event can be recorded for subsequent analysis and processing.

[0105] In some embodiments, multimodal fusion technology is used to combine information from different modalities for analysis. For example, the image information and audio information in the video can be combined to simultaneously identify the characters and conversation content in the video through face recognition and speech recognition technology. In the visual dimension, deep learning models can be used to extract image features in video frames; in the audio dimension, speech recognition technology can be used to convert audio into text and further extract text features; in the text dimension, features can be directly extracted from text information such as video titles, descriptions, or subtitles.

[0106] It is also possible to combine time information with user needs. For example, the target tracking algorithm can continuously track objects in the video and keep them consistent in different frames. At the same time, the scene recognition algorithm can also be used to identify different scenes in the video and understand the transition relationship between different scenes. If the user needs to search and retrieve specific people or objects in the video, the people and objects appearing in the video can be accurately identified and marked. If the user needs to understand the emotional expression or emotional changes in the video, the emotions and moods in the video can be described and marked.

[0107] However, the use of multimodal information may focus on one or two modal information, while the utilization of other modal information is low. For example, video annotation mainly performs annotation tasks through image information. Through the extraction of video frames and image processing technology, key objects, scenes or behaviors in the video are identified. Although text information exists, it may only be used as auxiliary information to verify or supplement the results of image recognition.

[0108] Video annotation mainly performs annotation tasks through text information. When the video content is mainly language communication, the image information is relatively secondary or in scenes that are difficult to directly identify. For example, in the annotation of video records or court trials, the voice in the video is mainly transcribed into text, and natural language processing technology is used to identify keywords, themes or emotional tendencies, and then generate annotation information based on this.

[0109] In other words, the use of multimodality is emphasized differently depending on the scenario, and different information needs to be selected for different scenarios. However, the scene may change at any time. When the scene changes, if the required information cannot be accurately selected, the video annotation content will be inaccurate.

[0110] Moreover, when annotating videos using a single modality, or using multi-modality to annotate image information, it is necessary to extract the current video frame, that is, to perform post-recognition annotation by capturing the image and then extracting the features of the video screen. However, the video image is in a real-time changing state, and the use of a single modality for annotation will result in inaccurate annotation of the video screen.

[0111] A multimodal retrieval system is used to perform information retrieval based on multimodal information. To obtain accurate retrieval, the construction of a multimodal retrieval system at least includes data collection and preprocessing, feature extraction and representation, retrieval model construction, and system deployment and testing.

[0112] During the data collection and preprocessing process, various types of video data can be collected, including but not limited to movies, TV series, educational videos, advertisements, etc. Meta information is extracted from the video data, such as title, author, description, tags, time, etc. The video images are processed by image recognition, video content analysis and other technical processing to extract key frames, image features, etc.

[0113] Use natural language processing technology to perform word segmentation and vectorization on text information such as video titles and descriptions, and convert them into computable numerical features. Use computer vision technology to extract image features of video key frames, such as color, texture, shape, object recognition, etc., and convert image features into vectors or semantic space representations. Fuse text features and image features to form a unified representation for cross-modal information retrieval.

[0114] Select the retrieval model to be trained, for example, algorithms based on vector space models, probability models or deep learning models, and perform model training and tuning. Build a multimodal retrieval engine that supports queries and retrieval based on video images, video meta-information and user questions. Deploy the trained model and retrieval engine to the server and provide an API interface for external calls. Conduct comprehensive testing, including functional testing, performance testing, stability testing, etc., to ensure that the system can provide multimodal retrieval services stably and accurately.

[0115] In some embodiments, a multimodal retrieval system may be constructed based on a multimodal information knowledge base. The multimodal information knowledge base includes multimodal information, and the multimodal information includes all information from the first request labeling instruction, the interface image, and the meta-information, and may also be keywords extracted from the first request labeling instruction, the interface image, and the meta-information. The construction process including but not limited to data collection and preprocessing, feature extraction and representation, retrieval model construction, system deployment and testing can all be constructed based on existing construction methods, which will not be elaborated here. In this embodiment, the multimodal retrieval system is a retrieval system that can retrieve related information based on the first request labeling instruction, the interface image, and the meta-information. In addition, the multimodal retrieval system can directly search based on the user's questions, and the keywords of the questions may not be extracted.

[0116] For the first request annotation instruction initiated by the display device 200, for example, the multimodal retrieval system can retrieve information such as the teams and athlete numbers of the two opponents from the multimodal information knowledge base based on the image and meta-information displayed in the current video screen, including the current game opponents and the game, time and other information indicated by the video title.

[0117] In response to a user input instruction, for example, the multimodal retrieval system retrieves information such as the filming location of the TV series from the multimodal information knowledge base based on the image displayed in the current video screen and information such as the TV series title, episode number, and time.

[0118] After the associated information is retrieved, the associated information can be displayed on the user interface. In some embodiments, the user can determine whether to continue searching, using the multimodal understanding model to analyze, etc. based on the associated information. That is, when the associated information is displayed on the user interface, the user can use the user interface to control the multimodal retrieval system to search again according to the needs, or if the associated information cannot meet the needs, the multimodal understanding model can be used to understand the associated information first.

[0119] like Fig. 9 As shown, errors may occur in the displayed associated information. To avoid errors in the understanding process, the user can correct the associated information. In some embodiments, the display 260 is controlled to display the associated information on the user interface; in response to a focus movement instruction input by the user, the display 260 is controlled to display a second interactive interface on the user interface; error correction information is generated according to an error correction instruction input by the user based on the second interactive interface; the error correction information, the associated information, the first request annotation instruction and the interface image are input into a multimodal understanding model to generate annotation feedback information through the multimodal understanding model.

[0120] When the user needs to perform further operations or view more information, the focus movement instruction can be issued in various ways, such as key operation, mouse movement, touch screen operation, remote control operation, etc. The display device 200 controls the display 260 to display the second interactive interface on the user interface by identifying the focus movement instruction.

[0121] The second interactive interface can be a drop-down menu, a pop-up interface, or a dialogue interface. If the user finds that the associated information is incorrect or needs to modify certain annotation settings, the user can enter a correction instruction through the second interactive interface. The display device 200 receives and parses the correction instruction and generates correction information including proposing a corrected text, instructing a re-search, or canceling a secondary search.

[0122] For example, the multimodal retrieval system retrieves information such as the filming location of the TV series, and the multimodal understanding model determines that the scene of the current interface image is a scenic spot in a certain city in China based on the above information. According to the travel intention requirements proposed by the user, the multimodal retrieval system is asked to retrieve information such as the recent weather conditions and travel suitability index of a certain city.

[0123] S300: Inputting the associated information, the first request annotation instruction, and the interface image into a multimodal understanding model to generate annotation feedback information through the multimodal understanding model.

[0124] The associated information includes associated information that has been displayed and corrected by a user, or associated information that has not been displayed, that is, associated information directly obtained through retrieval by a multimodal retrieval system.

[0125] The multimodal understanding model is used to encode, understand, and analyze the associated information, the first request annotation instruction, and the interface image, and generate annotation feedback information based on the analysis results. The construction of the multimodal understanding model may include data collection and preprocessing, feature extraction, and model training.

[0126] In the process of data collection and preprocessing, data of various modalities including video, text, image, and voice are collected. The collected data is then cleaned, labeled, and formatted for subsequent model training. For example, video data needs to be segmented and keyframes extracted; text data needs to be segmented and stop words removed.

[0127] Use algorithms or models to extract features from each modality of data. For example, use CNN to extract visual features from images, and use recurrent neural networks (RNN) or transformers to extract language features from text. Transform features from different modalities into the same feature space for cross-modal comparison and analysis.

[0128] The multimodal understanding model uses large models such as Transformer, BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-Trained) as the basic architecture, and customizes and optimizes them according to specific tasks. Design a suitable loss function to evaluate the difference between the model's prediction results and the actual annotations, and optimize the model parameters through the back-propagation algorithm. Train the model on a large-scale data set, and continuously adjust and optimize the model parameters to improve the accuracy and generalization ability of the model.

[0129] You can also use accuracy, recall, F1 score and other evaluation indicators to evaluate model performance. According to the evaluation results, you can tune the model, including adjusting the model architecture and optimizing the training strategy.

[0130] When the first request annotation instruction includes instructions generated by the display device 200 according to a preset cycle and instructions input by the user, the instructions may be sorted according to their priority or importance to process more important instructions first. For example, the multimodal understanding model may first understand the instructions input by the user so that the generated annotation feedback information better meets the user's needs.

[0131] The first request annotation instruction may be further processed, for example, semantic understanding and normalization. For natural language instructions, semantic analysis may be performed to understand the meaning of the instruction. Understanding includes identifying entities, relationships, actions, etc. in the instruction and understanding the logical relationship between them. If the instruction contains vague or ambiguous expressions, it may be disambiguated with the help of context information or a domain knowledge base.

[0132] By using the standardized language such as words and terms in the instructions as the standard vocabulary or term set used in model training, the multimodal understanding model can accurately identify and process the words involved in these instructions. For periodically generated instructions, time-related expressions can be converted into timestamps or time interval formats that the model can understand.

[0133] In order to facilitate the multimodal understanding model to understand the meta information of the user interface and improve the accuracy of the annotation feedback information, the meta information can be cleaned and normalized. By cleaning the meta information, useless or erroneous data can be removed, such as duplicates, dates with inconsistent formats, etc. The meta information is normalized into a format that the model can understand, for example, converting dates into a unified date format, and converting labels and topics into a standard vocabulary or term set used in model training.

[0134] Useful features are then extracted from the meta-information, such as keywords, sentiment, time trends, etc. The extracted features are encoded into numerical forms that the model can process, for example, by converting text information into vector representations through word embedding. The meta-information is associated with other modal data in the user interface, i.e., the first request annotation instructions and the interface image, so that the model can comprehensively consider multiple modal information for understanding and analysis.

[0135] For interface images, preprocessing can be performed on the interface images, such as image cleaning, image enhancement, and size adjustment. Low-level features such as edges, textures, and shapes can also be extracted from the interface images. Through further processing by the model, higher-level semantic features can be extracted from the low-level features, such as the category, location, and relationship of the object.

[0136] Fusion of image features with features of other modalities such as text and audio to form a multimodal feature representation helps the model understand the content of the user interface more comprehensively. Based on multimodal features, understanding is performed to improve the accuracy and depth of the model's understanding of the user interface.

[0137] After the first request annotation instruction, the interface image and the meta-information are processed, they can be understood through the multimodal understanding model to generate annotation feedback information.

[0138] In order to make the content of the annotation feedback information more accurate, the multimodal understanding model can be controlled to generate different annotation feedback information for different content types. In some embodiments, according to the content type, the emphasis information is set, and the emphasis information is used to characterize the emphasis of the annotation feedback of the video screen of the content type; the associated information, the first request annotation instruction, and the interface image are input into the multimodal understanding model to generate the annotation feedback information based on the emphasis information through the multimodal understanding model.

[0139] Different types of content have different annotation requirements. For example, educational videos may focus more on the annotation of knowledge points, while entertainment videos may focus more on the annotation of emotions or scenes.

[0140] In some embodiments, for videos of specific themes, such as science, history, art, etc., the emphasis information can be set to emphasize the key information annotations related to the theme. For videos with rich emotional expressions, such as movie clips, music videos, etc., the emphasis information can be set to focus on annotations of emotional tendencies, such as joy, sadness, anger, etc. For videos containing character movements or behaviors, such as sports games, dance performances, etc., the emphasis information can be set to identify and annotate key movements or behaviors. For videos containing specific objects, such as product displays, the animal world, etc., the emphasis information can be set to detect and annotate key objects in the video.

[0141] After the multimodal understanding model completes the understanding of the first request annotation instruction, the interface image, and the meta-information, it generates different annotation feedback information based on the emphasis information.

[0142] For the first request annotation instruction initiated by the display device 200, for example, the multimodal retrieval system can retrieve information such as the teams and athlete numbers of the two opponents from the multimodal information knowledge base based on the image and meta-information displayed in the current video screen, including the current game opponents and the game, time and other information indicated by the video title.

[0143] For instructions input by the user, for example, the multimodal retrieval system retrieves information such as the filming location of the TV series from the multimodal information knowledge base based on the image displayed in the current video screen and information such as the TV series title, episode number, and time.

[0144] For example, the multimodal retrieval system can retrieve information such as the teams and athlete numbers of the two opposing teams. The multimodal understanding model can mark the name of the player holding the ball and other information in the interface image based on the above information and the current interface image, and can also predict in-depth analysis information such as the winning rate comparison between the two sides.

[0145] For example, the multimodal retrieval system can retrieve information such as a city’s recent weather conditions and travel suitability index. After the multimodal understanding model understands the information, it provides answers to users.

[0146] S400: Control the display 260 to display the annotation feedback information on the user interface.

[0147] The user can view the annotation feedback information through the user interface. The annotation feedback information may include annotation information and feedback information, such as Fig.10 As shown, annotation information is defined as the content marked on the user interface, such as Fig.11 As shown, feedback information is defined as the response made during a dialogue with the user.

[0148] The annotation information is information generated after the multimodal retrieval system associates information and / or the multimodal understanding model understands the information, for example, the current score on the field, the comparison of the winning rates of both sides predicted by the model, and other in-depth analysis information are displayed non-invasively at the bottom of the screen.

[0149] The feedback information is the answer provided to the user based on the information fed back by the multimodal retrieval system and / or the information understood by the multimodal understanding model. For example, the answer content is that the scene shown in the current video screen is a certain scenic spot in a certain city, and the recent weather is suitable for travel.

[0150] In some embodiments, the video content analysis method of multimodal technology can mark the target area on the target visual content based on the question text and the target visual content through the multimodal understanding model. Through the multimodal understanding model, based on the target visual content of the marked target area and the question text, a reply text of the question text is generated. Based on the target reply text and the target visual content, the target tag of the target visual content is recalled, and the target tag is used to trigger the function related to the target visual content. The reply text is displayed to realize the reply to the question text, and the target tag is displayed at the same time, so that the user can quickly execute the corresponding function, enrich the way of interacting with the user, thereby meeting the user needs and improving the user experience.

[0151] After receiving the annotation feedback information, the user may think that the annotation feedback information is inaccurate or needs further annotation, such as Fig.12 As shown, in some embodiments, by obtaining a second request annotation instruction, the second request annotation instruction is used to perform supplementary annotation; in response to the second request annotation instruction, preference information is extracted; the preference information and the current interface image are re-retrieved through the multimodal retrieval system to generate supplementary information, and the supplementary information, the second request annotation instruction and the current interface image are understood through the multimodal understanding model to generate supplementary annotation feedback information.

[0152] The second request for annotation instruction is similar to the first request for annotation instruction, and may be a supplementary instruction generated by the display device 200 according to a preset period, or may be a supplementary instruction input by a user, for performing supplementary annotation.

[0153] The preference information can be obtained through a supplementary instruction input by the user, or can be obtained through historical interaction information. In some embodiments, in response to the second request annotation instruction, the display 260 is controlled to display a preference setting interface, wherein the preference setting interface includes preference setting items; the preference setting item selected by the user based on the preference setting interface is obtained; and the preference information is generated according to the selected preference setting item.

[0154] If the annotation setting parameters are not entered when setting the display device 200 to automatically annotate, that is, the generated annotation feedback information may not be the annotation feedback information required by the user, when the user enters the second request annotation instruction, the preference setting interface is displayed. The preference setting interface includes multiple preference setting items. The preference setting items can be the same as the content of the annotation setting parameters, and can also include correction directions, supplementary directions, etc. for the annotation feedback information.

[0155] In some embodiments, historical interaction information is retrieved according to the content type, the historical interaction information including the association information and / or the annotation feedback information generated in response to the first request annotation instruction; and preference information is extracted from the historical interaction information.

[0156] When generating the associated information and the annotation feedback information, a first annotation request instruction is input, and the first annotation request instruction is generated through user input.

[0157] The user's input command contains the user's preference information. For different content types, for example, for sports events and plots, the user's needs are different. Therefore, by extracting the preference information of users of different content types, when generating supplementary annotation feedback information, different types of preference information can be generated to improve the accuracy of the supplementary annotation feedback information, better meet user needs, and enhance user experience.

[0158] It is understandable that for each generated association information, annotation feedback information and supplementary annotation feedback information, preference information can be extracted from the content input by the user, and a mapping relationship is formed between the preference information and the content type to improve the annotation accuracy.

[0159] After the supplementary annotation feedback information is generated, the user will continue to communicate with the display device 200 through the supplementary annotation feedback information, but because the supplementary annotation feedback information can be provided to the user in the form of annotation information and feedback information, when the supplementary annotation feedback information is displayed in the form of annotation information, the user cannot communicate with the display device 200 in real time, or the user has other needs, resulting in a poor user experience. In some embodiments, a prompt interface is generated based on the supplementary annotation feedback information and the interface image; in response to an interaction instruction input by the user based on the prompt interface, the display 260 is controlled to display a first interaction interface on the user interface; query information is generated based on the query instruction input by the user based on the first interaction interface; the query information and the interface image are input into the multimodal retrieval system to retrieve reply information through the multimodal retrieval system; and the display 260 is controlled to display the reply information on the first interaction interface.

[0160] For example, after the supplementary annotation feedback information is generated, the display device 200 can display a prompt interface, prompting the user to wake up the movie-watching companion, i.e., the first interactive interface, through a predetermined button. The movie-watching companion includes but is not limited to smart speakers, TV companion boxes, smart TV applications, etc. Further questions can be asked about the supplementary annotation feedback information through the movie-watching companion, or other needs can be met through the movie-watching companion, such as the need for a viewing environment.

[0161] Based on the above display device 200, some embodiments of the present application further provide an interface content annotation method, including the following steps:

[0162] In response to a first request annotation instruction, acquiring the currently displayed interface image and meta information of the user interface, wherein the first request annotation instruction is an instruction generated by the display device according to a preset period and / or an instruction input by a user;

[0163] Inputting the first request annotation instruction, the interface image, and the meta-information into a multimodal retrieval system to retrieve associated information through the multimodal retrieval system, wherein the multimodal retrieval system is used to perform information retrieval according to multimodal information, wherein the multimodal information includes keywords extracted from the first request annotation instruction, the interface image, and the meta-information;

[0164] Inputting the associated information, the first request annotation instruction, and the interface image into a multimodal understanding model to generate annotation feedback information through the multimodal understanding model, wherein the multimodal understanding model is a model generated by training the multimodal information based on a deep learning algorithm;

[0165] The display is controlled to display the annotation feedback information on the user interface.

[0166] It can be seen from the above technical solutions that some embodiments of the present application provide a display device and an interface content annotation method, wherein the method obtains the currently displayed interface image and the meta-information of the user interface by responding to a first request annotation instruction, and then inputs the first request annotation instruction, the interface image, and the meta-information into a multimodal retrieval system to retrieve associated information, and then inputs the associated information, the first request annotation instruction, and the interface image into a multimodal understanding model to generate annotation feedback information, and controls the display to display the annotation feedback information on the user interface. The method performs information retrieval on multimodal information through a multimodal retrieval system, wherein by using the first request annotation instruction, the interface image, and the meta-information as multimodal information, the accuracy of information retrieval and understanding can be improved, thereby improving the accuracy of the annotated content.

[0167] The same and similar parts between the various embodiments in this specification can be referenced to each other and will not be described again here.

[0168] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention or certain parts of the embodiments.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0170] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, It is characterized in that include: a display configured to display a user interface; The controller is configured as: In response to a first request annotation instruction, acquiring the currently displayed interface image and meta information of the user interface, wherein the first request annotation instruction is an instruction generated by the display device according to a preset period and / or an instruction input by a user; Inputting the first request annotation instruction, the interface image, and the meta-information into a multimodal retrieval system to retrieve associated information through the multimodal retrieval system, wherein the multimodal retrieval system is used to perform information retrieval according to multimodal information, wherein the multimodal information includes keywords extracted from the first request annotation instruction, the interface image, and the meta-information; Inputting the associated information, the first request annotation instruction, and the interface image into a multimodal understanding model to generate annotation feedback information through the multimodal understanding model, wherein the multimodal understanding model is a model generated by training the multimodal information based on a deep learning algorithm; The display is controlled to display the annotation feedback information on the user interface.

2. The display device according to claim 1, It is characterized in that The controller executes the marking instruction in response to the first request, and is specifically configured to: Acquire annotation setting parameters input by a user, wherein the annotation setting parameters are annotation preference setting parameters of the interface image, and the annotation setting parameters include annotation content of the interface image; Extracting annotation preference information from the annotation setting parameters, wherein the annotation preference information includes at least one annotation type item; According to the labeling preference information, a first labeling request instruction is generated according to a preset period.

3. The display device according to claim 1, It is characterized in that The controller executes to obtain the currently displayed interface image and the meta information of the user interface, and is specifically configured as follows: Acquire the interface type of the interface image, where the interface type is a video interface or an image interface; If the interface type is a video interface, obtaining the content type of the current interface image, so as to obtain the image of the current video interface and the meta information of the video interface according to the content type; If the interface type is a graphic interface, an image of the graphic interface and meta information of the graphic interface are obtained.

4. The display device according to claim 3, It is characterized in that The controller executes inputting the associated information, the first request annotation instruction, and the interface image into a multimodal understanding model to generate annotation feedback information through the multimodal understanding model, and is specifically configured as follows: According to the content type, setting emphasis information, the emphasis information is used to characterize the emphasis of annotation feedback of the video screen of the content type; The associated information, the first request annotation instruction, and the interface image are input into a multimodal understanding model, so as to generate annotation feedback information through the multimodal understanding model based on the emphasis information.

5. The display device according to claim 3, It is characterized in that After the controller executes inputting the associated information, the first request annotation instruction, and the interface image into the multimodal understanding model to generate annotation feedback information through the multimodal understanding model, the controller is further configured to: Obtaining a second request annotation instruction, where the second request annotation instruction is used to perform supplementary annotation; extracting preference information in response to the second request marking instruction; Inputting the preference information and the interface image into a multimodal retrieval system to retrieve supplementary information through the multimodal retrieval system; Inputting the supplementary information, the second request annotation instruction, and the interface image into a multimodal understanding model to generate supplementary annotation feedback information through the multimodal understanding model; The display is controlled to display supplementary annotation feedback information on the user interface.

6. The display device according to claim 5, It is characterized in that The controller performs extraction of preference information and is specifically configured to: In response to the second request annotation instruction, controlling the display to display a preference setting interface, wherein the preference setting interface includes preference setting items; Obtaining the preference setting item selected by the user based on the preference setting interface; The preference information is generated according to the selected preference setting item.

7. The display device according to claim 5, It is characterized in that The controller performs extraction of preference information and is specifically configured to: Retrieving historical interaction information according to the content type, the historical interaction information including the association information and / or the annotation feedback information generated in response to the first request annotation instruction; Preference information is extracted from the historical interaction information.

8. The display device according to claim 6, It is characterized in that After the controller controls the display to display the supplementary annotation feedback information on the user interface, the controller is further configured to: Generate a prompt interface according to the supplementary annotation feedback information and the interface image; In response to an interaction instruction input by a user based on the prompt interface, controlling the display to display a first interaction interface on the user interface; Generate inquiry information according to an inquiry instruction input by the user based on the first interactive interface; Inputting the query information and the interface image into a multimodal retrieval system to retrieve reply information through the multimodal retrieval system; Control the display to display reply information on the first interactive interface.

9. The display device according to claim 1, It is characterized in that After the controller executes inputting the first request annotation instruction, the interface image, and the meta-information into the multimodal retrieval system to retrieve the associated information through the multimodal retrieval system, the controller is further configured to: Controlling the display to display the associated information on the user interface; In response to a focus movement instruction input by a user, controlling the display to display a second interactive interface on the user interface; Generate error correction information according to the error correction instruction input by the user based on the second interactive interface; The error correction information, the associated information, the first request annotation instruction, and the interface image are input into a multimodal understanding model to generate annotation feedback information through the multimodal understanding model.

10. A method for marking interface content, It is characterized in that Applied to the display device according to any one of claims 1 to 9, the interface content annotation method comprises: In response to a first request annotation instruction, acquiring the currently displayed interface image and meta information of the user interface, wherein the first request annotation instruction is an instruction generated by the display device according to a preset period and / or an instruction input by a user; Inputting the first request annotation instruction, the interface image, and the meta-information into a multimodal retrieval system to retrieve associated information through the multimodal retrieval system, wherein the multimodal retrieval system is used to perform information retrieval according to multimodal information, wherein the multimodal information includes keywords extracted from the first request annotation instruction, the interface image, and the meta-information; Inputting the associated information, the first request annotation instruction, and the interface image into a multimodal understanding model to generate annotation feedback information through the multimodal understanding model, wherein the multimodal understanding model is a model generated by training the multimodal information based on a deep learning algorithm; The control display displays the annotation feedback information on the user interface.

Images