Augmented reality device and interactive enhancement method
By detecting the electronic watermark information in the target image of the display device screen in the augmented reality device, the problem of low accuracy in traditional augmented reality devices when identifying the screen of the electronic device is solved, and a higher recognition accuracy is achieved.
Patent Information
- Application Number
- CN202110535801.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-17
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-05-17
AI Technical Summary
Traditional augmented reality devices have low accuracy when identifying screen images of electronic devices, mainly due to the difficulty of image recognition caused by picture brightness and image quality processing.
An augmented reality device and an interaction enhancement method are provided to generate identification object information to improve recognition accuracy by acquiring a target image of a display device screen and detecting electronic watermark information.
Directly obtaining the identification object information through electronic watermark information, avoiding the impact of picture brightness and image quality processing on the recognition process and improving the recognition accuracy.
Smart Images

Figure CN114363705B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of augmented reality devices, and in particular to an augmented reality device and an interactive enhancement method. Background Art
[0002] Augmented Reality (AR) technology is a technology that integrates virtual images with real scenes. Generated virtual objects, such as text, images, three-dimensional models, and video images can be simulated and applied to real scene images, giving users a viewing experience that combines virtual images with real images. An augmented reality device is an intelligent display device that uses augmented reality technology. It can obtain real scene images and virtual images in real time during use, and then integrate the virtual images with the real scene images to present the final integrated image to the user.
[0003] When presenting the final fused picture, the augmented reality device can also display some auxiliary information pictures in the fused picture to mark the real scene picture and / or virtual picture for user viewing and identification, thereby improving the user experience. The auxiliary information picture can be a combination of patterns and texts. For example, when the fused picture includes the content of an oil painting, the name, author and content of the oil painting can be marked by the speech bubble pattern and the text content in the bubble. The pattern content of the auxiliary information picture can be defined by a unified UI interface, and the text content of the auxiliary information picture can be obtained by identification through an artificial intelligence (AI) image classification algorithm.
[0004] The AI picture classification algorithm can perform image recognition on the image corresponding to the fused picture, identify the key elements contained in the image, and generate text content in the auxiliary information. However, when the fused picture includes the pictures presented by the screens of electronic devices such as televisions and advertising machines, due to the influence of the screen brightness and image quality processing, the displayed content is quite different from the training scene of the AI picture classification algorithm, which will reduce the accuracy of the AI picture classification algorithm in recognizing the fused picture with the screen of the electronic device. Summary of the invention
[0005] The present application provides an augmented reality device and an interactive enhancement method to solve the problem of low recognition accuracy of traditional augmented reality devices.
[0006] In one aspect, the present application provides an augmented reality device, comprising: a display, an image acquisition device, a communicator, and a controller. The display is configured as a user interface and displays a fused screen, wherein the fused screen includes a real scene screen and a virtual object screen. The image acquisition device is configured to obtain a real scene screen. The communicator is configured to connect to a display device. The controller is configured to execute the following program steps:
[0007] Receiving a control instruction input by a user for displaying an augmented reality screen;
[0008] In response to the control instruction, acquiring a target image in the augmented reality picture, the target image being an image displayed on a screen of the display device;
[0009] If the target image contains electronic watermark information, generating identification object information according to the electronic watermark information;
[0010] The display is controlled to display the identification object information.
[0011] On the other hand, the present application also provides an interactive enhancement method, which is applied to the above-mentioned augmented reality device, wherein the augmented reality device includes a display, an image acquisition device, a communicator, and a controller; the augmented reality device is connected to a display device via the communicator, and the interactive enhancement method includes the following steps:
[0012] Receiving a control instruction input by a user for displaying an augmented reality picture, wherein the fused picture includes a real scene picture and a virtual object picture;
[0013] In response to the control instruction, acquiring a target image in the augmented reality picture, the target image being an image displayed on a screen of the display device;
[0014] If the target image contains electronic watermark information, generating identification object information according to the electronic watermark information;
[0015] The display is controlled to display the identification object information.
[0016] It can be seen from the above technical solutions that the augmented reality device and interactive enhancement method provided by the present application can obtain the target image in the fused screen after the user inputs the control command to display the fused screen, that is, obtain the image currently displayed by the display device. Then, by performing electronic watermark detection on the target image, when the target image contains electronic watermark information, the identification object information is directly generated through the electronic watermark information to display the identification object information in the UI interface. The interactive enhancement method can obtain part of the identification object information through the electronic watermark information, so as to alleviate the influence of the screen brightness, image quality processing, etc. of the display device on the recognition process and improve the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0018] Figure 1 A schematic diagram of the structure of a display system including a virtual reality device in an embodiment of the present application;
[0019] Figure 2 This is a schematic diagram of the structure of an augmented reality device in an embodiment of the present application;
[0020] Figure 3 This is a schematic diagram of the structure of an augmented reality device with lenses in an embodiment of the present application;
[0021] Figure 4 This is a schematic diagram of the effect of presenting an augmented reality image through a projection device and a lens in an embodiment of the present application;
[0022] Figure 5 This is a schematic diagram of the global interface of the VR scene in the embodiment of the present application;
[0023] Figure 6 A schematic diagram of the recommended content area of the global interface in an embodiment of the present application;
[0024] Figure 7 This is a schematic diagram of the application quick operation entry area of the global interface in an embodiment of the present application;
[0025] Figure 8 A schematic diagram of suspended matter at a global interface in an embodiment of the present application;
[0026] Fig. 9 This is a schematic diagram of an AR screen in an embodiment of the present application;
[0027] Fig.10 This is a schematic diagram of the effect of displaying the information of the identified object in the AR screen in an embodiment of the present application;
[0028] Fig.11 This is a flow chart of the interactive enhancement method in the embodiment of the present application;
[0029] Fig.12 This is a schematic diagram of the process of detecting electronic watermark information on a target image in an embodiment of the present application;
[0030] Fig.13 A schematic diagram of a process for displaying an identification object according to whether it contains electronic watermark information in an embodiment of the present application;
[0031] Fig.14This is a schematic diagram of the superimposed display of the identification object information in the embodiment of the present application;
[0032] Fig.15 This is a schematic diagram of a process for detecting brightness gradient changes in an embodiment of the present application;
[0033] Fig.16 A schematic diagram of the process of querying electronic watermark information in an embodiment of the present application;
[0034] Fig.17 A schematic diagram of the process of obtaining identification object information in an embodiment of the present application;
[0035] Fig.18 A schematic diagram of a process for displaying identification object information in an embodiment of the present application;
[0036] Fig.19 A schematic diagram of the process of generating electronic watermark information in an embodiment of the present application. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solutions and advantages of the exemplary embodiments of the present application clearer, the technical solutions in the exemplary embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, not all of the embodiments.
[0038] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.
[0039] It should be understood that the terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented according to an order other than those given in the diagrams or descriptions of the embodiments of this application.
[0040] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.
[0041] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.
[0042] References throughout this specification to "multiple embodiments", "some embodiments", "one embodiment" or "an embodiment" etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in at least one embodiment. Therefore, phrases such as "in multiple embodiments", "in some embodiments", "in at least another embodiment" or "in an embodiment" etc. that appear throughout this specification do not necessarily all refer to the same embodiment. In addition, in one or more embodiments, specific features, structures or characteristics may be combined in any suitable manner. Therefore, without limitation, specific features, structures or characteristics shown or described in conjunction with one embodiment may be combined in whole or in part with features, structures or characteristics of one or more other embodiments. Such modifications and variations are intended to be included within the scope of this application.
[0043] In an embodiment of the present application, an augmented reality (AR) device is a display device that can be worn on the user's head to provide the user with an immersive experience. The AR device can acquire real-scene image pictures in real time, and fuse virtual object images into the real-scene pictures to form an augmented reality picture effect. The AR device includes but is not limited to AR head-mounted devices, AR glasses, wearable AR game consoles, etc. The augmented reality device can be an independent device, such as an AR head-mounted device. It can also be a combination of multiple devices and a combination of devices with an augmented reality picture display function. For example, an augmented reality device can be formed by adding a camera to a virtual reality (VR) device.
[0044] The augmented reality devices described in the embodiments of the present application are all described using AR head-mounted devices as an example of the technical solution. It should be understood that the provided technical solution can also be applied to other types of virtual reality devices. The virtual reality device 500 can operate independently or be connected to other smart display devices as an external device, such as Figure 1 The display device may be a smart TV, a computer, a tablet computer, a server, etc.
[0045] The augmented reality device 500 can be worn on the user's head, and after being worn on the user's face, the real scene picture is obtained in real time. By displaying the acquired real scene picture in real time, a close-up image is provided for the user's eyes, bringing an immersive experience. In order to present a specific display picture, the augmented reality device 500 may include a plurality of components for displaying pictures and for wearing. Taking the AR head-mounted device as an example, the augmented reality device 500 includes but is not limited to a housing, a position fixing part, an image acquisition device, an optical system, a display component, a posture detection circuit, an interface circuit and other components. In practical applications, the image acquisition device can be fixed on the housing or partially embedded in the housing to collect the real scene picture directly in front of the user. The optical system, the display component, the posture detection circuit and the interface circuit can be encapsulated in the housing to present a specific display picture; the position fixing parts are arranged on both sides of the housing to fix the entire AR device on the user's head.
[0046] The image acquisition device can replace the user's eyes to record the real scene in front of the video. Therefore, the image acquisition device can be one or more cameras that are compatible with the user's eye vision range. For example, Figure 2 As shown, the image acquisition device can be composed of two cameras arranged on the front shell. In use, the two cameras can capture the real scene picture in real time, and send the real scene picture to the display component in the form of a data stream for display. Among them, the data stream formed by the real scene picture includes multiple continuous frame images.
[0047] While the display component displays the real scene picture, the augmented reality device 500 can also add virtual objects to the real scene picture to finally merge and form an augmented reality picture. To this end, in some embodiments of the present application, the augmented reality device 500 can also include a data processing system such as a controller. After acquiring the real scene picture, the image acquisition device can first send the real scene picture to the controller, trigger the controller to call the virtual object, and render to obtain the virtual object picture. Then merge the virtual object picture with the real scene picture to form the final augmented reality picture. Finally, send the augmented reality picture to the display component for display.
[0048] The virtual objects added by the controller to the real scene screen include but are not limited to text, images, three-dimensional models, and video screens. Since the augmented reality device 500 can merge the real scene screen with the virtual object screen, in order to obtain a better fusion effect, the virtual object screen added to the real scene screen should also have a three-dimensional effect. For example, when a 3D model is added to the real scene screen, as the user's head movement changes, the screen presented by the 3D model can also change the viewing angle in real time.
[0049] To this end, in some embodiments, a posture detection circuit may be built into the augmented reality device 500. The posture detection circuit may be composed of a plurality of posture sensors, including but not limited to a gravity acceleration sensor, a gyroscope, etc. In use, the posture detection circuit may detect the head posture data of the user in use in real time, and send the head posture data to the controller, so that the controller may render the added virtual objects according to the head posture data, forming virtual object images under different viewing angles.
[0050] For example, the augmented reality device 500 can build a rendering scene based on the Unity 3D engine, and load a virtual object model into the rendering scene during use. The virtual rendering scene also includes virtual cameras such as a left display camera and a right display camera for presenting the final rendering effect. The augmented reality device 500 can capture images of virtual objects in the rendering scene through the left display camera and the right display camera to obtain the left eye picture and the right eye picture respectively, and output them to the display component for display. When the user's head rotates, the posture detection circuit can detect the corresponding posture data and send it to the controller. The controller adjusts the shooting angles of the left display camera and the right display camera in the rendering scene according to the posture data, so as to obtain virtual object pictures under different viewing angles.
[0051] In order to display the final augmented reality picture, the display component may include a display and a data transmission circuit related to the display process. Among them, for augmented reality devices with different display modes, the number of displays in the display component is also different. That is, in some embodiments, the augmented reality device 500 includes two displays, which present AR pictures for the user's eyes respectively. For example, for an AR head-mounted device, the presented AR picture includes a left eye picture and a right eye picture. The display component may be composed of a left display and a right display. When displaying the AR picture, the left display may display the left eye picture. The left eye picture is obtained by superimposing and fusion of the real scene picture taken by the left camera and the virtual object picture taken by the left display camera in the rendering scene. Similarly, the right display is used to display the right eye picture, and the right eye picture is obtained by superimposing and fusion of the real scene picture taken by the right camera and the virtual object picture taken by the right display camera in the rendering scene.
[0052] It should be noted that if Figure 3 As shown, for some augmented reality devices 500, their displays may not display the real scene images captured by the image acquisition device, but instead set the area in front of the user's eyes to be transparent so that the user can directly view the real scene images, and then add the virtual object images in front of the user's eyes through a transparent display or projection to form an augmented reality image.
[0053] In some embodiments, Figure 4As shown, the augmented reality device 500 may also include a projection device and a lens. The projection device is used to project a virtual object image; the lens may be composed of a plurality of polarizers to form a semi-transparent and semi-reflective lens. The semi-transparent and semi-reflective lens formed can reflect or transmit light incident from different directions, so that the real scene image in front of the user can enter the user's field of view through the lens on the one hand; on the other hand, it can reflect the light emitted by the projection device, so that the virtual object image projected by the projection device can enter the user's field of view through the reflection effect of the inner side of the lens.
[0054] In order to obtain a better image fusion effect, the thickness of the lens of the augmented reality device 500 gradually decreases from the top to the bottom, so that the reflected virtual object image light can be parallel to the user's horizon, alleviating the deformation of the virtual object image during the reflection process. In addition, the augmented reality device 500 that presents the augmented reality image through the projection device and the lens also includes a camera. It is used to shoot the real scene image, but the shot real scene image will not be displayed, and is only used to perform the fusion calculation process of the virtual object occlusion display method.
[0055] In some embodiments, the augmented reality device 500 may also include only one display. For example, the augmented reality device 500 may be composed of a widescreen display. The widescreen display is divided into two parts: a left area and a right area, wherein the left area is used to display the left eye image, and the right area is used to display the right eye image.
[0056] It should be noted that the above-mentioned real scene picture can be obtained by the camera in the augmented reality device 500 performing image capture on the current user's usage scene; it can also be obtained by other means. For example, the augmented reality device 500 can be connected to an image acquisition device or other device with an image acquisition device through an interface circuit, and the real scene picture can be obtained through the connected device.
[0057] Therefore, in some embodiments, the augmented reality device 500 can be connected to the display device 200, and a network-based display system can be constructed with the server 400, and data can be exchanged in real time between the augmented reality device 500, the display device 200 and the server 400. For example, the display device 200 can obtain media data from the server 400 and play it, and transmit specific screen content to the augmented reality device 500 for display. That is, the real scene screen has a broad meaning, which can refer to the real-time scene image captured by the image acquisition device of the augmented reality device 500, or the media screen sent to the augmented reality device 500 by the display device 200, the server 400, etc.
[0058] The display device 200 may be a liquid crystal display, an OLED display, or a projection display device. The specific display device type, size, and resolution are not limited. It is understood by those skilled in the art that the display device 200 may be modified in performance and configuration as needed. The display device 200 may provide a broadcast receiving television function, and may also provide an intelligent network television function that provides a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.
[0059] The display device 200 and the augmented reality device 500 also communicate data with the server 400 through a variety of communication methods. The display device 200 and the augmented reality device 500 may be allowed to communicate and connect via a local area network (LAN), a wireless local area network (WLAN) and other networks. The server 400 may provide various content and interactions to the display device 200. For example, the display device 200 receives software program updates or accesses a remotely stored digital media library by sending and receiving information and interacting with an electronic program guide (EPG). The server 400 may be a cluster or multiple clusters. It may include one or more types of servers. Other network service content such as video on demand and advertising services is provided through the server 400.
[0060] During the data interaction, the user can operate the display device 200 through the mobile terminal 300 and the remote controller 100. The mobile terminal 300 and the remote controller 100 can communicate with the display device 200 by direct wireless connection or by indirect connection. That is, in some embodiments, the mobile terminal 300 and the remote controller 100 can communicate with the display device 200 by direct connection such as Bluetooth or infrared. When sending a control command, the mobile terminal 300 and the remote controller 100 can directly send the control command data to the display device 200 by Bluetooth or infrared.
[0061] In other embodiments, the mobile terminal 300 and the remote controller 100 may also access the same wireless network as the display device 200 through a wireless router to establish indirect connection communication with the display device 200 through the wireless network. When sending a control command, the mobile terminal 300 and the remote controller 100 may first send the control command data to the wireless router, and then forward the control command data to the display device 200 through the wireless router.
[0062] In some embodiments, the user can also use the mobile terminal 300 and the remote controller 100 to interact with the augmented reality device 500. The connection mode established during the interaction process can also be a direct connection and an indirect connection. For example, the mobile terminal 300 and the remote controller 100 can be used as handles in the augmented reality scene to realize functions such as somatosensory interaction. For another example, the interaction instruction can be forwarded to the augmented reality device 500 through a wireless router or a display device 200.
[0063] In addition to the above-mentioned media data, the interface circuit of the augmented reality device 500 can be used to transmit interactive data. In practical applications, the augmented reality device 500 can also be connected to other display devices or peripherals through the interface circuit to achieve more complex functions by exchanging data with the connected devices. For example, the augmented reality device 500 can also be connected to a handle through the interface circuit, and the handle can be held by the user to perform related operations in the user interface.
[0064] The user interface can be presented in a variety of different types of UI layouts according to user operations. For example, the user interface can include a global interface. The global UI after the AR / VR terminal is started is as follows: Figure 5 As shown, the global UI can be displayed on the display screen of the AR / VR terminal or on the display of the display device. The global UI can include a recommended content area 1, a business classification extension area 2, an application quick operation entry area 3, and a floating object area 4.
[0065] The recommended content area 1 is used to configure different classification TAB columns; in the columns, media resources, special topics, etc. can be selected and configured; the media resources may include 2D film and television, educational courses, travel, 3D, 360-degree panoramic view, live broadcast, 4K film and television, program applications, games, travel and other businesses with media content, and the columns can select different template styles and support the simultaneous recommendation and arrangement of media resources and special topics, such as Figure 6 shown.
[0066] In some embodiments, a status bar may also be provided at the top of the recommended content area 1, and a plurality of display controls may be provided in the status bar, including common options such as time, network connection status, and battery level. The content included in the status bar may be customized by the user. For example, weather, user avatars, and other content may be added to the status bar. The content contained in the status bar may be selected by the user to perform the corresponding function. For example, when the user clicks on the time option, the augmented reality device 500 may display a time setting window in the current interface, or jump to the calendar interface. When the user clicks on the network connection status option, the augmented reality device 500 may display a WiFi list in the current interface, or jump to the network settings interface.
[0067] The content displayed in the status bar can be presented in different content forms according to the setting status of the specific project. For example, the time control can directly display specific time text information and display different texts at different times; the power control can display different battery patterns according to the current remaining power of the augmented reality device 500.
[0068] The status bar is used to enable the user to perform commonly used control operations and quickly set up the augmented reality device 500. Since the setting program for the augmented reality device 500 includes many items, it is usually not possible to display all commonly used setting options in the status bar. To this end, in some embodiments, an extended option may also be provided in the status bar. After the extended option is selected, an extended window may be presented in the current interface, and a plurality of setting options may be further provided in the extended window to implement other functions of the augmented reality device 500.
[0069] For example, in some embodiments, after the extension option is selected, a "Quick Center" option can be set in the extension window. After the user clicks the Quick Center option, the augmented reality device 500 can display the Quick Center window. The Quick Center window can include "Screenshot", "Screen Recording" and "Screen Casting" options for waking up the corresponding functions respectively.
[0070] The business category extension area 2 supports the configuration of extended categories of different categories. If there is a new business type, it supports the configuration of an independent TAB to display the corresponding page content. The extended categories in the business category extension area 2 can also be sorted and adjusted and offline business operations can be performed. In some embodiments, the business category extension area 2 may include the following content: film and television, education, travel, applications, and mine. In some embodiments, the business category extension area 2 is configured to display a large business category TAB and supports the configuration of more categories. Its icons support configuration, such as Figure 5 shown.
[0071] The application quick operation entry area 3 can specify pre-installed applications to be displayed in front for operation recommendation, and supports configuration of special icon styles to replace default icons. The pre-installed applications can be specified as multiple. In some embodiments, the application quick operation entry area 3 also includes a left-moving control and a right-moving control for moving the option target, which are used to select different icons, such as Figure 7 shown.
[0072] The floating object area 4 can be configured to be located at the upper left side or the upper right side of the fixed area, and can be configured as a replaceable image or a jump link. For example, after receiving a confirmation operation, the floating object jumps to a certain application or displays a specified function page, such as Figure 8 In some embodiments, the floating object may not be configured with a jump link and is simply used for image display.
[0073] In some embodiments, the global UI also includes a status bar at the top for displaying time, network connection status, battery status, and more quick operation entries. When an icon is selected using the handle of the AR / VR terminal, that is, a handheld controller, the icon will display a text prompt including left and right expansion, and the selected icon will be stretched left and right according to its position.
[0074] For example, after selecting the search icon, the search icon will display the text "Search" and the original icon. After further clicking the icon or text, the search icon will jump to the search page. For example, clicking the favorite icon will jump to the favorites TAB, clicking the history icon will display the history page by default, clicking the search icon will jump to the global search page, and clicking the message icon will jump to the message page.
[0075] In some embodiments, interaction can be performed through peripherals. For example, the handle of the AR / VR terminal can operate the user interface of the AR / VR terminal, including a back button; a home button, and a long press of the home button can realize a reset function; a volume plus or minus button; and a touch area, which can realize the focus click, slide, hold and drag functions.
[0076] In some embodiments, the user interface may also be provided with an "AR / VR" mode switching option, and the augmented reality device 500 may switch between the VR mode and the AR mode when the user clicks the switching option. For the VR mode, the augmented reality device 500 will not enable the image acquisition device, and only renders the specified media data by rendering the scene to form a virtual reality screen; while for the AR mode, the augmented reality device 500 needs to enable the image acquisition device to shoot in real time to obtain the real scene screen, and add virtual objects to the rendered scene to output the virtual object screen, so as to form an AR screen after fusion.
[0077] In some embodiments, in order to facilitate users to watch AR pictures, the AR device can perform image recognition on the AR picture, mark the recognized objects, and finally display them in the fused AR picture. For example, the AR device can identify the type of device in the real scene picture according to the appearance shape, such as identifying a flat-panel display device as a TV, and track and mark the TV image through anchor points to indicate to the user that the image is a TV.
[0078] While displaying the object recognition results, the annotated content can be further enriched according to the recognition results. For example, after marking the TV in the real scene, the AR device can further identify the specific model, device parameters and other information of the TV. All of this information can be displayed in the AR screen together with the name of the TV and other content. For the sake of ease of description, in the embodiment of the present application, the identified device type and additional description content are referred to as identification object information.
[0079] The identification object information can be displayed in the AR screen in a specific UI layout. The final display effect of the identification object information is different in different AR devices or different usage scenarios. For example, for shopping scenarios, the final display effect presented should describe the object features as detailed as possible, then the identification object information can be composed of guide lines and parameter lists. For navigation scenarios, the final display effect presented should try not to block the real scene viewed by the user, then the identification object information can be presented in the form of points and text names so that users can accurately distinguish.
[0080] like Fig. 9 As shown, when the real scene screen includes display devices such as televisions, since images can also be displayed on the screen of the display device, the images displayed on the display device may also be presented in the fused AR screen, forming a "picture-in-picture" effect. The AR device can also perform image recognition on the images displayed on the display device to determine the identification object information. For example, when a certain oil painting is displayed on a display device, the AR device can identify the displayed oil painting and identify the specific content of the oil painting, such as whether the scene of the oil painting is a banquet room. The identified information is then displayed in the form of an anchor point and a bubble, that is, the anchor point is located at a point in the oil painting displayed on the screen, and the bubble is in the upper right corner of the anchor point. The bubble includes the text "Oil Painting Banquet Room", thereby completing the marking of the content displayed on the display device, such as Fig.10 For the convenience of description, in the embodiment of the present application, the image displayed on the screen of the display device is referred to as the target image.
[0081] In the above object recognition process, the AR device can first build an image recognition model based on the AI image classification algorithm. Then, the fused screen image is obtained and the obtained image is input into the image recognition model. The image recognition model can calculate the classification probability of the input image according to the AI image classification algorithm, and can recognize the object in the image.
[0082] Since the image recognition model is a classification model obtained through training with a large number of sample images, the accuracy of the classification results of the image recognition model is affected by the sample images. When the scene of the image to be classified is similar to that of the sample image, the classification result is more accurate; when the scene of the image to be classified is very different from that of the sample image, the classification result is inaccurate.
[0083] As for the AR screen containing the target image, since the AR screen is affected by the user's posture, the target image finally presented in the fused screen may appear in a variety of uncertain postures, such as straight-on, squint, tilt, flip, etc. Such AR screen images are obviously very different from the sample images of the image recognition model. Therefore, after the AR screen images are input into the image recognition model, the accuracy of the recognition results obtained will be greatly reduced.
[0084] In order to improve the accuracy of object recognition, an augmented reality device is provided in some embodiments of the present application, including: a display, an image acquisition device, a communicator and a controller. The display is configured as a user interface and displays a fused screen, and the fused screen includes a real scene screen and a virtual object screen. The image acquisition device is configured to obtain a real scene screen. The communicator is configured to connect to a display device.
[0085] like Fig.11 As shown, the controller of the augmented reality device can be configured to perform an interactive enhancement method for improving the accuracy of object recognition, which specifically includes the following contents:
[0086] The controller of the augmented reality device 500 receives a control instruction input by the user for displaying an augmented reality screen. The controller of the augmented reality device 500 can receive multiple control instructions input by the user during the user's use. Each control instruction corresponds to a control function, and some control instructions can be used to display an augmented reality screen. The user can input a control instruction for displaying an augmented reality screen in different usage environments. For example, the control instruction for displaying an augmented reality screen can be input by opening an application with an AR function; the control instruction can also be input by opening a virtual model file in the augmented reality device 500.
[0087] The specific interactive actions can be presented in different action forms according to the different interactive modes supported by the augmented reality device 500. For example, for an AR device operated by an external handle, the user can control the focus cursor in the application interface through the handle, and when the focus cursor moves to the AR application position, click the "OK / Confirm" key on the handle to start the AR application. At this time, the controller can receive the control command input by the user for displaying the augmented reality screen.
[0088] For some augmented reality devices 500, an intelligent voice system may be built-in or externally connected, so the user may also input control instructions for displaying augmented reality images through the intelligent voice system. For example, the user may control the augmented reality device 500 to start an AR application or open an AR resource file by inputting voice content such as "open ××AR application" or "I want to see AR", and at this time, the controller may also receive control instructions for displaying augmented reality images.
[0089] After obtaining the control instruction for displaying the AR screen, the AR device can respond to the control instruction and extract the target image in the augmented reality screen. Since the target image is an image displayed on the screen of the display device 200, the AR device can obtain the target image from the display device 200.
[0090] To this end, after obtaining the control instruction for displaying the AR screen, the augmented reality device 500 can send an image acquisition request to the display device 200 through the communicator. After receiving the image acquisition request, the display device 200 can send the currently displayed image, that is, the target image, to the augmented reality device 500, so that the augmented reality device 500 can acquire the target image.
[0091] In order to transmit the request instruction and the target image, the display device 200 needs to establish a specific form of communication connection relationship with the augmented reality device 500. The communication connection relationship can be based on wired communication or wireless communication. For example, for wired communication, the communicator of the augmented reality device 500 can be a USB interface, and the augmented reality device 500 can be connected to the display device 200 via a USB data cable and establish a communication connection relationship based on the USB transmission protocol. For wireless communication, the communicator can have built-in communication function modules such as Bluetooth, infrared, and WiFi networks, and establish a wireless connection relationship through corresponding transmission protocols.
[0092] In the process of acquiring the target image, the request instruction and the target image data can be based on the same communication connection mode or on different communication connection modes. For example, the augmented reality device 500 can send an image acquisition request to the display device 200 via a Bluetooth connection, but since the transmission efficiency of the Bluetooth connection is low, after receiving the image acquisition request, the display device 200 sends the target image to the augmented reality device 500 via a WiFi network.
[0093] In some embodiments, when the image displayed by the display device 200 is not a local image but a network media resource, the display device 200 may not directly send the source file of the target image to the augmented reality device 500, but after receiving the image acquisition request from the augmented reality device 500, the display device 200 sends the uniform resource locator (URL) address corresponding to the target image to the augmented reality device 500, and the augmented reality device 500 then obtains the target image by accessing the URL address.
[0094] It should be noted that when acquiring the target image, the communication connection relationship between the augmented reality device 500 and the display device 200 can also be detected. When data can be exchanged between the augmented reality device 500 and the display device 200, an image acquisition request can be sent to the display device 200 through the communicator; and when data cannot be exchanged between the augmented reality device 500 and the display device 200, a connection interface can be displayed in the user interface of the display device 200 and the augmented reality device 500, respectively, for establishing a communication connection relationship.
[0095] In some embodiments, if the user does not control the augmented reality device 500 to establish a communication connection with the display device 200 within a set time after the connection interface is displayed, the augmented reality device 500 can also perform a capture operation on the AR screen, capture the content corresponding to the screen of the display device 200 in the AR screen, and thus obtain the target image.
[0096] After acquiring the target image, the augmented reality device 500 can also detect whether the target image contains electronic watermark information. The electronic watermark information can be a part of the image data and sound data that is not important to the user, that is, the so-called redundant part, which is mixed into the data as noise unrelated to the text, so as to serve as a signature or verification information.
[0097] For example, in order to reduce the impact of electronic watermarks on the user's sensory experience, the target image can be converted into a frequency domain graph, and then frequency domain features can be added to the frequency domain graph. Frequency domain features can be combined to form text information in a specific encoding method to represent signatures or verify information content. Since frequency domain features are modifications made to the frequency domain graph, the modified content can be used as noise data, so it will not be significantly reflected in the specific display content of the target image, that is, it will not affect the user's viewing effect.
[0098] In order to detect the electronic watermark information in the target image, a detection module for detecting the electronic watermark information can also be built into the AR device. After acquiring the target image, the detection module can perform watermark detection on the target image to obtain the specific content of the electronic watermark. Fig.12As shown, in some embodiments, after acquiring the target image, the AR device can use the detection module to convert the target image into a frequency domain graph, and then extract frequency domain feature data from the frequency domain graph, so as to read the content in the electronic watermark according to the frequency domain feature data.
[0099] It should be noted that the electronic watermark information can also be mixed into the target image in other ways. For example, information related to the target image content can be edited in the file description of the target image file as the electronic watermark information. Therefore, after acquiring the target image, the AR device can also read the file description corresponding to the target image to obtain the electronic watermark information.
[0100] In order to present the identification object information corresponding to the target image, such as Fig.13 As shown, the AR device can determine the identification object information to be displayed according to whether the target image contains electronic watermark information. If the target image contains electronic watermark information, the identification object information is generated according to the electronic watermark information. For example, when the target image contains electronic watermark information that describes the current image content as "oil painting banquet room", the AR device can directly extract the text content in the electronic watermark information and display it as the identification object information in the user interface.
[0101] It can be seen that for the target image containing electronic watermark information, the AR device can directly obtain the object recognition result by reading the electronic watermark information, without having to execute the AI image classification algorithm on the AR screen. Therefore, the influence of the display brightness of the display device 200 on the recognition result can be avoided, thereby improving the accuracy of the recognized object information.
[0102] If the target image does not contain electronic watermark information, the AI image classification algorithm can be executed on the target image to obtain the identification object information. That is, the target image can be input into the image recognition model to obtain the classification probability of the target image through the recognition model to obtain the identification object information.
[0103] Compared with executing the AI picture classification algorithm on the AR screen image, the present embodiment executes the AI picture classification algorithm on the target image. Since the target image is the original file of the image displayed in the display device 200 or a screenshot file with the same content as the original file, it will not be affected by the display process such as the screen brightness and image quality processing. It also reduces the picture deformation such as deflection caused by AR posture, so that the image in the input image recognition model is similar to the sample image, which can also improve the accuracy of identifying object information.
[0104] For example, when the screen of the display device 200 shows an oil painting image of a banquet room, the target image obtained is the oil painting image. Therefore, when the oil painting image is input into the image recognition model, the image recognition model is an AI picture classification algorithm executed on the oil painting image, so the influence of brightness on the recognition result can be alleviated. If the AI picture classification algorithm is executed on the AR screen image, the image input to the image recognition model includes images in the current real scene, such as TV shells, tables and chairs, etc., as well as oil painting images. Obviously, the oil painting image is only a small part of the AR screen image, and there are interference factors such as brightness and image quality processing, which will reduce the accuracy of the recognized object information.
[0105] Through the above-mentioned method of obtaining the identification object information, after obtaining the identification object information, the augmented reality device 500 can control the display to display the identification object information, that is, add display content containing the identification object information in the user interface. For example, when the identification object information obtained is that the current target image type is an oil painting and the target image content is a banquet room, the text "Oil Painting Banquet Room" can be displayed at the corresponding position of the screen of the display device 200 in the user interface to prompt the specific content currently displayed by the display device 200 in the AR screen.
[0106] In addition, the recognition object information can also be displayed in combination with the object information recognized from the entire AR screen image. For example, by performing object recognition on the AR screen image, it can be determined that the device type of the display device 200 contained in the current screen is a TV. Combined with the recognition object information of the above target image, a prompt such as "TV, the picture being demonstrated is an oil painting" can be finally presented to obtain more accurate recognition object information, such as Fig.14 shown.
[0107] According to the interactive enhancement method provided in the above embodiment, the AR device can obtain the identification object information through the target image when the AR screen includes the display content of the display device 200. Obviously, when the AR screen does not include the display content of the display device 200, the augmented reality device 500 cannot obtain the identification object information through the target image.
[0108] Therefore, if Fig.15 As shown, in some embodiments, after receiving the step of a control instruction for displaying an augmented reality screen, the augmented reality device 500 can obtain a key frame image corresponding to the augmented reality screen, and then detect the brightness change gradient of the key frame image to determine whether the display device 200 is included in the current AR screen based on the brightness change gradient.
[0109] If the brightness change gradient is less than the gradient threshold, it means that the current AR screen does not include the display device 200, or the display device 200 does not display an image, or the display brightness does not affect the accuracy of the identified object information. Therefore, the AI image classification algorithm can be executed on the key frame image to obtain the identified object information so that the identified object information can be displayed in the AR screen.
[0110] If the brightness change gradient is greater than or equal to the gradient threshold, it means that the current AR screen contains a display device 200, or the display brightness can affect the accuracy of the identified object information. Therefore, an image acquisition request can be sent to the display device through the communicator to obtain the target image in the augmented reality screen, and the identified object information can be obtained through the target image in the manner provided in the above embodiment to improve the accuracy of the identified object information.
[0111] Since when the target image contains electronic watermark information, the augmented reality device 500 can directly obtain the identification object information through the electronic watermark information, so in order to further improve the accuracy of the identification object information, the electronic watermark information can also be constructed according to a specific text format. That is, in some embodiments, the electronic watermark information includes an expiration date, a display position, and a label name. Among them, the expiration date is the time set by the operator according to the needs of AR display, which is used to enable the electronic watermark information of the target image to be read within the set time, and it will become invalid if it exceeds the set time, so as to ensure the real-time and effectiveness of the electronic watermark information. The display position is used to mark the reference point position of the identification object information when it is displayed, and can be set to a fixed point in the target image according to the actual application scenario of the target image. The label name is a text used to represent the actual identification object information, which can be a keyword, a descriptive sentence, and a paragraph.
[0112] In order to facilitate the mixing of electronic watermark information into the target image and reduce the perceptible interference of the electronic watermark information to the target image file after mixing, the text content contained in the electronic watermark information should be as small as possible. Therefore, the data in the electronic watermark information can be represented in a specific encoding method. For example, the electronic watermark information read is "J0120 06 04A", where "J0120" indicates that the validity period ends on January 20, 2023, "06" indicates the second area displayed in the field of view, and "04A" indicates the object name code, that is, the object name is "Oil Painting Banquet Room". Therefore, a database for parsing electronic watermark information can also be stored in the AR device. A variety of comparison tables can be cached in the database. After obtaining the electronic watermark information, the database can be called for matching to obtain the identification object information.
[0113] Obviously, when extracting frequency domain features from the frequency domain map corresponding to the target image, the frequency domain features can also be matched with the text content according to the agreed coding rules to determine the specific content of the electronic watermark information. Therefore, the corresponding relationship between the frequency domain features and the text content can be stored in the database. That is, after extracting the frequency domain features from the frequency domain map of the target image, the local database can be called and the electronic watermark information matching the frequency domain feature data can be queried in the local database.
[0114] Since the storage space of AR devices is limited, it is not conducive to maintaining a large-scale database. Therefore, AR devices can establish a communication connection with the cloud server through a communicator. Fig.16 As shown, if the electronic watermark information matching the frequency domain feature data is not found in the local data, a query request is generated and sent to the cloud server. After receiving the query request, the cloud server can query the database of the cloud server with the frequency domain feature data as an index according to the query request. The cloud server then feeds back the query result to the AR device so that the AR device can obtain the electronic watermark information.
[0115] In order to improve query efficiency, data can be classified in the database of the cloud server, and the classification standard can be based on the specific application field of the AR device. When the AR device enables any AR application, the application classification can be determined, so that the classification information can be attached when sending a query request to the cloud server, so that the cloud server can query according to the classification. It can be seen that matching electronic watermark information through the cloud server can not only facilitate the maintenance of large-scale databases and reduce the database's occupancy of AR device storage space, but also reduce the amount of data matching and improve the accuracy of matching results.
[0116] After obtaining the electronic watermark information through matching, the AR device can also read the validity period, display location, label name and other contents in the electronic watermark information, and generate identification object information based on the read content. Fig.17 As shown, in some embodiments, in the step of generating identification object information based on electronic watermark information, the AR device can first extract the validity period from the electronic watermark information, and obtain the current time, and determine whether the electronic watermark information of the current target image has expired by comparing the validity period and the current time.
[0117] If the current time is within the validity period, the current electronic watermark information is not invalid, so the display position and label name can be extracted from the electronic watermark information to generate the identification object information. If the current time is beyond the validity period, the current electronic watermark information is invalid, so it is necessary to perform an AI image classification algorithm on the target image to obtain the identification object information.
[0118] If the current time does not exceed the validity period, the AR device can further extract the display position and label name from the electronic watermark information, thereby controlling the display to display the identified object information according to the display position and label name. Fig.18 As shown, in some embodiments, the AR device can query the identification object information and the UI template applicable to the identification object information according to the tag name. Then, the UI template is applied to the identification object information to generate a to-be-displayed screen containing the identification object information, and finally, the to-be-displayed screen is added to the user interface according to the display position.
[0119] For example, the operator of the AR application and display device 200 can set the validity period of the target image to January 20, 2023 according to the operation and maintenance requirements, that is, before January 20, 2023, the identification object information can be obtained as "Oil Painting Banquet Room" through the electronic watermark information. When the AR device detects the frequency domain graph on January 20, 2021, since the current time has not exceeded the validity period, the obtained electronic watermark information is in a usable state, so the label name in the electronic watermark information can be further extracted, that is, by matching in the database, the label name corresponding to the text "Oil Painting Banquet Room" is obtained, and the display position in the electronic watermark information is read at the same time, that is, the display position is the area in the upper right corner of the oil painting. Therefore, the AR device can integrate the label name in the bubble dialog box according to the UI template, and display it in the upper right corner of the oil painting image in the user interface.
[0120] Since the AR device is a head-mounted device, the display content can be changed in real time according to the user's wearing posture. Therefore, in order to obtain a better indication effect, in some embodiments, after displaying the identification object information, the AR device can obtain the device appearance information of the display device 200. Obviously, different display devices 200 can have different device appearances. For example, a display device 200 such as a television has a device appearance of a rectangular flat structure with a width greater than the height; while a display device 200 such as a billboard has a device appearance of a rectangular flat structure with a width less than the height. After the AR device obtains the identification object information, the device type of the current display device 200 can be obtained through the connection status of the communicator, and then the device appearance information is obtained.
[0121] After obtaining the device appearance information, the AR device can also mark at least one device feature point in the AR screen according to the appearance shape of the device. The device feature point can be any point in the area where the display device 200 is located after edge extraction. In order to effectively mark the position of the display device 200, the device feature point can be a special point in the appearance shape of the display device 200, such as a vertex, a center point, etc. of a rectangular structure.
[0122] After marking at least one device feature point, the AR device also needs to display the identification object information based on the device feature point. For example, the AR device extends a guide line from a device feature point as the starting point to indicate the content displayed in the display device 200. If multiple device feature points are marked in the AR screen, the display reference point of the identification object information can also be switched as the AR perspective changes, so that the identification object information is always displayed in the appropriate position.
[0123] like Fig.19 As shown, in some embodiments, for a target image that does not contain electronic watermark information, after the AR device identifies the object-related information through an AI image classification algorithm, it can also generate electronic watermark information according to the identification result, and feed the electronic watermark information back to the display device 200 or to the cloud server, so that the subsequent AR screen display process, or other AR devices can directly obtain the identified object information through the generated electronic watermark information when displaying the target image.
[0124] To this end, the augmented reality device 500 can generate electronic watermark information based on the identification object information. Similarly, the electronic watermark information can be converted into a string form according to a specific encoding method. For example, through the AI picture classification algorithm, when the identification object information corresponding to the target image is obtained as "Oil Painting Banquet Room", the corresponding character of the label name can be obtained as "04A" according to the comparison table, and then according to the display position of the identification information in the current AR screen, the display position character can be generated. In addition, the validity period can be set according to the set validity time. If the validity period can be 1 year, the AR device can calculate the time one year later as the validity period based on the current time.
[0125] After the electronic watermark information is generated, the AR device can save the electronic watermark information in a locally stored database, or send the electronic watermark information and the target image to a cloud server for storage.
[0126] The AR device can also add electronic watermark information to the target image to generate an updated image. For example, after the target image is converted into a frequency domain image, frequency domain features can be added to the frequency domain graph according to the electronic watermark information to convert the target image into an updated image containing the electronic watermark information. Finally, the updated image is sent to the display device 200 so that the display device 200 displays and / or stores the updated image. It can be seen that when the display device 200 displays the updated image, the updated image can be used as the target image of the subsequent detection process in the subsequent AR scene. Since it contains electronic watermark information, the AR device can directly obtain the identification object information through the electronic watermark information, which greatly reduces the amount of data processing in the subsequent identification process and improves the operating efficiency of the AR device.
[0127] Based on the above-mentioned augmented reality device 500, some embodiments of the present application further provide an interactive enhancement method, including:
[0128] Receiving a control instruction input by a user for displaying an augmented reality picture, wherein the fused picture includes a real scene picture and a virtual object picture;
[0129] In response to the control instruction, acquiring a target image in the augmented reality picture, the target image being an image displayed on a screen of the display device;
[0130] If the target image contains electronic watermark information, generating identification object information according to the electronic watermark information;
[0131] The display is controlled to display the identification object information.
[0132] It can be seen from the above technical solution that the interactive enhancement method provided in the above embodiment can obtain the target image in the fused screen after the user inputs the control command to display the fused screen, that is, obtain the image currently displayed by the display device. Then, by performing electronic watermark detection on the target image, when the target image contains electronic watermark information, the identification object information is directly generated through the electronic watermark information to display the identification object information in the UI interface. The interactive enhancement method can obtain part of the identification object information through the electronic watermark information, so as to alleviate the influence of the screen brightness, image quality processing, etc. of the display device on the recognition process and improve the recognition accuracy.
[0133] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.
Claims
1. An augmented reality device, characterized in that: include: monitor; An image acquisition device is configured to acquire a real scene image; a communicator configured to connect to a display device; The controller is configured as: Receiving a control instruction input by a user for displaying an augmented reality screen; In response to the control instruction, acquiring a key frame image of the augmented reality picture; Detecting a brightness change gradient of the key frame image; In response to the brightness change gradient being greater than or equal to a gradient threshold, determining that the display device is included in the augmented reality picture; Acquire a target image displayed on the screen of the display device; Detecting electronic watermark information according to the acquired target image; If the target image contains the electronic watermark information, extracting a display position and a tag name in the electronic watermark information, wherein the display position is a fixed point of the target image; Generate identification object information and a UI template of the identification object information based on the tag name; The display is controlled to display the recognition object information applying the UI template at a fixed point of the target image based on the display position.
2. The augmented reality device according to claim 1, characterized in that After the step of acquiring the target image in the augmented reality screen, the controller is further configured to: If the target image does not contain the electronic watermark information, an AI picture classification algorithm is executed on the target image to obtain the identified object information.
3. The augmented reality device according to claim 2, characterized in that After obtaining the identification object information, the controller is further configured to: Generate electronic watermark information according to the identification object information; adding the electronic watermark information to the target image to generate an updated image; The updated image is sent to the display device, so that the display device displays and / or stores the updated image.
4. The augmented reality device according to claim 1, characterized in that After receiving the control instruction input by the user for displaying the augmented reality picture, the controller is further configured to: Obtain the key frame image corresponding to the fused image; Detecting a brightness change gradient of the key frame image; If the brightness change gradient is greater than or equal to the gradient threshold, sending an image acquisition request to the display device through the communicator to acquire the target image in the augmented reality screen; If the brightness change gradient is less than the gradient threshold, an AI picture classification algorithm is executed on the key frame image to obtain the identified object information.
5. The augmented reality device according to claim 1, characterized in that The electronic watermark information includes an expiration date, a display position, and a label name; in the step of generating identification object information according to the electronic watermark information, the controller is further configured to: Extracting the validity period from the electronic watermark information and obtaining the current time; comparing the expiration date with the current time; If the current time does not exceed the validity period, extracting the display position and the label name from the electronic watermark information to generate identification object information; If the current time exceeds the validity period, an AI picture classification algorithm is executed on the target image to obtain the identified object information.
6. The augmented reality device according to claim 5, characterized in that In the step of controlling the display to display the identified object information, the controller is further configured to: Query the identification object information and a UI template applicable to the identification object information according to the tag name; Applying the UI template to the identified object information to generate a to-be-displayed screen containing the identified object information; The picture to be displayed is added to the user interface according to the display position.
7. The augmented reality device according to claim 1, characterized in that: In the step of acquiring the target image in the augmented reality picture, the controller is further configured to: Converting the target image into a frequency domain image; Extracting frequency domain feature data from the frequency domain graph; Call the local database; The local database is searched for electronic watermark information matching the frequency domain feature data.
8. The augmented reality device according to claim 7, characterized in that The communicator is further configured to establish a communication connection with a cloud server. In the step of searching the local database for electronic watermark information matching the frequency domain feature data, the controller is further configured to: If no electronic watermark information matching the frequency domain feature data is found in the local data, generating a query request; Sending the query request to a cloud server; Receive the electronic watermark information fed back by the cloud server according to the query request.
9. The augmented reality device according to claim 1, characterized in that: In the step of controlling the display to display the identified object information, the controller is further configured to: Acquire device appearance information of the display device; Marking at least one device feature point in the augmented reality picture according to the device appearance information; The identification object information is displayed based on the device feature point.
10. An interaction enhancement method, characterized in that: Applied to an augmented reality device, the augmented reality device comprising a display, an image acquisition device, a communicator and a controller; The augmented reality device is connected to the display device via the communicator, and the interactive enhancement method includes: Receiving a control instruction input by a user for displaying an augmented reality screen; In response to the control instruction, acquiring a key frame image of the augmented reality picture; Detecting a brightness change gradient of the key frame image; In response to the brightness change gradient being greater than or equal to a gradient threshold, determining that the display device is included in the augmented reality picture; Acquire a target image displayed on the screen of the display device; Detecting electronic watermark information according to the acquired target image; If the target image contains the electronic watermark information, extracting a display position and a tag name in the electronic watermark information, wherein the display position is a fixed point in the target image; Generate identification object information and a UI template of the identification object information based on the tag name; and control the display to display the identification object information applying the UI template at a fixed point of the target image based on the display position.
