Display material automatic recognition method and system

Abstract

This invention provides an automatic display material identification method and system. Based on the real environment and historical display records of a head-mounted display device, it generates a set of real target display elements and a set of virtual target display elements, providing display material sources at both the real and virtual environment levels for subsequent adjustments to the displayed image content. It identifies permissible display change areas within the current display screen of the head-mounted display device, accurately locating subsequent content adjustments to ensure that the adjusted content does not affect the original display state and prevents interference. Furthermore, based on display control commands, it determines the set of display elements to be filtered, and identifies display elements that meet preset shape and contour conditions, ensuring that the content of the filtered display elements matches actual needs. Finally, after adjusting the visual characteristics of the filtered display elements, it integrates them into the screen area to improve the visual effect.

Description

Technical Field

[0001] This invention relates to the field of display control, and more particularly to a method and system for automatic identification of display materials. Background Technology

[0002] Head-mounted displays (HUDs) provide wearers with a VR experience. Existing HUDs receive video stream signals and then perform 3D conversion to present corresponding 3D dynamic images to the wearer. The content of the 3D dynamic images viewed by the wearer is determined by the video stream signal. Although video stream signals can be processed frame by frame to change the image state, these processing methods are limited to altering display parameters such as resolution, color tone, and contrast. They cannot substantially change the content displayed. HUDs cannot accurately target the display based on the wearer's real-world environment or the device's historical display information, reducing the variability and matching of the display to actual viewing needs. This results in a lack of suitable image content display and reduced flexibility and reliability in display changes. Summary of the Invention

[0003] The purpose of this invention is to provide an automatic display material identification method and system. Based on the real environment and historical display records of a head-mounted display device, it generates a set of real target display elements and a set of virtual target display elements, providing display material sources at both the real and virtual environment levels for subsequent adjustments to the displayed image content. It identifies permissible display change areas within the current display screen of the head-mounted display device, accurately locating subsequent content adjustments to ensure that the adjusted content does not affect the original display state and prevents interference. Furthermore, based on display control commands, it determines the set of display elements to be screened, and identifies display elements that meet preset shape and contour conditions, ensuring that the content of the screened elements matches actual needs. Finally, after adjusting the visual characteristics of the screened display elements, it integrates them into the screen area, ensuring good compatibility between the display elements and the original image, and improving the visual effect of the adjusted display.

[0004] This invention is achieved through the following technical solution:

[0005] Display material automatic identification methods include:

[0006] The system acquires real-world images corresponding to the head-mounted display device, identifies the real-world images, generates several real-world target display elements corresponding to the real-world environment, and forms a set of real-world target display elements.

[0007] Based on the historical display records of the head-mounted display device, several virtual target display elements are extracted from the virtual display platform and combined to form a set of virtual target display elements;

[0008] Based on the virtual image currently displayed by the head-mounted display device, determine the screen area that is allowed to be changed in the current display screen of the head-mounted display device;

[0009] Based on the display control command received by the head-mounted display device, a display change mode for the head-mounted display device is determined, thereby filtering display elements from the set of real target display elements or the set of virtual target display elements;

[0010] Perform element shape contour recognition on the set of real target display elements or the set of virtual target display elements, and filter out display elements that meet the preset shape contour conditions;

[0011] After adjusting the visual characteristics of the selected display elements, the display elements are then integrated into the screen area.

[0012] Optionally, a real-world image corresponding to the head-mounted display device is acquired, the real-world image is identified, and several real-world target display elements corresponding to the real-world environment are generated and combined into a set of real-world target display elements, including:

[0013] Binocular images of the real environment in which the head-mounted display device is located are captured by binoculars to obtain binocular real environment images;

[0014] Based on the binocular real environment image, generate a two-dimensional real environment image and a three-dimensional real environment image;

[0015] The two-dimensional real environment image and the three-dimensional real environment image are processed by real object contour recognition to obtain two-dimensional real target display elements and three-dimensional real target display elements corresponding to all real objects in the real environment, thereby forming a set of real target display elements.

[0016] Based on the historical display records of the head-mounted display device, several virtual target display elements are extracted from the virtual display platform and assembled into a set of virtual target display elements, including:

[0017] Based on the historical display records of the head-mounted display device, determine the address of the virtual display platform that the head-mounted display device was historically connected to and the type of the historically displayed image;

[0018] Based on the virtual display platform address and the historical display image type, several virtual target display elements are extracted from the corresponding virtual display platform, and all virtual target display elements are deduplicated to form a set of virtual target display elements.

[0019] Optionally, based on the virtual image currently displayed on the head-mounted display device, a screen area within the current display screen of the head-mounted display device that is allowed to be displayed differently is determined, including:

[0020] The virtual image currently displayed by the head-mounted display device is subjected to pixel contour recognition processing to obtain the blank area distribution status information of the virtual image screen; the blank area distribution status information includes the area of ​​each blank area in the virtual image screen and its position in the screen;

[0021] Based on the blank area distribution status information, the corresponding blank areas are designated as screen areas where display changes are permitted;

[0022] Based on the display control command received by the head-mounted display device, a display change mode for the head-mounted display device is determined, thereby filtering display elements from the set of real target display elements or the set of virtual target display elements, including:

[0023] The display control command received by the head-mounted display device is parsed to determine whether the head-mounted display device needs to switch to a virtual display mode or a real-world hybrid display mode. When it needs to switch to a virtual display mode, display elements are selected from the set of virtual target display elements. When it needs to switch to a real-world hybrid display mode, display elements are selected from the set of real-world target display elements.

[0024] Optionally, element shape contour recognition is performed on the set of real target display elements or the set of virtual target display elements to filter out display elements that meet preset shape contour conditions, including:

[0025] Perform element shape contour recognition on the real target display element set or the virtual target display element set to obtain the shape contour features of each display element contained in the real target display element set or the virtual target display element set;

[0026] Based on the shape contour features, select display elements that have the highest similarity to the preset target shape contour features;

[0027] After adjusting the visual characteristics of the selected display elements, the display elements are then integrated into the screen area, including:

[0028] Based on the color tone characteristics of the background portion of the screen area, the selected display elements are rendered with color tone; then, based on the area boundary of the screen area, the display elements are inserted into the corresponding positions of the screen area.

[0029] Display material automatic recognition system, including:

[0030] The real target display element set forming module acquires real environment images corresponding to the head-mounted display device, identifies the real environment images, generates several real target display elements corresponding to the real environment, and forms a real target display element set.

[0031] The virtual target display element set forming module extracts several virtual target display elements from the virtual display platform based on the historical display records of the head-mounted display device, and forms a virtual target display element set.

[0032] The image recognition module determines, based on the virtual image currently displayed on the head-mounted display device, the area of ​​the image that can be changed in the current display screen of the head-mounted display device;

[0033] The element set calibration module determines the display change mode of the head-mounted display device based on the display control command received by the head-mounted display device, thereby filtering display elements from the real target display element set or the virtual target display element set;

[0034] The display element filtering module performs element shape contour recognition on the real target display element set or the virtual target display element set, and filters out display elements that meet the preset shape contour conditions.

[0035] The display element visual adjustment and fusion module adjusts the visual characteristics of the selected display elements and then fuses the display elements into the screen area.

[0036] Optionally, the real target display element set forming module acquires real environment images corresponding to the head-mounted display device, identifies the real environment images, generates several real target display elements corresponding to the real environment, and forms a real target display element set, including:

[0037] Binocular images of the real environment in which the head-mounted display device is located are captured by binoculars to obtain binocular real environment images;

[0038] Based on the binocular real environment image, generate a two-dimensional real environment image and a three-dimensional real environment image;

[0039] The two-dimensional real environment image and the three-dimensional real environment image are processed by real object contour recognition to obtain two-dimensional real target display elements and three-dimensional real target display elements corresponding to all real objects in the real environment, thereby forming a set of real target display elements.

[0040] The virtual target display element set forming module extracts several virtual target display elements from the virtual display platform based on the historical display records of the head-mounted display device, and forms a virtual target display element set, including:

[0041] Based on the historical display records of the head-mounted display device, determine the address of the virtual display platform that the head-mounted display device was historically connected to and the type of the historically displayed image;

[0042] Based on the virtual display platform address and the historical display image type, several virtual target display elements are extracted from the corresponding virtual display platform, and all virtual target display elements are deduplicated to form a set of virtual target display elements.

[0043] Optionally, the image recognition module determines, based on the virtual image currently displayed on the head-mounted display device, a screen area within the current display screen of the head-mounted display device that is allowed to be displayed differently, including:

[0044] The virtual image currently displayed by the head-mounted display device is subjected to pixel contour recognition processing to obtain the blank area distribution status information of the virtual image screen; the blank area distribution status information includes the area of ​​each blank area in the virtual image screen and its position in the screen;

[0045] Based on the blank area distribution status information, the corresponding blank areas are designated as screen areas where display changes are permitted;

[0046] The element set calibration module determines the display change mode of the head-mounted display device based on the display control command received by the head-mounted display device, thereby filtering display elements from the real target display element set or the virtual target display element set, including:

[0047] The display control command received by the head-mounted display device is parsed to determine whether the head-mounted display device needs to switch to a virtual display mode or a real-world hybrid display mode. When it needs to switch to a virtual display mode, display elements are selected from the set of virtual target display elements. When it needs to switch to a real-world hybrid display mode, display elements are selected from the set of real-world target display elements.

[0048] Optionally, the display element filtering module performs element shape contour recognition on the real target display element set or the virtual target display element set to filter display elements that meet preset shape contour conditions, including:

[0049] Perform element shape contour recognition on the real target display element set or the virtual target display element set to obtain the shape contour features of each display element contained in the real target display element set or the virtual target display element set;

[0050] Based on the shape contour features, select display elements that have the highest similarity to the preset target shape contour features;

[0051] The display element visual adjustment and blending module adjusts the visual characteristics of the selected display elements and then blends the display elements into the screen area, including:

[0052] Based on the color tone characteristics of the background portion of the screen area, the selected display elements are rendered with color tone; then, based on the area boundary of the screen area, the display elements are inserted into the corresponding positions of the screen area.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The automatic display material identification method and system provided in this application generate a set of real target display elements and a set of virtual target display elements based on the real environment and historical display records of the head-mounted display device, respectively, providing display material sources at the real and virtual environment levels for subsequent adjustments to the display image content; it determines the screen area that is allowed to be changed in the current display screen of the head-mounted display device, accurately locates the subsequent screen content adjustments, and ensures that the adjusted screen content will not affect the display state of the original screen content, preventing interference with the original screen content; it also determines the set of display elements to be screened according to display control instructions, and screens that meet preset shape and contour conditions, ensuring that the content of the screened display elements matches the actual needs; and after adjusting the visual characteristics of the screened display elements, it merges the display elements into the screen area, ensuring that the display elements can achieve good compatibility with the original screen and improving the visual effect of the adjusted display screen. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0056] Figure 1 This is a flowchart illustrating the automatic identification method and system for display materials provided by the present invention.

[0057] Figure 2 This is a schematic diagram of the structure of the automatic identification method and system for display materials provided by the present invention. Detailed Implementation

[0058] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, it should be noted that, for ease of description, only the parts relevant to this application are shown in the accompanying drawings, not the entire structure. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0059] The terms “comprising” and “having”, and any variations thereof, used in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0060] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0061] Please see Figure 1 As shown, an embodiment of this application provides an automatic identification method for display materials, including:

[0062] The system acquires real-world images corresponding to the head-mounted display device, identifies these images, generates several real-world target display elements corresponding to the real-world environment, and assembles them into a set of real-world target display elements.

[0063] Based on the historical display records of the head-mounted display device, several virtual target display elements are extracted from the virtual display platform and combined into a set of virtual target display elements;

[0064] Based on the virtual image currently displayed on the head-mounted display device, determine the screen area that is allowed to be changed within the current display screen of the head-mounted display device;

[0065] Based on the display control commands received by the head-mounted display device, the display change mode of the head-mounted display device is determined, thereby filtering display elements from either the set of real target display elements or the set of virtual target display elements;

[0066] Perform element shape contour recognition on the set of real target display elements or the set of virtual target display elements, and filter out display elements that meet the preset shape contour conditions;

[0067] After adjusting the visual characteristics of the selected display elements, the display elements are then integrated into the screen area.

[0068] The beneficial effects of the above embodiments are as follows: the automatic display material identification method generates a set of real target display elements and a set of virtual target display elements based on the real environment and historical display records of the head-mounted display device, providing display material sources at both the real and virtual environment levels for subsequent adjustments to the display image content; it determines the screen area where display changes are allowed in the current display screen of the head-mounted display device, accurately locating subsequent adjustments to the screen content to ensure that the adjusted screen content does not affect the display state of the original screen content and prevents interference with the original screen content; it also determines the set of display elements to be screened according to display control instructions, and screens that meet preset shape and contour conditions, ensuring that the content of the screened display elements matches the actual needs; and after adjusting the visual characteristics of the screened display elements, it merges the display elements into the screen area, ensuring that the display elements can achieve good compatibility with the original screen and improving the visual effect of the adjusted display image.

[0069] In another embodiment, a real-world image corresponding to the head-mounted display device is acquired, the real-world image is identified, and several real-world target display elements corresponding to the real-world environment are generated and assembled into a set of real-world target display elements, including:

[0070] Binocular images of the real environment in which the head-mounted display device is located are captured by binoculars to obtain binocular real environment images;

[0071] Generate two-dimensional and three-dimensional realistic environment images based on binocular realistic environment images;

[0072] Real object contour recognition processing is performed on two-dimensional and three-dimensional real environment images to obtain two-dimensional and three-dimensional real target display elements corresponding to all real objects in the real environment, and thus form a set of real target display elements.

[0073] Based on the historical display records of the head-mounted display device, several virtual target display elements are extracted from the virtual display platform and assembled into a set of virtual target display elements, including:

[0074] Based on the historical display records of the head-mounted display device, determine the address of the virtual display platform that the head-mounted display device was connected to in the past and the types of images that were displayed in the past;

[0075] Based on the virtual display platform address and the historical image types, several virtual target display elements are extracted from the corresponding virtual display platform, and all virtual target display elements are deduplicated to form a set of virtual target display elements.

[0076] The beneficial effects of the above embodiments are that the head-mounted display device can perform two different display modes: virtual display and reality hybrid display. In the virtual display mode, the head-mounted display device provides dynamic virtual images to the wearer based on the received video stream signal. In the reality hybrid display mode, the head-mounted display device superimposes the dynamic virtual images corresponding to the received video stream signal onto real target images of the external environment, thereby providing the wearer with a display experience that simultaneously overlays and blends dynamic virtual images and real environment images. The head-mounted display device uses its built-in camera to capture binocular images of the external environment, obtaining binocular real images. These binocular real images are then converted into two-dimensional and three-dimensional images to obtain corresponding two-dimensional and three-dimensional real environment images, adapting to the superposition of real target images matched by the head-mounted display device in either two-dimensional or three-dimensional display modes. Specifically, the two-dimensional and three-dimensional real environment images undergo real object contour recognition processing to obtain the outlines of real people and / or real objects in the two-dimensional and three-dimensional real environment images. Based on these outlines, the real people and / or real objects are then graphically converted to obtain corresponding two-dimensional and three-dimensional real target display elements. Furthermore, the video stream signal received by the head-mounted display device comes from the corresponding virtual display platform. Based on the historical display records of the head-mounted display device, the address of the virtual display platform connected to the head-mounted display device during the historical display process and the types of historical displayed images (such as image content and image themes) are determined as the basis. Images of virtual targets such as different virtual people and / or virtual objects are extracted from the corresponding virtual display platform and used as virtual target display elements. Contour similarity recognition is performed on all extracted virtual target display elements. All virtual target display elements with contour similarity greater than a preset similarity threshold are merged and deduplicated to avoid the existence of similar or identical virtual target display elements in the set of virtual target display elements.

[0077] In another embodiment, determining a screen area within the current display screen of the head-mounted display device that is allowed to be changed, based on the virtual image currently displayed on the head-mounted display device, includes:

[0078] Pixel contour recognition processing is performed on the virtual image currently displayed by the head-mounted display device to obtain the blank area distribution status information of the virtual image screen; the blank area distribution status information includes the area of ​​each blank area in the virtual image screen and its position in the screen;

[0079] Based on the distribution status information of blank areas, the corresponding blank areas are designated as the screen areas where display changes are permitted.

[0080] Based on the display control commands received from the head-mounted display device, the display change mode of the head-mounted display device is determined, thereby filtering display elements from either a set of real target display elements or a set of virtual target display elements, including:

[0081] The system parses the display control commands received by the head-mounted display device to determine whether the head-mounted display device needs to switch to a virtual display mode or a real-world hybrid display mode. When it needs to switch to a virtual display mode, it filters display elements from the set of virtual target display elements. When it needs to switch to a real-world hybrid display mode, it filters display elements from the set of real-world target display elements.

[0082] The beneficial effects of the above embodiments are that pixel contour recognition processing is performed on the virtual image currently displayed by the head-mounted display device to obtain the location and area range of all objects in the currently displayed virtual image, thereby obtaining the distribution status information of blank areas in the image. Each blank area refers to an area in the image that is not occupied by objects and whose area is greater than or equal to a first preset area threshold. This makes it convenient to use the blank areas as areas for subsequent display insertion and fusion of the virtual image currently displayed by the head-mounted display device (i.e., as image areas that allow display changes). Specifically, based on the blank area distribution status information, including the area of ​​each blank area in the virtual image and its position in the image, the actual area of ​​each blank area and its position in the image are determined. If the actual area is greater than or equal to a second preset area threshold (the second preset area threshold is greater than the first preset area threshold) and there are no objects within a predetermined radius of the position in the image, then the corresponding blank area is used as an image area that allows display changes. Furthermore, the display control commands received by the head-mounted display device are parsed to determine whether the head-mounted display device needs to switch to a virtual display mode (i.e., a pure virtual image display mode) or a reality hybrid display mode (i.e., a display mode that overlays virtual and real images). When switching to a virtual display mode, display elements are selected from the set of virtual target display elements; when switching to a reality hybrid display mode, display elements are selected from the set of real target display elements. This ensures the richness of the displayed images in both the virtual and reality hybrid display modes.

[0083] In another embodiment, element shape contour recognition is performed on the set of real target display elements or the set of virtual target display elements to filter out display elements that meet preset shape contour conditions, including:

[0084] Perform element shape contour recognition on the set of real target display elements or the set of virtual target display elements to obtain the shape contour features of each display element contained in the set of real target display elements or the set of virtual target display elements;

[0085] Based on shape and outline features, select display elements that have the highest similarity to the preset target shape and outline features;

[0086] After adjusting the visual characteristics of the selected display elements, the display elements are then integrated into the screen area, including:

[0087] Based on the color tone characteristics of the background area of ​​the screen region, the selected display elements are rendered with color tone; then, based on the area boundary of the screen region, the display elements are inserted into the corresponding positions of the screen region.

[0088] The beneficial effects of the above embodiments are as follows: Shape and contour recognition is performed on either the set of real or virtual target display elements, enabling the calibration of the shape and contour features of each display element contained in each set. The shape and contour features of each display element are then compared with preset target shape and contour features to obtain the display element with the highest similarity, which is then used for subsequent fusion and insertion into the screen area of ​​the virtual image currently displayed on the head-mounted display device. Furthermore, based on the color tone features of the background portion of the screen area, the selected display elements are color-rendered to ensure that the color tone of the rendered display element is not the same as or similar to the color tone of the background portion, preventing the display element from being submerged by the background portion and thus unable to be clearly identified after insertion into the screen area. Finally, based on the area boundaries of the screen area, the display element is inserted into the corresponding position within the screen area, ensuring that the display element is inserted into the centered position of the screen area.

[0089] Please see Figure 2 As shown, an embodiment of this application provides an automatic display material recognition system, including:

[0090] The module for forming a set of real target display elements acquires real environment images corresponding to the head-mounted display device, identifies the real environment images, generates several real target display elements corresponding to the real environment, and forms a set of real target display elements.

[0091] The virtual target display element set forming module extracts several virtual target display elements from the virtual display platform based on the historical display records of the head-mounted display device, and forms a virtual target display element set.

[0092] The image recognition module determines the area of ​​the image that can be changed within the current display of the head-mounted display device, based on the virtual image currently displayed on the head-mounted display device.

[0093] The element set calibration module determines the display change mode of the head-mounted display device based on the display control instructions received by the head-mounted display device, thereby filtering display elements from the real target display element set or the virtual target display element set;

[0094] The display element filtering module performs element shape and contour recognition on the set of real target display elements or the set of virtual target display elements, and filters out display elements that meet the preset shape and contour conditions.

[0095] The Display Element Visual Adjustment and Blending module adjusts the visual characteristics of the selected display elements and then blends them into the screen area.

[0096] The beneficial effects of the above embodiments are as follows: the automatic display material identification system generates a set of real target display elements and a set of virtual target display elements based on the real environment and historical display records of the head-mounted display device, providing display material sources at both the real and virtual environment levels for subsequent adjustments to the display image content; it determines the screen area in the current display screen of the head-mounted display device that allows display changes, accurately positioning subsequent screen content adjustments to ensure that the adjusted screen content does not affect the display state of the original screen content and prevents interference with the original screen content; it also determines the set of display elements to be screened according to display control commands, and screens that meet preset shape and contour conditions, ensuring that the content of the screened display elements matches the actual needs; and after adjusting the visual characteristics of the screened display elements, it merges the display elements into the screen area, ensuring that the display elements can achieve good compatibility with the original screen and improving the visual effect of the adjusted display image.

[0097] In another embodiment, the real target display element set forming module acquires real environment images corresponding to the head-mounted display device, identifies the real environment images, generates several real target display elements corresponding to the real environment, and forms a real target display element set, including:

[0098] Binocular images of the real environment in which the head-mounted display device is located are captured by binoculars to obtain binocular real environment images;

[0099] Generate two-dimensional and three-dimensional realistic environment images based on binocular realistic environment images;

[0100] Real object contour recognition processing is performed on two-dimensional and three-dimensional real environment images to obtain two-dimensional and three-dimensional real target display elements corresponding to all real objects in the real environment, and thus form a set of real target display elements.

[0101] The virtual target display element set forming module extracts several virtual target display elements from the virtual display platform based on the historical display records of the head-mounted display device, and assembles them into a virtual target display element set, including:

[0102] Based on the historical display records of the head-mounted display device, determine the address of the virtual display platform that the head-mounted display device was connected to in the past and the types of images that were displayed in the past;

[0103] Based on the virtual display platform address and the historical image types, several virtual target display elements are extracted from the corresponding virtual display platform, and all virtual target display elements are deduplicated to form a set of virtual target display elements.

[0104] The beneficial effects of the above embodiments are that the head-mounted display device can perform two different display modes: virtual display and reality hybrid display. In the virtual display mode, the head-mounted display device provides dynamic virtual images to the wearer based on the received video stream signal. In the reality hybrid display mode, the head-mounted display device superimposes the dynamic virtual images corresponding to the received video stream signal onto real target images of the external environment, thereby providing the wearer with a display experience that simultaneously overlays and blends dynamic virtual images and real environment images. The head-mounted display device uses its built-in camera to capture binocular images of the external environment, obtaining binocular real images. These binocular real images are then converted into two-dimensional and three-dimensional images to obtain corresponding two-dimensional and three-dimensional real environment images, adapting to the superposition of real target images matched by the head-mounted display device in either two-dimensional or three-dimensional display modes. Specifically, the two-dimensional and three-dimensional real environment images undergo real object contour recognition processing to obtain the outlines of real people and / or real objects in the two-dimensional and three-dimensional real environment images. Based on these outlines, the real people and / or real objects are then graphically converted to obtain corresponding two-dimensional and three-dimensional real target display elements. Furthermore, the video stream signal received by the head-mounted display device comes from the corresponding virtual display platform. Based on the historical display records of the head-mounted display device, the address of the virtual display platform connected to the head-mounted display device during the historical display process and the types of historical displayed images (such as image content and image themes) are determined as the basis. Images of virtual targets such as different virtual people and / or virtual objects are extracted from the corresponding virtual display platform and used as virtual target display elements. Contour similarity recognition is performed on all extracted virtual target display elements. All virtual target display elements with contour similarity greater than a preset similarity threshold are merged and deduplicated to avoid the existence of similar or identical virtual target display elements in the set of virtual target display elements.

[0105] In another embodiment, the image recognition module determines, based on the virtual image currently displayed on the head-mounted display device, a screen area within the current display screen of the head-mounted display device that is allowed to be changed, including:

[0106] Pixel contour recognition processing is performed on the virtual image currently displayed by the head-mounted display device to obtain the blank area distribution status information of the virtual image screen; the blank area distribution status information includes the area of ​​each blank area in the virtual image screen and its position in the screen;

[0107] Based on the distribution status information of blank areas, the corresponding blank areas are designated as the screen areas where display changes are permitted.

[0108] The element set calibration module determines the display change mode for the head-mounted display device based on the display control commands received from the head-mounted display device. This allows for the selection of display elements from either the real target display element set or the virtual target display element set, including:

[0109] The system parses the display control commands received by the head-mounted display device to determine whether the head-mounted display device needs to switch to a virtual display mode or a real-world hybrid display mode. When it needs to switch to a virtual display mode, it filters display elements from the set of virtual target display elements. When it needs to switch to a real-world hybrid display mode, it filters display elements from the set of real-world target display elements.

[0110] The beneficial effects of the above embodiments are that pixel contour recognition processing is performed on the virtual image currently displayed by the head-mounted display device to obtain the location and area range of all objects in the currently displayed virtual image, thereby obtaining the distribution status information of blank areas in the image. Each blank area refers to an area in the image that is not occupied by objects and whose area is greater than or equal to a first preset area threshold. This makes it convenient to use the blank areas as areas for subsequent display insertion and fusion of the virtual image currently displayed by the head-mounted display device (i.e., as image areas that allow display changes). Specifically, based on the blank area distribution status information, including the area of ​​each blank area in the virtual image and its position in the image, the actual area of ​​each blank area and its position in the image are determined. If the actual area is greater than or equal to a second preset area threshold (the second preset area threshold is greater than the first preset area threshold) and there are no objects within a predetermined radius of the position in the image, then the corresponding blank area is used as an image area that allows display changes. Furthermore, the display control commands received by the head-mounted display device are parsed to determine whether the head-mounted display device needs to switch to a virtual display mode (i.e., a pure virtual image display mode) or a reality hybrid display mode (i.e., a display mode that overlays virtual and real images). When switching to a virtual display mode, display elements are selected from the set of virtual target display elements; when switching to a reality hybrid display mode, display elements are selected from the set of real target display elements. This ensures the richness of the displayed images in both the virtual and reality hybrid display modes.

[0111] In another embodiment, the display element filtering module performs element shape contour recognition on the set of real target display elements or the set of virtual target display elements, and filters out display elements that meet preset shape contour conditions, including:

[0112] Perform element shape contour recognition on the set of real target display elements or the set of virtual target display elements to obtain the shape contour features of each display element contained in the set of real target display elements or the set of virtual target display elements;

[0113] Based on shape and outline features, select display elements that have the highest similarity to the preset target shape and outline features;

[0114] The display element visual adjustment and blending module adjusts the visual characteristics of the selected display elements and then blends them into the screen area, including:

[0115] Based on the color tone characteristics of the background area of ​​the screen region, the selected display elements are rendered with color tone; then, based on the area boundary of the screen region, the display elements are inserted into the corresponding positions of the screen region.

[0116] The beneficial effects of the above embodiments are as follows: Shape and contour recognition is performed on either the set of real or virtual target display elements, enabling the calibration of the shape and contour features of each display element contained in each set. The shape and contour features of each display element are then compared with preset target shape and contour features to obtain the display element with the highest similarity, which is then used for subsequent fusion and insertion into the screen area of ​​the virtual image currently displayed on the head-mounted display device. Furthermore, based on the color tone features of the background portion of the screen area, the selected display elements are color-rendered to ensure that the color tone of the rendered display element is not the same as or similar to the color tone of the background portion, preventing the display element from being submerged by the background portion and thus unable to be clearly identified after insertion into the screen area. Finally, based on the area boundaries of the screen area, the display element is inserted into the corresponding position within the screen area, ensuring that the display element is inserted into the centered position of the screen area.

[0117] In summary, the automatic display material identification method and system generate both real and virtual target display element sets based on the real environment and historical display records of the head-mounted display device. This provides display material sources at both the real and virtual environment levels for subsequent adjustments to the displayed image content. It identifies permissible display change areas within the current display screen of the head-mounted display device, accurately locating subsequent content adjustments to ensure that the adjusted content does not affect the original display state and prevents interference. Furthermore, based on display control commands, it determines the set of display elements to be filtered, and identifies those that meet preset shape and contour conditions, ensuring that the content of the filtered elements matches actual needs. After adjusting the visual characteristics of the filtered elements, it integrates them into the screen area, ensuring good compatibility between the elements and the original image and improving the visual effect of the adjusted display.

[0118] The above is only one specific embodiment of the present invention, and any improvements made based on the concept of the present invention shall be considered within the scope of protection of the present invention.

Claims

1. A method for automatic identification of display materials, characterized in that, include: The system acquires real-world images corresponding to the head-mounted display device, identifies the real-world images, generates several real-world target display elements corresponding to the real-world environment, and forms a set of real-world target display elements. Based on the historical display records of the head-mounted display device, several virtual target display elements are extracted from the virtual display platform and combined to form a set of virtual target display elements; Based on the virtual image currently displayed by the head-mounted display device, determine the screen area that is allowed to be changed in the current display screen of the head-mounted display device; Based on the display control command received by the head-mounted display device, a display change mode for the head-mounted display device is determined, thereby filtering display elements from the set of real target display elements or the set of virtual target display elements; Perform element shape contour recognition on the set of real target display elements or the set of virtual target display elements, and filter out display elements that meet the preset shape contour conditions; After adjusting the visual characteristics of the selected display elements, the display elements are then integrated into the screen area.

2. The automatic identification method for display materials as described in claim 1, characterized in that: Acquire real-world images corresponding to the head-mounted display device, identify the real-world images, generate several real-world target display elements corresponding to the real-world environment, and form a set of real-world target display elements, including: The head-mounted display device is subjected to binocular imaging of the real environment to obtain binocular real environment images; based on the binocular real environment images, two-dimensional real environment images and three-dimensional real environment images are generated. The two-dimensional and three-dimensional real-environment images are processed for real object contour recognition to obtain two-dimensional and three-dimensional real target display elements corresponding to all real objects in the real environment, thus forming a set of real target display elements. Based on the historical display records of the head-mounted display device, several virtual target display elements are extracted from the virtual display platform and combined to form a set of virtual target display elements, including: Based on the historical display records of the head-mounted display device, determine the address of the virtual display platform that the head-mounted display device was historically connected to and the type of the historically displayed image; Based on the virtual display platform address and the historical display image type, several virtual target display elements are extracted from the corresponding virtual display platform, and all virtual target display elements are deduplicated to form a set of virtual target display elements.

3. The automatic identification method for display materials as described in claim 1, characterized in that: Based on the virtual image currently displayed on the head-mounted display device, determine the screen area within the current display screen of the head-mounted display device that is allowed to be changed, including: The virtual image currently displayed by the head-mounted display device is subjected to pixel contour recognition processing to obtain the blank area distribution status information of the virtual image screen; the blank area distribution status information includes the area of ​​each blank area in the virtual image screen and its position in the screen; Based on the blank area distribution status information, the corresponding blank areas are designated as screen areas where display changes are permitted; Based on the display control command received by the head-mounted display device, a display change mode for the head-mounted display device is determined, thereby filtering display elements from the set of real target display elements or the set of virtual target display elements, including: The display control command received by the head-mounted display device is parsed to determine whether the head-mounted display device needs to switch to a virtual display mode or a real-world hybrid display mode. When it needs to switch to a virtual display mode, display elements are selected from the set of virtual target display elements. When it needs to switch to a real-world hybrid display mode, display elements are selected from the set of real-world target display elements.

4. The automatic identification method for display materials as described in claim 1, characterized in that: Perform element shape contour recognition on the set of real target display elements or the set of virtual target display elements, and filter out display elements that meet preset shape contour conditions, including: Perform element shape contour recognition on the real target display element set or the virtual target display element set to obtain the shape contour features of each display element contained in the real target display element set or the virtual target display element set; Based on the shape contour features, select display elements that have the highest similarity to the preset target shape contour features; After adjusting the visual characteristics of the selected display elements, the display elements are then integrated into the screen area, including: Based on the color tone characteristics of the background portion of the screen area, the selected display elements are rendered with color tone; then, based on the area boundary of the screen area, the display elements are inserted into the corresponding positions of the screen area.

5. An automatic material recognition system, characterized in that, include: The real target display element set forming module acquires real environment images corresponding to the head-mounted display device, identifies the real environment images, generates several real target display elements corresponding to the real environment, and forms a real target display element set. The virtual target display element set forming module extracts several virtual target display elements from the virtual display platform based on the historical display records of the head-mounted display device, and forms a virtual target display element set. The image recognition module determines, based on the virtual image currently displayed on the head-mounted display device, the area of ​​the image that can be changed in the current display screen of the head-mounted display device; The element set calibration module determines the display change mode of the head-mounted display device based on the display control command received by the head-mounted display device, thereby filtering display elements from the real target display element set or the virtual target display element set; The display element filtering module performs element shape contour recognition on the real target display element set or the virtual target display element set, and filters out display elements that meet the preset shape contour conditions. The display element visual adjustment and fusion module adjusts the visual characteristics of the selected display elements and then fuses the display elements into the screen area.

6. The automatic display material identification system as described in claim 5, characterized in that: The real target display element set formation module acquires real environment images corresponding to the head-mounted display device, identifies the real environment images, generates several real target display elements corresponding to the real environment, and forms a real target display element set, including: The head-mounted display device is subjected to binocular imaging of the real environment to obtain binocular real environment images; based on the binocular real environment images, two-dimensional real environment images and three-dimensional real environment images are generated. The two-dimensional real environment image and the three-dimensional real environment image are processed by real object contour recognition to obtain two-dimensional real target display elements and three-dimensional real target display elements corresponding to all real objects in the real environment, thereby forming a set of real target display elements. The virtual target display element set forming module extracts several virtual target display elements from the virtual display platform based on the historical display records of the head-mounted display device, and forms a virtual target display element set, including: Based on the historical display records of the head-mounted display device, determine the address of the virtual display platform that the head-mounted display device was historically connected to and the type of the historically displayed image; Based on the virtual display platform address and the historical display image type, several virtual target display elements are extracted from the corresponding virtual display platform, and all virtual target display elements are deduplicated to form a set of virtual target display elements.

7. The automatic display material recognition system as described in claim 5, characterized in that: The image recognition module determines, based on the virtual image currently displayed on the head-mounted display device, the permitted display area for changes within the current display screen of the head-mounted display device, including: The virtual image currently displayed by the head-mounted display device is subjected to pixel contour recognition processing to obtain the blank area distribution status information of the virtual image screen; the blank area distribution status information includes the area of ​​each blank area in the virtual image screen and its position in the screen; Based on the blank area distribution status information, the corresponding blank areas are designated as screen areas where display changes are permitted; The element set calibration module determines the display change mode of the head-mounted display device based on the display control command received by the head-mounted display device, thereby filtering display elements from the real target display element set or the virtual target display element set, including: The display control command received by the head-mounted display device is parsed to determine whether the head-mounted display device needs to switch to a virtual display mode or a real-world hybrid display mode. When it needs to switch to a virtual display mode, display elements are selected from the set of virtual target display elements. When it needs to switch to a real-world hybrid display mode, display elements are selected from the set of real-world target display elements.

8. The automatic display material identification system as described in claim 5, characterized in that: The display element filtering module performs element shape contour recognition on the real target display element set or the virtual target display element set, and filters out display elements that meet preset shape contour conditions, including: Perform element shape contour recognition on the real target display element set or the virtual target display element set to obtain the shape contour features of each display element contained in the real target display element set or the virtual target display element set; Based on the shape contour features, select display elements that have the highest similarity to the preset target shape contour features; The display element visual adjustment and blending module adjusts the visual characteristics of the selected display elements and then blends the display elements into the screen area, including: Based on the color tone characteristics of the background portion of the screen area, the selected display elements are rendered with color tone; then, based on the area boundary of the screen area, the display elements are inserted into the corresponding positions of the screen area.

Images