Augmented reality display processing method and device and electronic equipment

Abstract

The embodiment of the invention discloses an augmented reality display processing method and device and electronic equipment, and the method comprises the steps: displaying an augmented reality scene picture which is obtained through the combination of a real scene picture and a virtual display picture; if it is detected that the real scene picture meets the preset condition, a scene image corresponding to the real scene picture is acquired, and the scene image comprises a target object; according to a preset condition, performing display transformation processing on a target object in the scene image to obtain a target image; and displaying the target image through the virtual display picture, so that the target object in the virtual display picture corresponds to the position of the target object in the real scene picture. When the current real scene meets the preset condition, the target object is displayed by utilizing the virtual display picture to replace the target object in the real scene picture, and the target object is subjected to display transformation processing, so that the visual effect of changing the target object through the virtual display picture is realized, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to an extended reality display processing method, device, and electronic device. Background Art

[0002] Extended reality technology expands human perception and interaction methods. For example, augmented reality (AR) technology is a technology that combines the real world with virtual information based on real-time computer computing and multi-sensor fusion. This technology simulates and re-outputs human vision, hearing, smell, touch and other senses, and superimposes virtual information on real information, providing people with an experience beyond the real world. At present, augmented reality technology has been widely used in production line inspections in the industrial field, surgical simulations in the medical field, scene reproduction in the cultural and tourism field, space display in the marketing field, etc., and there is a trend of gradually expanding and penetrating into the field of personal smart mobile consumer terminals.

[0003] In the field of personal smart mobile consumer terminals, wearable AR display terminal products of different specifications and models have emerged from different manufacturers, such as Vision Pro released by Apple, Quest released by Meta, Holoens released by Microsoft, MUYU released by StarJi Meizu, etc. The technical routes adopted by these wearable AR display terminals are mainly the following two: 1. Video-see-through 2. Optical-see-through. Video-see-through uses a camera to collect environmental images, adds the information images to be added to the environmental images, and presents them in the form of virtual images through optical display components; optical-see-through directly presents the information images to be added in the real field of view through the optical display module. In order to achieve more accurate and clear visual observation effects, the optical display module can be finely controlled to optimize the brightness of the virtual image.

[0004] However, controlling the optical display module to adjust the brightness of the virtual image does not change people's visual observation of the real environment, especially the optical perspective type, which is that the light from the real world passes through the device and directly enters the human eye. This restricts the normal use of current AR terminal devices in different scenarios. Summary of the invention

[0005] The embodiments of the present application provide an extended reality display processing method, device, and electronic device, which can improve the user experience of using the extended reality device in different scenarios.

[0006] In a first aspect, an embodiment of the present application provides an extended reality display processing method, the method comprising:

[0007] Displaying an extended reality scene screen through the extended reality device, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen;

[0008] If it is detected that the real scene picture meets the preset condition, a scene image corresponding to the real scene picture is acquired, and the scene image contains the target object;

[0009] According to the preset conditions, performing display transformation processing on the target object in the scene image to obtain a target image;

[0010] The target image is displayed through the virtual display screen so that the position of the target object in the virtual display screen corresponds to the position of the target object in the real scene screen.

[0011] In a second aspect, an embodiment of the present application further provides an extended reality display processing device, the device comprising:

[0012] A display module, used to display an extended reality scene screen through the extended reality device, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen;

[0013] A collection module, for collecting and obtaining a scene image corresponding to the real scene picture if it is detected that the real scene picture meets a preset condition, wherein the scene image contains a target object;

[0014] A processing module, configured to perform display transformation processing on the target object in the scene image according to the preset conditions to obtain a target image;

[0015] The display module is used to display the target image through the virtual display screen so that the position of the target object in the virtual display screen corresponds to the position of the target object in the real scene screen.

[0016] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the above-mentioned extended reality display processing method when executed by the processor.

[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned extended reality display processing method are implemented.

[0018] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0019] The extended reality device of the embodiment of the present application displays an extended reality scene screen, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen. If it is detected that the real scene screen meets a preset condition, a scene image corresponding to the real scene screen is acquired, and the scene image contains a target object. According to the preset condition, a display transformation process is performed on the target object in the scene image to obtain a target image, and the target image is displayed through the virtual display screen so that the position of the target object in the virtual display screen corresponds to that of the target object in the real scene screen.

[0020] Among them, by displaying the target image on the virtual display screen, the target object is displayed in the virtual display screen. By setting the position of the target object in the virtual display screen to correspond to the position of the target object in the real scene screen, the target object in the virtual display screen is used to cover or highlight the target object, so that the user can browse the target object through the virtual display screen, that is, browsing the target object is no longer dependent on the real scene screen.

[0021] Furthermore, by performing display transformation processing on the target object in the scene image based on the preset conditions when the real scene screen meets the preset conditions, it is ensured that the target object displayed in the virtual display screen can meet the user's needs, and the visual effect of changing the target object by using the virtual display screen is achieved, thereby improving the display effect of the target image and target object in the virtual display screen and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a scene schematic diagram of the extended reality display processing method provided in an embodiment of the present application;

[0024] Figure 2 is a flowchart of an extended reality display processing method provided in an embodiment of the present application;

[0025] Figure 3 It is a visual effect diagram of a night scene provided by an embodiment of the present application;

[0026] Figure 4 It is a visual effect diagram after processing of a night scene provided in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of the structure of an extended reality display processing device provided in an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0030] The embodiments of the present application provide an extended reality display processing method, device and electronic device. Specifically, the embodiments of the present application provide an extended reality display processing device suitable for an electronic device (extended reality device), which is used to display a target object in a real scene in a virtual display screen of the extended reality device, and the extended reality device includes but is not limited to a head-mounted display and wearable glasses. The extended reality device can be an integrated extended reality device built into a computing processing unit, or a split extended reality device external to the computing processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, automobiles, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, a handheld mobile device such as a mobile phone, a desktop computer, a laptop computer, a tablet, etc., a wearable near-eye display system such as a head-mounted display, smart glasses, etc. When the extended reality device is a wearable head-mounted display or smart glasses, it can be an integrated extended reality device built into a computing processing unit, or a split extended reality device external to the computing processing unit.

[0031] See also Figure 1 , Figure 1 : is a schematic diagram of a scenario in which an extended reality device according to an embodiment of the present application performs the extended reality display processing method, wherein the specific execution process of the extended reality device performing the extended reality display processing method is as follows:

[0032] The extended reality device 10 displays an extended reality scene screen, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen. If it is detected that the real scene screen meets a preset condition, a scene image corresponding to the real scene screen is acquired, and the scene image contains a target object. According to the preset condition, the target object in the scene image is subjected to display transformation processing to obtain a target image, and the target image is displayed through the virtual display screen so that the position of the target object in the virtual display screen corresponds to that of the target object in the real scene screen.

[0033] like Figure 1 As shown, the extended reality device is an optical see-through glasses, which at least includes a glasses body, an optical display module, electronic circuit components, sensors, etc. The sensors built into the glasses include a camera, an eye tracker, a microphone, etc.

[0034] Among them, by displaying the target image on the virtual display screen, the target object is displayed in the virtual display screen. By setting the position of the target object in the virtual display screen to correspond to the position of the target object in the real scene screen, the target object in the virtual display screen is used to cover or highlight the real scene target object, so that the user can browse the target object through the virtual display screen, that is, browsing the target object is no longer dependent on the real scene screen.

[0035] Among them, by performing display transformation processing on the target object in the scene image based on the preset conditions when the real scene screen meets the preset conditions, it is ensured that the target object displayed in the virtual display screen can meet the user's needs, the display effect of the target image and target object in the virtual display screen is improved, and the user experience is improved.

[0036] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0037] See also Figure 2 , Figure 2 A flowchart of an extended reality display processing method provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. Specifically, the specific process of the extended reality display processing method is as follows:

[0038] 101. Display an extended reality scene screen, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen.

[0039] Among them, the extended reality scene image is provided based on the optical display module of the extended reality device, and the optical display module can be a combination of a series of optical elements. For example, the optical display module can include a micro display screen, a sensor, a combiner, a lens and a transmitter, etc.

[0040] It is understandable that the extended reality scene picture is a combination of the real scene picture and the virtual display picture. For example, the virtual display picture is superimposed on the real scene picture to obtain the extended reality scene picture. Therefore, the extended reality scene picture provided by the optical display module of the extended reality device includes the real scene picture and the virtual display picture. Among them, the real scene picture is the picture of the real scene browsed through the optical lens (such as the physical screen), and the virtual display picture is displayed on the virtual display screen through comprehensive processing of the transmitter, sensor and micro display screen.

[0041] Among them, the virtual display screen is not the optical lens of the extended reality device. The virtual display screen is a screen with a certain resolution (such as 1280*960) that the user can see at a preset distance in front of the eyes (such as close to 4m) through the direction of the optical lenses and sensory perception after wearing the smart glasses corresponding to the extended reality device. The screen with a certain resolution is the virtual display screen. For example, a virtual display screen is displayed on the virtual display screen of the extended reality device, and at the same time, the real scene screen corresponding to the real scene is browsed through the optical lenses of the extended reality device, and then, the effect of superimposing the virtual display screen on the real scene screen is presented, and an extended reality scene screen is obtained. Among them, the real scene screen is a screen formed by part of the real scene, and the virtual display screen is a screen composed of virtual content.

[0042] 102. If it is detected that the real scene picture meets a preset condition, a scene image corresponding to the real scene picture is acquired, and the scene image contains a target object.

[0043] The preset condition is a pre-set condition, which can also be understood as a trigger condition for capturing a scene image and performing a display transformation process on the scene image.

[0044] Among them, whether the real scene picture meets the preset conditions can be analyzed according to the current ambient illumination and brightness. Specifically: the current ambient illumination is obtained. If the current ambient illumination belongs to the preset illumination range, it is determined that the real scene picture meets the preset conditions; and / or the brightness of the real scene picture is extracted. If the brightness belongs to the preset brightness range, it is determined that the real scene picture meets the preset conditions.

[0045] Among them, the current ambient illumination can be acquired through sensors, and the brightness of the real scene picture can be obtained through image recognition processing, for example, an image corresponding to the real scene picture is captured, and the image is analyzed to determine the brightness corresponding to the real scene picture.

[0046] The preset illumination range is a predefined illumination range. If the current environment illumination is within the preset illumination range, it is considered that the real scene image meets the preset condition. In some cases, the preset illumination range is a range indicating that the environment illumination is relatively low. For example, if the current environment illumination is within the preset illumination range, it is considered that the current environment illumination is relatively low, that is, the brightness in the scene is relatively low, or the visibility of the target object in the scene is relatively low, such as the inability to see the vehicle door handles, road obstacles, etc. clearly at night.

[0047] In some embodiments of the present application, the contrast ratio may be used to analyze whether the real scene image meets the preset condition. For example, if the contrast ratio is low, the real scene image is considered to meet the preset condition. The low contrast ratio means that the target object in the scene is not prominent enough, such as the door handle of a vehicle and a road obstacle at night are basically the same brightness or color.

[0048] Optionally, in an embodiment of the present application, a scene image may be collected according to a target object of interest to the user, that is, optionally, in some embodiments of the present application, the step of “if it is detected that the real scene screen meets a preset condition, then collecting and obtaining a scene image corresponding to the real scene screen” includes:

[0049] If it is detected that the real scene image meets the preset conditions, collecting the eye gaze information of the current user;

[0050] A target object is determined from the real scene image according to the eye gaze information, and a scene image containing the target object is acquired.

[0051] Among them, the eye gaze information refers to the gaze position information or gaze direction information of the user's eyes. Based on the position information corresponding to the eye gaze information, the target object can be determined from the real scene, and then, the scene image can be collected based on the target object as the center.

[0052] Optionally, in some scenarios, when the ambient brightness is low and the target object cannot be directly located from the real scene, a gaze area may be determined based on eye gaze information, and a scene image may be acquired based on the gaze area.

[0053] Optionally, in order to improve the accuracy of target object analysis, the target object may be located in combination with the user's gesture information or voice information. That is, optionally, in some embodiments of the present application, the step of "if it is detected that the real scene screen meets a preset condition, then acquiring a scene image corresponding to the real scene screen" includes:

[0054] If it is detected that the real scene image meets the preset conditions, collecting the eye gaze information and indication feature information of the current user;

[0055] Determine a target object from the real scene image according to the eye gaze information and the indication feature information, and collect a scene image containing the target object;

[0056] The indication feature information includes at least one of gesture information and voice information.

[0057] Among them, the accuracy of target object recognition can be improved through eye gaze information, gesture information and voice information.

[0058] In some cases, eye gaze information, gesture information and voice information can also be combined in pairs to identify the target object in the real scene. For example, eye gaze information and gesture information can be combined to identify the target object in the real scene, and eye gaze information and voice information can be combined to identify the target object in the real scene. For example, when the voice clearly indicates the target object (such as a road sign in the scene, a vehicle door handle, etc.), the target object can be directly located. When the voice does not clearly indicate the target object, the eye gaze information or gesture information is combined to locate the target object from the real scene.

[0059] 103. According to the preset condition, perform display transformation processing on the target object in the scene image to obtain a target image.

[0060] The scene image is subjected to display transformation processing according to preset conditions currently satisfied by the real scene picture, so that the processed target image meets the requirements of the current real scene picture.

[0061] Optionally, in some cases, the display transformation processing of the scene image (i.e., the target object) is not necessarily just display enhancement processing, but can also be display weakening processing. For example, for the target object in the night scene, in order to improve the brightness or display contrast or display effect of the target object, the scene image can be subjected to display enhancement processing. For daytime scenes with higher brightness, in order to reduce the brightness of the target object and avoid glare and other problems, the display brightness of the target object can be appropriately reduced, that is, the target object is subjected to display weakening processing. That is, optionally, in some embodiments of the present application, the step of "performing display transformation processing on the target object in the scene image to obtain the target image according to the preset conditions" includes:

[0062] Determining adjustment information according to the preset conditions;

[0063] If the adjustment information is display enhancement, performing display enhancement processing on the target object in the scene image to obtain a target image;

[0064] If the adjustment information is display attenuation, display attenuation processing is performed on the target object in the scene image to obtain a target image.

[0065] In the embodiment of the present application, the processing of the scene image can be implemented by an image generation model or an image processing technology. For example, the step of "performing a display transformation process on the target object in the scene image to obtain a target image" includes:

[0066] Generate a target image corresponding to the scene image by using an image generation model; and / or,

[0067] Segmenting the target object from the scene image, and performing display transformation processing on the target object to obtain a target image;

[0068] The display transformation processing includes at least one of exposure compensation, color correction, detail enhancement, brightness enhancement, and contrast improvement for the target object.

[0069] The image generation model is a graph-to-graph model in deep learning, that is, the input is an image and the output is also an image. For example, the input is an image with low brightness and the output is an image with high brightness.

[0070] Optionally, in an embodiment of the present application, in order to improve the accuracy of the target image output by the image generation model, optical parameters for scene image processing may also be designed, and the image generation model may be controlled to process the scene image according to the optical parameters. That is, optionally, in some embodiments of the present application, the step of "generating a target image corresponding to the scene image by using the image generation model" includes:

[0071] Identifying location information of the target object in the scene image;

[0072] Determining optical parameters corresponding to the target object;

[0073] The optical parameters and the position information are input into the image generation model, so as to process the scene image through the image generation model with the optical parameters and the position information as generation conditions to generate a target image.

[0074] The position information also refers to range information, which is the area range of the target object in the scene image.

[0075] Among them, the position information or range information corresponding to the target object in the scene image can be obtained through an image recognition algorithm, for example, by recognizing the scene image through an artificial intelligence large model to identify the target object and the position information or range information of the target object from the scene image.

[0076] Among them, in an embodiment of the present application, after obtaining the scene image, the recognition of the target object can also be determined based on the user's interaction instructions, that is, in response to the user's interaction instructions, the target object is determined from the scene image. For example, the user's interaction instruction can be a voice instruction (for example, the voice instruction can specifically be: perform display transformation processing on the door handle), then specifically: in response to the user's voice instruction, the target object is identified from the scene image based on the indication information corresponding to the voice instruction, for example, the door handle is identified from the scene image.

[0077] The optical parameters refer to the target optical parameters of the target image to be generated, such as image brightness, image illumination, etc. The optical parameters can be obtained by querying the optical parameters pre-set by the user to match the user's observation habits. For example, when the optical parameters are image illumination, determining the optical parameters corresponding to the target object includes: detecting the ambient illumination of the current real environment, querying the observation contrast pre-set by the user, and calculating the target illumination of the target image through the ambient illumination of the real environment and the observation contrast set by the user. The observation contrast is the ratio of the illumination of the object to be observed to the illumination of the surrounding environment. A moderate contrast helps to improve the observation effect, for example, more prominently displaying the target object from the scene image.

[0078] 104. Display the target image through the virtual display screen, so that the position of the target object in the virtual display screen corresponds to the position of the target object in the real scene screen.

[0079] By displaying the target image in the virtual display screen and setting the position object, the target object in the real scene is enhanced by the target object in the virtual display screen, so that the user can observe the target object more accurately.

[0080] Optionally, after the target image is processed, the display of the target image in the virtual display screen may be controlled based on the display position corresponding to the scene image. That is, optionally, in some embodiments of the present application, the step of “displaying the target image through the virtual display screen” includes:

[0081] Determining a display position of the scene image;

[0082] The display of the target image in the virtual display screen is controlled according to the display position.

[0083] The following example illustrates the extended reality display processing method of the embodiment of the present application, specifically:

[0084] Scenario 1:

[0085] In an outdoor parking lot at night, a user wearing AR glasses walks to the front of the car and prepares to open the door to drive. However, due to the insufficient light intensity at night, the user can only see the overall outline of the car in traditional cases, and judge the rough position of the door handle (where the door handle is the target object in the current example, also called the observation object) based on the overall outline. However, due to the night situation, the user cannot find the door handle or finds it very hard to open the door after touching and exploring near the door handle for a long time, which is more serious when the door handle is hidden.

[0086] Based on the extended reality display processing method of the embodiment of the present application, the enhancement system of the AR glasses is awakened and starts working at this time. The eye tracker tracks the user's line of sight, calculates the area where the user's line of sight falls most frequently in the last 1 minute (can be customized to other durations), and determines the object in this line of sight landing area as the user's target observation object. The camera captures images near the line of sight landing area, and transmits the scene image near the line of sight landing area to the cloud server. The cloud server runs a target detection artificial intelligence large model (used to identify the target object in the scene image and determine the location information of the target object) and a large image generation artificial intelligence model. The target detection artificial intelligence large model uses reasoning calculations to deduce that the target observation object near the line of sight landing area at this time is a door handle. After reasoning calculations, the output is generated to determine the image of this object under the lighting environment and returned to the AR glasses. After receiving the image, the AR glasses determine the display position coordinates of the image, that is, the position of the target object in the real environment, through three-dimensional space registration. The optical display module displays the image in the form of a virtual image at the target position coordinates corresponding to the display position coordinates. In terms of visual effect, the virtual image position of the door handle image coincides with the real position of the door handle entity. The user cannot see the door handle of the real scene at this position, but can see the door handle of the virtual scene displayed at this position clearly. In this way, the user can easily find the door handle in dim light, such as Figure 3 and Figure 4 As shown, Figure 3 is a visual effect diagram of a night scene. Based on the processing of the extended reality display method of the embodiment of the present application, the following is obtained: Figure 4 The processed visual effect of the night scene shown in the figure is from the Figure 4 The door handle in the generated virtual scene can be seen.

[0087] In some cases, the object in the sight area is closely connected with other objects or the object is part of a larger object, such as a door handle is part of a vehicle. In order to more accurately determine the user's observation object, the result of voice recognition is combined to determine that the observation object is only the door handle and not the entire vehicle. For example, in the above scenario, there are objects such as reflectors near the user's sight area. AR glasses receive the "show door handle" command sent by the user in voice, and use the door handle as the object of target detection. The target detection artificial intelligence large model calculates and processes the image near the user's sight area to identify the door handle and obtain the image corresponding to the door handle of the target detection object.

[0088] Scenario 2:

[0089] The user wears AR glasses and walks from the bright outdoor to the dim indoor, looking for a power adapter to charge his phone.

[0090] AR glasses are equipped with an ambient light detector (sensor). When it detects that the ambient light intensity has significantly weakened and reaches an effect that is not conducive to human eye observation, the sunshade or electrochromic layer of the AR glasses automatically opens to allow more external light to pass through the glasses lens into the eye. When the human eye scans the indoor environment, the tracking results of the line of sight viewpoint will be displayed on the virtual display screen. The user selects the range to be observed with the viewpoint as the center. Based on the selection made by the user on the virtual screen, the range of the actual field of view to be observed is determined. The built-in camera of the AR glasses shoots the selected range, converts the captured image into an image under a certain illumination, and displays the converted image on the virtual display screen. In this way, the user can browse to the image directly from the virtual display screen, that is, see the power adapter.

[0091] In some cases, AR glasses are equipped with lighting. If the photos taken are not clear and are not enough to convert image to image, you can turn on the lighting. The light emitted by the lighting will fall in line with the user's line of sight, so that the photos taken will be clearer, and the image conversion effect will be better. The display coordinates of the image are determined by three-dimensional space registration technology, and the image is presented in the real field of view in the form of a virtual image through the optical display module of the AR glasses. Among them, in this case, the light provided by the lighting is not enough to directly and clearly illuminate the power adapter, but it appropriately increases the brightness of the scene to avoid collecting photos in which the power adapter is completely invisible.

[0092] In order to facilitate better implementation of the extended reality display processing method of the present application, the present application also provides an extended reality display processing device based on the above extended reality display processing method. The meanings of the terms are the same as those in the above extended reality display processing method, and the specific implementation details can refer to the description in the method embodiment.

[0093] See also Figure 5 , Figure 5 : is a schematic diagram of the structure of an extended reality display processing device provided in an embodiment of the present application, wherein the extended reality display processing device may be specifically as follows:

[0094] A display module 201 is used to display an extended reality scene screen, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen;

[0095] The acquisition module 202 is used for acquiring a scene image corresponding to the real scene picture if it is detected that the real scene picture meets a preset condition, wherein the scene image contains a target object;

[0096] The processing module 203 is used to perform display transformation processing on the target object in the scene image to obtain a target image according to the preset conditions;

[0097] The display module 204 is used to display the target image through the virtual display screen, so that the position of the target object in the virtual display screen corresponds to the position of the target object in the real scene screen.

[0098] Optionally, in some embodiments of the present application, if it is detected that the real scene picture meets a preset condition, acquiring a scene image corresponding to the real scene picture includes:

[0099] If it is detected that the real scene image meets the preset conditions, collecting the eye gaze information of the current user;

[0100] A target object is determined from the real scene image according to the eye gaze information, and a scene image containing the target object is acquired.

[0101] Optionally, in some embodiments of the present application, if it is detected that the real scene picture meets a preset condition, acquiring a scene image corresponding to the real scene picture includes:

[0102] If it is detected that the real scene image meets the preset conditions, collecting the eye gaze information and indication feature information of the current user;

[0103] Determine a target object from the real scene image according to the eye gaze information and the indication feature information, and collect a scene image containing the target object;

[0104] The indication feature information includes at least one of gesture information and voice information.

[0105] Optionally, in some embodiments of the present application, before detecting that the real scene image satisfies a preset condition, the method further includes:

[0106] Acquiring the current environment illumination, and if the current environment illumination belongs to a preset illumination range, determining that the real scene image meets a preset condition; and / or,

[0107] The brightness of the real scene picture is extracted, and if the brightness belongs to a preset brightness range, it is determined that the real scene picture meets a preset condition.

[0108] Optionally, in some embodiments of the present application, performing display transformation processing on the target object in the scene image to obtain the target image according to the preset condition includes:

[0109] Determining adjustment information according to the preset conditions;

[0110] If the adjustment information is display enhancement, performing display enhancement processing on the target object in the scene image to obtain a target image;

[0111] If the adjustment information is display attenuation, display attenuation processing is performed on the target object in the scene image to obtain a target image.

[0112] Optionally, in some embodiments of the present application, performing display transformation processing on the target object in the scene image to obtain the target image includes:

[0113] Generate a target image corresponding to the scene image by using an image generation model; and / or,

[0114] Segmenting the target object from the scene image, and performing display transformation processing on the target object to obtain a target image;

[0115] The display transformation processing includes at least one of exposure compensation, color correction, detail enhancement, brightness enhancement, and contrast improvement for the target object.

[0116] Optionally, in some embodiments of the present application, generating a target image corresponding to the scene image by using an image generation model includes:

[0117] Identifying location information of the target object in the scene image;

[0118] Determining optical parameters corresponding to the target object;

[0119] The optical parameters and the position information are input into the image generation model, so as to process the scene image through the image generation model with the optical parameters and the position information as generation conditions to generate a target image.

[0120] In the embodiment of the present application, the display module 201 first displays an extended reality scene screen, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen. Then, if the acquisition module 202 detects that the real scene screen meets a preset condition, it acquires a scene image corresponding to the real scene screen, wherein the scene image contains a target object. Subsequently, the processing module 203 performs display transformation processing on the target object in the scene image according to the preset condition to obtain a target image. Then, the display module 204 displays the target image through the virtual display screen so that the position of the target object in the virtual display screen corresponds to that of the target object in the real scene screen.

[0121] The embodiment of the present application displays the target object in the virtual display screen by displaying the target image on the virtual display screen. By setting the position of the target object in the virtual display screen to correspond to the position of the target object in the real scene screen, the target object in the virtual display screen is used for enhancement, so that the user can observe the target object more accurately, that is, the browsing of the target object is no longer dependent on the real scene screen.

[0122] Among them, by performing display transformation processing on the target object in the scene image based on the preset conditions when the real scene screen meets the preset conditions, it is ensured that the target object displayed in the virtual display screen can meet the user's needs, the display effect of the target image and target object in the virtual display screen is improved, and the user experience is improved.

[0123] In addition, the present application also provides an electronic device, such as Figure 6As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0124] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0125] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0126] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0127] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0128] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0129] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby implementing the steps in any one of the augmented reality display processing methods provided in the embodiments of the present application.

[0130] The extended reality device of the embodiment of the present application displays an extended reality scene screen, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen. If it is detected that the real scene screen meets a preset condition, a scene image corresponding to the real scene screen is acquired, and the scene image contains a target object. According to the preset condition, a display transformation process is performed on the target object in the scene image to obtain a target image, and the target image is displayed through the virtual display screen so that the position of the target object in the virtual display screen corresponds to that of the target object in the real scene screen.

[0131] Among them, by displaying the target image on the virtual display screen, the target object is displayed in the virtual display screen. By setting the position of the target object in the virtual display screen to correspond to the position of the target object in the real scene screen, the target object in the virtual display screen is used to cover or highlight the target object, so that the user can browse the target object through the virtual display screen, that is, browsing the target object is no longer dependent on the real scene screen.

[0132] Among them, by performing display transformation processing on the target object in the scene image based on the preset conditions when the real scene screen meets the preset conditions, it is ensured that the target object displayed in the virtual display screen can meet the user's needs, the display effect of the target image and target object in the virtual display screen is improved, and the user experience is improved.

[0133] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0134] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0135] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any of the extended reality display processing methods provided in the present application.

[0136] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0137] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0138] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the extended reality display processing methods provided in the present application, the beneficial effects that can be achieved by any of the extended reality display processing methods provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0139] The above is a detailed introduction to an extended reality display processing method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An extended reality display processing method, characterized in that: Applied to an extended reality device, the method comprises: Displaying an extended reality scene screen, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen; If it is detected that the real scene picture meets the preset condition, a scene image corresponding to the real scene picture is acquired, and the scene image contains the target object; According to the preset conditions, performing display transformation processing on the target object in the scene image to obtain a target image; The target image is displayed through the virtual display screen so that the position of the target object in the virtual display screen corresponds to the position of the target object in the real scene screen.

2. The method for processing extended reality display according to claim 1, characterized in that: If it is detected that the real scene picture meets a preset condition, acquiring a scene image corresponding to the real scene picture includes: If it is detected that the real scene image meets the preset conditions, collecting the eye gaze information of the current user; A target object is determined from the real scene image according to the eye gaze information, and a scene image containing the target object is acquired.

3. The method for processing extended reality display according to claim 1, characterized in that: If it is detected that the real scene picture meets a preset condition, acquiring a scene image corresponding to the real scene picture includes: If it is detected that the real scene image meets the preset conditions, collecting the eye gaze information and indication feature information of the current user; Determine a target object from the real scene image according to the eye gaze information and the indication feature information, and collect a scene image containing the target object; The indication feature information includes at least one of gesture information and voice information.

4. The method for processing extended reality display according to claim 1, characterized in that: If it is detected that the real scene image meets the preset condition, the method further includes: Acquiring the current environment illumination, and if the current environment illumination belongs to a preset illumination range, determining that the real scene image meets a preset condition; and / or, The brightness of the real scene picture is extracted, and if the brightness belongs to a preset brightness range, it is determined that the real scene picture meets a preset condition.

5. The method for processing extended reality display according to claim 1, characterized in that: The step of performing display transformation processing on the target object in the scene image to obtain the target image according to the preset condition includes: Determining adjustment information according to the preset conditions; If the adjustment information is display enhancement, performing display enhancement processing on the target object in the scene image to obtain a target image; If the adjustment information is display attenuation, display attenuation processing is performed on the target object in the scene image to obtain a target image.

6. The method for processing extended reality display according to claim 1, characterized in that: The performing display transformation processing on the target object in the scene image to obtain the target image includes: Generate a target image corresponding to the scene image by using an image generation model; and / or, Segmenting the target object from the scene image, and performing display transformation processing on the target object to obtain a target image; The display transformation processing includes at least one of exposure compensation, color correction, detail enhancement, brightness enhancement, and contrast improvement for the target object.

7. The method for processing extended reality display according to claim 6, characterized in that: The step of generating a target image corresponding to the scene image by using an image generation model includes: Identifying location information of the target object in the scene image; Determining optical parameters corresponding to the target object; The optical parameters and the position information are input into the image generation model, so as to process the scene image through the image generation model with the optical parameters and the position information as generation conditions to generate a target image.

8. An extended reality display processing device, characterized in that: The device includes: A display module, used to display an extended reality scene screen, wherein the extended reality scene screen is obtained by combining a real scene screen and a virtual display screen; A collection module, used for collecting and obtaining a scene image corresponding to the real scene picture if it is detected that the real scene picture meets a preset condition, wherein the scene image contains a target object; A processing module, configured to perform display transformation processing on the target object in the scene image according to the preset conditions to obtain a target image; The display module is used to display the target image through the virtual display screen so that the position of the target object in the virtual display screen corresponds to the position of the target object in the real scene screen.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the extended reality display processing method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the extended reality display processing method according to any one of claims 1 to 7 are implemented.

Images