Image content anchoring method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a picture content anchoring method and device, electronic equipment and a readable storage medium, and belongs to the field of augmented reality. The method is applied to augmented reality equipment, and a user wears the augmented reality equipment in a movable carrier. The method comprises the following steps: acquiring inertial navigation data and a visual image acquired by augmented reality equipment; the inertial navigation data is data reflecting the acceleration of the augmented reality equipment; correcting the inertial navigation data by using the visual image to obtain corrected inertial navigation data; performing attitude estimation on the corrected inertial navigation data and the visual image, and fusing attitude estimation results to obtain attitude data reflecting the attitude of the user; and anchoring the picture content of the augmented reality equipment according to the attitude data.

Description

Technical Field

[0001] The present application belongs to the field of extended reality, and specifically relates to a method, device, electronic device and readable storage medium for anchoring picture content. Background Art

[0002] Extended Reality (XR) combines the real environment with the virtual environment through computers to create a virtual environment that can be interacted with by humans and computers, giving users an immersive experience of seamless transition between the virtual world and the real world.

[0003] In related technologies, short-term posture tracking results are usually provided by visual inertial odometry (VIO), which includes two modes, vision and inertial navigation, and achieves posture tracking through a fusion algorithm (such as a Kalman filter).

[0004] However, when the user is on a movable vehicle and posture tracking is achieved using methods in related technologies, the virtual content may first move in the opposite direction of the movable vehicle, and then trigger a tracking loss state, thereby causing the spatial position presentation relationship of all virtual content to become invalid. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device and readable storage medium for anchoring screen content, which can solve the problem in the related art that VIO cannot track virtual content when the user is in a movable carrier.

[0006] In a first aspect, an embodiment of the present application provides a method for anchoring screen content, which is applied to an extended reality device, where a user wears the extended reality device in a movable carrier; the method includes:

[0007] Acquire inertial navigation data and a visual image collected by the extended reality device; the inertial navigation data is data reflecting the acceleration of the extended reality device;

[0008] Correcting the inertial navigation data using the visual image to obtain corrected inertial navigation data;

[0009] Performing posture estimation on the corrected inertial navigation data and the visual image respectively, and fusing the posture estimation results to obtain posture data reflecting the user's posture;

[0010] Anchor the screen content of the extended reality device according to the posture data.

[0011] In a second aspect, an embodiment of the present application provides an anchoring device for screen content, wherein the device is located in an extended reality device, and a user wears the extended reality device in a movable carrier; the device includes:

[0012] A first acquisition module, used to acquire inertial navigation data and a visual image collected by the extended reality device; the inertial navigation data is data reflecting the acceleration of the extended reality device;

[0013] A first correction module, used for correcting the inertial navigation data using the visual image to obtain corrected inertial navigation data;

[0014] A first posture estimation module, used to perform posture estimation on the corrected inertial navigation data and the visual image respectively, and fuse the posture estimation results to obtain posture data reflecting the user's posture;

[0015] The first anchoring module is used to anchor the screen content of the extended reality device according to the posture data.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0020] In an embodiment of the present application, inertial navigation data and a visual image collected by an extended reality device are obtained, and the inertial navigation data is corrected by using the visual image to obtain the corrected inertial navigation data, so that the error caused by the movement of the movable carrier in the inertial navigation data can be eliminated, and then the corrected inertial navigation data and the visual image are respectively subjected to posture estimation, and the posture estimation results of the two are fused to obtain posture data reflecting the user's posture, and finally the screen content of the extended reality device is anchored according to the posture data. In an embodiment of the present application, the inertial navigation data is corrected by the visual image. Since the error caused by the movement of the movable carrier can be eliminated in the corrected inertial navigation data, when the corrected inertial navigation data and the visual image are respectively subjected to posture estimation, the posture estimation results of the two are consistent with each other, and by fusing the posture estimation results of the two, more robust user posture data can be obtained. Finally, the screen content of the extended display device is anchored by the user posture data, so that the posture estimation of the user in the moving state of the carrier can be realized, so that the state of virtual content tracking loss can be avoided, and the virtual content can be presented in the correct spatial position relationship, which improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a typical VIO framework provided in the embodiments of the present application;

[0022] Figure 2 This is a schematic diagram of an application scenario of a method for anchoring screen content provided by an embodiment of the present application;

[0023] Figure 3 It is a flowchart of the steps of a method for anchoring screen content provided by an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of a method for dividing screen content provided in an embodiment of the present application;

[0025] Figure 5 It is an improved VIO framework for the movable carrier scenario provided by the embodiment of the present application;

[0026] Figure 6 is a schematic diagram of an image quality assessment model provided in an embodiment of the present application;

[0027] Figure 7 It is a structural schematic diagram of a device for anchoring picture content provided by an embodiment of the present application;

[0028] Figure 8 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0029] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0032] The Simultaneous Localization and Mapping (SLAM) algorithm is achieved by fusing the VIO front-end and the mapping back-end. The VIO front-end is responsible for providing the results of posture tracking in a short period of time, and the mapping back-end is responsible for jointly optimizing the historical posture and key points of the environment. Specifically, the VIO front-end combines the two modes of vision and inertial navigation, and realizes the posture tracking of six degrees of freedom (6DOF) in space through the fusion algorithm (such as Kalman filter). For details, see Figure 1 In the pure vision module, the image is obtained through the visual sensor, and the image features are extracted. The features of each frame of the current image are matched with the features of the previous frame of the image. Finally, the visual mileage module is used to obtain the result of the posture tracking based on vision. In the pure inertial navigation module, the acceleration of the image is measured by the inertial sensor (IMU, Inertial Measurement Unit) through the accelerometer, and the data is filtered to remove the high-frequency noise in the accelerometer output. The angular velocity measured by the gyroscope is used for inertial navigation fusion to reduce the drift error. Finally, the inertial navigation-based posture tracking result is obtained through the inertial navigation mileage module. By fusing the posture tracking results based on vision and the posture tracking results based on inertial navigation, higher accuracy and robustness can be provided.

[0033] In general scenarios, the posture tracking results based on vision and the posture tracking results based on inertial navigation are consistent with each other. The posture tracking results of the two are then fused through the VIO fusion algorithm to obtain a more robust posture tracking result.

[0034] However, when the user is in a movable carrier, the acceleration measured by the IMU is not only driven by the user's own movements, but also the value after the acceleration of the movable carrier is superimposed. At this time, if the general VIO algorithm is still used for processing, the virtual content will move in the opposite direction of the movable carrier due to the inconsistency between the posture tracking results based on vision and the posture tracking results based on inertial navigation, and then the tracking loss state will be triggered, which will cause all the spatial position presentation relationships of all virtual content to become invalid.

[0035] Figure 2 This is a schematic diagram of an application scenario of a method for anchoring screen content provided in an embodiment of the present application, see Figure 2 , Figure 2 The vehicle 20, the XR device 21, and the screen content 22 on the XR device display screen are included. The screen content is a five-pointed star, and can also be other content. Figure 2 It may also include an image processing system, which is not shown in the figure. It should be noted that the embodiment of the present application only provides an application scenario and electronic device, and in fact, it can also be applied to other scenarios and electronic devices, and the embodiment of the present application is not specifically limited here.

[0036] XR devices are one of the key tools for implementing extended reality technology, including helmet displays, controllers, sensors, etc., which work together to provide high-quality immersive experience. Among them, the helmet display is one of the most important parts of XR devices, which can display virtual environments or augmented reality content, allowing users to experience an immersive experience. Controllers and sensors are used to capture user movements and inputs to enable interaction with the virtual world.

[0037] Specifically, the graphics processing system obtains the display information wirelessly transmitted by the display screen, fuses the display information with the virtual scene, and then sends the fused information to the display of the XR device, which presents the fused information on the display lens. In this way, users wearing XR devices can obtain the fused picture content. Since the vehicle is movable, the acceleration measured by the IMU on the XR device is not only driven by the user's own movements, but is the value after the acceleration of the movable carrier is superimposed. At this time, if it is still processed according to the general VIO algorithm, the virtual content will move in the opposite direction of the movable carrier due to the inconsistency between the vision-based posture tracking results and the inertial navigation-based posture tracking results, and then trigger the tracking loss state, which will cause all the spatial position presentation relationships of all virtual content to become invalid.

[0038] In order to solve the above-mentioned problems in the related art, the embodiment of the present application corrects the inertial navigation data through the visual image. Since the error caused by the movement of the movable carrier can be eliminated in the corrected inertial navigation data, when the corrected inertial navigation data and the visual image are respectively estimated, the posture estimation results of the two are consistent with each other, and by fusing the posture estimation results of the two, more robust user posture data can be obtained. Finally, by anchoring the screen content of the extended display device with the user posture data, the posture estimation of the user when the carrier is in motion can be realized, thereby avoiding the state of virtual content tracking loss, and can correctly present the virtual content in spatial position relationship, thereby improving the user experience.

[0039] The following describes in detail the method for anchoring screen content provided by the embodiment of the present application through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0040] Figure 3 This is a flowchart of a method for anchoring screen content provided by an embodiment of the present application. The method for anchoring screen content is applied to an extended reality device, and the user wears the extended reality device in a movable carrier. The method may include the following steps:

[0041] Step 101, acquiring inertial navigation data and a visual image collected by the extended reality device; the inertial navigation data is data reflecting the acceleration of the extended reality device.

[0042] In an embodiment of the present application, the visual image may be an image of the user's surroundings measured by an extended reality device.

[0043] In an embodiment of the present application, the inertial navigation data may be the acceleration measured by the extended reality device, and posture tracking may be performed based on the acceleration subsequently.

[0044] Step 102: Correct the inertial navigation data using the visual image to obtain corrected inertial navigation data.

[0045] In the embodiment of the present application, the movable carrier may be a vehicle, a subway, or other movable carrier.

[0046] Since the acceleration measured by the extended reality device is not only driven by the user's own actions, but also superimposed with the acceleration of the movable carrier, the error caused by the movement of the movable carrier in the inertial navigation data can be eliminated by correcting the inertial navigation data with visual images. This makes the posture tracking results based on vision and the posture tracking results based on inertial navigation consistent. Finally, the posture tracking results of the two are fused through the VIO fusion algorithm to obtain a more robust posture tracking result.

[0047] In one embodiment, step 102 may include the following sub-steps:

[0048] Sub-step 1021, dividing the visual image into a plurality of areas, and determining a first area from the plurality of divided areas; the first area includes picture content outside the movable carrier.

[0049] In an embodiment of the present application, a plurality of regions correspond to different areas of an image, and each area of ​​the image carries different information. For example, the regions may include a first region, a second region, and other regions except the first region and the second region. Among them, the first region is a region outside the movable carrier, which is used to carry information outside the movable carrier, so as to track the motion state of the movable carrier. The second region is a region inside the movable carrier, which is used to carry information inside the movable carrier, so as to track the posture of the user inside the movable carrier. Other regions are regions other than the first region and the second region, which have uncertainty and are not suitable for posture tracking.

[0050] See also Figure 4 , Figure 4 The image content is divided into three areas, namely area 31 outside the vehicle, area 32 inside the vehicle, and uncertain area 33. Among them, the area outside the vehicle is the area outside the vehicle porthole and the window, which is used to carry information outside the vehicle and to subsequently track the movement state of the vehicle. The uncertain area is the area where high dynamic lighting changes with the environment (such as reflectors, display screens, etc.), that is, the part including the lens. Since reflection or active display has a high uncertainty, this area is not suitable for posture tracking. Other areas are areas inside the vehicle, which are used to carry information inside the vehicle and to subsequently track the posture of users in the vehicle. Area 32 inside the vehicle includes the user wearing the XR device and the area where the XR device displays content.

[0051] In one embodiment, sub-step 1021 may include the following sub-steps:

[0052] Sub-step 10211, obtaining a region of a key frame in the visual image.

[0053] In the embodiment of the present application, the visual image may be an image sequence, and the key frame may be selected based on the quality of the image in the image sequence. There are many methods for selecting key frames, and the embodiment of the present application does not limit this. For example, the first frame and the last frame in the image sequence may be used as key frames. This method is simple and feasible and is suitable for shots with low activity or unchanged content. For another example, the key frames are extracted based on the semantics of the image through a neural network model. The redundancy of the key frames extracted by this method is relatively small.

[0054] There are many methods for partitioning image key frames, which are not specifically limited in the embodiments of the present application. For example, the key frames can be partitioned manually. For another example, the key frames can be partitioned by a semantic segmentation model.

[0055] The embodiment of the present application can obtain key frames in an image and partition the key frames, so that the image other than the key frames can be partitioned based on the partitions of the key frames, and more image information can be extracted.

[0056] In one embodiment, sub-step 10211 may include the following sub-steps:

[0057] Sub-step 102111, performing image segmentation on the key frame through a semantic segmentation model to obtain a partition outside the movable carrier, a partition inside the movable carrier, and other partitions except the partition outside the movable carrier and the partition inside the movable carrier corresponding to the key frame.

[0058] In an embodiment of the present application, the semantic segmentation model may be a deep neural network model, and the partitioning of key frames may be achieved by selecting a suitable neural network model.

[0059] In order to ensure a certain frame rate, the semantic segmentation model can be a bottom-up end-to-end model. For example, the network structure of the semantic segmentation model may include: a backbone feature network and a decoding network. Among them, the backbone network can be MobileNetV3, which is a model that can be applied to movable carriers and balances accuracy and efficiency by combining an automatic search algorithm and a network architecture. The decoding network can be an efficient segmentation decoder (LRASPP, LiteReducedAtrusSpacePyramidPooling) network, which is a decoder structure that can reduce the amount of calculation while ensuring a certain performance, so that the model can better adapt to the hardware limitations of mobile devices.

[0060] For example, by inputting the key frame into the semantic segmentation model, three classification masks can be output, corresponding to the partition outside the movable carrier, the partition inside the movable carrier, and other partitions except the partition outside the movable carrier and the partition inside the movable carrier. These three types of masks can control the range and method of segmentation according to the grayscale and color information of the image, and divide the image into different regions, thereby realizing automatic analysis and processing of the image.

[0061] Sub-step 10212, partitioning the visual images other than the key frames by the regions of the key frames to obtain respective regions of each of the visual images.

[0062] In the embodiment of the present application, images other than the key frame may be images captured in the same time period as the key frame, or images tracked after the time period of the key frame.

[0063] For example, after obtaining the partition outside the movable carrier, the partition inside the movable carrier, and other partitions except the partition outside the movable carrier and the partition inside the movable carrier corresponding to the key frame through the semantic segmentation model, the key frame is applied to the tracked image, and the partition of the image corresponding to the partition outside the movable carrier in the key frame is the partition outside the movable carrier corresponding to the image. The partition of the image corresponding to the partition inside the movable carrier in the key frame is the partition inside the movable carrier corresponding to the image.

[0064] In the embodiment of the present application, by extracting key frames and partitioning the image using the partitions of the key frames, more information can be extracted from the image and the accuracy of the image partitioning can be improved.

[0065] Sub-step 1022, performing external attitude estimation on the content of the first area and the inertial navigation data respectively, and fusing the attitude estimation results to obtain the external attitude acceleration of the movable carrier.

[0066] In the embodiment of the present application, the content of the first area is the content of the picture in the image that is outside the movable carrier. Figure 5 ,After partitioning the image, feature extraction is performed to determine the ,features corresponding to the content of the first area, so as to perform external ,feature matching on it, and obtain the external attitude estimation ,result through the external odometer.

[0067] See also Figure 5When the inertial navigation data is used for external attitude estimation, the IMU measures the acceleration of the image through the accelerometer, and performs data filtering to remove the high-frequency noise in the accelerometer output, and performs inertial navigation fusion with the angular velocity measured by the gyroscope to reduce the drift error. Finally, the inertial navigation-based attitude tracking result is obtained through the inertial navigation mileage module. It is then fused with the above external attitude result and a robust external attitude tracking result is obtained through the external VIO module.

[0068] In the embodiment of the present application, the content of the first area and the inertial navigation data are respectively subjected to external attitude estimation, and the attitude estimation results are fused to obtain a robust external attitude tracking result. Based on the robust external attitude tracking result, the external attitude acceleration of the movable carrier is obtained.

[0069] Sub-step 1023, using the external attitude acceleration, correcting the inertial navigation data to obtain corrected inertial navigation data.

[0070] In the embodiment of the present application, since the external attitude acceleration is an estimate of the acceleration of the movable carrier, the inertial navigation data is corrected by the external attitude acceleration, which can eliminate the error caused by the movement of the movable carrier, so that the corrected inertial navigation data and the acceleration caused by the user drive can be consistent, and the virtual content can be tracked.

[0071] In one embodiment, a gyroscope is provided on the extended reality device, and sub-step 1022 may include the following sub-steps:

[0072] Sub-step 10221, matching the image features of the first area of ​​each frame with the image features of the first area of ​​the previous frame to obtain an external posture estimation result of the movable carrier.

[0073] In the embodiment of the present application, if the current frame is the first frame, there is no image of the previous frame, and the frame is only feature matched with the subsequent image.

[0074] The image features can be obtained by a feature point method, by extracting image feature points and tracking them in an image sequence. The image features can also be directly using the pixel intensity features in the image sequence. The current image features can also be obtained by other methods, which are not specifically limited in the embodiments of the present application.

[0075] The embodiment of the present application obtains an external posture estimation result of the movable carrier by matching the image features of the outside of the movable carrier in each frame with the image features of the outside of the movable carrier in the previous frame. This method does not depend on the driving surface of the movable carrier and can improve the accuracy of external posture estimation.

[0076] Sub-step 10222, angular velocity data is measured by a gyroscope, and attitude estimation is performed based on the inertial navigation data and the angular velocity data to obtain a comprehensive attitude estimation result.

[0077] In the embodiment of the present application, the angular velocity measured by the gyroscope provides information about the angular velocity of the movable carrier, and the inertial navigation data provides information about the tilt of the movable carrier. Combining the information provided by the inertial navigation data and the angular velocity data for attitude estimation can reduce attitude estimation errors.

[0078] See also Figure 5 When the inertial navigation data is used for external attitude estimation, the IMU obtains the acceleration of the image through the accelerometer, and performs data filtering to remove the high-frequency noise in the accelerometer output, and performs inertial navigation fusion with the angular velocity measured by the gyroscope to reduce the drift error. Finally, the inertial navigation-based attitude tracking result is obtained through the inertial navigation mileage module.

[0079] Sub-step 10223, fusing the external attitude estimation result with the comprehensive attitude estimation result, and performing an inverse operation on the fusion result to obtain the external attitude acceleration.

[0080] In the embodiment of the present application, the external attitude estimation result is integrated with the comprehensive attitude estimation result to obtain a robust attitude estimation result. Thus, the external VIO can perform an inverse operation based on the robust attitude estimation result to obtain the external attitude acceleration, which can improve the accuracy of the external attitude acceleration estimation.

[0081] In one embodiment, sub-step 1023 may include the following sub-steps:

[0082] Sub-step 10231, taking the difference between the inertial navigation data and the external attitude acceleration as the corrected inertial navigation data.

[0083] It should be noted that the reading of the accelerometer in the IMU is usually considered to be the value of the total external force after gravity is removed. For example, in the northeast celestial coordinate system, when the XR device and the movable carrier are both stationary, the acceleration reading is g.

[0084] accreadou t =accsu m -(-g)=0+g=g(1)

[0085] Among them, ACCSU m is the acceleration of the XR device when both the XR device and the movable carrier are stationary, t is the actual accelerometer reading.

[0086] When the movable carrier accelerates sWhen the XR device moves with the movable carrier (i.e., is stationary relative to the movable carrier), the accelerometer reading should be:

[0087] accreadou t =accchassi s -(-g)=accchassi s +g(2)

[0088] Among them, accchassi s It is the acceleration of the movable carrier during movement.

[0089] When the movable carrier accelerates s Movement, and the XR device uses acceleration acchm d When in motion, the accelerometer readings should be:

[0090] accreadou t =accchassi s +acchm d +g(3)

[0091] Among them, acchm d The acceleration of the XR device during movement.

[0092] Obviously, in the scenarios corresponding to equations (2) and (3), if the general inertial navigation fusion formula (4) is still used for processing, the inversely derived XR device acceleration estimate will be Contains ingredients that can move a moving carrier.

[0093]

[0094] in, is an estimate of the acceleration of the XR device when the movable carrier moves with the acceleration accchassis.

[0095] Therefore, it is necessary to use the formula (3) as Figure 5 External attitude acceleration output by external VIO By performing a prior subtraction, we can obtain an estimate of the acceleration of the XR device itself, namely:

[0096]

[0097] in, External attitude acceleration estimate output by external VIO.

[0098] In the embodiment of the present application, the difference between the inertial navigation data and the external posture acceleration is used as the corrected inertial navigation data, which can eliminate the error caused by the movement of the movable carrier, so that the corrected inertial navigation data and the acceleration caused by the user drive can be consistent, and the virtual content can be tracked.

[0099] Step 103, perform attitude estimation on the corrected inertial navigation data and the visual image, and

[0100] The estimation results are fused to obtain posture data reflecting the user's posture.

[0101] In the embodiment of the present application, the visual image is the content of the picture in the image that is inside the movable carrier. Figure 5 ,After partitioning the image, feature extraction is performed to determine the ,features corresponding to the content inside the movable carrier, so as to perform ,intrinsic matching on it, and obtain the internal attitude ,estimation result through the internal odometer.

[0102] See also Figure 5 ,When the corrected inertial navigation data is performed attitude estimation, the corrected inertial navigation data is re-fused with the angular velocity measured by the gyroscope to reduce the drift error, and the internal attitude estimation result based on the corrected inertial navigation data is obtained.

[0103] In an embodiment of the present application, the corrected inertial navigation data and the visual image are respectively subjected to posture estimation, and the posture estimation results are fused to obtain posture data reflecting the user posture, which can improve the robustness of the user posture data.

[0104] In one embodiment, step 103 may include the following sub-steps:

[0105] Sub-step 1031, angular velocity data is measured by a gyroscope, and attitude estimation is performed based on the corrected inertial navigation data and the angular velocity data to obtain a first internal attitude estimation result.

[0106] See also Figure 5 , the corrected inertial navigation data is re-fused with the angular velocity measured by the gyroscope to reduce the drift error and obtain the first internal attitude estimation result.

[0107] Sub-step 1032, divides the visual image into multiple areas, determines a second area from the divided multiple areas, and matches the image features of the second area in each frame with the image features of the second area in the previous frame to obtain a second internal posture estimation result; the second area contains the picture content inside the movable carrier.

[0108] See also Figure 4, the image is divided into three areas, wherein the area 32 inside the vehicle is a second area inside the movable carrier, so that a second internal posture estimation is performed based on the content of the second area.

[0109] See also Figure 5 After partitioning the image, feature extraction is performed to determine the features corresponding to the content of the second area, so that the image features of the second area in each frame are matched with the image features of the second area in the previous frame to obtain the second internal posture estimation result. This method does not depend on the running surface of the movable carrier and can improve the accuracy of external posture estimation.

[0110] Sub-step 1033, fusing the first internal posture estimation result with the second internal posture estimation result to obtain posture data reflecting the user's posture.

[0111] In the embodiment of the present application, the first internal posture estimation result and the second internal posture estimation result are fused to obtain posture data reflecting the user's posture with robustness.

[0112] In one embodiment, sub-step 1032 may include the following sub-steps:

[0113] Sub-step 10321, obtaining a region of a key frame in the visual image.

[0114] The sub-step 10321 can refer to the above-mentioned sub-step 10211 and will not be elaborated here.

[0115] The embodiment of the present application can obtain key frames in an image and partition the key frames, so that the image other than the key frames can be partitioned based on the partitions of the key frames, and more image information can be extracted.

[0116] Sub-step 10322, partitioning the visual images other than the key frames by the regions of the key frames to obtain respective regions of each of the visual images.

[0117] The sub-step 10322 can refer to the above-mentioned sub-step 10212 and will not be elaborated here.

[0118] In the embodiment of the present application, by extracting key frames and partitioning the image using the partitions of the key frames, more information can be extracted from the image and the accuracy of the image partitioning can be improved.

[0119] In one embodiment, sub-step 10321 may include the following sub-steps:

[0120] Sub-step 103211, performing image segmentation on the key frame through a semantic segmentation model to obtain a partition outside the movable carrier, a partition inside the movable carrier, and other partitions except the partition outside the movable carrier and the partition inside the movable carrier corresponding to the key frame.

[0121] This sub-step 103211 can refer to the above-mentioned sub-step 102111 and will not be elaborated here.

[0122] In one embodiment, before sub-step 102111 and sub-step 103211, the method further includes:

[0123] Sub-step A1, obtaining images around the user at preset time intervals, and preprocessing the images to obtain preprocessed images.

[0124] In the embodiment of the present application, the image may be the original image of the perspective camera that is buffered and saved for a period of time when the XR device is started. The preprocessing of the image may be to count the brightness changes of the image and adjust the exposure parameters of the camera, such as frequency, duration, delay, etc. The embodiment of the present application preprocesses the acquired image to reduce the distortion rate of the lens and improve the stability of the image captured during the movement of the movable carrier.

[0125] The preset time interval may be a time interval set by the user, for example, 10 seconds. After the preset time interval of 10 seconds, the image around the user continues to be acquired and the preprocessing operation is performed, thereby adjusting the exposure parameters to improve the adaptability to the motion environment during the motion process.

[0126] Sub-step A2, extracting features from the preprocessed image to obtain features of the preprocessed image.

[0127] In the embodiment of the present application, the feature extraction of the preprocessed image can be performed by a neural network model. Figure 6The feature extraction model adopts the feature extraction part of the No Reference Image Quality Assessment in the Spatial Domain (BRISQUE). Specifically, the local normalized brightness coefficient (MSCN) is extracted from the preprocessed image. Since the MSCN coefficient has no strong dependence on the strength of the texture, the extracted features are more applicable. Then, the MSCN coefficient is fitted into a generalized Gaussian distribution (GGD) to obtain a two-dimensional feature vector. Since the structural information of the image is of great value, the statistical features between adjacent MSCN coefficients are used as the structural information of the image. In the horizontal direction, vertical direction, main diagonal direction and sub-diagonal direction, the adjacent MSCN coefficients are fitted into a generalized Gaussian distribution (AGGD) to obtain a four-dimensional vector respectively, and finally an eighteen-dimensional feature vector is obtained.

[0128] Sub-step A3, performing image quality assessment on the features of the preprocessed image through a neural network model to obtain a key frame.

[0129] For example, see Figure 6 Image quality assessment uses the basic framework of the BRISQUE algorithm, which is a two-stage model of "feature extraction" + "scoring network". In the feature extraction stage, the model of the feature extraction part of the BRISQUE algorithm is used. In the scoring stage, a small convolution-based multilayer perceptron (MLP) is used for scoring to obtain the quality score of the image quality.

[0130] Step 104: anchor the screen content of the extended reality device according to the posture data.

[0131] In the embodiment of the present application, the posture data is obtained by performing posture estimation on the corrected inertial navigation data and the visual image respectively, and fusing the posture estimation results of the two, and the corrected inertial navigation parameters eliminate the error caused by the movement of the movable carrier. Therefore, by anchoring the screen content of the extended display device with the user posture data, the posture estimation of the user when the carrier is in motion can be realized, thereby avoiding the state of virtual content tracking loss, and presenting the virtual content in the correct spatial position relationship, thereby improving the user experience.

[0132] In one embodiment, step 104 may include the following sub-steps:

[0133] Sub-step 1041, determining the position coordinates of the user in the coordinate system of the extended reality device according to the posture data.

[0134] In an embodiment of the present application, after obtaining the user's posture data, the user's coordinates in the extended reality device coordinate system can be determined based on the user's posture data, which is conducive to anchoring virtual content.

[0135] Sub-step 1042, determining the relative position relationship between the user and the extended reality device in the coordinate system of the extended reality device based on the position coordinates of the user.

[0136] In the embodiment of the present application, the relative position relationship can be that the extended reality device is at the position of the user in the extended reality device coordinate system, for example, 30 degrees diagonally below, 45 degrees diagonally above, etc. Thus, the screen content of the extended reality device can be displayed based on the relative position relationship.

[0137] Sub-step 1043, anchoring the screen content of the extended reality device according to the relative position relationship.

[0138] The embodiments of the present application can display the screen content of the extended reality device through relative position relationships, so that the virtual content can be presented with the correct spatial position relationship, thereby improving the user experience.

[0139] The embodiment of the present application corrects the inertial navigation data through the visual image. Since the error caused by the movement of the movable carrier can be eliminated in the corrected inertial navigation data, when the corrected inertial navigation data and the visual image are respectively estimated, the attitude estimation results of the two are consistent with each other, and by fusing the attitude estimation results of the two, more robust user attitude data can be obtained. Finally, by anchoring the screen content of the extended display device with the user attitude data, the attitude estimation of the user when the carrier is in motion can be realized, thereby avoiding the state of virtual content tracking loss, and can correctly present virtual content in spatial position relationship, thereby improving the user experience.

[0140] Combine the following Figure 4 and Figure 5 , the anchoring method of the screen content proposed in the embodiment of the present application is described in detail.

[0141] IMU measures the acceleration of the image through the accelerometer, and performs data filtering to remove the high-frequency noise in the accelerometer output. It also performs inertial navigation fusion with the angular velocity measured by the gyroscope to reduce the drift error. Finally, the inertial navigation-based attitude tracking result is obtained through the inertial navigation mileage module.

[0142] When the XR device is started, it will buffer and save the original images of the perspective camera for a period of time. It will achieve two main functions by counting the image characteristics: counting the changes in image brightness, adjusting the exposure parameters (frequency, duration, delay) and selecting key frames according to the image quality, which will be used as input for subsequent semantic segmentation. In the subsequent operation process, key frames will also be selected at settable intervals (i.e. Figure 5 The image quality assessment uses the basic framework of the BRISQUE algorithm, which is a two-stage model of "feature extraction" + "scoring network". The scoring network uses a small MLP based on a convolutional network for scoring. The key frame is fed into the AI ​​segmentation model, and the output of the model is a three-classification mask, corresponding to the three categories of "first area, second area, and uncertain" (i.e. Figure 5 b in the figure). To ensure a certain frame rate, AI segmentation models can generally select a bottom-up end-to-end method. The general structure of the network is: backbone feature network + decoder network, such as MobileNetV3+LRASPP. In this reference model, MobileNetV3 is the backbone feature network and LRASPP is the decoder structure. The three types of masks obtained above are applied to the images obtained by the tracking camera to partition the images (i.e. Figure 5 The first area corresponds to the image outside the movable carrier, and the second area corresponds to the image inside the movable carrier.

[0143] The feature extraction of the image outside the movable carrier is performed, and after the key points are obtained, they are matched with the key points of the image outside the movable carrier in the historical frame to confirm the corresponding relationship (i.e. Figure 5 The external odometer is used to obtain the attitude tracking result of the external movable carrier. The attitude tracking result based on the inertial navigation odometer is fused with the external VIO to obtain the estimated value of the external acceleration of the movable carrier.

[0144] The corrected inertial navigation acceleration (i.e. Figure 5 The f part in ) is then integrated with the angular velocity to obtain the user posture tracking result based on inertial navigation.

[0145] The image inside the movable carrier is feature extracted, and after the key points are obtained, they are matched with the key points of the image inside the movable carrier in the historical frame (i.e. Figure 5 The internal odometer is used to obtain the posture tracking result of the user inside the movable carrier. It is then combined with the user posture tracking result based on inertial navigation (i.e. Figure 5 g part), and realize the user's posture tracking through internal VIO (i.e. Figure 5After obtaining the user's posture tracking result, the relative position relationship between the virtual content and the user is locked, thereby achieving natural anchoring of terminal applications such as playback and browsing (i.e. Figure 5 (i part in the ).

[0146] The anchoring method of the picture content provided in the embodiment of the present application can be executed by a picture content anchoring device. In the embodiment of the present application, the anchoring method of the picture content performed by the picture content anchoring device is taken as an example to illustrate the picture content anchoring device provided in the embodiment of the present application.

[0147] Figure 7 : is a structural diagram of a screen content anchoring device provided in an embodiment of the present application. The device is located in an extended reality device, and a user wears the extended reality device in a movable carrier.

[0148] Reference Figure 7 , the apparatus 700 may include:

[0149] A first acquisition module 701 is used to acquire inertial navigation data and a visual image collected by the extended reality device; the inertial navigation data is data reflecting the acceleration of the extended reality device;

[0150] A first correction module 702, configured to correct the inertial navigation data using the visual image to obtain corrected inertial navigation data, so as to eliminate errors in the inertial navigation data caused by the movement of the movable carrier;

[0151] A first posture estimation module 703, used to perform posture estimation on the corrected inertial navigation data and the visual image respectively, and fuse the posture estimation results to obtain posture data reflecting the user's posture;

[0152] The first anchoring module 704 is used to anchor the screen content of the extended reality device according to the posture data.

[0153] Optionally, the first correction module includes:

[0154] A first division submodule is used to divide the visual image into a plurality of regions, and determine a first region from the plurality of divided regions; the first region includes picture content outside the movable carrier;

[0155] A first attitude estimation submodule, used for performing external attitude estimation on the content of the first area and the inertial navigation data respectively, and fusing the attitude estimation results to obtain the external attitude acceleration of the movable carrier;

[0156] The first correction submodule is used to correct the inertial navigation data by using the external attitude acceleration to obtain corrected inertial navigation data.

[0157] Optionally, a gyroscope is provided on the extended reality device; and the first posture estimation submodule includes:

[0158] A matching unit, used for matching the image features of the first area in each frame with the image features of the first area in the previous frame to obtain an external posture estimation result of the movable carrier;

[0159] an attitude estimation unit, configured to obtain angular velocity data measured by the gyroscope, perform attitude estimation based on the inertial navigation data and the angular velocity data, and obtain a comprehensive attitude estimation result;

[0160] A fusion unit is used to fuse the external attitude estimation result with the comprehensive attitude estimation result, and perform an inverse operation on the fusion result to obtain the external attitude acceleration.

[0161] Optionally, the first correction submodule includes:

[0162] The computing unit is used to use the difference between the inertial navigation data and the external attitude acceleration as the corrected inertial navigation data.

[0163] Optionally, the extended reality device is provided with a gyroscope, and the first posture estimation module includes:

[0164] A second correction submodule is used to measure angular velocity data through the gyroscope, perform attitude estimation based on the corrected inertial navigation data and the angular velocity data, and obtain a first internal attitude estimation result;

[0165] A second division submodule is used to divide the visual image into a plurality of regions, determine a second region from the divided plurality of regions, and match image features of the second region in each frame with image features of the second region in a previous frame to obtain a second internal posture estimation result; the second region contains picture content inside the movable carrier;

[0166] The fusion submodule is used to fuse the first internal posture estimation result with the second internal posture estimation result to obtain posture data reflecting the user posture.

[0167] Optionally, the first division submodule and the second division submodule both include:

[0168] An acquisition unit, used for acquiring a slice of a key frame in the visual image;

[0169] A partitioning unit is used to partition the visual image except the key frame by the slices of the key frame to obtain the slices of each visual image.

[0170] Optionally, the acquisition unit includes:

[0171] The segmentation subunit is used to perform image segmentation on the key frame through a semantic segmentation model to obtain the partition outside the movable carrier, the partition inside the movable carrier, and other partitions except the partition outside the movable carrier and the partition inside the movable carrier corresponding to the key frame.

[0172] Optionally, the first anchoring module includes:

[0173] A first determination submodule, configured to determine the position coordinates of the user in the coordinate system of the extended reality device according to the posture data;

[0174] A second determination submodule, configured to determine a relative position relationship between the user and the extended reality device in the coordinate system of the extended reality device based on the position coordinates of the user;

[0175] The display submodule is used to anchor the screen content of the extended reality device according to the relative position relationship.

[0176] The embodiment of the present application corrects the inertial navigation data through the visual image. Since the error caused by the movement of the movable carrier can be eliminated in the corrected inertial navigation data, when the corrected inertial navigation data and the visual image are respectively estimated, the attitude estimation results of the two are consistent with each other, and by fusing the attitude estimation results of the two, more robust user attitude data can be obtained. Finally, by anchoring the screen content of the extended display device with the user attitude data, the attitude estimation of the user when the carrier is in motion can be realized, thereby avoiding the state of virtual content tracking loss, and can correctly present virtual content in spatial position relationship, thereby improving the user experience.

[0177] The anchoring device of the screen content in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0178] The anchoring device of the screen content in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0179] The anchoring device for the screen content provided in the embodiment of the present application can achieve Figures 3 to 6 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0180] Alternatively, if Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the embodiment of the anchoring method of the above-mentioned screen content are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0181] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0182] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0183] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910 and other components.

[0184] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0185] The embodiment of the present application corrects the inertial navigation data through the visual image. Since the error caused by the movement of the movable carrier can be eliminated in the corrected inertial navigation data, when the corrected inertial navigation data and the visual image are respectively estimated, the attitude estimation results of the two are consistent with each other, and by fusing the attitude estimation results of the two, more robust user attitude data can be obtained. Finally, by anchoring the screen content of the extended display device with the user attitude data, the attitude estimation of the user when the carrier is in motion can be realized, thereby avoiding the state of virtual content tracking loss, and can correctly present virtual content in spatial position relationship, thereby improving the user experience.

[0186] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0187] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0188] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0189] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned embodiment of the anchoring method of the screen content is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0190] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0191] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned Huam content display method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0192] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0193] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned screen content anchoring method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0194] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0196] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for anchoring picture content, characterized in that: Applied to an extended reality device, a user wears the extended reality device in a movable carrier; the method comprises: Acquire inertial navigation data and a visual image collected by the extended reality device; the inertial navigation data is data reflecting the acceleration of the extended reality device; Correcting the inertial navigation data using the visual image to obtain corrected inertial navigation data; Performing posture estimation on the corrected inertial navigation data and the visual image respectively, and fusing the posture estimation results to obtain posture data reflecting the user's posture; Anchor the screen content of the extended reality device according to the posture data.

2. The method according to claim 1, characterized in that The method of correcting the inertial navigation data by using the visual image to obtain the corrected inertial navigation data includes: Dividing the visual image into a plurality of slices, and determining a first slice from the plurality of divided slices; the first slice contains picture content outside the movable carrier; Performing external attitude estimation on the content of the first area and the inertial navigation data respectively, and fusing the attitude estimation results to obtain the external attitude acceleration of the movable carrier; The inertial navigation data is corrected using the external attitude acceleration to obtain corrected inertial navigation data.

3. The method according to claim 2, characterized in that The extended reality device is provided with a gyroscope; the content of the first area and the inertial navigation data are respectively subjected to external attitude estimation, and the attitude estimation results are fused to obtain the external attitude acceleration of the movable carrier, including: Matching the image features of the first area in each frame with the image features of the first area in the previous frame to obtain an external posture estimation result of the movable carrier; Angular velocity data is measured by the gyroscope, and attitude estimation is performed based on the inertial navigation data and the angular velocity data to obtain a comprehensive attitude estimation result; The external attitude estimation result is fused with the comprehensive attitude estimation result, and the fusion result is inversely calculated to obtain the external attitude acceleration.

4. The method according to claim 2, characterized in that: The method of using the external attitude acceleration to correct the inertial navigation data to obtain the corrected inertial navigation data includes: The difference between the inertial navigation data and the external attitude acceleration is used as the corrected inertial navigation data.

5. The method according to claim 1, characterized in that The extended reality device is provided with a gyroscope, and the posture estimation is performed on the corrected inertial navigation data and the visual image respectively, and the posture estimation results are fused to obtain posture data reflecting the user posture, including: Angular velocity data is measured by the gyroscope, and attitude estimation is performed based on the corrected inertial navigation data and the angular velocity data to obtain a first internal attitude estimation result; Dividing the visual image into a plurality of regions, determining a second region from the divided plurality of regions, and matching image features of the second region in each frame with image features of the second region in a previous frame to obtain a second internal posture estimation result; the second region includes picture content inside the movable carrier; The first internal posture estimation result and the second internal posture estimation result are fused to obtain posture data reflecting the user's posture.

6. The method according to claim 2 or 5, characterized in that: The step of dividing the visual image into a plurality of regions comprises: Obtaining a slice of a key frame in the visual image; The visual image other than the key frame is partitioned by the slices of the key frame to obtain a slice of each visual image.

7. The method according to claim 6, characterized in that The step of obtaining a region of a key frame in the visual image includes: The key frame is subjected to image segmentation through a semantic segmentation model to obtain a partition outside the movable carrier, a partition inside the movable carrier, and other partitions except the partition outside the movable carrier and the partition inside the movable carrier corresponding to the key frame.

8. The method according to claim 1, characterized in that Anchoring the screen content of the extended reality device according to the posture data includes: Determine the position coordinates of the user in the coordinate system of the extended reality device according to the posture data; Determine the relative position relationship between the user and the extended reality device in the coordinate system of the extended reality device based on the position coordinates of the user; According to the relative position relationship, the screen content of the extended reality device is anchored.

9. A device for anchoring picture content, characterized in that: The device is located in an extended reality device, and a user wears the extended reality device in a movable carrier; the device includes: A first acquisition module, used to acquire inertial navigation data and a visual image collected by the extended reality device; the inertial navigation data is data reflecting the acceleration of the extended reality device; A first correction module, used for correcting the inertial navigation data using the visual image to obtain corrected inertial navigation data; A first posture estimation module, used to perform posture estimation on the corrected inertial navigation data and the visual image respectively, and fuse the posture estimation results to obtain posture data reflecting the user's posture; The first anchoring module is used to anchor the screen content of the extended reality device according to the posture data.

10. The device according to claim 9, characterized in that The first correction module comprises: A first division submodule is used to divide the visual image into a plurality of regions, and determine a first region from the divided plurality of regions; the first region includes picture content outside the movable carrier; A first attitude estimation submodule, used for performing external attitude estimation on the content of the first area and the inertial navigation data respectively, and fusing the attitude estimation results to obtain the external attitude acceleration of the movable carrier; The first correction submodule is used to correct the inertial navigation data by using the external attitude acceleration to obtain corrected inertial navigation data.

11. The device according to claim 10, characterized in that The extended reality device is provided with a gyroscope; the first posture estimation submodule includes: A matching unit, used for matching the image features of the first area in each frame with the image features of the first area in the previous frame to obtain an external posture estimation result of the movable carrier; an attitude estimation unit, configured to obtain angular velocity data measured by the gyroscope, perform attitude estimation based on the inertial navigation data and the angular velocity data, and obtain a comprehensive attitude estimation result; A fusion unit is used to fuse the external attitude estimation result with the comprehensive attitude estimation result, and perform an inverse operation on the fusion result to obtain the external attitude acceleration.

12. The device according to claim 10, characterized in that The first correction submodule comprises: The computing unit is used to use the difference between the inertial navigation data and the external attitude acceleration as the corrected inertial navigation data.

13. The device according to claim 9, characterized in that The extended reality device is provided with a gyroscope, and the first posture estimation module includes: A second correction submodule is used to measure angular velocity data through the gyroscope, perform attitude estimation based on the corrected inertial navigation data and the angular velocity data, and obtain a first internal attitude estimation result; A second division submodule is used to divide the visual image into a plurality of regions, determine a second region from the divided plurality of regions, and match image features of the second region in each frame with image features of the second region in a previous frame to obtain a second internal posture estimation result; the second region contains picture content inside the movable carrier; The fusion submodule is used to fuse the first internal posture estimation result with the second internal posture estimation result to obtain posture data reflecting the user posture.

14. The device according to claim 10 or 13, characterized in that The first division submodule and the second division submodule both include: An acquisition unit, used for acquiring a slice of a key frame in the visual image; A partitioning unit is used to partition the visual image except the key frame by the slices of the key frame to obtain the slices of each visual image.

15. The device according to claim 14, characterized in that The acquisition unit comprises: The segmentation subunit is used to perform image segmentation on the key frame through a semantic segmentation model to obtain the partition outside the movable carrier, the partition inside the movable carrier, and other partitions except the partition outside the movable carrier and the partition inside the movable carrier corresponding to the key frame.

16. The device according to claim 9, characterized in that The first anchoring module comprises: A first determination submodule, configured to determine the position coordinates of the user in the coordinate system of the extended reality device according to the posture data; A second determination submodule, configured to determine a relative position relationship between the user and the extended reality device in the coordinate system of the extended reality device based on the position coordinates of the user; The display submodule is used to anchor the screen content of the extended reality device according to the relative position relationship.

17. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for anchoring the picture content according to any one of claims 1 to 8 are implemented.

18. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the method for anchoring the picture content according to any one of claims 1 to 8 are implemented.

Images