Virtual-real fusion method, device, equipment and storage medium

Abstract

This application discloses a virtual-real fusion method, apparatus, device, and storage medium. The virtual-real fusion method includes: determining a first projection position of a virtual object in the current captured image of the target device based on a first positioning result of the target device before calibration, wherein the first projection position is determined as the display position of the virtual object in the current captured image; in response to obtaining a second positioning result of the target device after calibration, and the virtual object meeting a first preset condition, further determining the first projection position as the display position of the virtual object in the current captured image. This solution can reduce the adverse effects caused by the abrupt changes in the virtual object's position after the target device's positioning result is calibrated.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to a method, apparatus, device and storage medium for virtual-real fusion. Background Art

[0002] With the development of science and technology, augmented reality (AR) technology is becoming increasingly widespread. AR technology combines information from the real world and the virtual world, displaying virtual visual information within real-world images through devices.

[0003] During the virtual-real fusion process, cumulative errors occur during device positioning. Therefore, the device's position needs to be updated periodically to correct these accumulated errors. Due to these accumulated errors, the original virtual visual information will change abruptly on the device's screen after the position is updated. These periodic changes in virtual visual information can lead to a poor visual experience. Summary of the Invention

[0004] This application provides at least one method, apparatus, device, and storage medium for virtual-real fusion.

[0005] This application provides a virtual-real fusion method, comprising: determining a first projection position of a virtual object in the current shooting frame of the target device based on a first positioning result of the target device before calibration, wherein the first projection position is determined as the display position of the virtual object in the current shooting frame; in response to obtaining a second positioning result of the target device after calibration, and the virtual object meeting a first preset condition, further determining the first projection position as the display position of the virtual object in the current shooting frame.

[0006] Therefore, after obtaining the second positioning result of the target device after correction, in response to the virtual object meeting the first preset condition, the virtual object is kept displayed at the first projection position of the current shooting screen, which can reduce the visual impact caused by the virtual object changing its display position.

[0007] After determining the display location, the method further includes: displaying the currently captured image and displaying the virtual object at the display location in the currently captured image.

[0008] Therefore, by ensuring that the virtual object meets the first preset condition after the target device is calibrated, and keeping the virtual object displayed in the current shooting frame position, the visual impact caused by the change in display position can be reduced.

[0009] The first preset condition includes at least one of the following: the virtual object meets the second preset condition, the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirements, and the second preset condition includes the virtual object belonging to the first type and / or the first changed spatial position of the virtual object belonging to a reasonable position; wherein the first changed spatial position is the spatial position determined based on the second positioning result of the target device, and the first changed spatial position can be projected onto the spatial position of the first projection position.

[0010] Therefore, by considering the type of virtual object, the fact that the first changed spatial position of the virtual object is in a reasonable position, and the fact that the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirements, the visual impact caused by a certain deviation in the display position is smaller compared to the change in the display position of the virtual object.

[0011] Among them, the preset position requirements include that the position deviation is less than a preset threshold.

[0012] Therefore, if the positional deviation between the first changed spatial position of the virtual object and the original spatial position is less than a preset threshold, the deviation between the display positions of the two positions on the current shooting screen will not be large. Thus, keeping the display position of the virtual object unchanged has a smaller visual impact than changing the display position of the virtual object.

[0013] The preset position requirement also includes that the display change process of the virtual object is visible, wherein the display change process is the process by which the display of the virtual object changes from the first projection position to the target projection position.

[0014] Therefore, by keeping the display position of the virtual object unchanged while it remains visible during the display change process and the position deviation is less than a preset threshold, the former has a smaller visual impact than the latter.

[0015] The first preset condition includes that the virtual object meets the second preset condition, and the second preset condition includes that the first changed spatial position of the virtual object is a reasonable position; wherein, the reasonable position includes at least one of a non-dangerous area and a preset height range; and / or, before further determining the first projection position as the display position of the virtual object in the current shooting image, the method further includes: determining whether the first changed spatial position is a reasonable position based on the semantic information of the preset map.

[0016] Therefore, by combining the semantic information of the preset map, it is possible to determine whether the first changed spatial location is a reasonable location, thus making the determination of the reasonable location more accurate.

[0017] The method further includes: in response to obtaining the second positioning result of the target device after correction, and the virtual object does not meet the first preset condition, adopting a first correction strategy or a second correction strategy to change the display position of the virtual object from the first projection position to the target projection position.

[0018] Therefore, by changing the display position of the virtual object when the virtual object does not meet the first preset condition, the bad visual experience caused by the deviation of the virtual object's projection position can be reduced.

[0019] Specifically, in response to obtaining the second positioning result of the target device after correction, and the virtual object does not meet the first preset condition, the display position of the virtual object is changed from the first projection position to the target projection position using a first correction strategy or a second correction strategy, including: in response to the virtual object belonging to the second type, the display position of the virtual object is changed from the first projection position to the target projection position using the first correction strategy; in response to the virtual object belonging to the third type, the display position of the virtual object is changed from the first projection position to the target projection position using the second correction strategy; in response to the first change spatial position not being a reasonable position, the display position of the virtual object is changed from the first projection position to the target projection position using either the first correction strategy or the second correction strategy.

[0020] Therefore, by applying different correction strategies to different types of virtual objects, the flexibility of the correction strategy is improved.

[0021] The preset position requirements include that the display change process of the virtual object is visible and the position deviation is less than a preset threshold. The display change process is the process by which the display of the virtual object changes from a first projection position to a target projection position. In response to obtaining the second positioning result after correction of the target device, and the virtual object does not meet the first preset condition, the display position of the virtual object is changed from the first projection position to the target projection position using a first correction strategy or a second correction strategy. This includes: in response to the virtual object meeting the second preset condition, the display change process of the virtual object being visible, and the position deviation not less than the preset threshold, the display position of the virtual object is changed from the first projection position to the target projection position using a first correction strategy; and in response to the virtual object meeting the second preset condition, and the display change process of the virtual object being invisible, the display position of the virtual object is changed from the first projection position to the target projection position using a second correction strategy.

[0022] Therefore, by correcting the display position of virtual objects that meet the second preset condition but not the preset position requirement on the current shooting interface, the adverse effects caused by the projection deviation of virtual objects on the current shooting interface can be reduced.

[0023] Before using the first correction strategy or the second correction strategy to change the display position of the virtual object from the first projection position to the target projection position, the method further includes: determining the second projection position of the virtual object in the current shooting frame based on the second positioning result and the original spatial position, as the target projection position; or, if the first changed spatial position is not a reasonable position, adjusting the first changed spatial position to a second changed spatial position that is a reasonable position, and determining the third projection position of the virtual object in the current shooting frame based on the second positioning result and the second changed spatial position, as the target projection position.

[0024] Therefore, by determining the target projection position based on the second positioning result and the original spatial position, or by adjusting the second changed spatial position to a reasonable position based on the first changed spatial position and determining the target projection position based on the second changed spatial position, the adverse effects caused by projection errors are greatly reduced after the display position of the virtual object is corrected.

[0025] The first correction strategy is to render the virtual object directly to the target projection position in the next display rendering, and the second correction strategy is to move the virtual object from the first projection position to the target projection position at a preset speed.

[0026] Therefore, by setting different correction strategies, the correction process becomes more flexible.

[0027] The method further includes, in response to obtaining the corrected second positioning result of the target device and the virtual object meeting the first preset condition, and before determining the first projection position as the display position of the virtual object in the current shooting frame, the method further includes: using the first projection position, the projection parameters of the target device, and the first transformation parameters between the world coordinate system and the camera coordinate system of the target device to obtain the first changed spatial position in the world coordinate system, wherein the first transformation parameters are obtained based on the second positioning result of the target device.

[0028] Therefore, by determining the first changed spatial position of the virtual object using the second positioning result of the target device and the first projection position of the virtual object, it is possible to keep the projection position of the virtual object in the current shooting frame at the first changed spatial position unchanged.

[0029] The method of determining the first projection position of a virtual object in the current shooting frame of the target device based on the first positioning result of the target device before correction includes: obtaining the first projection position by using the original spatial position of the virtual object, the projection parameters of the target device, and the second transformation parameters between the world coordinate system and the camera coordinate system of the target device, wherein the second transformation parameters are obtained based on the first positioning result of the target device; in response to obtaining the second positioning result of the target device after correction and the virtual object meeting the first preset condition, before further determining the first projection position as the display position of the virtual object in the current shooting frame, the method further includes: performing positioning matching between the target device and a preset map to obtain the second positioning result of the target device in the preset map.

[0030] Therefore, by matching the target device with the preset map, a second positioning result of the target device in the preset map is obtained, which makes the second positioning result more accurate than the first positioning result before correction.

[0031] This application provides a virtual-real fusion device, comprising: a determining module, configured to determine a first projection position of a virtual object in the current captured image of the target device based on a first positioning result of the target device before calibration, wherein the first projection position is determined as the display position of the virtual object in the current captured image; and a fusion module, configured to, in response to obtaining a second positioning result of the target device after calibration and the virtual object meeting a first preset condition, further determine the first projection position as the display position of the virtual object in the current captured image.

[0032] This application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described virtual-real fusion method.

[0033] This application provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the above-described virtual-real fusion method.

[0034] The above solution, after obtaining the second positioning result of the target device after correction, responds to the virtual object meeting the first preset condition and keeps the virtual object displayed at the first projection position of the current shooting screen, which can reduce the visual impact caused by the virtual object changing its display position.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0037] Figure 1 This is a flowchart illustrating an embodiment of the virtual-real fusion method of this application;

[0038] Figure 2 This is a schematic diagram of a portion of the sub-process of step S12 in an embodiment of the virtual-real fusion method of this application;

[0039] Figure 3 This is another flowchart illustrating an embodiment of the virtual-real fusion method of this application;

[0040] Figure 4 This is a schematic diagram of the structure of an embodiment of the virtual-real fusion device of this application;

[0041] Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0042] Figure 6 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0043] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0044] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0045] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0046] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the virtual-real fusion method of this application.

[0047] Specifically, the virtual-real fusion method may include the following steps:

[0048] Step S11: Based on the first positioning result of the target device before correction, determine the first projection position of the virtual object in the current shooting screen of the target device, wherein the first projection position is determined as the display position of the virtual object in the current shooting screen.

[0049] Among them, the method of determining the first projection position of the virtual object in the current shooting screen of the target device based on the first positioning result of the target device before correction can be determined according to the first positioning result of the target device and the original empty position of the virtual object.

[0050] In this embodiment, the target device can be the execution device of the method provided in this disclosure, specifically a device with augmented reality functionality, such as AR glasses or a mobile phone. In other embodiments, the target device may not be the execution device of the method provided in this disclosure. Specifically, the target device may only perform the tasks of capturing environmental images and displaying a virtual-real fusion image. Specifically, after capturing the environmental image, the target device transmits the environmental image to the execution device. The execution device locates the target device to obtain a first positioning result before correction. Then, based on the first positioning result and the original spatial position of the virtual object, it determines the first projection position of the virtual object in the current captured image of the target device. Then, it merges the virtual object with the current captured image to obtain a fused image, and transmits the fused image to the target device so that the target device can display the fused image. This disclosure embodiment uses the target device as the execution device of the method provided in this disclosure embodiment as an example.

[0051] Virtual objects include visible display information or effects, and may also include invisible virtual objects used to prevent collisions, such as visible text, graphics, models, and cartoon characters. For example, virtual objects can be directional, navigational, points of interest, or virtual avatars.

[0052] The initial positioning result of the target device before calibration can be its position or pose in the world coordinate system. Pose includes both position and orientation. The world coordinate system is aligned with the virtual space where the virtual object resides.

[0053] Step S12: In response to obtaining the second positioning result of the target device after correction, and the virtual object meeting the first preset condition, the first projection position is further determined as the display position of the virtual object in the current shooting screen.

[0054] In virtual-real fusion, visual-inertial positioning or other 6-DOF tracking algorithms, which have low processing requirements, are generally used to obtain the target device's positioning result in the world coordinate system. However, these positioning methods may have positioning errors, so a certain amount of cumulative error is inevitable during continuous positioning. Therefore, it is necessary to periodically correct the target device's positioning result to reduce the cumulative error during the positioning process. The specific time for correcting the target device's positioning result can be set by the user or the factory setting of the target device can be used. For example, the first positioning result of the target device can be corrected every 30 seconds.

[0055] The first preset condition can be that the accuracy requirement for setting virtual objects is less than a preset accuracy requirement. For example, the accuracy requirement for some indicator-type or navigation-type virtual objects is relatively lenient compared to the accuracy requirement for label-type virtual objects. Indicator-type and navigation-type virtual objects can specifically be virtual arrows, while label-type virtual objects can specifically be descriptions of various products in a store (e.g., prices, product descriptions, etc.). The accuracy requirement for such indicator-type and navigation-type virtual objects can be set to be less than the preset accuracy requirement. The accuracy requirement for label-type virtual objects can be set to be greater than or equal to the preset accuracy requirement. Of course, in other embodiments, the first preset condition can be adjusted according to the allowable error requirement for setting virtual objects.

[0056] If the second positioning result of the target device after correction is obtained and the virtual object meets the first preset condition, the first projection position is still determined as the display position of the virtual object in the current shooting screen. This means that the display position of the virtual object in the current shooting screen does not change before and after the target device is corrected.

[0057] The above solution, after obtaining the second positioning result of the target device after correction, responds to the virtual object meeting the first preset condition and keeps the virtual object displayed at the first projection position of the current shooting screen, which can reduce the visual impact caused by the virtual object changing its display position.

[0058] In some disclosed embodiments, after determining the display position, the virtual-real fusion method further includes the following steps: displaying the currently captured image and displaying a virtual object at the first projection position of the currently captured image.

[0059] Specifically, the captured image and the virtual object are merged to obtain a merged image. The specific merging method involves projecting the virtual object onto the first projection position of the current captured image. Then, the merged image is displayed on the target device's display interface, thus displaying the current captured image and the virtual object at the first projection position of the current captured image. That is, after determining the first projection position of the virtual object on the target device's current captured image based on the first positioning result of the parent device, and defining this first projection position as the display position of the virtual object in the current captured image, the virtual object is displayed at the display position of the current captured image, and the current captured image is also displayed. After executing step S12, the first projection position is again defined as the display position of the virtual object in the current captured image, and the current captured image continues to be displayed, with the virtual object continuing to be displayed at the display position of the current captured image.

[0060] Once the display position of the virtual object in the current shooting frame is initially determined, the virtual object can be continuously displayed at that determined position, and the display position of the virtual object can be updated during the display process. For example, before the positioning result of the target device is corrected, the display position of the virtual object in the current shooting frame is the first projection position. After the positioning result of the target device is corrected, and if the virtual object meets a first preset condition, the display position of the virtual object in the current shooting frame remains unchanged. If the virtual object does not meet the first preset condition, the display position of the virtual object in the current shooting frame may change, and the virtual object will continue to be displayed at the changed display position.

[0061] In some disclosed embodiments, step S11 includes: obtaining a first projection position using the original spatial position of the virtual object, the projection parameters of the target device, and the second transformation parameters between the world coordinate system and the camera coordinate system of the target device.

[0062] The original spatial position of the virtual object is preset, referring to its position in the virtual space. The second transformation parameter between the world coordinate system and the camera coordinate system of the target device is specifically the view matrix of the target device. Specifically, this second transformation parameter is the first and second transformation parameters for transforming from the world coordinate system to the camera coordinate system of the target device. The first positioning result of the target device can be its pose in the world coordinate system. The pose of the target device includes its position and orientation (also known as facing). In this embodiment, the pose of the target device's imaging component in the world coordinate system is equated to the target device's pose in the world coordinate system. Of course, in other embodiments, the pose of a preset component on the target device can be used as the target device's pose. Then, based on the relative pose relationship between the imaging component on the target device and a preset position on the target device, and the pose of the preset component, the pose of the imaging component is determined. The position of the imaging component can be considered as the origin of the camera coordinate system. For details on how to determine the transformation relationship between the world coordinate system and the camera coordinate system of the target device based on the pose of the imaging component, please refer to generally known techniques, which will not be elaborated here.

[0063] The projection parameters are in the form of a projection matrix. These projection parameters are internal parameters of the camera component and are mainly constructed based on the projection principle. They are used to realize the transformation from three-dimensional spatial coordinates to two-dimensional image coordinates.

[0064] Specifically, the second transformation parameter is used to determine the three-dimensional spatial coordinates of the virtual object in the camera coordinate system of the target device based on the virtual object's original spatial position, and the projection parameter is used to determine the first projection position of the virtual object in the current shooting frame based on the virtual object's three-dimensional spatial coordinates in the camera coordinate system. By matching the target device with a preset map, a second positioning result of the target device in the preset map is obtained, making the second positioning result more accurate than the first positioning result before correction.

[0065] Prior to step S12, the following steps are also included:

[0066] The target device is matched with a preset map to obtain a second positioning result of the target device in the preset map. The preset map can be a high-precision 3D map, such as a dense 3D map. Matching the target device with the preset map can also be referred to as obtaining a more accurate pose of the target device in the preset map. Specifically, the method of matching the target device with the preset map can be to obtain the second positioning result of the target device in the preset map based on a precise positioning component in the target device. This precise positioning component can be GPS. Of course, the method of obtaining a more accurate positioning result of the target device in the preset map can also be to use visual inertial positioning. Feature points of the currently captured image are extracted and matched with feature points in the preset map, and a second positioning result is obtained based on the matching result. In other disclosed embodiments, the currently captured image can be sent to the cloud so that the cloud can extract feature points of the currently captured image and match the extracted feature points with feature points in the preset map to obtain the second positioning result of the target device, and then receive the second positioning result sent by the cloud.

[0067] In other disclosed embodiments, the following steps may be included before step S12:

[0068] The system detects whether there is a preset change between the second positioning result and the first positioning result after the target device has been calibrated. If a preset change exists between the second positioning result and the first positioning result, step S12 is executed.

[0069] The preset variation can be defined as the error between the first and second positioning results being greater than or equal to a preset error. The preset error can be determined based on the user's requirements for the accuracy of virtual-real fusion. If higher accuracy is required, the preset error can be set to be smaller, approaching 0; otherwise, the preset error can be set to be larger.

[0070] In some disclosed embodiments, the first preset condition includes at least one of the following: the virtual object meets the second preset condition, and the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets a preset positional requirement. In some application scenarios, the first preset condition includes the virtual object meeting the second preset condition. In other application scenarios, the first preset condition includes the positional deviation between the first changed spatial position and the original spatial position of the virtual object meeting a preset positional requirement. In still other application scenarios, the first preset condition includes the virtual object meeting the second preset condition, and the positional deviation between the first changed spatial position and the original spatial position of the virtual object meeting a preset positional requirement.

[0071] The second preset condition includes that the virtual object belongs to the first type and / or the first changed spatial position of the virtual object is a reasonable position. The first changed spatial position is determined based on the second positioning result of the target device, and the first changed spatial position can be projected onto the first projection position. That is, when the target device is located at the second positioning result, the virtual object in the first changed space can be projected onto the first projection position of the currently captured image. Specifically, before executing step S12, the method for obtaining the first changed spatial position of the virtual object can be:

[0072] Using the first projection position, the projection parameters of the target device, and the first transformation parameters between the world coordinate system and the camera coordinate system of the target device, the first changed spatial position in the world coordinate system is obtained. The first transformation parameters are obtained based on the second positioning result of the target device. Specifically, the method for obtaining the first transformation parameters is the same as the method for obtaining the second transformation parameters described above, and will not be elaborated further here.

[0073] By using projection parameters, a virtual object can be back-projected from its first projection position in the current captured image to the camera coordinate system, obtaining the virtual object's three-dimensional coordinates in the camera coordinate system. Then, based on the first transformation parameters, the virtual object can be transformed from the camera coordinate system to the world coordinate system, obtaining the virtual object's first changed spatial position in the world coordinate system. By determining the first changed spatial position of the virtual object using the target device's second positioning result and the virtual object's first projection position, the projected position of the virtual object in the current captured image at the first changed spatial position can be kept unchanged.

[0074] In some disclosed embodiments, the second preset condition includes that the virtual object belongs to the first type. Optionally, an instruction from the user to classify virtual objects is received, and the virtual objects are divided into several types, such as three types. The fusion accuracy corresponding to each type of virtual object is different. The classification principle of virtual objects can be determined based on the balance between the visual impact caused by the virtual object's jump and the impact caused by the virtual object's projection error. For example, if the projection error of a virtual object is large, but the impact of its projection error can be ignored if it is not jumped, it can be classified as the first type. If the visual impact of a virtual object's jump is less than the impact caused by its projection error, it can be classified as the second type. If the visual impact of a virtual object's jump is greater than the impact caused by its projection error, it can be classified as the third type.

[0075] In some disclosed embodiments, the second preset condition includes that the first changed spatial position of the virtual object is a reasonable position. Whether the first changed spatial position of the virtual object is a reasonable position can be determined based on the semantic information at the first changed spatial position in the preset map. Optionally, a reasonable position includes at least one of a non-dangerous area and a position within a preset height range. Specifically, dangerous areas can be pre-marked on the preset map. Areas other than dangerous areas can be considered non-dangerous areas. For example, in a city navigation scenario, the virtual object is a navigation arrow, and the dangerous area could be a lake located beside a road. The preset height range can be set according to the requirements for projection accuracy. To better understand the preset height range, the following specific application example is shown. If the preset map is a map of a room, the virtual object is a vase, and the original spatial position of the virtual object is on the surface of a table in the room. If the first changed spatial position of the virtual object is below the floor of the room, this is obviously illogical; at least the base of the vase should be above the floor of the room, that is, the height of the vase's base should be higher than the floor height. The preset height range is the maximum and / or minimum height allowed in the vertical direction of the first changed spatial position in the preset map. By combining the semantic information of the preset map, it is determined whether the first changed spatial location is a reasonable location, making the determination of reasonable locations more accurate.

[0076] In some disclosed embodiments, the second preset condition includes that the virtual object belongs to the first type and that the first changed spatial position of the virtual object is a reasonable position. That is, in response to the virtual object belonging to the first type and the first changed spatial position of the virtual object being a reasonable position, the virtual object is kept displayed at the first projection position of the currently captured image.

[0077] In some disclosed embodiments, the first preset condition includes that the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets a preset position requirement. The positional deviation between the first changed spatial position and the original spatial position may or may not include deviation in the height direction. For example, in some application scenarios, the positional deviation may only include the deviation between the two positions in a plane perpendicular to the height direction. Specifically, it may be the distance deviation in that plane. The preset position requirement may include that the positional deviation is less than a preset threshold. That is, the deviation between the original spatial position and the first changed spatial position is small. The impact of this positional deviation is less than the impact of a jump in the virtual object's display position in the current captured image. The preset threshold can be user-defined or set to the default settings of the target device. Because the positional deviation between the first changed spatial position and the original spatial position of the virtual object is less than the preset threshold, the deviation between the displayed positions of the two positions in the current captured image will not be large. Therefore, keeping the virtual object's display position unchanged results in a smaller visual impact compared to a jump in the virtual object's display position.

[0078] Optionally, depending on whether the display change process of the virtual object is visible or invisible, the preset position requirement may also include that the display change process of the virtual object is visible. That is, the display change process of the virtual object is visible and the position deviation is less than a preset threshold. The display change process of the virtual object refers to the process by which the virtual object changes from a first position to a target projection position for display. Specifically, a joint bounding box formed by the virtual object's original spatial position and the first changed spatial position is obtained. This joint bounding box is a three-dimensional bounding box, which can be considered as a cuboid that encloses the virtual object located at the original spatial position and the first changed position. The process of the virtual object changing in the current captured image can be specifically considered as the projection of the virtual object onto the current display interface during its movement from the first changed position to the original spatial position. In the current captured image, this is mainly reflected in the interpolation process during the movement of the virtual object from the first projection position to the target projection position. The target projection position refers to the projection position of the virtual object in the current captured image obtained based on the virtual object's original spatial position, the aforementioned first transformation parameters, and the projection parameters. "Display change process invisible" means that the interpolation process is not visible in the current captured image, while "display change process visible" means that the interpolation process is visible in the current captured image. Whether the interpolation process is visible can be determined by whether there is a target projection position. Optionally, if there is no target projection position corresponding to the virtual object in the current captured image, the interpolation process is considered invisible. Specifically, if the target projection position is not located in the current captured image, that is, the target device cannot observe the virtual object at all based on the corrected position, the virtual object cannot obtain its projection position in the current captured image based on its original spatial position and the second transformation parameters and projection parameters corresponding to the target device. In this case, the projection of the virtual object during its movement within the bounding box cannot be reflected in the current captured image. Alternatively, whether the interpolation process is visible can be determined by whether the interpolation process is complete. Optionally, if the interpolation process is complete, it is considered visible; if the interpolation process is incomplete, it is considered invisible. For example, during the movement of the virtual object within the bounding box, the target device cannot observe the middle part of the position, and therefore it cannot be displayed in the current captured image, so the interpolation at that position is considered invisible.

[0079] In some application scenarios, if the display change process of a virtual object is not visible, it can be assumed that the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset positional requirements. In other application scenarios, precisely because the display change process of a virtual object is not visible, even if the virtual object changes abruptly, the impact on visual perception is relatively small, and the projection position of the virtual object can be corrected to reduce the impact of positional deviation.

[0080] In some disclosed embodiments, the first preset condition includes the virtual object meeting a second preset condition, and the positional deviation between the virtual object's first changed spatial position and its original spatial position meeting a preset positional requirement. Please also refer to... Figure 2 , Figure 2 This diagram illustrates a portion of a sub-process of step S12 in an embodiment of the virtual-real fusion method of this application. Figure 2 As shown, step S12 above includes the following steps:

[0081] Step S121: Determine whether the virtual object belongs to the first type.

[0082] Specifically, the type of virtual object is determined based on pre-set parameter information. This pre-set parameter information includes type information corresponding to at least one virtual object. This pre-set parameter information can be customized by the user. As described above, it will not be repeated here.

[0083] If the virtual object does not belong to the first type, proceed to step S122; if the virtual object belongs to the first type, proceed to step S123.

[0084] Step S122: Determine that the virtual object does not meet the first preset condition.

[0085] Step S123: Determine whether the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset positional requirements.

[0086] The specific requirements for the preset position are detailed above and will not be repeated here. If the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirements, proceed to step S124; otherwise, proceed to step S122.

[0087] Step S124: Determine whether the first change in spatial location of the virtual object is a reasonable location.

[0088] Please refer to the above definition for a reasonable location. If the first changed spatial location falls within the range of reasonable locations, the first changed spatial location of the virtual object is considered a reasonable location. If the first changed spatial location of the virtual object does not fall within a reasonable location, the first changed spatial location of the virtual object is considered a reasonable location. If the first space of the virtual object does not fall within a reasonable location, proceed to step S122. If the first space of the virtual object falls within a reasonable location, proceed to step S125.

[0089] Step S125: Determine that the virtual object meets the first preset condition.

[0090] The execution order of steps S123 and S124 can be interchanged, and the execution is not strictly in accordance with the above steps.

[0091] By considering the type of virtual object, the fact that the first changed spatial position of the virtual object is in a reasonable position, and the fact that the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirements, the visual impact caused by a certain deviation in the display position is smaller compared to the change in the display position of the virtual object.

[0092] In some disclosed embodiments, the virtual-real fusion method may further include the following steps:

[0093] In response to obtaining the second, calibrated positioning result of the target device, and if the virtual object does not meet the first preset condition, a first correction strategy or a second correction strategy is used to change the display position of the virtual object from the first projection position to the target projection position. The target projection position can be obtained from the original spatial position of the virtual object and the aforementioned first transformation parameters and projection parameters, or it can be determined based on the setting parameters of the target device. The determination based on the setting parameters of the target device can be based on the reasonableness of the first change in the virtual object's spatial position. By changing the display position of the virtual object when it does not meet the first preset condition, the unpleasant visual experience caused by the deviation in the virtual object's projection position can be reduced.

[0094] Specifically, before using the first correction strategy or the second correction strategy to change the display position of the virtual object from the first projection position to the target projection position, the target projection position is first obtained. There are several ways to obtain the target projection position:

[0095] The first method involves determining the second projection position of the virtual object in the current captured image based on the second positioning result and the original spatial position, using this as the target projection position. The specific process for determining the second projection position of the virtual object in the current captured image based on the second positioning result and the original spatial position is as follows: Based on the second positioning result of the target device, obtain the first transformation parameters between the camera coordinate system and the world coordinate system of the target device. The specific method for obtaining the first transformation parameters is described above and will not be repeated here. Then, use the first transformation parameters to transform the original spatial position of the virtual object to the camera coordinate system of the target device, and then use the projection parameters to project the virtual object in the camera coordinate system onto the current captured image, obtaining the second projection position of the virtual object in the current captured image. This second projection position of the virtual object in the current captured image is then used as the target projection position. A projection position at a preset distance from this second projection position can also be used as the target projection position. For example, a projection position at a distance of 2 pixels from the second projection position can also be used as the target projection position.

[0096] The second approach is to adjust the first changed spatial position to a second changed spatial position that is considered reasonable if the first changed spatial position is not a reasonable position. Based on the second positioning result and the second changed spatial position, the third projection position of the virtual object in the current shooting frame is determined as the target projection position. The method to adjust the first changed spatial position to a second changed spatial position that is considered reasonable can be to obtain the relative position between the first changed spatial position and the reasonable position, and then adjust the first changed spatial position based on this relative position. Continuing the previous example, the bottom of the virtual vase should ideally be above the ground. However, when the vase is in the first changed spatial position, the bottom of the vase is 0.5m below the ground, meaning the relative position between the vase and the reasonable position is 0.5m. Therefore, the vase should be moved upwards by at least 0.5m from the first changed position so that the bottom of the vase is in contact with or above the ground. This yields the second changed spatial position where the virtual object is considered reasonable. Then, based on the second positioning result of the target device, the first transformation parameters between the camera coordinate system and the world coordinate system of the target device are obtained. The specific method for obtaining the first transformation parameters is described above and will not be repeated here. Then, the second spatial position of the virtual object is transformed to the camera coordinate system of the target device using the first transformation parameters. Next, the virtual object in the camera coordinate system is projected onto the current shooting frame using projection parameters, obtaining the second projected position of the virtual object in the current shooting frame. This second projected position is then used as the target projection position. This method allows for adjustment of the alignment between the virtual object and the shooting frame. By determining the target projection position based on the second positioning result and the original spatial position, or by adjusting the second spatial position to a reasonable position based on the first spatial position and determining the target projection position based on the second spatial position, the adverse effects caused by projection errors are effectively reduced after correcting the display position of the virtual object.

[0097] In some disclosed embodiments, the first correction strategy is to directly render the virtual object to the target projection position in the next display rendering. That is, the target projection position of the virtual object on the target device is determined directly based on the position of the virtual object in its original spatial position, and then the virtual object is displayed at the target projection position on the target device. The second correction strategy is to move the virtual object from the first projection position to the target projection position at a preset speed. Specifically, the first correction strategy can be considered as immediately changing the display position of the virtual object from the first projection position to the target projection position, without displaying the interpolation process between the first projection position and the target projection position, while the second correction strategy can be considered as gradually moving the display position of the virtual object from the first projection position to the target projection position. Specifically, the second correction strategy is to move the virtual object from the first changed spatial position to the original spatial position or the corresponding second changed spatial position at a uniform speed, and sequentially determine the corresponding projection position based on the current position of the virtual object, and display it at the corresponding projection position, so as to achieve moving the virtual object from the first projection position to the target projection position for display at a preset speed. Similarly, the first correction strategy is to directly switch the virtual object's first modified spatial position back to its original spatial position or second modified spatial position, so that the virtual object can be rendered directly to the target projection position in the next display rendering. By setting different correction strategies, the correction process becomes more flexible.

[0098] Specifically, in response to the virtual object belonging to the second type, a first correction strategy is used to change the display position of the virtual object from the first projection position to the target projection position. Alternatively, in response to the virtual object belonging to the third type, a second correction strategy is used to change the display position of the virtual object from the second projection position to the target projection position. In response to the first change in spatial position being unreasonable, either the first or second correction strategy is selected to change the display position of the virtual object from the first projection position to the target projection position. Specifically, even if the virtual object does not have high requirements for projection accuracy and belongs to the first type, if the first change in spatial position of the virtual object is not reasonable, either the first or second correction strategy can be selected to change the display position of the virtual object from the first projection position to the target projection position. As mentioned above, the classification of virtual object types can be determined based on the projection accuracy requirements corresponding to the virtual object. In this case, the second type has higher requirements for projection position accuracy than the third type, and the third type has higher requirements for projection position accuracy than the first type. That is, for the second type, if a correction strategy is not implemented, it will result in a worse user experience, so a correction strategy needs to be implemented immediately. For the first type, the projection deviation of the virtual object in the current shooting frame has little impact on the virtual-real fusion. In certain situations, implementing a correction strategy might even cause abrupt changes, affecting the user experience. Therefore, it's acceptable not to correct the deviation. These "certain situations" refer to the virtual object meeting a first preset condition. For the third type, the impact of the projection deviation of the virtual object in the current shooting frame is not significantly different from the abrupt changes caused by implementing a correction strategy. Therefore, a correction strategy can be implemented for this type. By applying different correction strategies to different types of virtual objects, the flexibility of the correction strategy is improved.

[0099] In some disclosed embodiments, when the preset position requirements include the visibility of the virtual object's display change process and the position deviation being less than a preset threshold, in response to the obtained corrected second positioning result of the target device, and the virtual object does not meet the first preset condition, the method of changing the display position of the virtual object from the first projection position to the target projection position using a first correction strategy or a second correction strategy may be as follows:

[0100] In response to the virtual object meeting the second preset conditions, the display change process of the virtual object being visible, and the position deviation not less than a preset threshold, a first correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position. Here, the position deviation is the positional deviation between the virtual object's original spatial position and the first changed spatial position. That is, if the virtual object belongs to the first type and / or the first changed spatial position of the virtual object is a reasonable position, the display change process of the virtual object is visible, and the positional deviation between the virtual object's original spatial position and the first changed spatial position is not less than the preset threshold, then the first correction strategy is adopted to change the display position of the virtual object.

[0101] In response to a virtual object meeting a second preset condition and the display change process of the virtual object being invisible, a second correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position. Here, the position deviation is the position deviation between the virtual object's original spatial position and the first changed spatial position. That is, if the virtual object belongs to the first type and / or the first changed spatial position of the virtual object is a reasonable position, and the display change process of the virtual object is invisible, then the second correction strategy is adopted to change the display position of the virtual object. In other disclosed embodiments, in response to a virtual object meeting the second preset condition and the display change process of the virtual object being invisible, the first correction strategy can also be selected to change the display position of the virtual object. By correcting the display position of virtual objects that meet the second preset condition but do not meet the preset position requirements on the current shooting interface, the adverse effects caused by the projection deviation of the virtual object on the current shooting interface can be reduced.

[0102] In some disclosed embodiments, in response to the virtual object meeting the first preset condition, the way to keep the virtual object displayed on the first projection position of the current shooting screen can be to keep the screen displayed by the target device unchanged. Another way is to obtain the first changed spatial position of the virtual object according to the above method, and then obtain the projection position of the virtual object on the current shooting screen based on the first changed spatial position of the virtual object and the first transformation parameters and projection parameters between the camera coordinate system and the world coordinate system of the target device, and use it as the target projection position, thereby realizing the virtual object being kept displayed on the first projection position of the current shooting screen.

[0103] To better understand the above scheme, please refer to the following example. Please also refer to... Figure 3 , Figure 3 This is another schematic flowchart of an embodiment of the virtual-real fusion method of this application. The virtual-real fusion method provided in this disclosure includes the following steps:

[0104] Step S21: Based on the first positioning result of the target device before correction, determine the first projection position of the virtual object in the current shooting screen of the target device.

[0105] The specific determination method is as described above and will not be repeated here.

[0106] Step S22: Display the current shooting screen and display the virtual object at the first projection position of the current shooting screen.

[0107] Step S23: Obtain the second positioning result of the target device after calibration.

[0108] The method for obtaining the corrected second positioning result of the target device is as described above and will not be repeated here. This second positioning result can be used to determine the target projection position of the virtual object.

[0109] Step S24: Determine whether the virtual object meets the first preset condition.

[0110] The first preset condition is as described above. If the virtual object meets the first preset condition, step S25 is executed; otherwise, the corresponding steps S26, S27, or S28 are executed.

[0111] Step S25: In response to the virtual object meeting the first preset condition, keep the virtual object displayed at the first projection position in the current shooting screen.

[0112] The method of keeping the virtual object displayed at the first projection position in the current shooting screen is as described above, and will not be repeated here.

[0113] Step S26: In response to the virtual object belonging to the second type, the first correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position.

[0114] The method of changing the display position of the virtual object from the first projection position to the target projection position using the first correction strategy is as described above and will not be repeated here.

[0115] Step S27: In response to the virtual object belonging to the third type, the second correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position.

[0116] The method of changing the display position of the virtual object from the first projection position to the target projection position using the second correction strategy is as described above and will not be repeated here.

[0117] Step S28: In response to the first change of spatial position being an unreasonable position, select a first correction strategy or a second correction strategy to change the display position of the virtual object from the first projection position to the target projection position.

[0118] Specifically, in response to the virtual object belonging to the first type and the virtual object not meeting the first preset condition, a first correction strategy or a second correction strategy is selected to change the display position of the virtual object from the first projection position to the target projection position.

[0119] The above solution, after obtaining the second positioning result of the target device after correction, responds to the virtual object meeting the first preset condition and keeps the virtual object displayed at the first projection position of the current shooting screen, which can reduce the visual impact caused by the virtual object changing its display position.

[0120] The entity executing the virtual-real fusion method can be a virtual-real fusion device. For example, the virtual-real fusion method can be executed by a terminal device, a server, or other processing devices. The terminal device can be an augmented reality (AR) device, such as augmented reality or virtual reality glasses, helmets, etc., user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, and autonomous vehicles, robots, etc. In some possible implementations, the virtual-real fusion method can be implemented by a processor calling computer-readable instructions stored in memory.

[0121] Please see Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the virtual-real fusion device of this application. The virtual-real fusion device 40 includes a determining module 41 and a fusion module 42. The determining module 41 is used to determine a first projection position of a virtual object in the current shooting frame of the target device based on a first positioning result of the target device before correction, wherein the first projection position is determined as the display position of the virtual object in the current shooting frame; the fusion module 42 is used to, in response to obtaining a second positioning result of the target device after correction, and the virtual object meeting a first preset condition, continue to determine the first projection position as the display position of the virtual object in the current shooting frame.

[0122] The above solution, after obtaining the second positioning result of the target device after correction, responds to the virtual object meeting the first preset condition and keeps the virtual object displayed at the first projection position of the current shooting screen, which can reduce the visual impact caused by the virtual object changing its display position.

[0123] In some disclosed embodiments, after determining the display position, the fusion module 42 is further configured to: display the currently captured image and display the virtual object at the display position in the currently captured image.

[0124] The above solution reduces the visual impact caused by changes in display position by ensuring that the virtual object meets the first preset condition after the target device is calibrated.

[0125] In some disclosed embodiments, the first preset condition includes at least one of the following: the virtual object meets the second preset condition, the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirements, and the second preset condition includes the virtual object belonging to a first type and / or the first changed spatial position of the virtual object belonging to a reasonable position; wherein, the first changed spatial position is a spatial position determined based on the second positioning result of the target device, and the first changed spatial position can be projected onto the spatial position of the first projection position.

[0126] The above solution, by ensuring that the type of virtual object, the first changed spatial position of the virtual object is in a reasonable position, and the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirements, has a smaller visual impact than a jump in the display position of the virtual object.

[0127] In some disclosed embodiments, the preset position requirement includes a position deviation that is less than a preset threshold.

[0128] The above solution ensures that the positional deviation between the first changed spatial position of the virtual object and its original spatial position is less than a preset threshold. Therefore, the deviation between the display positions of the two positions on the current shooting screen will not be large. Thus, keeping the display position of the virtual object unchanged has a smaller visual impact than keeping the display position of the virtual object changed.

[0129] In some disclosed embodiments, the first preset condition includes that the virtual object meets the second preset condition, and the second preset condition includes that the first changed spatial position of the virtual object is a reasonable position; wherein, the reasonable position includes at least one of a non-dangerous area and a preset height range; and / or, before the first projection position is determined as the display position of the virtual object in the current shooting image in response to the virtual object meeting the first preset condition, the fusion module 42 is further configured to: determine whether the first changed spatial position is a reasonable position based on the semantic information of the preset map.

[0130] The above scheme, by combining the semantic information of the preset map, determines whether the first changed spatial location is a reasonable location, making the determination of reasonable locations more accurate.

[0131] In some disclosed embodiments, the fusion module 42 is further configured to: in response to obtaining the second positioning result of the target device after correction, and the virtual object does not meet the first preset condition, adopt the first correction strategy or the second correction strategy to change the display position of the virtual object from the first projection position to the target projection position.

[0132] The above solution reduces the unpleasant visual experience caused by the deviation in the projection position of virtual objects by changing the display position of virtual objects when the virtual objects do not meet the first preset condition.

[0133] In some disclosed embodiments, in response to obtaining the corrected second positioning result of the target device and the virtual object not meeting the first preset condition, the fusion module 42 adopts a first correction strategy or a second correction strategy to change the display position of the virtual object from the first projection position to the target projection position, including: in response to the virtual object belonging to the second type, adopting the first correction strategy to change the display position of the virtual object from the first projection position to the target projection position; in response to the virtual object belonging to the third type, adopting the second correction strategy to change the display position of the virtual object from the first projection position to the target projection position; in response to the first change spatial position not being a reasonable position, selecting the first correction strategy or the second correction strategy to change the display position of the virtual object from the first projection position to the target projection position.

[0134] The above scheme improves the flexibility of correction strategies by applying different correction strategies to different types of virtual objects.

[0135] In some disclosed embodiments, the preset position requirements include that the display change process of the virtual object is visible and the position deviation is less than a preset threshold. The display change process is the process by which the display of the virtual object changes from a first projection position to a target projection position. In response to obtaining the corrected second positioning result of the target device and the virtual object not meeting the first preset condition, the fusion module 42 uses a first correction strategy or a second correction strategy to change the display position of the virtual object from the first projection position to the target projection position. This includes: in response to the virtual object meeting the second preset condition, the display change process of the virtual object being visible, and the position deviation not less than the preset threshold, using the first correction strategy to change the display position of the virtual object from the first projection position to the target projection position; and in response to the virtual object meeting the second preset condition and the display change process of the virtual object being invisible, using the second correction strategy to change the display position of the virtual object from the first projection position to the target projection position.

[0136] The above solution corrects the display position of virtual objects that meet the second preset condition but not the preset position requirement on the current shooting interface, thereby reducing the adverse effects caused by the projection deviation of virtual objects on the current shooting interface.

[0137] In some disclosed embodiments, before changing the display position of the virtual object from the first projection position to the target projection position using the first correction strategy or the second correction strategy, the fusion module 42 further includes: determining the second projection position of the virtual object in the current shooting frame based on the second positioning result and the original spatial position, as the target projection position; or, if the first changed spatial position is not a reasonable position, adjusting the first changed spatial position to a second changed spatial position that is a reasonable position, and determining the third projection position of the virtual object in the current shooting frame based on the second positioning result and the second changed spatial position, as the target projection position.

[0138] The above scheme determines the target projection position based on the second positioning result and the original spatial position, or determines the target projection position based on the second changed spatial position which is a reasonable position according to the first changed spatial position. This effectively reduces the adverse effects caused by projection errors after correcting the display position of the virtual object.

[0139] In some disclosed embodiments, the first correction strategy is to render the virtual object directly to the target projection position in the next display rendering, and the second correction strategy is to move the virtual object from the first projection position to the target projection position at a preset speed.

[0140] The above scheme makes the correction process more flexible by setting different correction strategies.

[0141] In some disclosed embodiments, before determining the first projection position as the display position of the virtual object in the current shooting frame in response to obtaining the corrected second positioning result of the target device and the virtual object meeting the first preset condition, the determining module 41 is further configured to: obtain the first changed spatial position in the world coordinate system by using the first projection position, the projection parameters of the target device, and the first transformation parameters between the world coordinate system and the camera coordinate system of the target device, wherein the first transformation parameters are obtained based on the second positioning result of the target device.

[0142] The above solution determines the first changed spatial position of the virtual object by using the second positioning result of the target device and the first projection position of the virtual object, thus keeping the projection position of the virtual object in the current shooting frame at the first changed spatial position unchanged.

[0143] In some disclosed embodiments, the determining module 41 determines the first projection position of the virtual object in the current shooting frame of the target device based on the first positioning result of the target device before correction, including: obtaining the first projection position by using the original spatial position of the virtual object, the projection parameters of the target device, and the second transformation parameters between the world coordinate system and the camera coordinate system of the target device, wherein the second transformation parameters are obtained based on the first positioning result of the target device; after obtaining the second positioning result of the target device after correction, in response to obtaining the second positioning result of the target device after correction and the virtual object meeting the first preset condition, before continuing to determine the first projection position as the display position of the virtual object in the current shooting frame, the fusion module 42 is further configured to: perform positioning matching between the target device and the preset map to obtain the second positioning result of the target device in the preset map.

[0144] The above scheme obtains a second positioning result of the target device in the preset map by matching the target device with the preset map, making the second positioning result more accurate than the first positioning result before correction.

[0145] The above solution, after obtaining the second positioning result of the target device after correction, responds to the virtual object meeting the first preset condition and keeps the virtual object displayed at the first projection position of the current shooting screen, which can reduce the visual impact caused by the virtual object changing its display position.

[0146] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 50 includes a memory 51 and a processor 52. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps in the above-described embodiment of the virtual-real fusion method. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, AR devices, microcomputers, and servers. In addition, the electronic device 50 may also include mobile devices such as laptops and tablets, which are not limited here.

[0147] Specifically, processor 52 controls itself and memory 51 to implement the steps in the above-described virtual-real fusion method embodiments. Processor 52 can also be called a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0148] The above solution, after obtaining the second positioning result of the target device after correction, responds to the virtual object meeting the first preset condition and keeps the virtual object displayed at the first projection position of the current shooting screen, which can reduce the visual impact caused by the virtual object changing its display position.

[0149] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 601, which, when executed by a processor, are used to implement the steps in the above-described embodiment of the virtual-real fusion method.

[0150] The above solution, after obtaining the second positioning result of the target device after correction, responds to the virtual object meeting the first preset condition and keeps the virtual object displayed at the first projection position of the current shooting screen, which can reduce the visual impact caused by the virtual object changing its display position.

[0151] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0152] This disclosure relates to the field of augmented reality (AR). It involves acquiring image information of target objects in a real-world environment and then using various visual algorithms to detect or identify the relevant features, states, and attributes of these objects, thereby achieving an AR effect that combines virtual and real elements to suit specific applications. For example, target objects may include human features such as faces, limbs, gestures, and movements; objects such as signs and markers; or venues such as sand tables, display areas, or displayed items. Visual algorithms may include visual localization, SLAM, 3D reconstruction, image registration, background segmentation, keypoint extraction and tracking of objects, and pose or depth detection. Specific applications can include interactive scenarios related to real-world scenes or objects, such as guided tours, navigation, explanations, reconstruction, and virtual effect overlay displays, as well as human-related special effects processing, such as makeup enhancement, body enhancement, special effects displays, and virtual model displays.

[0153] Convolutional neural networks (CNNs) can be used to detect or identify the relevant features, states, and attributes of target objects. The aforementioned CNNs are network models obtained through training using deep learning frameworks.

[0154] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0156] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for integrating virtual and real worlds, characterized in that, include: Based on the first positioning result of the target device before calibration, the first projection position of the virtual object in the current shooting screen of the target device is determined, wherein the first projection position is determined as the display position of the virtual object in the current shooting screen; In response to obtaining the second positioning result of the target device after correction, and the virtual object meeting the first preset condition, the first projection position is further determined as the display position of the virtual object in the current shooting screen; The first preset condition includes setting the accuracy requirement of the virtual object to be less than a preset accuracy requirement, or the first preset condition includes at least one of the following: the virtual object meets the second preset condition, the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirement, and the second preset condition includes the virtual object belonging to a first type and / or the first changed spatial position of the virtual object belonging to a reasonable position; wherein the first changed spatial position is a spatial position determined based on the second positioning result of the target device, and the first changed spatial position can be projected onto the first projection position.

2. The method according to claim 1, characterized in that, After determining the display position, the method further includes: The currently captured image is displayed, and the virtual object is displayed at the display position in the currently captured image.

3. The method according to claim 1, characterized in that, The preset position requirement includes that the position deviation is less than a preset threshold.

4. The method according to claim 3, characterized in that, The preset position requirement also includes the visibility of the display change process of the virtual object, wherein the display change process is the process of the virtual object's display changing from a first projection position to a target projection position.

5. The method according to claim 1, characterized in that, The first preset condition includes that the virtual object meets the second preset condition, and the second preset condition includes that the first change in the spatial position of the virtual object is a reasonable position; wherein, The reasonable location includes at least one of a non-dangerous area and a location within a preset height range; and / or, before further determining the first projection position as the display position of the virtual object in the current captured image, the method further includes: Based on the semantic information of the preset map, it is determined whether the first changed spatial location belongs to the reasonable location.

6. The method according to claim 1, characterized in that, The method further includes: In response to obtaining the second positioning result of the target device after correction, and the virtual object does not meet the first preset condition, the display position of the virtual object is changed from the first projection position to the target projection position using a first correction strategy or a second correction strategy.

7. The method according to claim 6, characterized in that, The step of responding to obtaining the corrected second positioning result of the target device, and the virtual object not meeting the first preset condition, and using a first correction strategy or a second correction strategy to change the display position of the virtual object from the first projection position to the target projection position, includes: In response to the virtual object belonging to the second type, a first correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position; In response to the virtual object belonging to the third type, a second correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position; In response to the first change of spatial position being an unreasonable position, a first correction strategy or a second correction strategy is selected to change the display position of the virtual object from the first projection position to the target projection position.

8. The method according to claim 6, characterized in that, The preset position requirements include that the display change process of the virtual object is visible and that the position deviation is less than a preset threshold, wherein the display change process is the process by which the display of the virtual object changes from a first projection position to a target projection position; The step of responding to obtaining the corrected second positioning result of the target device, and the virtual object not meeting the first preset condition, and using a first correction strategy or a second correction strategy to change the display position of the virtual object from the first projection position to the target projection position, includes: In response to the virtual object meeting the second preset condition, the display change process of the virtual object being visible, and the position deviation not being less than the preset threshold, a first correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position; In response to the virtual object meeting the second preset condition and the display change process of the virtual object being invisible, a second correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position.

9. The method according to any one of claims 6 to 8, characterized in that, Before employing a first correction strategy or a second correction strategy to change the display position of the virtual object from the first projection position to the target projection position, the method further includes: Based on the second positioning result and the original spatial position, determine the second projection position of the virtual object in the current captured image, and use it as the target projection position; or, If the first changed spatial position is not a reasonable position, the first changed spatial position is adjusted to a second changed spatial position that is a reasonable position. Based on the second positioning result and the second changed spatial position, the third projection position of the virtual object in the current shooting screen is determined as the target projection position.

10. The method according to any one of claims 6 to 8, characterized in that, The first correction strategy is to render the virtual object directly to the target projection position in the next display rendering, and the second correction strategy is to move the virtual object from the first projection position to the target projection position at a preset speed.

11. The method according to any one of claims 1 to 8, characterized in that, Before determining the first projection position as the display position of the virtual object in the current captured image in response to obtaining the corrected second positioning result of the target device and the virtual object meeting the first preset condition, the method further includes: Using the first projection position, the projection parameters of the target device, and the first transformation parameters between the world coordinate system and the camera coordinate system of the target device, the first changed spatial position on the world coordinate system is obtained, wherein the first transformation parameters are obtained based on the second positioning result of the target device.

12. The method according to any one of claims 1 to 8, characterized in that, The step of determining the first projection position of the virtual object in the current captured image of the target device based on the first positioning result of the target device before calibration includes: The first projected position is obtained by using the original spatial position of the virtual object, the projection parameters of the target device, and the second transformation parameters between the world coordinate system and the camera coordinate system of the target device, wherein the second transformation parameters are obtained based on the first positioning result of the target device; Before determining the first projection position as the display position of the virtual object in the current captured image, in response to obtaining the corrected second positioning result of the target device and the virtual object meeting the first preset condition, the method further includes: The target device is matched with a preset map to obtain the second positioning result of the target device in the preset map.

13. A virtual-real fusion device, characterized in that, include: The determining module is used to determine the first projection position of the virtual object in the current shooting frame of the target device based on the first positioning result of the target device before correction, wherein the first projection position is determined as the display position of the virtual object in the current shooting frame; The fusion module is used to respond to obtaining the second positioning result of the target device after correction, and the virtual object meeting the first preset condition, to further determine the first projection position as the display position of the virtual object in the current shooting screen; The first preset condition includes setting the accuracy requirement of the virtual object to be less than a preset accuracy requirement, or the first preset condition includes at least one of the following: the virtual object meets the second preset condition, the positional deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirement, and the second preset condition includes the virtual object belonging to a first type and / or the first changed spatial position of the virtual object belonging to a reasonable position; wherein the first changed spatial position is a spatial position determined based on the second positioning result of the target device, and the first changed spatial position can be projected onto the first projection position.

14. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 12.

Images