An information display method, apparatus, device, and storage medium

Abstract

This application discloses an information display method, apparatus, device, and storage medium. The method includes: acquiring pose change information of the device between a first time moment and a second time moment; using the pose change information and a first position information of an identified object stored in a preset storage space at the first time moment, determining a second position information of the identified object at the second time moment; if the second position information meets a preset condition, acquiring object information of the identified object stored in the preset storage space and sending it to a target display area determined based on the second position information on the device display screen for display. Through the above technical solution, this application can reduce the time consumption caused by feature point extraction and matching, effectively reducing the display delay of object information in the field of view, thereby improving the display speed of object information in the field of view.

Description

Technical Field

[0001] This invention relates to the field of smart device technology, and in particular to an information display method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, with the rapid development of smart wearable devices, smart devices that can overlay virtual information onto real-world scenes have gradually entered people's daily lives and work. These devices integrate sensing technology, 3D modeling, and optical display, enabling them to simulate and display computer-generated text, images, audio, and video information in the real world as a supplement to real-world information. Examples include AR (Augmented Reality) devices and MR (Mixed Reality) devices, both of which can achieve the effect of merging virtual information with the real world.

[0003] The aforementioned smart devices currently identify target object information and the target display area within the field of view by extracting feature points of the target object and performing feature point matching. The target object information is then displayed within this target display area. However, if the smart device moves after identifying the current scene, the target object's position relative to the field of view will shift. In this case, feature points need to be extracted and matched again to identify the target object information and its corresponding display area. However, extracting and matching feature points is a very time-consuming operation, resulting in a noticeable delay in the display of object information within the field of view. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an information display method, apparatus, device, and storage medium that can effectively reduce the display delay of object information in the field of view. The specific solution is as follows:

[0005] In a first aspect, this application provides an information display method, applied to a device that overlays virtual information onto a real scene, comprising:

[0006] Acquire the pose change information of the device between the first time point and the second time point;

[0007] Using the pose change information and the first position information of the identified object at the first moment stored in the preset storage space, the second position information of the identified object at the second moment is determined;

[0008] If the second location information meets the preset conditions, the object information of the identified object stored in the preset storage space is obtained and sent to the target display area determined based on the second location information on the device display screen for display.

[0009] Optionally, the information display method further includes:

[0010] Capture real-world scene footage at any given moment;

[0011] The real scene image is sent to a preset image recognition platform so that the image recognition platform can identify the object information in the real scene image and detect the position information of the scene objects in the real scene image to obtain the corresponding image recognition result.

[0012] The image recognition results returned by the image recognition platform are saved to the preset storage space.

[0013] Optionally, the image recognition platform identifies object information in the real-world scene, including:

[0014] Using the image recognition model pre-built based on artificial intelligence algorithms in the image recognition platform, the object type of the scene object in the real scene image is identified, and based on the object type, the specific description information corresponding to the scene object is searched from the preset information database.

[0015] Optionally, detecting the position information of scene objects in the real scene image through the image recognition platform includes:

[0016] The image recognition platform detects the size information of the scene object in the real scene image and the two-dimensional coordinate information of the scene object in the image coordinate system corresponding to the real scene image.

[0017] Optionally, the acquisition of real-world scene images at any given time includes:

[0018] Capture real-world scene images and image depth information at any given moment;

[0019] Accordingly, the real-scene image is sent to a preset image recognition platform to detect the position information of objects in the real-scene image, including:

[0020] The real scene image and the image depth information are sent to a preset image recognition platform. The image recognition platform detects the size information of the scene object in the image area of ​​the real scene image, and detects the two-dimensional coordinate information of the scene object in the image coordinate system corresponding to the real scene image. Based on the image depth information, the two-dimensional coordinate information is mapped to the camera coordinate system to obtain the three-dimensional coordinate information of the scene object in the camera coordinate system.

[0021] Optionally, saving the image recognition result returned by the image recognition platform to the preset storage space includes:

[0022] The object information in the image recognition result is compared with the object information currently stored in the preset storage space to filter out the object information of the target static object that is not currently stored in the preset storage space from the image recognition result.

[0023] The object information and corresponding location information of the target static object in the image recognition result are saved to the preset storage space.

[0024] Optionally, after determining the second location information of the identified object at the second time moment, the method further includes:

[0025] The first location information in the preset storage space is updated using the second location information.

[0026] Optionally, if the second location information meets a preset condition, then acquiring the object information of the identified object stored in the preset storage space and sending it to the target display area determined based on the second location information on the device display screen for display includes:

[0027] Based on the second location information, the screen area corresponding to the identified object in the device display screen is determined, and a first ratio between the size of the object corresponding to the screen area and the full size of the identified object is determined.

[0028] If the first ratio is not less than the first preset threshold, then a second ratio is determined between the area of ​​the screen region and the total area of ​​the screen displayed by the device.

[0029] If the second ratio is less than the second preset threshold, the object information of the identified object stored in the preset storage space is sent to the target display area determined based on the screen area on the device display screen for display.

[0030] Optionally, sending the object information of the identified object stored in the preset storage space to the target display area determined based on the screen area on the device display screen for display includes:

[0031] A target display area with a fixed relative position to the screen area is determined from the device display screen, and the object information of the identified object stored in the preset storage space is sent to the target display area for display.

[0032] Secondly, this application provides an information display device, used for overlaying virtual information onto a real scene, comprising:

[0033] The pose acquisition module is used to acquire pose change information of the device between a first time moment and a second time moment;

[0034] The position determination module is used to determine the second position information of the identified object at the second time using the pose change information and the first position information of the identified object at the first time stored in the preset storage space.

[0035] The information display module is used to obtain the object information of the identified object stored in the preset storage space when the second location information meets the preset conditions, and send it to the target display area determined based on the second location information on the device display screen for display.

[0036] Thirdly, this application provides an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute the computer program to implement the steps of the aforementioned information display method.

[0039] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the steps of the aforementioned information display method.

[0040] In this application, the first position information of the identified object at a first moment and the corresponding object information are pre-stored in a preset storage space. When the device's pose change information between the first and second moments is obtained, the position information of the identified object at the second moment is determined using the pose change information and the position information of the identified object at the first moment stored in the preset storage space. If the position information of the identified object at the second moment meets a preset condition, the object information of the identified object stored in the preset storage space is sent to the target display area determined based on the second position information on the device display screen for display. Thus, this application can effectively display the object information on the device display screen at the current moment. When displaying, the system first determines the second position information of the identified object at the current moment based on the first position information of the identified object at the previous moment and the pose change information of the device between the two moments. If the second position information meets the preset conditions, the corresponding target display area can be determined based on the second position information, and the object information of the identified object can be directly obtained from the preset storage space and displayed in the target display area. This avoids the need to re-extract and match feature points to obtain the object information and the corresponding target display area, thereby reducing the time consumption caused by feature point extraction and matching, effectively reducing the display delay of object information in the field of view, and improving the display speed of object information in the field of view. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 A flowchart of an information display method provided in this application;

[0043] Figure 2 An image recognition flowchart is provided for this application;

[0044] Figure 3 A flowchart illustrating a specific information display method provided in this application;

[0045] Figures 4 to 9 These are schematic diagrams showing the display of object information under different circumstances;

[0046] Figure 10 A timing diagram of an information display method provided in this application;

[0047] Figure 11 A schematic diagram of an information display device provided in this application;

[0048] Figure 12 This application provides a structural diagram of an electronic device. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In recent years, with the rapid development of smart wearable devices, smart devices capable of overlaying virtual information onto real-world scenes have gradually entered people's daily lives and work. After recognizing the current scene, if the smart device moves, the position of the target object relative to the field of view will also shift. At this point, it is necessary to re-extract feature points and perform feature point matching to identify the target object information and its corresponding display area in the current field of view. Because this process consumes a significant amount of time, it results in a noticeable delay in the display of object information in the field of view.

[0051] To this end, this application provides an information display solution that can effectively reduce the display delay of object information in the field of view. Specifically, the aforementioned information display solution is applied to devices capable of overlaying virtual information onto real-world scenes, including but not limited to augmented reality devices and mixed reality devices.

[0052] In this application, the aforementioned device captures real-scene images through its own camera module and sends the captured images to a preset image recognition platform via its own communication interface. The image recognition platform then identifies object information and detects the positional information of objects in the real-scene image, obtaining the corresponding image recognition result. It should be noted that the aforementioned image recognition platform typically refers to an image recognition platform located on a cloud server with strong computing power, leveraging the high computing power of the cloud server to improve the recognition speed of the real-scene image. Of course, this application does not preclude the use of image recognition components integrated in the aforementioned device or integrated into a computer connected to the aforementioned device for short-range communication to perform image recognition of the real-scene image. Although the computing power of the aforementioned device or computer is not as high as that of a cloud server, it saves time in communication transmission by avoiding remote communication requests and responses.

[0053] After obtaining the image recognition result containing the corresponding object information and location information, the image recognition platform returns the image recognition result to the device through the corresponding API (Application Programming Interface). The device then sends the received image recognition result to a preset storage space for storage. This means that the object information and location information stored in the preset storage space are the object information and location information of the recognized object. Furthermore, it should be noted that the preset storage space typically refers to the storage space integrated within the device. This allows data to be retrieved directly from the device's local storage whenever data stored in the preset storage space is needed, effectively reducing data retrieval time. Alternatively, the preset storage space can also refer to the storage space in a computer connected to the device via short-range communication. In this case, data can be retrieved from the computer via short-range data access whenever data stored in the preset storage space is needed.

[0054] During user operation of the aforementioned device, the device uses its internally integrated pose detection component to detect pose changes between the current and previous moments. This pose detection component can perform pose detection based on IMU data integration (IMU, Inertial Measurement Unit), or it can employ SLAM (Simultaneous Localization and Mapping) pose estimation technology based on structured light and IMU, or TOF (Time of Flight) pose detection technology. No specific limitation is made to the pose detection method used in this application.

[0055] Upon detecting a pose change between the current and previous moments, the device acquires the pre-stored position information of the identified object at the previous moment from a preset storage space. Then, using its processor and the aforementioned pose change information along with the previously stored position information, it calculates the current position of the identified object. If this position information meets preset conditions, it determines a target display area from the device's display screen based on this position information. Finally, it sends the object information of the identified object stored in the preset storage space to the target display area for display. This application, through the above technical solution, can reduce the time consumption associated with feature point extraction and matching, thereby effectively reducing the display delay of object information in the field of view and improving the display speed of object information in the field of view.

[0056] For details, see Figure 1 As shown, this embodiment of the invention discloses an information display method, applied to a device that overlays virtual information with a real scene, comprising:

[0057] Step S11: Obtain the pose change information of the device between the first time and the second time.

[0058] It is understandable that during the use of the above-mentioned device, the position and posture of the device will change due to the movement of the user. At this time, the posture detection component integrated inside the device can collect the posture change information of the device between the first moment and the second moment, which specifically includes the turning angle and displacement distance of the device between the two moments.

[0059] In this embodiment, the first moment and the second moment generally refer to two consecutive moments. Of course, in some scenarios, the first moment and the second moment may be separated by a number of moments. In addition, the time interval between any two adjacent moments in this embodiment can be determined based on the actual situation in the application process. For example, the time interval between any two adjacent moments in this embodiment can be determined based on the pose information sampling frequency of the pose detection component, or based on the display frame rate of the device display screen. No specific limitation is imposed here.

[0060] Step S12: Using the pose change information and the first position information of the identified object at the first moment stored in the preset storage space, determine the second position information of the identified object at the second moment.

[0061] In this embodiment, the first position information of the identified object at a first moment is stored in a preset storage space. When the pose change information of the device between the first and second moments is obtained, the first position information of the identified object at the first moment is retrieved from the preset storage space using the corresponding time index information. Then, the second position information of the identified object at the second moment is calculated using the pose change information and the first position information. In this embodiment, the time index information can be constructed based on the timestamp information corresponding to the first moment. In this way, the first position information of the identified object corresponding to the first moment can be retrieved from the preset storage space using the time index information.

[0062] It should be noted that, after determining the second position information of the identified object at the second moment, this embodiment can further update the first position information in the preset storage space using the second position information, that is, replace the first position information corresponding to the first moment stored in the preset storage space with the second position information corresponding to the second moment. Similarly, the first position information of the identified object originally stored in the preset storage space at the first moment can also be calculated using the pose change information between the first moment and the previous moment and the position information of the identified object stored in the preset storage space at the previous moment. Of course, the first position information can also be obtained by a dedicated image recognition platform performing image recognition on the real scene image captured at the first moment.

[0063] Step S13: If the second location information meets the preset conditions, the object information of the identified object stored in the preset storage space is obtained and sent to the target display area determined based on the second location information on the device display screen for display.

[0064] In this embodiment, the aforementioned second location information must meet preset conditions before the object information of the identified object stored in the preset storage space is obtained and sent to the target display area determined based on the second location information on the device display screen for display. It is understood that the aforementioned preset conditions must include at least one condition: after the second location information is projected onto the screen coordinate system of the device display screen, the screen area coordinates corresponding to the second location information cannot be completely outside the device display screen; otherwise, obtaining and displaying the object information of the identified object from the preset storage space is prohibited.

[0065] In this embodiment, the object information stored in the preset storage space is obtained by recognizing scene objects in real-world scene images. Typically, after the object information of a scene object is first recognized, it is saved to the aforementioned preset storage space, and it is usually unnecessary to save the object information of the same scene object recognized in other real-world scene images to the preset storage space thereafter. However, considering that the specific description information of the same scene object may change significantly over time, such as the price information of an item, in order to avoid the problem of outdated object information displayed on the device screen, when the object information of the same scene object is recognized from the current real-world scene image, it is necessary to determine whether the specific description information in the currently recognized object information is consistent with the specific description information of the same scene object previously saved in the preset storage space. If they are consistent, the currently recognized object information is discarded and not saved; if they are inconsistent, the object information of the same scene object saved in the preset storage space is replaced and updated using the currently recognized object information.

[0066] In this embodiment, the preset storage space can bind and save the location information of the identified object with the object information of the identified object. After finding the first location information of the identified object at the first moment from the preset storage space using the time index information, if the calculated second location information also meets the preset conditions, the object information that is bound to the first location information can be found from the preset storage space. Based on the second location information, the target display area is determined from the device display screen, and then the found object information is sent to the target display area for display.

[0067] Therefore, when displaying object information in the device display screen at the current moment, this application embodiment first determines the second position information of the identified object at the current moment based on the first position information of the identified object at the previous moment and the pose change information of the device between the two moments. If the second position information meets the preset conditions, the corresponding target display area can be determined based on the second position information, and the object information of the identified object can be directly obtained from the preset storage space and displayed in the target display area. This avoids the need to re-extract and match feature points to obtain the object information and the corresponding target display area, thereby reducing the time consumption caused by feature point extraction and matching, effectively reducing the display delay of object information in the field of view, and improving the display speed of object information in the field of view.

[0068] As can be seen from the previous embodiment, this application can obtain object information and location information in a preset storage space by recognizing real-world scene images. Therefore, this embodiment will now describe the recognition process of real-world scene images in detail. See [link to previous embodiment]. Figure 2 As shown, this embodiment discloses an image recognition process, including:

[0069] Step S21: Collect real scene images at any time.

[0070] Step S22: Send the real scene image to a preset image recognition platform to identify object information in the real scene image and detect the position information of scene objects in the real scene image, so as to obtain the corresponding image recognition result.

[0071] Step S23: Save the image recognition result returned by the image recognition platform to the preset storage space.

[0072] In this embodiment, the device can capture images of the real scene using its own camera module to obtain the aforementioned real scene image.

[0073] In one specific implementation, the aforementioned camera module is a common camera module used for acquiring two-dimensional images. In this embodiment, the two-dimensional real-world scene image captured by the camera module at any given time is sent to a preset image recognition platform. The platform utilizes an image recognition model pre-built based on artificial intelligence algorithms to identify the object type of objects in the real-world scene image. Based on the object type, the platform searches a preset information database for specific descriptive information corresponding to the scene object. Furthermore, the platform detects the size information of the scene object's image area in the real-world scene image and its two-dimensional coordinates in the image coordinate system corresponding to the real-world scene image. In other words, the object information identified by the image recognition platform in this embodiment can include the object type and specific descriptive information of the scene object. The position information detected by the image recognition platform includes the size information of the scene object's image area in the real-world scene image and its two-dimensional coordinates in the image coordinate system corresponding to the real-world scene image. The image area size information can include the width and height of the image area corresponding to the scene object, and the two-dimensional coordinate information includes the distance from the upper left corner of the image area corresponding to the scene object to the vertical axis and the distance from the upper left corner to the horizontal axis.

[0074] Corresponding to the above implementation, when calculating the position information of the target scene object at the current moment using the current pose change information of the device and the position information of the target scene object in the preset storage space at the previous moment, the specific process may include transforming the screen area size information and two-dimensional coordinate information stored in the preset storage space at the previous moment based on the pose change information, so as to obtain the transformed screen area size information and the transformed two-dimensional coordinate information, that is, to obtain the position information of the target scene object at the current moment.

[0075] Furthermore, the preset information database in this embodiment is specifically used to collect the latest detailed description information corresponding to various scene objects. Each time the image recognition platform is used to recognize a received real-world scene image, the latest detailed description information corresponding to the scene object can be retrieved from the preset information database and compared with the detailed description information of the scene object currently stored in the preset storage space. If a discrepancy is found, the detailed description information retrieved from the preset information database can be used to replace and update the detailed description information in the preset storage space. It is understood that the preset information database can be a database pre-integrated into a third-party server, capable of dynamically collecting the latest detailed description information corresponding to various scene objects.

[0076] In another specific embodiment, the aforementioned camera module refers to a camera module that includes a depth sensor. In this embodiment, the camera module acquires two-dimensional real-world scene images and image depth information at any given time, and sends these images and depth information to a preset image recognition platform. The platform utilizes an image recognition model pre-built based on artificial intelligence algorithms to identify the object type of objects in the real-world scene image. Based on the object type, it searches a preset information database for specific description information corresponding to the scene object. The platform also detects the image area size information of the scene object in the real-world scene image and its two-dimensional coordinate information in the image coordinate system corresponding to the real-world scene image. Then, based on the image depth information, the two-dimensional coordinate information is mapped to the camera coordinate system to obtain the three-dimensional coordinate information of the scene object in the camera coordinate system. The image area size information may include the width and height of the image area corresponding to the scene object.

[0077] Corresponding to the above implementation, when calculating the position information of the target scene object at the current moment using the current pose change information of the device and the position information of the target scene object in the preset storage space at the previous moment, the specific process may include transforming the screen area size information and three-dimensional coordinate information stored in the preset storage space at the previous moment based on the pose change information, so as to obtain the transformed screen area size information and the transformed three-dimensional coordinate information, that is, to obtain the position information of the target scene object at the current moment.

[0078] In this embodiment, saving the image recognition result returned by the image recognition platform to the preset storage space may specifically include: comparing the object information in the image recognition result with the object information currently saved in the preset storage space to filter out the object information of the target static object that is not currently saved in the preset storage space from the image recognition result; and saving the object information of the target static object and its corresponding location information in the image recognition result to the preset storage space. In this embodiment, to improve information comparison efficiency, various data information can be saved in the preset storage space in the form of a list, and corresponding keyword index information based on object information, such as object type keywords, can be added to it. In this way, after obtaining the object information in the image recognition result, the object information in the image recognition result is compared with the object information currently saved in the preset storage space using the keyword index information to filter out the object information of the target static object that is not currently saved in the preset storage space from the image recognition result.

[0079] In this embodiment, when the device detects that the image recognition result of the real scene image returned by the image recognition platform at any given time includes the object information and location information of a new object, it can send the object information of the new object to the area above the screen area where the new object is located for display. It is understood that there is a certain time delay between acquiring the real scene image of the new object and sending the object information of the new object to the device display screen for display. For example, if a real scene image with a new object is acquired at time T2, the object information of the new object cannot be displayed at time T2. Instead, it needs to be sent to the image recognition platform for image recognition and the image recognition result needs to be returned to the device processor for information comparison and analysis to determine that a new object exists in the scene at time T2. Then, the object information of the new object is transmitted to the screen for display. The time delay for this entire process is Δ, meaning that the time corresponding to displaying the object information of the new object on the screen is already T2+Δ.

[0080] It is understood that in this embodiment, by comparing the object information in the image recognition result with the object information stored in the preset storage space, it can be determined whether the corresponding scene objects are the same object. If they are the same object, it can be further determined whether the scene object is a static object. If it is a static object, the object information of the static object and the corresponding position information in the image recognition result are saved to the preset storage space.

[0081] Furthermore, in order to determine whether a target scene object in a real scene image is a static object, this embodiment can perform a unified analysis on multiple real scene images collected at several past moments to detect whether the relative position of the target scene object with respect to a fixed reference object in the scene has changed in these real scene images. If it has changed, the target scene object is determined to be a dynamic object; if it has not changed, the target scene object is determined to be a static object.

[0082] It should be noted that if the target scene object in the real scene is not a static object but a dynamic object, then the depth sensor on the device needs to be used to detect the pose change information of the target scene object between the first time and the second time. Based on the pose change information of the device, the pose change of the object, and the first position information of the target scene object at the first time stored in the preset storage space, the second position information of the target scene object at the second time is determined.

[0083] See Figure 3 As shown in the figure, this application discloses a specific information display method, including:

[0084] Step S31: Obtain the pose change information of the device between the first time and the second time.

[0085] Step S32: Using the pose change information and the first position information of the identified object at the first moment stored in the preset storage space, determine the second position information of the identified object at the second moment.

[0086] Step S33: Based on the second location information, determine the screen area corresponding to the identified object in the device display screen, and determine the first ratio between the object size corresponding to the screen area and the full size of the identified object.

[0087] In this embodiment, the second location information includes the size information of the image area of ​​the identified object in the real scene and the corresponding coordinate information. The coordinate information may be the two-dimensional coordinate information of the identified object in the image coordinate system corresponding to the real scene, or the three-dimensional coordinate information of the identified object in the camera coordinate system.

[0088] Step S34: If the first ratio is not less than the first preset threshold, then determine the second ratio between the area of ​​the screen region and the total screen area of ​​the device display.

[0089] In this embodiment, if the ratio between the size of the object corresponding to the area of ​​the identified object in the device display screen and the full size of the object is less than a first preset threshold, it indicates that most or all of the area corresponding to the currently identified object is outside the device display screen. In this case, it is not suitable to send the object information of the identified object stored in the preset storage space to the device display screen for display, which is beneficial to improving the user experience. If the ratio between the size of the object corresponding to the area of ​​the identified object in the device display screen and the full size of the object is not less than the first preset threshold, it indicates that most of the area of ​​the currently identified object is within the device display screen. In this case, it is necessary to further determine a second ratio between the area of ​​the screen region and the entire screen area of ​​the device display screen.

[0090] Step S35: If the second ratio is less than the second preset threshold, the object information of the identified object stored in the preset storage space is sent to the target display area determined based on the screen area on the device display screen for display.

[0091] In this embodiment, if the second ratio is greater than or equal to the second preset threshold, it indicates that the screen area corresponding to the currently identified object covers most of the entire device display screen. In this case, it is not suitable to send the object information of the identified object stored in the preset storage space to the device display screen for display. If the second ratio is less than the second preset threshold, the object information of the identified object stored in the preset storage space is sent to the target display area determined based on the screen area on the device display screen for display.

[0092] It should be noted that the first preset threshold and the second preset threshold can be set differently based on the actual needs of the scenario. For example, the first preset threshold can be set to 30% and the second preset threshold can be set to 70%.

[0093] In this embodiment, through Figures 4 to 9 The diagrams show the display of object information under different circumstances.

[0094] See Figure 4 As shown, assuming there is a carton of milk in the real scene at time T1, and the product type, price information and corresponding location information of this carton of milk have been obtained in advance, the product type MILK and the price tag information $2 can be displayed in the upper area of ​​this carton of milk based on the location information of this carton of milk.

[0095] See Figure 5 As shown, assuming that after time T1, only a small portion of the milk carton remains on the device's display screen, its price tag information can be omitted from display.

[0096] See Figure 6 As shown, assuming that after time T1, a small portion of the milk carton has been removed from the device display screen, but most of the milk carton remains within the device display screen, the display of its object type MILK and price tag information $2 can still be retained.

[0097] See Figure 7 As shown, assuming that after time T, although most of the milk carton remains on the device display screen, since its corresponding screen area covers most of the entire device display screen, the price tag information is not displayed at this time.

[0098] See Figure 8As shown, assuming a new cup appears in the real scene at time T2, since the relevant information about this cup has not yet been obtained, the object information of the cup cannot be displayed at time T2. Instead, it needs to be sent to the image recognition platform for image recognition, and the image recognition results need to be returned to the device for information comparison and analysis to determine that a new object, a cup, exists in the scene at time T2. Then, the product type and price tag information of this cup are transmitted to the screen for display. The time delay for the entire process is △, which means that the time corresponding to displaying the object information of the cup on the screen is T2 + △. Figure 9 As shown.

[0099] In this embodiment, sending the object information of the identified object stored in the preset storage space to a target display area determined based on the screen area on the device display screen for display may specifically include: determining a target display area on the device display screen that has a fixed relative positional relationship with the screen area, and sending the object information of the identified object stored in the preset storage space to the target display area for display. In this embodiment, the target display area has a fixed relative positional relationship with the screen area. For example, the target display area may refer to the display area directly above the screen area, or the display area to the left of the screen area, etc. The fixed relative positional relationship is not specifically limited here.

[0100] The following is combined with Figure 10 The timing diagrams disclosed herein provide a detailed explanation of the technical solutions of the embodiments of this application.

[0101] In this embodiment, the device used to overlay virtual information onto a real scene is an AR wearable device, and the cloud server integrating an image recognition platform is used for image recognition. The AR wearable device integrates a communication interface, a camera module (not shown in the figure), a pose detection component (not shown in the figure), preset storage space, and a display module.

[0102] First, the AR wearable device captures the real scene image of the new object A appearing at time T1 through the camera module, and sends the real scene image at time T1 to the cloud server through its own communication interface. The image recognition platform in the cloud server identifies the object information in the real scene image at time T1 and detects the position information of the scene objects in the real scene image at time T1, and obtains the corresponding first image recognition result.

[0103] The AR wearable device obtains the first image recognition result returned by the cloud server, and extracts the object information and first location information of object A from the first image recognition result through the processor, and saves the object information and first location information of object A to the preset storage space.

[0104] When time T2 arrives, the AR wearable device detects the pose change information between time T1 and time T2 through the pose detection component, and captures the real scene image of the new object B appearing at time T2 through the camera module. Then, it sends the real scene image at time T2 to the cloud server through the communication interface, so that the image recognition platform in the cloud server can identify the object information in the real scene image at time T2 and detect the position information of the scene objects in the real scene image at time T2, and obtain the corresponding second image recognition result.

[0105] The AR wearable device retrieves the first position information of object A at time T1 from the preset storage space, and uses the device's pose change information between time T1 and time T2, along with the first position information, to determine the second position information of object A at time T2. It then determines whether the second position information satisfies the preset conditions constructed based on the proportion of object A outside the edge of the screen and the area of ​​object A in the screen. If the conditions are met, the device retrieves the object information of object A stored in the preset storage space and sends it to the area above object A in the device's display screen for display.

[0106] The AR wearable device obtains the second image recognition result returned by the cloud server, and compares the second image recognition result with the information saved in the current preset storage space. When it is determined that object B in the second image recognition result is a new object in the scene, the object information and position information of object B are saved to the preset storage space.

[0107] Accordingly, see Figure 11 As shown in the illustration, this application also discloses an information display device, which is used to overlay virtual information with a real scene for display, including:

[0108] The pose acquisition module 11 is used to acquire pose change information of the device between a first time and a second time.

[0109] The position determination module 12 is used to determine the second position information of the identified object at the second moment by using the pose change information and the first position information of the identified object at the first moment stored in the preset storage space.

[0110] The information display module 13 is used to obtain the object information of the identified object stored in the preset storage space when the second location information meets the preset conditions, and send it to the target display area determined based on the second location information on the device display screen for display.

[0111] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0112] Therefore, when displaying object information in the device display screen at the current moment, this application embodiment first determines the second position information of the identified object at the current moment based on the first position information of the identified object at the previous moment and the pose change information of the device between the two moments. If the second position information meets the preset conditions, the corresponding target display area can be determined based on the second position information, and the object information of the identified object can be directly obtained from the preset storage space and displayed in the target display area. This avoids the need to re-extract and match feature points to obtain the object information and the corresponding target display area, thereby reducing the time consumption caused by feature point extraction and matching, effectively reducing the display delay of object information in the field of view, and improving the display speed of object information in the field of view.

[0113] Furthermore, embodiments of this application also provide an electronic device. Figure 12 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0114] Figure 12 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, a communication bus 27, and a sensor 28. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the information display method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0115] In this embodiment, the power supply 26 provides operating voltage to the various hardware devices on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, without specific limitations; the input / output interface 24 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, without specific limitations. The sensor 28 is used to detect information about the surrounding environment, including but not limited to depth sensors, inertial measurement units, etc.

[0116] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon may include computer programs 221, and the storage method may be temporary storage or permanent storage. The computer programs 221 may include, in addition to computer programs capable of performing the information display method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, computer programs capable of performing other specific tasks.

[0117] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed information display method.

[0118] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0119] The above provides a detailed description of the information display method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An information display method, characterized in that, Devices used to overlay virtual information onto real-world scenes include: The pose change information of the device between the first and second time moments is obtained by integrating IMU data. Using the pose change information and the first position information of the identified object at the first moment stored in the preset storage space, the second position information of the identified object at the second moment is determined; the identified object is a static object. If the second location information meets the preset conditions, the object information of the identified object stored in the preset storage space is obtained and sent to the target display area determined based on the second location information on the device display screen for display. Also includes: Capture real-world scene footage at any given moment; The real scene image is sent to a preset image recognition platform so that the image recognition platform can identify the object information in the real scene image and detect the position information of the scene objects in the real scene image to obtain the corresponding image recognition result. The image recognition result returned by the image recognition platform is saved to the preset storage space; The image recognition platform detects the position information of objects in the real scene, including: The image recognition platform detects the size information of the scene object in the real scene image and the two-dimensional coordinate information of the scene object in the image coordinate system corresponding to the real scene image.

2. The information display method according to claim 1, characterized in that, The image recognition platform identifies object information in the real-world scene, including: Using the image recognition model pre-built based on artificial intelligence algorithms in the image recognition platform, the object type of the scene object in the real scene image is identified, and based on the object type, the specific description information corresponding to the scene object is searched from the preset information database.

3. The information display method according to claim 1, characterized in that, The acquisition of real-world scene images at any given time includes: Capture real-world scene images and image depth information at any given moment; Accordingly, the real-scene image is sent to a preset image recognition platform to detect the position information of objects in the real-scene image, including: The real scene image and the image depth information are sent to a preset image recognition platform. The image recognition platform detects the size information of the scene object in the image area of ​​the real scene image, and detects the two-dimensional coordinate information of the scene object in the image coordinate system corresponding to the real scene image. Based on the image depth information, the two-dimensional coordinate information is mapped to the camera coordinate system to obtain the three-dimensional coordinate information of the scene object in the camera coordinate system.

4. The information display method according to claim 1, characterized in that, Saving the image recognition result returned by the image recognition platform to the preset storage space includes: The object information in the image recognition result is compared with the object information currently stored in the preset storage space to filter out the object information of the target static object that is not currently stored in the preset storage space from the image recognition result. The object information and corresponding location information of the target static object in the image recognition result are saved to the preset storage space.

5. The information display method according to claim 1, characterized in that, After determining the second location information of the identified object at the second time moment, the method further includes: The first location information in the preset storage space is updated using the second location information.

6. The information display method according to any one of claims 1 to 5, characterized in that, If the second location information meets a preset condition, then the object information of the identified object stored in the preset storage space is obtained and sent to the target display area determined based on the second location information on the device display screen for display, including: Based on the second location information, the screen area corresponding to the identified object in the device display screen is determined, and a first ratio between the size of the object corresponding to the screen area and the full size of the identified object is determined. If the first ratio is not less than the first preset threshold, then a second ratio is determined between the area of ​​the screen region and the total area of ​​the screen displayed by the device. If the second ratio is less than the second preset threshold, the object information of the identified object stored in the preset storage space is sent to the target display area determined based on the screen area on the device display screen for display.

7. The information display method according to claim 6, characterized in that, The step of sending the object information of the identified object stored in the preset storage space to the target display area determined based on the screen area on the device display screen for display includes: A target display area with a fixed relative position to the screen area is determined from the device display screen, and the object information of the identified object stored in the preset storage space is sent to the target display area for display.

8. An information display device, characterized in that, Devices used to overlay virtual information onto real-world scenes include: The pose acquisition module is used to acquire the pose change information of the device between the first time and the second time by integrating IMU data; The position determination module is used to determine the second position information of the identified object at the second time using the pose change information and the first position information of the identified object at the first time stored in the preset storage space; the identified object is a static object. The information display module is used to obtain the object information of the identified object stored in the preset storage space when the second location information meets the preset conditions, and send it to the target display area determined based on the second location information on the device display screen for display. Also includes: Capture real-world scene footage at any given moment; The real scene image is sent to a preset image recognition platform so that the image recognition platform can identify the object information in the real scene image and detect the position information of the scene objects in the real scene image to obtain the corresponding image recognition result. The image recognition result returned by the image recognition platform is saved to the preset storage space; The image recognition platform detects the position information of objects in the real scene, including: The image recognition platform detects the size information of the scene object in the real scene image and the two-dimensional coordinate information of the scene object in the image coordinate system corresponding to the real scene image.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the information display method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the steps of the information display method as described in any one of claims 1 to 7.

Images