AR image display method and device and storage medium

Abstract

The invention discloses an AR image display method and device and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: firstly obtaining a background image collected by a camera device, then determining the pose information of the camera device by recognizing a positioning identifier in the background image, and then sending the pose information to a main device, the method comprises the following steps: acquiring pose information of a main device, receiving a foreground image sent by the main device, finally splicing the foreground image into a background image based on the pose information, generating an AR image, and displaying the AR image in a display screen. Therefore, the corresponding pose information can be determined based on the background image through the mobile terminal, then the pose information is fed back to the main device, the foreground image sent by the main device is received, and finally the AR image is generated based on the foreground image. And thus, the bystanders can obtain the VR content matched with the viewing angle of the bystanders.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to an AR image display method, device, and storage medium. Background Art

[0002] Augmented Reality (AR) is a technology that combines virtual information with the real world. By superimposing virtual content such as text, images, and 3D models generated by a computer onto the real environment, it provides users with a richer and more intuitive interactive experience.

[0003] Currently, mainstream VR display solutions are mainly optimized for the main user's perspective of a head-mounted device (HMD). Specifically, a VR device obtains the user's perspective through head tracking technology, renders the corresponding 3D scene image based on this, and finally outputs a single-perspective image to the display screen. At this time, if a bystander views the display screen from other physical angles, they can only obtain a 2D image without perspective correction, and its content has no direct association with the main user's real-time perspective.

[0004] However, due to the traditional solution only rendering for the single perspective of the main user, bystanders cannot directly synchronously obtain VR content that matches their physical perspective through the display screen. The image presented on the display screen is still a 2D projection of a fixed perspective or the main user's perspective, which makes it difficult for bystanders to directly obtain the complete content of a clear VR scene.

[0005] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] This application provides an AR image display method, device, and storage medium, aiming to solve the problem of inaccurate AR image display on a mobile terminal in the traditional solution.

[0007] To achieve the above objective, an AR image display method provided by this application is applied to a mobile terminal. The AR image display method includes the following steps:

[0008] Obtain a background image collected by a camera device;

[0009] Determine the pose information of the camera device by identifying a positioning identifier in the background image;

[0010] Send the pose information to a main device, and receive a foreground image sent by the main device;

[0011] Based on the pose information, splice the foreground image into the background image to generate an AR image;

[0012] Display the AR image on a display screen.

[0013] In one embodiment, the step of determining the pose information of the imaging device by identifying the positioning identifier in the background image includes:

[0014] Obtain preset marker information, and in the background image, identify the visual marker ArUco code that matches the marker information;

[0015] Determine the pixel coordinate position of the ArUco code in the background image;

[0016] According to the pixel coordinate position and the marker information, calculate the rotation matrix and translation vector of the imaging device corresponding to the plane where the background image is located;

[0017] According to the rotation matrix and translation vector, determine the pose information of the imaging device.

[0018] In one embodiment, the step of splicing the foreground image into the background image based on the pose information to generate an AR image includes:

[0019] Obtain the display screen information associated with the identifier corresponding to the display screen;

[0020] According to the display screen information and the pose information, determine the corresponding display screen area in the background image;

[0021] Use the display screen area as the image splicing area, and overlap the foreground image into the image splicing area to generate the AR image.

[0022] In one embodiment, after the step of determining the pose information of the imaging device by identifying the positioning identifier in the background image, the following is further included:

[0023] When it is detected that the position of the imaging device has changed, obtain the current background image collected by the imaging device;

[0024] According to the positioning identifier in the current background image, determine the current pose information of the imaging device;

[0025] Send the current pose information to the master device.

[0026] In one embodiment, after the step of sending the current pose information to the master device, the following is further included:

[0027] Receive the target foreground image sent by the master device;

[0028] Based on the image stitching area corresponding to the current pose information in the current background image, stitch the target foreground image to the current background image to generate a target AR image;

[0029] Display the target AR image on the display screen.

[0030] In addition, to achieve the above object, an AR image display method provided in this application is applied to a master device, and the AR image display method includes the following steps:

[0031] Receive the pose information sent by the mobile terminal;

[0032] Construct a virtual camera according to the pose information, and capture a foreground image of the virtual scene through the virtual camera;

[0033] Send the foreground image to the mobile terminal.

[0034] In one embodiment, the pose information is the pose information of the imaging device, the pose information includes position information and attitude information, and the step of constructing a virtual camera according to the pose information and capturing a foreground image of the virtual scene through the virtual camera includes:

[0035] Map the position information and attitude information to the virtual scene, and create a virtual camera at the corresponding position in the virtual scene;

[0036] Capture the foreground image corresponding to the virtual scene through the virtual camera.

[0037] In one embodiment, the step of capturing the foreground image corresponding to the virtual scene through the virtual camera includes:

[0038] Determine the viewing angle information and window information corresponding to the virtual camera according to the pose information;

[0039] Render the virtual scene according to the corresponding viewing angle information of the virtual camera to obtain a rendered image;

[0040] Crop the rendered image according to the window information to obtain the foreground image.

[0041] In addition, to achieve the above object, this application also provides an AR image display device, and the AR image display device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the AR image display method based on the Android system as described above.

[0042] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the AR image display method described above are implemented.

[0043] The present application provides an AR image display method, an AR image display device, and a storage medium. First, a background image collected by a camera device is obtained, then the pose information of the camera device is determined by identifying a positioning identifier in the background image, and then the pose information is sent to a master device, and a foreground image sent by the master device is received. Finally, based on the pose information, the foreground image is spliced into the background image to generate an AR image, and the AR image is displayed on a display screen. Thus, a mobile terminal can determine corresponding pose information based on a background image, then feedback the pose information to the master device, and receive the foreground image sent by the master device. Finally, an AR image is generated based on the foreground image. Thereby enabling bystanders to obtain VR content that matches their own perspective. Description of the Drawings

[0044] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0045] To more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a schematic flowchart of the first embodiment of the AR image display method of the present application;

[0047] Figure 2 It is a schematic flowchart of the second embodiment of the AR image display method of the present application;

[0048] Figure 3 It is a schematic flowchart of the third embodiment of the AR image display method of the present application;

[0049] Figure 4 It is a schematic flowchart of the fourth embodiment of the AR image display method of the present application;

[0050] Figure 5 It is a schematic architecture diagram of the hardware operating environment of the AR image display device involved in the embodiments of the present application.

[0051] The implementation, functional features, and advantages of the object of the present application will be further described in combination with the embodiments with reference to the drawings. Detailed implementation manners

[0052] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] In order to better understand the above technical solutions, the exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0054] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0055] The main solution of the present application is: acquiring a background image collected by a camera device; determining the pose information of the camera device by identifying a positioning identifier in the background image; sending the pose information to a master device and receiving a foreground image sent by the master device; based on the pose information, splicing the foreground image into the background image to generate an AR image; and displaying the AR image on a display screen.

[0056] Augmented Reality (AR) is a technology that combines virtual information with the real world. By superimposing virtual contents such as texts, images, and three-dimensional models generated by a computer onto the real environment, it provides users with a richer and more intuitive interactive experience. Currently, the mainstream VR display solutions are mainly optimized for the main user's perspective of a head-mounted device (HMD). Specifically, a VR device acquires the user's perspective through head tracking technology, renders a corresponding three-dimensional scene image based on this, and finally outputs a single-perspective image to a display screen. At this time, if a bystander views the display screen from other physical angles, only a two-dimensional image without perspective correction can be obtained, and its content has no direct association with the real-time perspective of the main user.

[0057] However, due to the traditional solution only rendering for the single perspective of the main user, bystanders cannot directly synchronously obtain VR content that matches their physical perspectives through the display screen. The image presented on the display screen is still a two-dimensional projection of a fixed perspective or the main user's perspective, which makes it difficult for bystanders to directly obtain the complete content of a clear VR scene.

[0058] This application obtains the background image collected by the imaging device, then determines the pose information of the imaging device by identifying the positioning identifier in the background image, and then sends the pose information to the master device, and receives the foreground image sent by the master device. Finally, based on the pose information, the foreground image is spliced into the background image to generate an AR image, and the AR image is displayed on the display screen. Thus, the mobile terminal can determine the corresponding pose information based on the background image, then feedback the pose information to the master device, and receive the foreground image sent by the master device. Finally, an AR image is generated based on the foreground image. Thus, bystanders can obtain VR content that matches their own perspective.

[0059] It should be noted that the execution subject of this embodiment can be an AR image display system, or a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a mobile terminal that can implement the above functions. This embodiment does not make specific limitations in this regard. Hereinafter, taking the mobile terminal as the execution subject as an example, this embodiment and the following embodiments will be described.

[0060] Embodiment 1

[0061] Based on this, an embodiment of this application provides an AR image display method, which is applied to a mobile terminal. Refer to Figure 1 , Figure 1 is a schematic flowchart of the first embodiment of the AR image display method of this application. The AR image display method includes steps S10 to S40:

[0062] Step S10: Obtain the background image collected by the imaging device.

[0063] In this embodiment, the mobile terminal performs the processing action. The mobile terminal can be based on the Android system or the iOS system, which is not limited here. The master device is the interaction end between the user and the virtual scene, and is used to share the VR image of the master device. Among them, the master device can be a terminal device or a mobile terminal. The mobile terminal is communicatively connected to the imaging device, and the imaging device is used to capture the background image corresponding to the mobile terminal.

[0064] Step S20: Determine the pose information of the imaging device by identifying the positioning identifier in the background image.

[0065] In this embodiment, the positioning identifier in the background image is used to determine the pose information of the imaging device, and the pose information includes position information and attitude information.

[0066] Specifically, the pose information includes the position information and the attitude information of the imaging device in space. Among them, the position information is composed of three-dimensional coordinates or a translation vector, representing the position of the imaging device, and the attitude information can be described by a rotation matrix or Euler angles, representing the rotation state of the imaging device based on a certain reference coordinate. The background image collected by the imaging device is a real image, which is used as the real information part of the augmented reality image. The background image is the background image collected by the imaging device corresponding to the mobile terminal based on the background of the mobile terminal.

[0067] Optionally, in this embodiment, the step S20 includes:

[0068] Obtain preset marker information, and in the background image, identify the visual marker ArUco code that matches the marker information; determine the pixel coordinate position of the ArUco code in the background image; calculate the rotation matrix and translation vector of the imaging device corresponding to the plane where the background image is located according to the pixel coordinate position and the marker information; determine the pose information of the imaging device according to the rotation matrix and the translation vector.

[0069] It should be noted that the ArUco code (Augmented Reality University of Cordoba) is a marker system for computer vision, which can detect and identify specific markers in an image. The ArUco code consists of a black border and a binary matrix. Among them, the black border of the ArUco code helps its rapid detection in the image, while the internal binary coding is used to identify the marker and provide error detection and correction. The rotation matrix is a 3×3 matrix, which is used to describe the rotation state of the camera in space. The translation vector is a 3×1 vector, which represents the position of the camera in space.

[0070] In this embodiment, the background image contains an ArUco code, which is used to assist the image display system in identifying the relative pose of the imaging device and the plane where the ArUco code is located to determine the camera pose information. Among them, the ArUco code can be displayed on the target display screen of the main device, or other positions such as the display screen for displaying virtual reality images. A preset ArUco code dictionary is defined in the main device. After the main device identifies and determines that there is an ArUco code in the background image collected by the imaging device, it loads the preset ArUco code dictionary and detects the ArUco code in the background image. Among them, the detection process includes identifying the black border and internal coding of the marker. The main device will store the pixel coordinate position of the detected ArUco code, and calculate the rotation matrix and translation vector of the imaging device according to the mapping relationship between the pixel coordinate position and the camera pose information, and combine the rotation matrix and the translation vector to generate the complete camera pose information.

[0071] Step S30: Send the pose information to the master device and receive the foreground image sent by the master device.

[0072] In this embodiment, after determining the pose information, the pose information is sent to the master device through wireless communication. The master device generates a foreground image according to the pose information and feeds back the foreground image to the mobile terminal.

[0073] Step S40: Based on the pose information, splice the foreground image into the background image to generate an AR image.

[0074] In this embodiment, based on the mapping relationship between the pose information and the image splicing area in the background image, the mobile terminal can determine the pixel point coordinate range where the object is located in the background image according to the pose information and the pose information of the object to be covered in the actual scene in the background picture, and use this pixel point coordinate range as the image splicing area of the foreground image in the background image.

[0075] Optionally, in this embodiment, the step of splicing the foreground image into the background image based on the pose information to generate an AR image includes:

[0076] Obtain the display screen information associated with the identifier corresponding to the display screen; determine the corresponding display screen area in the background image according to the display screen information and the pose information; use the display screen area as the image splicing area, and overlap the foreground image onto the image splicing area to generate the AR image.

[0077] Specifically, the background image contains a display screen image, which is a virtual reality image displayed on the display screen collected by the camera device from the perspective corresponding to the camera pose information. During the process of splicing the foreground image by the slave device, the pixel point area of the display screen image in the background image will be used as the covering area of the foreground image.

[0078] Furthermore, by splicing the cropped foreground image with the display screen image in the background image, that is, the content on the original screen of the master device is replaced by the foreground image, so as to reflect the visual perception that the foreground image extends from the display screen image.

[0079] Optionally, the mobile terminal can also adjust and scale different pieces of information to be displayed in the foreground image, and then splice them again to generate a corresponding foreground image. By displaying a virtual reality image in the display screen image in the background technology and splicing the foreground image with the display screen image in a coincident manner, the display effect of the augmented reality image is improved.

[0080] Step S50: Display the AR image on the display screen.

[0081] In this embodiment, the display screen is a smart large screen equipped with the Android system, or it can also be the screen of the mobile terminal itself. The mobile terminal transmits the data of the image to be displayed to the display controller of the display screen, and the display controller renders the image data into the graphics memory of the display screen. The display screen displays the AR image according to the data in the graphics memory.

[0082] In the technical solution provided in this embodiment, first, obtain the background image collected by the imaging device, then determine the pose information of the imaging device by identifying the positioning identifier in the background image, and then send the pose information to the master device, and receive the foreground image sent by the master device. Finally, based on the pose information, splice the foreground image into the background image to generate an AR image, and display the AR image on the display screen. Thus, the mobile terminal can determine the corresponding pose information based on the background image, then feedback the pose information to the master device, and receive the foreground image sent by the master device. Finally, generate an AR image based on the foreground image. So that bystanders can obtain VR content that matches their own perspective.

[0083] Embodiment Two

[0084] Please refer to Figure 2 , in the second embodiment, after step S40, steps S41 to S43 are further included:

[0085] Step S41: When it is detected that the position of the imaging device has changed, obtain the current background image collected by the imaging device.

[0086] Step S42: Determine the current pose information of the imaging device according to the positioning identifier in the current background image.

[0087] Step S43: Send the current pose information to the master device.

[0088] In this embodiment, the mobile terminal can monitor the position change of the imaging device in real time through sensors (such as accelerometers, gyroscopes or vision sensors). When it is detected that the position or orientation has changed, the sensor sends an instruction to the imaging device, which will trigger the shooting operation of the imaging device and shoot the current background image of the mobile terminal. Then, according to the positioning identifier in the current background image, determine the current pose information of the imaging device, and feedback the current pose information to the master device so that the master device can generate a target foreground image according to the current pose information.

[0089] Optionally, in this embodiment, after the step of sending the current pose information to the master device, the following is further included:

[0090] Receive the target foreground image sent by the master device; based on the image stitching area corresponding to the current pose information in the current background image, stitch the target foreground image into the current background image to generate a target AR image; display the target AR image on the display screen.

[0091] In this embodiment, when the position of the imaging device associated with the mobile terminal changes, or the position of the mobile terminal changes, it is necessary to re-determine the current background image captured by the imaging device and the current pose information. Thus, the mobile terminal receives the target foreground image updated by the master device according to the current pose information. The mobile terminal receives the target foreground image sent by the master device through wireless communication, where the wireless communication can be in the form of wifi, and sends the current background image and the current pose information to the mobile terminal.

[0092] Then, the mobile terminal determines the stitching position and area of the target foreground image in the background image according to the pose information. And fuse the target foreground image with the background image to generate a target AR image. The processes of generating the target foreground image and stitching to generate the target AR image have been discussed in Embodiment 1 and will not be elaborated here. Finally, the mobile terminal transmits the generated target AR image to the display screen for display.

[0093] In the technical solution provided in this embodiment, when it is detected that the position of the imaging device changes, obtain the current background image captured by the imaging device, then determine the current pose information of the imaging device according to the positioning identifier in the current background image, and then send the current pose information to the master device. It is possible to adjust the background image in real time when the position of the imaging device changes, thereby improving the accuracy of AR image sharing. And, by receiving the target foreground image sent by the master device, and then based on the image stitching area corresponding to the current pose information in the current background image, stitch the target foreground image into the current background image to generate a target AR image, so as to display the target AR image on the display screen. The solution of this embodiment generates a target AR image according to the changed current pose information and the target foreground image when the position information of the imaging device changes, so as to complete the real-time sharing of VR images.

[0094] Embodiment 3

[0095] Please refer to Figure 3 , in the third embodiment, applied to the master device, the AR image display method includes steps S60 to S80:

[0096] Step S60: Receive the pose information sent by the mobile terminal.

[0097] Step S70: Construct a virtual camera according to the pose information, and capture a foreground image of the virtual scene through the virtual camera.

[0098] Step S80: Send the foreground image to the mobile terminal.

[0099] In this embodiment, the main device performs the processing actions. The main device can be a terminal device, such as a PC or other devices, or a mobile terminal, such as a mobile phone, a tablet computer or other devices. The main device is communicatively connected to the imaging device, and the imaging device is used to capture a background image corresponding to the main device. The positioning identifier in the background image is used to determine the pose information of the imaging device, and the pose information includes position information and attitude information. After determining the pose information, the pose information and the background image are sent to the mobile terminal through wireless communication.

[0100] The foreground image is the virtual image information in the augmented reality image. The virtual camera is used to simulate a real camera in computer graphics, and defines the viewing angle and parameters for observing the virtual scene. The main device (rendering is all completed by the main device, and the main work of the secondary device is imaging, positioning, and AR image synthesis) sets the parameters of the virtual camera according to the pose information of the imaging device, including position, direction, field of view angle, etc., and uses a rendering engine to render the virtual model. During the rendering process, the virtual camera converts the three-dimensional objects in the virtual model into two-dimensional images, and the generated image is the foreground image, which contains elements in the virtual scene, such as virtual characters, objects and other information.

[0101] Specifically, the pose information is data describing the position and direction of the virtual camera in the three-dimensional space, and usually includes the coordinates (x, y, z) of the virtual camera in the three-dimensional space. The orientation of the virtual camera is usually represented by a quaternion or Euler angles. Specifically, the position and direction data can be extracted from the pose information received by the main device. The virtual camera is the observation point in the virtual scene, and its parameters determine how to project the three-dimensional virtual scene onto a two-dimensional image.

[0102] As an alternative implementation of constructing a virtual camera, first determine the internal parameter matrix of the virtual camera, including the focal length, pixel size, principal point, etc., and then construct it according to the position and orientation in the pose information, which is used to describe the position and orientation of the virtual camera in the world coordinate system. Use the position and orientation in the pose information to construct the external parameter matrix (usually a 4×4 homogeneous transformation matrix). Combine the internal parameter matrix to construct a complete virtual camera model. The virtual scene is a pre-designed 3D model or virtual content generated by a program. The generation of the foreground image requires the virtual camera to render the virtual scene. Then use the external parameter matrix to transform the virtual scene from the world coordinate system to the view coordinate system of the virtual camera, apply the projection matrix to project the 3D scene onto a 2D plane, and generate the foreground image according to the preset rendering technology.

[0103] Optionally, in this embodiment, the step of constructing a virtual camera according to the pose information and capturing a foreground image of the virtual scene by the virtual camera includes:

[0104] Map the position information and attitude information to the virtual scene, and create a virtual camera at the corresponding position in the virtual scene; capture the foreground image corresponding to the virtual scene through the virtual camera.

[0105] Specifically, the pose information is the pose information of the imaging device, and the pose information includes position information and attitude information. First, it is necessary to convert the position information and attitude information sent by the master device into coordinates and directions in the virtual scene. The position information is represented as three-dimensional coordinates (x, y, z), and the attitude information is represented as rotation angles (Euler angles) or quaternions. These information are used to determine the specific position and orientation of the virtual camera in the virtual scene. Then, in the virtual scene, create a virtual camera according to the mapped position and attitude information. The virtual scene is a pre-constructed 3D model that contains virtual elements (such as virtual objects, special effects, etc.) that need to be displayed. After creating the virtual camera, it is necessary to ensure that the virtual scene has been loaded and is ready for rendering. The elements in the virtual scene will be rendered according to the position and direction of the virtual camera. Render the virtual scene through the virtual camera to generate the foreground image.

[0106] The specific process of rendering is to capture the virtual scene at the determined position according to the position and direction of the virtual camera to generate the foreground image, where the projection method uses perspective projection.

[0107] Specifically, after generating the foreground image, the host device will crop the rendered foreground image and only retain the information to be displayed that is needed. For example, the foreground image before cropping is usually a complete rectangular image, containing information such as virtual objects and virtual scenes. Through cropping, the virtual objects in the image can be retained while the virtual scene information is removed. It is also possible to crop the image outside the screen to produce a natural and realistic AR effect.

[0108] In the technical solution provided in this embodiment, by receiving the pose information sent by the mobile terminal, then constructing a virtual camera according to the pose information, capturing the foreground image of the virtual scene through the virtual camera, and finally sending the foreground image to the mobile terminal. In the solution of this embodiment, the host device receives the pose information and generates the corresponding foreground image, so that the mobile terminal obtains the foreground image of the VR scene, thereby improving the accuracy of VR image sharing.

[0109] Embodiment 4

[0110] Please refer to Figure 4 , based on any one of the embodiments, the step of capturing the foreground image corresponding to the virtual scene by the virtual camera includes steps S90 to S110:

[0111] Step S90: Determine the view angle information and viewport information corresponding to the virtual camera according to the pose information.

[0112] Step S100: Render the virtual scene according to the corresponding view angle information by the virtual camera to obtain a rendered image.

[0113] Step S110: Crop the rendered image according to the viewport information to obtain the foreground image.

[0114] In this embodiment, after obtaining the camera pose information of the imaging device in the real space, the host device will create a virtual camera with the same pose as the real camera in the virtual scene, so that the virtual camera can capture and render the virtual model.

[0115] Furthermore, based on the relative pose between the real camera and the VR display screen, that is, the camera pose information of the imaging device, the mobile terminal can obtain the line-of-sight range extending from the perspective of the imaging device to the virtual scene in the perspective of the imaging device, that is, the VR content that can be seen through the VR display screen, so as to determine the viewport information of the virtual camera. After the virtual camera completes the rendering of the virtual model, the virtual camera will further determine the visual area in the rendered image based on the viewport information, and crop the rendered image based on the visual area to generate the foreground image.

[0116] Necessarily, the master device can obtain the foreground image generated by rendering the virtual model and determine the image information to be displayed in the foreground image. By cropping the foreground image, the target image information other than the image information to be displayed in the foreground image is removed. Among them, the master device can determine the displayable image information according to the camera device perspective corresponding to the position relationship between the camera device and the display screen, or can also determine the image information to be displayed for AR imaging in the scene according to the model information of the virtual model. The master device can also crop different image information to be displayed respectively, and scale and re-stitch the images after cropping to form foreground images with different perspective effects.

[0117] In the technical solution provided in this embodiment, according to the pose information, the perspective information and viewport information corresponding to the virtual camera are determined, then the virtual scene is rendered according to the virtual camera with the corresponding perspective information to obtain a rendered image, and finally, according to the viewport information, the rendered image is cropped to obtain the foreground image. By cropping the rendered image with the viewport information, the accuracy of the visual effect of the foreground image is improved.

[0118] Since the system introduced in the embodiments of the present application is the system adopted for implementing the methods in the embodiments of the present application, based on the methods introduced in the embodiments of the present application, those skilled in the art can understand the specific structure and variations of the system, so it will not be elaborated here. Any system adopted by the methods in the embodiments of the present application falls within the scope of protection of the present application.

[0119] The present application provides an AR image display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the AR image display method in the first embodiment above.

[0120] Next, refer to Figure 5 , which shows a schematic structural diagram of an AR image display device suitable for implementing the embodiments of the present application. The AR image display device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5The AR image display device shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0121] As Figure 5 shown, the AR image display device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the random access memory RAM 1004, various programs and data required for the operation of the AR image display device are also stored. The processing device 1001, the read-only memory ROM 1002, and the random access memory RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the AR image display device to communicate with other devices wirelessly or wiredly to exchange data. Although the AR image display device with various systems is shown in the figure, it should be understood that it is not required to implement or include all the shown systems. More or fewer systems may be alternatively implemented or included.

[0122] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device, or installed from the storage device 1003, or installed from the read-only memory ROM 1002. When the computer program is executed by the processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are performed.

[0123] The AR image display device provided by this application adopts the AR image display method in the above-mentioned embodiment, and can solve the technical problem that the AR image display on the mobile terminal in the traditional solution is inaccurate. Compared with the prior art, the beneficial effects of the AR image display device provided by this application are the same as those of the AR image display method provided by the above-mentioned embodiment, and other technical features in the AR image display device are the same as those disclosed in the method of the previous embodiment, which will not be elaborated here.

[0124] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0125] As mentioned above, it is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0126] This application provides a computer-readable storage medium, which has computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the AR image display method in the above-mentioned embodiment.

[0127] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device or component. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination of the above.

[0128] The above computer-readable storage medium may be included in an AR image display device; or it may exist independently and not be assembled into the AR image display device.

[0129] The above computer-readable storage medium stores one or more programs. When the one or more programs are executed by an AR image display device, the AR image display device is caused to: obtain a background image collected by an imaging device; determine the pose information of the imaging device by recognizing a positioning identifier in the background image; send the pose information to a host device, and receive a foreground image sent by the host device; based on the pose information, splice the foreground image into the background image to generate an AR image; and display the AR image on a display screen.

[0130] Computer program code for performing the operations of the present application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java and C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, execute as a stand-alone software package, execute partially on the user's computer and partially on a remote computer, or execute entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, by connecting through an Internet service provider using the Internet).

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0132] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0133] The readable storage medium provided in the present application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above AR image display method, and can solve the technical problem that the AR image display on a mobile terminal is inaccurate in the traditional solution. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as those of the AR image display method provided in the above embodiments, and will not be elaborated here.

[0134] An embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the AR image display method as described above are implemented.

[0135] The computer program product provided in the present application can solve the technical problem that the AR image display on a mobile terminal is inaccurate in the traditional solution. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of the present application are the same as those of the AR image display method provided in the above embodiments, and will not be elaborated here.

[0136] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structural or equivalent process transformation made by using the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent scope of the present application.

Claims

1. An AR image display method, characterized in that, Applied to a mobile terminal, the AR image display method includes the following steps: Obtain a background image collected by a camera device; Determine the pose information of the camera device by identifying a positioning identifier in the background image; Send the pose information to a master device and receive a foreground image sent by the master device; Based on the pose information, splice the foreground image into the background image to generate an AR image; Display the AR image on a display screen.

2. The method according to claim 1, wherein The step of determining the pose information of the camera device by identifying a positioning identifier in the background image includes: Obtain preset marker information and identify a visual marker ArUco code that matches the marker information in the background image; Determine the pixel coordinate position of the ArUco code in the background image; According to the pixel coordinate position and the marker information, calculate a rotation matrix and a translation vector of the camera device corresponding to the plane where the background image is located; Determine the pose information of the camera device according to the rotation matrix and the translation vector.

3. The method according to claim 1, wherein The step of splicing the foreground image into the background image based on the pose information to generate an AR image includes: Obtain display screen information associated with an identifier corresponding to the display screen; Determine a corresponding display screen area in the background image according to the display screen information and the pose information; Use the display screen area as an image splicing area and overlap the foreground image onto the image splicing area to generate the AR image.

4. The method according to claim 1, characterized in that After the step of determining the pose information of the camera device by identifying a positioning identifier in the background image, it further includes: When it is detected that the position of the camera device has changed, obtain the current background image collected by the camera device; Determine the current pose information of the camera device according to the positioning identifier in the current background image; Send the current pose information to the master device.

5. The method according to claim 4, wherein After the step of sending the current pose information to the master device, it further includes: Receive a target foreground image sent by the master device; Based on the image splicing area corresponding to the current pose information in the current background image, splice the target foreground image into the current background image to generate a target AR image; Display the target AR image on the display screen.

6. An AR image display method, characterized in that, Applied to a master device, the AR image display method includes the following steps: Receive pose information sent by a mobile terminal; Construct a virtual camera according to the pose information and capture a foreground image of a virtual scene through the virtual camera; Send the foreground image to the mobile terminal.

7. The method according to claim 6, characterized in that, The pose information is the pose information of a camera device, and the pose information includes position information and attitude information. The step of constructing a virtual camera according to the pose information and capturing a foreground image of a virtual scene through the virtual camera includes: Map the position information and the attitude information to the virtual scene and create a virtual camera at the corresponding position in the virtual scene; Capture the foreground image corresponding to the virtual scene through the virtual camera.

8. The method according to claim 7, wherein The step of capturing the foreground image corresponding to the virtual scene through the virtual camera includes: Determine the viewing angle information and window information corresponding to the virtual camera according to the pose information; Render the virtual scene according to the corresponding viewing angle information by the virtual camera to obtain a rendered image; Crop the rendered image according to the window information to obtain the foreground image.

9. An AR image display device, characterized in that, The AR image display device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the AR image display method based on the Android system according to any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the AR image display method according to any one of claims 1 to 8 are implemented.

Images