Control method and device of image display system and storage medium

Abstract

The invention discloses a control method and device of an image display system and a storage medium, and belongs to the technical field of data processing. The method comprises the steps of receiving a background image collected by camera equipment, obtaining camera pose information of the camera equipment, rendering a virtual model based on the camera pose information through a virtual camera, generating a foreground image, splicing the foreground image into the background image based on a corresponding image splicing area of the camera pose information in the background image, and displaying the foreground image in the background image. And generating an augmented reality image, and outputting the augmented reality image to the target display screen. According to the method and the device, the virtual reality image is converted into the augmented reality image based on the camera pose information, so that in the interaction process of an operator and a virtual scene through the virtual display screen, bystanders can also watch the virtual scene and the interaction process from different visual angles through the augmented reality image.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a control method, device, and storage medium for an image display system. Background Art

[0002] A virtual reality (VR, Virtual Reality) display screen can use VR technology to present a 3D virtual scene to users, thereby enhancing the sense of interaction between the user and the information displayed on the screen.

[0003] In the related art, based on the differences in the perspectives of a person's two eyes, the VR display screen determines the left-eye view and the right-eye view by obtaining the position information of the user's eyes, and then uses an interleaving algorithm to restore and display the left-eye rendered image and the right-eye rendered image.

[0004] However, since the images rendered by the VR display screen are based on the parallax between the operator's left and right eyes for VR image rendering and display to produce a 3D effect, bystanders cannot directly obtain clear VR content when viewing the images on the VR display screen from different perspectives.

[0005] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a control method, device, and storage medium for an image display system, aiming to solve the technical problem that bystanders cannot view the VR content on the VR display screen during the interaction process of the VR display screen.

[0007] To achieve the above objective, this application provides a control method for an image display system. The method includes the following steps:

[0008] Receive the background image collected by the imaging device, and obtain the camera pose information of the imaging device;

[0009] Render the virtual model through a virtual camera based on the camera pose information to generate a foreground image;

[0010] Based on the image stitching area corresponding to the camera pose information in the background image, stitch the foreground image into the background image to generate an augmented reality image;

[0011] Output the augmented reality image to the target display screen.

[0012] In one embodiment, the step of stitching the foreground image into the background image based on the image stitching area corresponding to the camera pose information in the background image to generate an augmented reality image includes:

[0013] Obtain pre-stored display screen information;

[0014] Determine the display screen image area in the background image according to the display screen information and the camera pose information;

[0015] Use the display screen image area as the image stitching area, and overlap the foreground image onto the image stitching area to generate the augmented reality image.

[0016] In one embodiment, before the step of receiving the background image collected by the imaging device and obtaining the camera pose information of the imaging device, it further includes:

[0017] Obtain the display screen viewing angle information of the user;

[0018] Render the virtual model according to the display screen viewing angle information to generate a virtual reality image of the virtual model;

[0019] Display the virtual reality image on the display screen.

[0020] In one embodiment, the step of rendering the virtual model based on the camera pose information through a virtual camera to generate a foreground image includes:

[0021] Determine the viewing angle information and window information of the virtual camera according to the camera pose information of the imaging device;

[0022] Render the virtual model through the virtual camera based on the viewing angle information to obtain a rendered image;

[0023] Crop the rendered image based on the window information to obtain the foreground image.

[0024] In one embodiment, the step of rendering the virtual model based on the camera pose information through a virtual camera to generate a foreground image further includes:

[0025] Obtain the foreground image generated by rendering the virtual model, and determine the image information to be displayed in the foreground image;

[0026] Crop the foreground image to remove the target image information other than the image information to be displayed in the foreground image.

[0027] In one embodiment, the step of receiving the background image collected by the imaging device and obtaining the camera pose information of the imaging device includes:

[0028] Receive the background image collected by the imaging device;

[0029] Obtain the preset marker information, and in the background image, identify the visual marker ArUco code that matches the marker information;

[0030] Determine the pixel coordinate position of the ArUco code in the background image;

[0031] According to the pixel coordinate position and the marker information, calculate the rotation matrix and translation vector of the imaging device corresponding to the plane where the background image is located;

[0032] According to the rotation matrix and translation vector, determine the camera pose information of the imaging device.

[0033] In one embodiment, the step of rendering the virtual model based on the camera pose information through the virtual camera to generate the foreground image includes:

[0034] In one embodiment, after the step of outputting the augmented reality image to the target display screen, the following is further included:

[0035] According to the interaction information between the virtual model and the user, real-time update the model pose information of the virtual model;

[0036] Based on the updated model pose information and the camera pose information, render the virtual model to generate the target foreground image;

[0037] Stitch the target foreground image into the background image to generate a target augmented reality image, and output the target augmented reality image to the target display screen in real time.

[0038] In addition, to achieve the above object, the present application further provides an image display system, which is applied to the control method of the image display system as described above. The image display system includes:

[0039] An imaging device, configured to collect a background image containing the main device image and transmit the background image to the main device;

[0040] A main device, configured to receive the background image collected by the imaging device, and through a virtual camera, render a virtual scene based on the camera pose information to generate a foreground image, and send the background image and the foreground image to the secondary device;

[0041] A secondary device, configured to receive the foreground image and the background image sent by the main device, and according to the camera pose information of the imaging device, in the corresponding image stitching area of the background image, stitch the foreground image into the scene image to generate an augmented reality image, and display the augmented reality image.

[0042] In addition, to achieve the above object, the present application further provides a control device for an image display system, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the control method of the image display system as described above.

[0043] In addition, to achieve the above object, the present application further provides a storage medium, the storage medium being a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the control method of the image display system as described above are implemented.

[0044] One or more technical solutions proposed by the present application have at least the following technical effects:

[0045] In the process of generating an augmented reality image, the present application renders the foreground image of the virtual model through a virtual camera with the same pose information as that of the real camera device, and based on this pose information in the background image, the image splicing area corresponding to the virtual reality image, and splices the foreground image into the background image, so as to realize the fusion of the virtual foreground image and the real background image, and thus convert the virtual reality image into an augmented reality image based on the camera pose information, so that when the operator interacts with the virtual scene through the virtual display screen, bystanders can also view the virtual scene and the interaction process from different perspectives through the augmented reality image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0047] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0048] Figure 1 It is a schematic flowchart of the first embodiment of the control method of the image display system of the present application;

[0049] Figure 2 It is a schematic flowchart of the second embodiment of the control method of the image display system of the present application;

[0050] Figure 3 It is a schematic flowchart of the third embodiment of the control method of the image display system of the present application;

[0051] Figure 4Schematic flowchart of the fourth embodiment of the control method for the image display system of the present application;

[0052] Figure 5 Schematic flowchart of the fifth embodiment of the control method for the image display system of the present application;

[0053] Figure 6 Schematic diagram of the structure of the control device of the image display system in the hardware operating environment involved in the solution of the embodiment of the present application.

[0054] The realization, functional features and advantages of the object of the present application will be further described in conjunction with the embodiments with reference to the accompanying drawings. Detailed implementation manners

[0055] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0057] The main solution of the embodiment of the present application is: receiving a background image collected by a camera device, and obtaining the camera pose information of the camera device; through a virtual camera, based on the camera pose information, rendering a virtual model to generate a foreground image; based on an image stitching area corresponding to the camera pose information in the background image, stitching the foreground image to the background image to generate an augmented reality image; outputting the augmented reality image to a target display screen.

[0058] In the related art, due to the differences in the perspectives of the human binoculars, the VR display screen determines the left-eye line of sight and the right-eye line of sight by obtaining the position information of the user's eyeballs, and then through an interleaving algorithm, restores and displays the left-eye rendered image and the right-eye rendered image. However, since the images rendered by the VR display screen are rendered and displayed based on the parallax of the operator's left and right eyes to produce a 3D effect, therefore, when a bystander views the images of the VR display screen from different perspectives, they cannot directly obtain clear VR content.

[0059] In the process of generating the augmented reality image in the present application, a foreground image of a virtual model is rendered through a virtual camera with the same pose information as that of a real camera device, and based on the pose information in the background image, in the image stitching area corresponding to the virtual reality image, the foreground image is stitched to the background image, realizing the fusion of the virtual foreground image and the real background image, and thus converting the virtual reality image into an augmented reality image based on the camera pose information, so that during the interaction process between the operator and the virtual scene through the virtual display screen, the bystander can also view the virtual scene and the interaction process from different perspectives through the augmented reality image.

[0060] To better understand the above technical solution, the exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0061] It should be noted that the execution subject of this embodiment can be an image display system, or a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of implementing the above functions, a control device of an image display system, etc. This embodiment does not make specific limitations in this regard. The following takes an image display system as an example to illustrate this embodiment and the following embodiments.

[0062] Based on this, the embodiment of the present application provides a control method for an image display system, referring to Figure 1 , Figure 1 which is a schematic flowchart of the first embodiment of the control method for the image display system of the present application.

[0063] In this embodiment, the control method of the image display system includes steps S10 to S40:

[0064] Step S10: Receive the background image collected by the imaging device, and obtain the camera pose information of the imaging device;

[0065] In this embodiment, the camera pose information of the imaging device includes the position information and attitude information of the imaging device in space. Among them, the position information is composed of three-dimensional coordinates or a translation vector, representing the position of the imaging device, and the attitude information can be described by a rotation matrix or Euler angles, representing the rotation state of the imaging device based on a certain reference coordinate. The background image collected by the imaging device is a real image, which is used as the real information part of the augmented reality image.

[0066] As an alternative implementation manner for obtaining the camera pose information, step S10 includes steps S11 to S15:

[0067] Step S11: Receive the background image collected by the imaging device;

[0068] Step S12: Obtain the preset marker information, and in the background image, identify the visual marker ArUco code that matches the marker information;

[0069] Step S13: Determine the pixel coordinate position of the ArUco code in the background image;

[0070] Step S14: Calculate the rotation matrix and translation vector of the imaging device corresponding to the plane where the background image is located according to the pixel coordinate position and the marker information;

[0071] Step S15: Determine the camera pose information of the imaging device according to the rotation matrix and the translation vector.

[0072] It should be noted that the ArUco code (Augmented Reality University of Cordoba) is a marker system for computer vision, which can detect and recognize specific markers in images. The ArUco code consists of a black border and a binary matrix. Among them, the black border of the ArUco code helps its rapid detection in the image, while the internal binary coding is used to identify the marker and provide error detection and correction. The rotation matrix is a 3×3 matrix used to describe the rotation state of the camera in space. The translation vector is a 3×1 vector representing the position of the camera in space.

[0073] In this embodiment, the background image contains ArUco codes, which are used to assist the image display system in identifying the relative pose between the imaging device and the plane where the ArUco codes are located to determine the camera pose information. Among them, the ArUco codes can be displayed on the target display screen or other positions such as the display screen for displaying virtual reality images (VR, Virtual Reality). A preset ArUco code dictionary is defined in the image display system. After the image display system recognizes and determines that there are ArUco codes in the background image collected by the imaging device, it loads the preset ArUco code dictionary and detects the ArUco codes in the background image. Among them, the detection process includes identifying the black border and internal coding of the marker. The image display system will store the pixel coordinate positions of the detected ArUco codes and calculate the rotation matrix and translation vector of the imaging device according to the mapping relationship between the pixel coordinate positions and the camera pose information, and combine the rotation matrix and the translation vector to generate the complete camera pose information.

[0074] Further, the image display system can optionally determine the mapping relationship between the pixel coordinate position and the camera pose information through methods such as Perspective-n-Point (PnP), multi-point calculation, or pinhole algorithm, and save it in the image display system, or adopt a real-time calculation method to calculate the mapping relationship between the pixel coordinate position and the camera pose information during the process of determining the camera pose information.

[0075] Exemplarily, taking the calculation of the rotation matrix and translation vector of a camera device using the PnP algorithm as an example, the internal parameter matrix and distortion coefficients of the camera are stored in the image display system to assist in calculating the camera pose information. When the PnP algorithm recognizes the ArUco code in the image, it will store the corner coordinates and ID information therein. For example, the detected corner coordinates of the marker with ID 1 are [(x1, y1), (x2, y2), (x3, y3), (x4, y4)], and it is determined that the corner coordinates of the marker are the pixel coordinate positions in the image. At the same time, the coordinates of the four corners of the marker in the world coordinate system are [(X1, Y1, Z1), (X2, Y2, Z2), (X3, Y3, Z3), (X4, Y4, Z4)]. The PnP algorithm constructs a camera model and an error function, and by transforming the three-dimensional points into the camera coordinate system and projecting them onto the image plane, the difference from the actually observed two-dimensional points is calculated. By continuously iterating the rotation matrix and translation vector of the camera model, the rotation matrix R and translation vector t with the smallest difference are determined as the rotation matrix and translation vector of the camera device.

[0076] As another alternative implementation, the image display system can pre-store the camera pose information of the camera device in the system, and when receiving the background image collected by the camera device, retrieve the currently stored camera pose information from the local.

[0077] Furthermore, the image display system can detect the camera pose information and update the camera pose information stored locally. The image display system can obtain the camera pose information of the camera device through sensors such as gyroscopes and accelerometers or external devices, or can also update the camera pose information by collecting images to recognize ArUco codes.

[0078] It should be noted that through the relatively fixed pose relationship between the camera and the main device, the image display system can determine the correct position and direction of the virtual object in the real world based on the position and direction of the camera in the three-dimensional space provided by the pre-stored camera pose information, avoiding the complex feature point processing process, simplifying the geometric transformation and fusion operations during the stitching process, and reducing the computational amount. At the same time, by directly obtaining the camera pose information of the camera device to locate the virtual camera, the virtual object can be more accurately aligned with the real scene, reducing the dependence on environmental features and improving the robustness and adaptability of the system. The main device is a terminal device with computing and rendering capabilities, such as a computer or a mobile phone. Among them, the main device optionally includes a VR display screen.

[0079] Optionally, the image display system can periodically detect and update the camera pose information based on a preset time interval, or can also perform real-time monitoring of the camera pose information through sensors and update the camera pose information when the camera pose information changes.

[0080] Step S20: Render the virtual model based on the camera pose information through a virtual camera to generate a foreground image;

[0081] In this embodiment, the foreground image is the virtual image information in the augmented reality image. The virtual camera is used to simulate a real camera in computer graphics and define the viewing angle and parameters for observing the virtual scene. The image display system sets the parameters of the virtual camera according to the camera pose information of the imaging device, including position, orientation, field of view, etc., and uses a rendering engine to render the virtual model. During the rendering process, the virtual camera converts the three-dimensional objects in the virtual model into two-dimensional images, and the generated image is the foreground image, which contains elements in the virtual scene, such as virtual characters, objects, etc.

[0082] Exemplarily, the virtual model includes a virtual scene model, a virtual object model, etc. According to the camera pose information of the imaging device, the viewing angle of the virtual camera is set to be the same as that of the real camera. The rendering engine renders the virtual model into a foreground image, which has the same viewing angle as the background image but contains virtual element information.

[0083] Optionally, the image display system renders the image captured by the virtual camera based on the viewing angle of the imaging device to generate a corresponding foreground image, so that the visual effect presented by the virtual model is only within the screen window.

[0084] Step S30: Stitch the foreground image to the background image in the image stitching area corresponding to the camera pose information in the background image to generate an augmented reality image;

[0085] In this embodiment, based on the mapping relationship between the camera pose information and the image stitching area in the background image, the image display system can determine the pixel point coordinate range where the object is located in the background image according to the camera pose information and the pose information of the object to be covered in the actual scene in the background picture, and use this pixel point coordinate range as the image stitching area of the foreground image in the background image. The image display system can use image stitching techniques to align the foreground image with the background image, such as feature point matching, homography transformation, etc., and fuse the aligned foreground image into the background image to generate an augmented reality image.

[0086] Exemplarily, assume that the background image is a panoramic view of a room, and the foreground image is a virtual character and furniture rendered by a virtual camera. By calculating the projection position of the foreground image in the background image, image stitching techniques are used to fuse the virtual character and furniture into the room panoramic view to generate a displayable image containing the real background and virtual elements.

[0087] Step S40: Output the augmented reality image to the target display screen.

[0088] In this embodiment, the target display screen is an AR display screen. The image display system transmits the data of the image to be displayed to the display controller of the target display screen, and the display controller renders the image data into the graphic memory of the display screen, and the display screen displays the image according to the data in the graphic memory.

[0089] In the embodiment of the present application, in the process of generating an augmented reality image, a foreground image of a virtual model is rendered through a virtual camera with the same pose information as that of a real camera device, and in the background image based on the pose information, an image splicing area corresponding to a virtual reality image is used to splice the foreground image to the background image, so as to realize the fusion of the virtual foreground image and the real background image. Thus, based on the camera pose information, the virtual reality image is converted into an augmented reality image, so that when an operator interacts with a virtual scene through a virtual display screen, a bystander can also view the virtual scene and the interaction process from different perspectives through the augmented reality image.

[0090] Based on the same inventive concept, the present application also provides a second embodiment. Refer to Figure 2 , Figure 2 which is a schematic flowchart of the second embodiment of the control method of the image display system of the present application.

[0091] In this embodiment, before receiving the background image collected by the camera device as described in step S10 and obtaining the camera pose information of the camera device, steps S01 to S03 are further included:

[0092] Step S01: Obtain the display screen viewing angle information of the user;

[0093] Step S02: Render the virtual model according to the display screen viewing angle information to generate a virtual reality image of the virtual model;

[0094] Step S03: Display the virtual reality image on the display screen.

[0095] In this embodiment, before the image display system generates an augmented reality image, a virtual reality image of the same virtual model as the augmented reality image will be displayed on another display screen outside the target display screen. Among them, the virtual reality image is displayed on the display screen, and the display screen is a VR display screen. The display screen can obtain the display screen viewing angle information of the user relative to the display screen, and render the virtual model based on the display screen viewing angle information to generate a virtual reality image.

[0096] Furthermore, the display screen perspective information includes the left eye perspective and the right eye perspective of the user when watching the VR display screen, and the complete virtual scene in the virtual model is rendered through the virtual cameras of the left eye perspective and the right eye perspective of the display screen respectively to generate the left eye image of the display screen and the right eye of the display screen, and through the interleaving algorithm, the left eye image of the display screen and the right eye perspective of the display screen are fused to form a naked-eye 3D virtual reality image.

[0097] Specifically, after obtaining the camera pose information including the rotation matrix and the translation vector, the image display system calculates the specific parameters of the corresponding left-eye virtual camera and right-eye virtual camera. The positions of the left-eye virtual camera and the right-eye virtual camera will be offset in the horizontal direction to simulate the parallax of the human eye. The translation vector in the user's perspective information is offset to the left and right sides by a certain distance, and the rotation matrix is ​​kept unchanged, so as to obtain the rendering perspectives of the left and right eyes. According to the determined left-eye and right-eye rendering perspectives, the parameters of the left-eye virtual camera and the right-eye virtual camera are set respectively, and the virtual model is rendered. The geometric data and material information of the virtual model are combined with the perspective of the virtual camera to render and generate the corresponding left-eye image and right-eye image.

[0098] Among them, the user's left eye perspective and right eye perspective can be obtained through sensing devices such as optical sensors, which are used to calculate the interleaving weight of the left eye image and the right eye image. The weight is determined by comparing the user's perspective with the perspective of the virtual camera according to the difference in perspective. Generally, the images with closer perspectives have a greater weight in the interleaving process. The system applies an interleaving algorithm to fuse the two images pixel by pixel according to the image interleaving weight to generate a virtual reality VR image. Among them, the VR image generated based on the different perspectives of the left eye and the right eye can be a naked eye 3D image.

[0099] Exemplarily, for each pixel position, the system calculates the pixel value of the position in the final foreground image according to the weight. Generally, the calculation formula of the pixel value is: Cforeground = wleft×Cleft+wright×Cright, where Cforeground is the pixel value of the foreground image, Cleft and Cright are the pixel values ​​of the left eye image and the right eye image respectively, and wleft and wright are the corresponding image interlacing weights.

[0100] Optionally, the user can also watch the VR image by wearing VR glasses or naked-eye 3D. This embodiment does not specifically limit this.

[0101] Optionally, the display screen can obtain the visual marker ArUco code and display the ArUco code together with the virtual reality image on the display screen, so that when the camera device captures the background image including the display screen image, it can identify the ArUco code in the display screen image and calculate the camera posture information.

[0102] Since the system introduced in the second embodiment of the present application is a system used to implement the method of the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems used in the method of the first embodiment of the present application belong to the scope of protection of this application.

[0103] Based on the same inventive concept, the present application also provides a third embodiment, referring to Figure 3 , Figure 3 This is a flow chart of a third embodiment of a control method for an image display system of the present application.

[0104] In this embodiment, as described in step S30, based on the image stitching area corresponding to the camera posture information in the background image, the foreground image is stitched into the background image, and the generation of the augmented reality image includes steps S31 to S33:

[0105] Step S31: Obtaining pre-stored display screen information;

[0106] Step S32: determining the display screen image area in the background image according to the display screen information and the camera posture information;

[0107] Step S33: using the display screen image area as an image stitching area, and overlapping the foreground image into the image stitching area to generate the augmented reality image.

[0108] In this embodiment, the background image includes a display screen image, which is a virtual reality image displayed on the display screen captured by the camera device at a viewing angle corresponding to the camera posture information. When the image display system splices the foreground image, the pixel area of ​​the display screen image in the background image is used as the coverage area of ​​the foreground image.

[0109] In one embodiment, the image display system crops the rendered foreground image before stitching the foreground image, retaining only the information to be displayed. For example, the foreground image before cropping is usually a complete rectangular image, including information such as virtual objects and virtual scenes. By cropping, only the virtual objects in the image can be retained, while the virtual scene information can be removed.

[0110] Furthermore, due to the lack of cropping of the foreground image, part of the VR image in the display screen image will be displayed as the background. Therefore, by splicing the cropped foreground image with the display screen image in the background image, the VR display screen image overlaps and connects with the cropped foreground image, reflecting the visual perception that the foreground image extends from the display screen image.

[0111] Exemplarily, when the virtual model includes a dolphin model and an ocean scene model, the virtual reality image displayed on the display screen is an image of a dolphin swimming in the ocean. The image display system removes the part of the ocean scene model in the foreground image through cropping, and connects the foreground image to the background image, creating a visual effect that the dolphin swims out of the display screen on the virtual reality image displayed on the display screen.

[0112] Optionally, the image display system can also adjust and scale different information to be displayed in the foreground image, and then splice them again to generate the corresponding foreground image.

[0113] In the embodiment of the present application, by displaying a virtual reality image in the display screen image of the background technology, and splicing the cropped foreground image with the display screen image in a coincident manner, the virtual reality image contains both the display screen image and the foreground image, so as to improve the display effect of the augmented reality image.

[0114] Since the system introduced in the third embodiment of the present application is the system adopted for implementing the method of the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, those skilled in the art can understand the specific structure and deformation of the system, so it will not be elaborated here. Any system adopted by the method of the first embodiment of the present application falls within the scope of protection of the present application.

[0115] Based on the same inventive concept, the present application also provides a fourth embodiment. Refer to Figure 4 , Figure 4 which is a schematic flowchart of the fourth embodiment of the control method of the image display system of the present application.

[0116] In this embodiment, as described in step S20, by using a virtual camera and based on the camera pose information, rendering the virtual model to generate a foreground image includes steps S21 to S23:

[0117] Step S21: Determine the viewing angle information and window information of the virtual camera according to the camera pose information of the imaging device;

[0118] Step S22: Render the virtual model through the virtual camera based on the viewing angle information to obtain a rendered image;

[0119] Step S23: Crop the rendered image based on the window information to obtain the foreground image.

[0120] In this embodiment, after obtaining the camera pose information of the imaging device in the real space, the image display system will create a virtual camera with the same pose as the real camera in the virtual scene, so that the virtual camera can capture and render the virtual model.

[0121] Further, based on the relative pose between the real camera and the VR display screen, i.e., the camera pose information of the imaging device, the image display system can obtain the line-of-sight range extending from the perspective of the imaging device to the virtual scene under the perspective of the imaging device, i.e., the VR content that can be seen through the VR display screen, so as to determine the window information of the virtual camera. After the virtual camera finishes rendering the virtual model, the virtual camera will further determine the visualization area in the rendered image based on the window information, and crop the rendered image based on the visualization area to generate a foreground image.

[0122] Optionally, the image display system can obtain the foreground image generated by rendering the virtual model and determine the image information to be displayed in the foreground image. By cropping the foreground image, the target image information other than the image information to be displayed in the foreground image is removed. Among them, the image display system can determine the displayable image information according to the perspective of the imaging device corresponding to the position relationship between the imaging device and the display screen, or can determine the image information to be displayed for AR imaging in the scene according to the model information of the virtual model. The image display system can also crop different image information to be displayed respectively, and scale and re-stitch the images after cropping to form foreground images with different perspective effects.

[0123] Since the system introduced in the fourth embodiment of the present application is the system adopted for implementing the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, those skilled in the art can understand the specific structure and variations of the system, so it will not be elaborated here. Any system adopted for the method in the first embodiment of the present application falls within the scope of protection of the present application.

[0124] Based on the same inventive concept, the present application also provides a fifth embodiment, referring to Figure 5 , Figure 5 which is a schematic flowchart of the fifth embodiment of the control method of the image display system of the present application.

[0125] In this embodiment, the control method of the image display system includes steps S51 to S54:

[0126] Step S51: Update the model pose information of the virtual model in real time according to the interaction information between the virtual model and the user;

[0127] Step S52: Render the virtual model based on the updated model pose information and the camera pose information to generate a target foreground image;

[0128] Step S53: Stitch the target foreground image into the background image to generate a target augmented reality image, and output the target augmented reality image to the target display screen in real time.

[0129] In this embodiment, the image display system can determine the user's interaction information based on the user's gesture, posture or other motion information, or the control signal of the control device, and control the delicate model to perform corresponding interaction actions based on the interaction information. Among them, based on the model pose information that changes during the interaction execution of the virtual model, the image display system can update the augmented reality image in real time.

[0130] Since the system introduced in the fifth embodiment of the present application is the system adopted for implementing the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, those skilled in the art can understand the specific structure and variations of the system, so it will not be elaborated here. Any system adopted by the method in the first embodiment of the present application falls within the scope of protection of the present application.

[0131] Based on the same inventive concept, the present application also provides an image display system and a control method applied to the image display system. The image display system includes a camera device, a main device and a secondary device. Among them, the main device is equipped with a VR display screen for displaying virtual reality images, and the secondary device is equipped with an AR display screen, that is, the target display screen, for displaying augmented reality images.

[0132] Specifically, the main device continuously displays the virtual reality image rendered by the virtual model. The camera device captures the background image including the image of the main device and transmits the background image to the main device. After receiving the background image, the main device obtains the camera pose information, and based on the camera pose information, renders the foreground image of the virtual scene through the virtual camera, crops the target image information that does not need to be displayed in the foreground image, and sends the background image and the cropped foreground image to the secondary device. After receiving the foreground image and the background image sent by the main device, the secondary device stitches the foreground image into the background image in the image stitching area corresponding to the main device image in the background image according to the camera pose information, connects the main device image and the foreground image to generate an augmented reality image, and displays the augmented reality image on the target display screen.

[0133] The present application provides a control device for an image display system. The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the control method of the image display system in the first embodiment above.

[0134] Next, refer to Figure 6, which shows a schematic structural diagram of a control device suitable for implementing the image display system according to the embodiments of the present application. The control device of the image display system in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The control device of the image display system shown is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0135] As Figure 6 shown, the control device of the image display system may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the control device of the image display system are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the control device of the image display system to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a control device of an image display system with various systems, it should be understood that it is not required to implement or have all the systems shown. Instead, more or fewer systems may be implemented or had.

[0136] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0137] The control device of the image display system provided by the present application adopts the control method of the image display system in the above embodiments, and can solve the technical problem that continuous image processing will occupy high computing resources, resulting in a high cost of real-time rendering of AR images. Compared with the prior art, the beneficial effects of the control device of the image display system provided by the present application are the same as those of the control method of the image display system provided in the above embodiments, and other technical features in the control device of the image display system are the same as the features disclosed in the method of the previous embodiment, and will not be elaborated here.

[0138] It should be understood that the various parts disclosed in the present application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0139] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0140] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the control method of the image display system in the above embodiments.

[0141] The computer-readable storage medium provided by the present application may, for example, be a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0142] The above computer-readable storage medium may be included in the control device of the image display system; or it may exist independently and not be assembled into the control device of the image display system.

[0143] The above computer-readable storage medium carries one or more programs. When the one or more programs are executed by the control device of the image display system, the control device of the image display system is caused to: receive the background image collected by the imaging device and obtain the camera pose information of the imaging device; through a virtual camera, based on the camera pose information, render a virtual model to generate a foreground image; based on the image stitching area corresponding to the camera pose information in the background image, stitch the foreground image into the background image to generate an augmented reality image; and output the augmented reality image to a target display screen.

[0144] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java and C++. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments described in this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0147] The readable storage medium provided in this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for performing the control method of the above-mentioned image display system, and can solve the technical problem that continuous image processing will occupy high computing resources, resulting in a high cost of real-time rendering of AR images. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the control method of the image display system provided in the above embodiments, and will not be elaborated here.

[0148] The above are only some embodiments of the present application, and thus do not limit the patent scope of the present application. Any equivalent structural transformation made under the technical concept of the present application by using the content of the specification and drawings of the present application, or any direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A control method for an image display system, characterized in that: The method comprises the following steps: Receiving a background image captured by a camera device and obtaining camera position information of the camera device; Rendering the virtual model through a virtual camera based on the camera pose information to generate a foreground image; Based on the image stitching area corresponding to the camera posture information in the background image, stitching the foreground image into the background image to generate an augmented reality image; The augmented reality image is output to a target display screen.

2. The method according to claim 1, characterized in that The step of stitching the foreground image into the background image based on the image stitching area corresponding to the camera posture information in the background image to generate an augmented reality image comprises: Get the pre-stored display screen information; Determine a display screen image area in the background image according to the display screen information and the camera posture information; The display screen image area is used as an image stitching area, and the foreground image is overlapped in the image stitching area to generate the augmented reality image.

3. The method according to claim 1, characterized in that Before the step of receiving the background image captured by the camera device and obtaining the camera pose information of the camera device, the method further includes: Get the user's display viewing angle information; Rendering the virtual model according to the display screen viewing angle information to generate a virtual reality image of the virtual model; The virtual reality image is displayed on a display screen.

4. The method according to claim 1, characterized in that The step of rendering the virtual model by the virtual camera based on the camera posture information to generate a foreground image comprises: Determining the viewing angle information and window information of the virtual camera according to the camera posture information of the camera device; Rendering the virtual model based on the viewing angle information through the virtual camera to obtain a rendered image; Based on the viewport information, the rendered image is cropped to obtain the foreground image.

5. The method according to claim 1, characterized in that The step of rendering the virtual model by the virtual camera based on the camera pose information to generate a foreground image also includes: Acquire the foreground image generated by rendering the virtual model, and determine the image information to be displayed in the foreground image; The foreground image is cropped to remove target image information other than the image information to be displayed in the foreground image.

6. The method according to claim 1, characterized in that The step of receiving a background image captured by a camera device and obtaining camera pose information of the camera device comprises: Receiving a background image captured by a camera device; Acquire preset marking information, and identify the visual marking ArUco code matching the marking information in the background image; Determine the pixel coordinate position of the ArUco code in the background image; Calculating the rotation matrix and translation vector of the camera device corresponding to the plane where the background image is located according to the pixel coordinate position and the marking information; The camera position information of the camera device is determined according to the rotation matrix and the translation vector.

7. The method according to claim 1, characterized in that After the step of outputting the augmented reality image to the target display screen, the method further includes: According to the interaction information between the virtual model and the user, updating the model posture information of the virtual model in real time; Based on the updated model pose information and the camera pose information, the virtual model is rendered to generate a target foreground image; The target foreground image is spliced ​​into the background image to generate a target augmented reality image, and the target augmented reality image is output to the target display screen in real time.

8. An image display system, characterized in that: The control method for the image display system according to claim 1, wherein the image display system comprises: A camera device, used for capturing a background image including an image of a main device, and transmitting the background image to the main device; A main device, configured to receive the background image captured by the camera device, and render the virtual scene through a virtual camera based on the camera pose information to generate a foreground image, and send the background image and the foreground image to the secondary device; The secondary device is used to receive the foreground image and the background image sent by the primary device, and according to the camera posture information of the camera device, stitch the foreground image into the scene image in the corresponding image stitching area in the background image to generate an augmented reality image, and display the augmented reality image.

9. A control device for an image display system, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the control method of the image display system according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the control method of the image display system according to any one of claims 1 to 7 are implemented.

Images