Augmented reality method, device, equipment and storage medium
By collecting user facial images to determine transformation parameters, transforming real-world images and rendering virtual information, the problem of fixed image perspective in augmented reality methods is solved, improving the realism of images and user experience.
Patent Information
- Application Number
- CN201910721941.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-08-06
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-05-05
AI Technical Summary
In existing augmented reality methods, the image viewpoint presented on the display screen is fixed to the camera's shooting perspective, resulting in a low degree of realism and a poor user experience.
By capturing the user's facial images in real time, determining transformation parameters based on the facial images, transforming the real image, generating a transformed image, rendering virtual information on the transformed image, and finally displaying the rendered transformed image on the screen.
It improves the realism of the images displayed on the screen, enhances the interactivity between the user and the images, and optimizes the user experience.
Smart Images

Figure CN112306222B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to augmented reality technology, and more particularly to an augmented reality method, apparatus, device and storage medium. Background Art
[0002] Augmented reality (AR) technology is a technology that superimposes virtual information (such as text, images, videos, 3D models, etc.) provided by a computer system on the real scene to enhance the user's perception of the real world.
[0003] The existing augmented reality method usually directly renders virtual information based on the image captured by the camera, and displays the rendered image on the display screen. The shortcomings of the existing technology at least include: the viewing angle of the image presented on the display screen is fixed to the camera shooting angle, resulting in a low sense of reality and a poor experience when the user views the image presented on the display screen. Summary of the invention
[0004] In view of this, the embodiments of the present disclosure provide an augmented reality method, apparatus, device and storage medium for improving the realism of images presented on a display screen to optimize the user experience.
[0005] In a first aspect, an embodiment of the present disclosure provides an augmented reality method, including:
[0006] Collect user's facial image in real time;
[0007] Determining transformation parameters for transforming a real image based on the facial image;
[0008] Transforming the real image according to the transformation parameters to obtain a transformed image;
[0009] rendering virtual information on the transformed image;
[0010] Displays the rendered rollover image.
[0011] In a second aspect, an embodiment of the present disclosure provides an augmented reality device, including:
[0012] A collection module, used for collecting the user's facial image in real time;
[0013] A transformation parameter determination module, used to determine the transformation parameters used to transform the real image according to the facial image;
[0014] A transformation module, used for transforming the real image according to the transformation parameters to obtain a transformed image;
[0015] A rendering module, used for rendering virtual information on the transformed image;
[0016] The display module is used to display the rendered transformed image.
[0017] In a third aspect, an embodiment of the present disclosure provides a terminal device, including:
[0018] one or more processors;
[0019] A memory for storing one or more programs;
[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the augmented reality method as described in any embodiment of the present disclosure.
[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the augmented reality method as described in any embodiment of the present disclosure is implemented.
[0022] The embodiments of the present disclosure provide an augmented reality method, device, equipment and storage medium. The camera of the terminal device collects facial images in real time, determines the transformation parameters used to transform the real image according to the facial images; transforms the real image according to the transformation parameters to obtain a transformed image; renders virtual information on the transformed image, and displays the rendered transformed image. The real image is transformed by the facial image collected by the camera, so that the transformed image is more consistent with the user's viewing angle, thereby improving the realism of the image presented on the display screen to optimize the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0024] Figure 1 A flowchart of an augmented reality method provided by Embodiment 1 of the present disclosure is shown;
[0025] Figure 2 A flowchart of an augmented reality method provided by Embodiment 2 of the present disclosure is shown;
[0026] Figure 3 A schematic diagram of the structure of an augmented reality device provided in Embodiment 3 of the present disclosure is shown;
[0027] Figure 4 A schematic diagram of the hardware structure of a terminal device provided in Embodiment 4 of the present disclosure is shown. DETAILED DESCRIPTION
[0028] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0030] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0031] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0032] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0033] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0034] Embodiment 1
[0035] Figure 1 A flow chart of an augmented reality method provided by the first embodiment of the present disclosure is shown. The embodiment of the present disclosure can be applied to the augmented reality situation. The method can be implemented by a terminal, and specifically can be implemented by software and / or hardware in the terminal.
[0036] like Figure 1 As shown, the augmented reality method provided in the embodiment of the present disclosure includes the following steps:
[0037] S110: Collect the user's facial image in real time.
[0038] The terminal may include a camera for photographing the user's face, and the camera for photographing the user's face may be integrated on a display screen for displaying the transformed image. The camera for photographing the user's face may collect the user's facial image in real time after the terminal monitors a preset trigger event; the preset trigger event may be, for example, a terminal power-on event, or an event for turning on the camera for photographing the face. The transformation parameters may be determined in real time according to the facial image collected in real time, that is, the transformation parameters may vary according to different facial images collected.
[0039] S120. Determine, based on the facial image, transformation parameters for transforming the real image.
[0040] When users view objects in real scenes from different positions, they can see the objects in different viewing angles. Therefore, the viewing position of the user can be obtained based on the collected facial image of the user, and then the transformation parameters can be determined based on the viewing position of the user, so that the transformed image is more consistent with the viewing angle of the user, thereby improving the realism of the image presented on the display screen, increasing the interaction between the displayed image and the user, and optimizing the user experience.
[0041] Optionally, determining transformation parameters for transforming a real image based on a facial image includes: detecting a facial position in the facial image; determining shearing transformation parameters for performing a shearing transformation on the real image based on the facial position; and / or, detecting a facial size in the facial image; and determining scaling transformation parameters for performing a scaling transformation on the real image based on the facial size.
[0042] The viewing position of the user can be represented by a three-dimensional spatial position. When the camera used to shoot the face is a single camera, the camera can determine the two-dimensional plane position based on the collected two-dimensional facial image, but cannot determine the depth position. At this time, the depth position can be estimated based on the size of the face in the facial image. The position of the user's face in the two-dimensional facial image can be used as the facial position, and the facial size can be determined based on the number of pixels occupied by the face. The position of the user's face in the image can be the position of the facial feature points in the image, and the facial feature points can be, for example, the eyes or the center of the two eyes. After determining the facial position, the shearing transformation parameters for shearing the real image can be determined, and after determining the facial size, the scaling transformation parameters for shearing the real image can be determined.
[0043] Furthermore, shearing transformation parameters for shearing the real image are determined according to the facial position, including: determining the shearing transformation parameters for shearing the real image according to the distance between the facial position and the center position of the facial image.
[0044] The shear transformation can be understood as a plane figure obtained by scaling the directed distance from each point of the figure to a straight line parallel to the direction in a certain direction according to a certain ratio, which can include horizontal shear transformation and vertical shear transformation. The center position of the facial image can be predefined; the distance between the facial position and the center position of the facial image can include a horizontal distance and a vertical distance, and the vertical shear transformation parameters of the image can be determined according to the horizontal distance, and the horizontal shear transformation parameters can be determined according to the vertical distance.
[0045] Furthermore, scaling transformation parameters for performing shearing transformation on the real image are determined according to the face size, including: determining the scaling transformation parameters for performing shearing transformation on the real image according to the ratio of the face size to a preset face size.
[0046] The preset face size can be determined based on a large amount of face data collected at a fixed depth position by a camera used to shoot the user's face. When the ratio of the face size to the preset face size is greater than 1, it can be understood that the user is close to the display screen, and the zoom-in transformation parameter can be determined based on the ratio to zoom in on the real image; when the ratio of the face size to the preset face size is less than 1, it can be understood that the user is far away from the display screen, and the zoom-out transformation parameter can be determined based on the ratio to zoom out on the real image; when the ratio of the face size to the preset face size is equal to 1, no zoom-in transformation can be performed on the real image.
[0047] S130, transforming the real image according to the transformation parameters to obtain a transformed image.
[0048] Among them, transforming the real image according to the transformation parameters may include: performing a shear transformation on the real image according to the shear transformation parameters, and / or performing a scaling transformation on the displayed image according to the scaling transformation parameters. By transforming the real image, the transformed image can be made more consistent with the user's viewing angle, thereby improving the realism of the image presented on the display screen.
[0049] Optionally, the real image is a real image collected in real time, or a real image stored in advance.
[0050] The terminal may further include a camera for photographing a real scene. The camera for photographing a real scene may collect a real image in real time after the terminal monitors a preset trigger event. The preset trigger event may be, for example, a terminal power-on event or a camera for photographing a real scene opening event. In addition, the real image may also be an image pre-stored in the terminal, for example, a real scene image pre-photographed by a camera for photographing a real scene and stored in the terminal. The transformation parameters may be used to transform both the real image collected in real time and the real image stored in advance.
[0051] S140: Rendering virtual information on the transformed image.
[0052] The virtual information rendered on the transformed image may be text, image, video, 3D model or other information.
[0053] Optionally, rendering the virtual information on the transformed image includes: transforming the virtual information according to the transformation parameters; and rendering the transformed virtual information on the transformed image.
[0054] Among them, transforming the virtual information according to the transformation parameters may include: performing a shear transformation on the virtual information according to the shear transformation parameters, and / or performing a scaling transformation on the virtual information according to the scaling transformation parameters. By transforming the virtual information, the superposition effect of the transformed virtual information and the transformed image can be better, thereby improving the realism of the image presented on the display screen.
[0055] S150: Display the rendered transformed image.
[0056] After the virtual information is rendered on the transformed image, the rendered transformed image can be displayed on a display screen.
[0057] The embodiment of the present disclosure provides an augmented reality method, in which a camera of a terminal device collects a facial image in real time, determines a transformation parameter for transforming a real image according to the facial image, transforms the real image according to the transformation parameter to obtain a transformed image, renders virtual information on the transformed image, and displays the rendered transformed image. The real image is transformed by the facial image collected by the camera, so that the transformed image is more consistent with the user's viewing angle, thereby improving the realism of the image presented on the display screen to optimize the user experience.
[0058] Embodiment 2
[0059] Based on the above embodiments, this embodiment describes a specific application scenario of the augmented reality method, which can be applied to tutoring equipment, for example. It belongs to the same inventive concept as the augmented reality method proposed in the above embodiments. For technical details not described in detail in this embodiment, please refer to the above embodiments.
[0060] Optionally, the facial image is a teacher's facial image, the real image is a student's homework image, and the virtual information is annotation information.
[0061] Figure 2 A flowchart of an augmented reality method provided in the second embodiment of the present disclosure is shown. The augmented reality method provided in the embodiment of the present disclosure includes the following steps:
[0062] S210, collecting teacher's facial image in real time;
[0063] S220, determining a transformation parameter for transforming the student homework image according to the teacher's facial image;
[0064] S230, transforming the student homework image according to the transformation parameters to obtain a transformed image;
[0065] S240, rendering annotation information on the transformed image;
[0066] S250: Display the rendered transformed image.
[0067] The tutoring device may include a first camera for capturing images of student homework, a second camera for capturing images of the teacher's face, and a display screen for presenting a transformed image after rendering. After the student completes the homework, the first camera may capture images of the student's homework in real time, or may capture images of the student's homework in advance. The transformation parameters may be determined based on the teacher's face image captured by the second camera, and the student's homework image may be transformed based on the transformation parameters, so that the transformed student's homework image is more in line with the teacher's viewing angle. The teacher may view the transformed image after rendering on the display screen, and add annotation information (such as text information of reviewing the homework) to the transformed image, and the added annotation information may also be transformed based on the transformation parameters, so that the superposition effect of the annotation information and the student's homework information is more realistic and natural, thereby improving the realism of the image presented on the display screen.
[0068] In addition, the augmented reality method can also be applied to other scenarios, such as when a user reads an e-book, the e-book page is transformed according to the transformation parameters; when playing a game, the image in the game is transformed according to the transformation parameters, etc. In the above scenarios, the transformation parameters can be determined according to the collected facial image, and the image can be transformed according to the transformation parameters, so that the transformed image is more in line with the user's viewing angle, thereby improving the realism of the image presented on the display screen and optimizing the user experience.
[0069] Based on the above embodiments, this embodiment describes a specific application scenario of the augmented reality method. The above augmented reality method can be applied to tutoring equipment. When the teacher is checking the student's homework, the tutoring equipment can transform the student's homework image displayed on the display screen according to the teacher's facial image taken by the camera, so that the transformed image is more in line with the teacher's perspective, thereby improving the user experience of the tutoring equipment. In addition, the augmented reality method proposed in the above embodiments belongs to the same inventive concept, and the technical details not described in detail in this embodiment can be referred to the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.
[0070] Embodiment 3
[0071] Figure 3 The third embodiment of the present disclosure provides a schematic diagram of the structure of an augmented reality device, which is applicable to augmented reality. The augmented reality device provided by the present disclosure can implement the augmented reality method provided by the above embodiment.
[0072] like Figure 3 As shown, the augmented reality device in the embodiment of the present disclosure includes:
[0073] The acquisition module 310 is used to acquire the user's facial image in real time;
[0074] A transformation parameter determination module 320, for determining, based on the facial image, transformation parameters for transforming the real image;
[0075] A transformation module 330, configured to transform the real image according to the transformation parameters to obtain a transformed image;
[0076] A rendering module 340, configured to render virtual information on the transformed image;
[0077] The display module 350 is used to display the rendered transformed image.
[0078] Optionally, the transformation parameter determination module is specifically used to: detect the facial position in the facial image; determine the shearing transformation parameters for performing a shearing transformation on the real image based on the facial position; and / or detect the facial size in the facial image; determine the scaling transformation parameters for performing a scaling transformation on the real image based on the facial size.
[0079] Furthermore, the shearing transformation parameters for shearing the real image are determined according to the facial position, including: determining the shearing transformation parameters for shearing the real image according to the distance between the facial position and the center position of the facial image.
[0080] Furthermore, scaling transformation parameters for performing shearing transformation on the real image are determined according to the face size, including: determining the scaling transformation parameters for performing shearing transformation on the real image according to the ratio of the face size to a preset face size.
[0081] Optionally, the real image is a real image collected in real time, or a real image stored in advance.
[0082] Optionally, the rendering module is specifically used to: transform the virtual information according to the transformation parameters; and render the transformed virtual information on the transformed image.
[0083] Optionally, the augmented reality device can be integrated into the tutoring equipment, and accordingly, the facial image is the teacher's facial image, the real image is the student's homework image, and the virtual information is the annotation information.
[0084] The augmented reality device provided in the embodiment of the present disclosure belongs to the same inventive concept as the augmented reality method provided in the above embodiment. The technical details not fully described in the embodiment of the present disclosure can be referred to the above embodiment, and the embodiment of the present disclosure has the same beneficial effects as the above embodiment.
[0085] Embodiment 4
[0086] Figure 4 The hardware structure diagram of a terminal device provided by the fourth embodiment of the present disclosure is shown. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0087] like Figure 4 As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0088] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0089] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0090] The terminal provided in the embodiment of the present disclosure and the augmented reality method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in the embodiment of the present disclosure can be referred to the above embodiment, and the embodiment of the present disclosure has the same beneficial effects as the above embodiment.
[0091] Embodiment 5
[0092] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the augmented reality method provided by the above embodiments is implemented.
[0093] It should be noted that the computer storage medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory (FLASH), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0094] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0095] The computer storage may be included in the electronic device; or it may exist independently without being installed in the electronic device.
[0096] The computer storage carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: collects the user's facial image in real time; determines the transformation parameters for transforming the real image based on the facial image; transforms the real image according to the transformation parameters to obtain a transformed image; renders virtual information on the transformed image; and displays the rendered transformed image.
[0097] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: collects the user's facial image in real time; determines the transformation parameters for transforming the real image based on the facial image; transforms the real image according to the transformation parameters to obtain a transformed image; renders virtual information on the transformed image; and displays the rendered transformed image.
[0098] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0099] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0100] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.
[0101] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0102] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0103] According to one or more embodiments of the present disclosure, [Example 1] provides an augmented reality method, including:
[0104] Collect user's facial image in real time;
[0105] Determining transformation parameters for transforming the real image based on the facial image;
[0106] Transform the real image according to the transformation parameter to obtain a transformed image;
[0107] rendering virtual information on the transformed image;
[0108] Displays the rendered rollover image.
[0109] According to one or more embodiments of the present disclosure, [Example 2] provides an augmented reality method, further comprising:
[0110] Optionally, determining a transformation parameter for transforming the real image according to the facial image includes:
[0111] Detecting facial locations in facial images;
[0112] Determining shearing transformation parameters for shearing the real image according to the facial position; and / or,
[0113] Detecting face size in facial images;
[0114] A scaling transformation parameter for performing a scaling transformation on the real image is determined according to the face size.
[0115] According to one or more embodiments of the present disclosure, [Example 3] provides an augmented reality method, further comprising:
[0116] Optionally, determining shearing transformation parameters for shearing the real image according to the facial position includes: determining the shearing transformation parameters for shearing the real image according to the distance between the facial position and the center position of the facial image.
[0117] According to one or more embodiments of the present disclosure, [Example 4] provides an augmented reality method, further comprising:
[0118] Optionally, determining scaling transformation parameters for performing a shearing transformation on a real image based on the face size includes: determining scaling transformation parameters for performing a shearing transformation on a real image based on a ratio of the face size to a preset face size.
[0119] According to one or more embodiments of the present disclosure, [Example 5] provides an augmented reality method, further comprising:
[0120] Optionally, the real image is a real image collected in real time, or a real image stored in advance.
[0121] According to one or more embodiments of the present disclosure, [Example 6] provides an augmented reality method, further comprising:
[0122] Optionally, rendering the virtual information on the transformed image includes: transforming the virtual information according to the transformation parameters; and rendering the transformed virtual information on the transformed image.
[0123] According to one or more embodiments of the present disclosure, [Example 7] provides an augmented reality method, further comprising:
[0124] Optionally, the facial image is a teacher's facial image, the real image is a student's homework image, and the virtual information is annotation information.
[0125] According to one or more embodiments of the present disclosure, [Example 8] provides an augmented reality device, including:
[0126] A collection module, used for collecting the user's facial image in real time;
[0127] A transformation parameter determination module, used to determine the transformation parameters for transforming the real image according to the facial image;
[0128] A transformation module, used for transforming the real image according to the transformation parameters to obtain a transformed image;
[0129] A rendering module, for rendering virtual information on the transformed image;
[0130] The display module is used to display the rendered transformed image.
[0131] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0132] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0133] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. An augmented reality method, It is characterized in that include: Collect user's facial image in real time; Determining transformation parameters for transforming a real image based on the facial image; Transforming the real image according to the transformation parameters to obtain a transformed image; rendering virtual information on the transformed image; Display the rendered transformed image; The step of determining, according to the facial image, a transformation parameter for transforming the real image comprises: detecting a face size in the face image; Determining a scaling transformation parameter for performing a scaling transformation on the real image according to the face size; and, detecting a facial position in the facial image; Determining shearing transformation parameters for performing shearing transformation on the real image according to the facial position; The step of determining the shearing transformation parameters for performing shearing transformation on the real image according to the facial position includes: Determining a shearing transformation parameter for performing a shearing transformation on the real image according to a distance between the facial position and a center position of the facial image; The detecting the face size in the face image comprises: Determining the size of the face according to the number of pixels occupied by the face in the facial image; The step of determining a scaling transformation parameter for performing a scaling transformation on the real image according to the face size includes: According to the ratio of the face size to a preset face size, scaling transformation parameters for performing a shear transformation on the real image are determined, wherein the preset face size is determined based on face data collected at a fixed depth position by a camera used to photograph the user's face.
2. The method according to claim 1, It is characterized in that The real image is a real image collected in real time, or a real image stored in advance.
3. The method according to claim 1, It is characterized in that The rendering of virtual information on the transformed image comprises: transforming the virtual information according to the transformation parameters; The transformed virtual information is rendered on the transformed image.
4. The method according to any one of claims 1 to 3, It is characterized in that The facial image is a teacher's facial image, the real image is a student's homework image, and the virtual information is annotation information.
5. An augmented reality device, It is characterized in that include: A collection module, used for collecting the user's facial image in real time; A transformation parameter determination module, used to determine the transformation parameters used to transform the real image according to the facial image; A transformation module, used for transforming the real image according to the transformation parameters to obtain a transformed image; A rendering module, used for rendering virtual information on the transformed image; A display module, used for displaying the rendered transformed image; The transformation parameter determination module is specifically used to: detect the face size in the face image; determine the scaling transformation parameters for performing scaling transformation on the real image according to the face size; and, detect the face position in the face image; determine the shearing transformation parameters for performing shearing transformation on the real image according to the face position; The step of determining the shearing transformation parameters for performing shearing transformation on the real image according to the facial position includes: Determining a shearing transformation parameter for performing a shearing transformation on the real image according to a distance between the facial position and a center position of the facial image; The detecting the face size in the face image comprises: Determining the size of the face according to the number of pixels occupied by the face in the facial image; The step of determining a scaling transformation parameter for performing a scaling transformation on the real image according to the face size includes: According to the ratio of the face size to a preset face size, scaling transformation parameters for performing a shear transformation on the real image are determined, wherein the preset face size is determined based on face data collected at a fixed depth position by a camera used to photograph the user's face.
6. A terminal device, It is characterized in that The terminal comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the augmented reality method as described in any one of claims 1 to 4.
7. A computer storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the augmented reality method as described in any one of claims 1 to 4 is implemented.
