Augmented reality system, augmented reality display method and device, equipment and medium
By configuring a head-mounted display device on the vehicle and converting the view angle, the problem that the existing AR-HUD virtual reality screen is limited to the driver's view is solved, and the virtual reality screen sharing between the occupants and the driver is realized, and the augmented reality screen is improved. interactive.
Patent Information
- Application Number
- CN202310597700.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-11-26
AI Technical Summary
The virtual reality picture projected by the existing augmented reality head-up display (AR-HUD) is limited to the driver's view that the occupants in the passenger seat or other passenger seats cannot share the information of the virtual reality picture with the driver, and the interaction is poor. .
By configuring a head-mounted display device on the vehicle, obtain the virtual reality image output from the augmented reality head-up display, and perform the viewing angle conversion based on the perspective transformation relationship between the head-mounted display device and the augmented reality head-up display, output the virtual reality image suitable for the occupant's perspective. , allowing occupants to see the virtual reality images seen by the driver from their own perspective.
It realizes the sharing of virtual reality images between the occupants and the driver, improves the interactiveness of augmented reality images, and overcomes the technical shortcomings that the virtual reality images projected by AR-HUD are limited to the driver's view.
Smart Images

Figure CN119024556A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of augmented reality technology, and in particular to an augmented reality system, an augmented reality display method, a device, an equipment and a medium. Background Art
[0002] AR (Augmented Reality) technology is to apply virtual information (visual information, sound, taste, touch, etc.) that is difficult to experience in a certain time and space range in the real world to the real world through computer and other scientific technologies, and then simulate and superimpose it, so that the virtual information can be perceived by human senses, thereby achieving a sensory experience beyond reality. Augmented Reality Head Up Display (AR-HUD) is a driving assistance device currently used in passenger cars. It can display prompt information such as vehicle speed, speed, navigation and fault warning in the driver's front sight range, and combine it with the surrounding real scene to reduce the frequency of the driver looking down at the instrument to improve driving safety and enhance the driver's perception of the surrounding environment. However, the virtual reality screen projected by AR-HUD is limited to the driver. Passengers in the co-pilot seat or other passenger seats cannot share the information of the virtual reality screen with the driver, and the interactivity is poor. Summary of the invention
[0003] The main purpose of the present application is to provide an augmented reality system, an augmented reality display method, an apparatus, a device and a medium, aiming to solve the technical problem of poor interactivity in displaying augmented reality images through an augmented reality head-up display in the prior art.
[0004] To achieve the above object, the present application provides an augmented reality system, the augmented reality system comprising a head mounted display device and a first inertial sensor, wherein:
[0005] The augmented reality head-up display is used to project a first virtual reality image in a display area corresponding to the main driving seat of the vehicle;
[0006] The head mounted display device is mounted on the vehicle, and is used to output a second virtual reality image, wherein the second virtual reality image is obtained by performing a perspective conversion on the first virtual reality image.
[0007] Furthermore, the present application also provides an augmented reality display method, which is applied to the augmented reality system as described above, and comprises the following steps:
[0008] Acquire a first virtual reality image output by the augmented reality head-up display and a current viewing angle transformation relationship between the head mounted display device and the augmented reality head-up display;
[0009] According to the current perspective transformation relationship, the first virtual reality image is transformed in perspective to obtain a second virtual reality image;
[0010] The second virtual reality image is outputted through the head mounted display device.
[0011] The present application also provides an augmented reality display device, the augmented reality display device comprising:
[0012] An acquisition module, configured to acquire a first virtual reality image output by the augmented reality head-up display and a current viewing angle transformation relationship between the head mounted display device and the augmented reality head-up display;
[0013] A perspective conversion module, configured to perform perspective conversion on the first virtual reality image according to the current perspective conversion relationship to obtain a second virtual reality image;
[0014] An output module is used to output the second virtual reality image through the head mounted display device.
[0015] The present application also provides an electronic device, which is a physical device, and includes: a memory, a processor, and a program of the augmented reality display method stored in the memory and executable on the processor. When the program of the augmented reality display method is executed by the processor, the steps of the augmented reality display method as described above can be implemented.
[0016] The present application also provides a storage medium, which is a computer-readable storage medium, and a program for implementing an augmented reality display method is stored on the computer-readable storage medium. When the program of the augmented reality display method is executed by a processor, the steps of the augmented reality display method as described above are implemented.
[0017] The present application provides an augmented reality system, an augmented reality display method, an apparatus, a device and a medium, wherein the augmented reality system includes a head-mounted display device and an augmented reality head-up display, wherein the augmented reality head-up display is used to project a first virtual reality image in a display area corresponding to the main driving seat of the vehicle; the head-mounted display device is mounted on the vehicle, and the head-mounted display device is used to output a second virtual reality image, wherein the second virtual reality image is obtained by converting the perspective of the first virtual reality image. By configuring the augmented reality head-up display on the vehicle, the driver can view the driving information at any time without lowering his head. At the same time, by configuring the head-mounted display device and outputting the second virtual display image obtained by converting the perspective of the first virtual reality image, the passengers in the vehicle can see the virtual reality picture seen by the driver through the augmented reality head-up display from their own perspective by wearing the head-mounted display device, thereby realizing the sharing of the virtual reality picture between the passengers and the driver, overcoming the technical defects that the virtual reality picture projected by the AR-HUD is limited to the driver, and the passengers in the co-pilot seat or other passenger seats cannot share the information of the virtual reality picture with the driver, and the interactivity is poor, thereby improving the interactivity of the augmented reality picture display. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0020] Figure 1 A schematic diagram of the structure of an embodiment of an augmented reality system in the present application;
[0021] Figure 2 A schematic diagram of a scene with different viewing angles in an embodiment of the present application;
[0022] Figure 3 A schematic diagram of a flow chart of an embodiment of an augmented reality display method in the present application;
[0023] Figure 4 It is a flowchart of another embodiment of the augmented reality display method in the present application;
[0024] Figure 5 A schematic diagram of a scenario of a possible implementation method of the seat position in an embodiment of the present application;
[0025] Figure 6 It is a structural schematic diagram of an embodiment of an augmented reality display device in the present application;
[0026] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the augmented reality display method in the embodiment of the present application.
[0027] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0028] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.
[0029] Embodiment 1
[0030] The present application embodiment provides an augmented reality system. In the first embodiment of the augmented reality system of the present application, refer to Figure 1 , the augmented reality system includes a head mounted display device and an augmented reality head up display, wherein,
[0031] The augmented reality head-up display is used to project a first virtual reality image in a display area corresponding to the main driving seat of the vehicle;
[0032] The head mounted display device is mounted on the vehicle, and is used to output a second virtual reality image, wherein the second virtual reality image is obtained by performing a perspective conversion on the first virtual reality image.
[0033] In this embodiment, it should be noted that the Augmented Reality HeadUp Display (AR-HUD) is a driving assistance device currently used in passenger cars. Prompt information such as vehicle speed, rotation speed, navigation and fault warning can be displayed in the driver's front sight range and combined with the surrounding real scene to reduce the frequency of the driver looking down at the instrument to improve driving safety and enhance the driver's perception of the surrounding environment. However, the visible range of the image projected by the AR-HUD is small, and the virtual reality screen projected by the AR-HUD is limited to the driver. The passengers in the co-pilot seat or other passenger seats cannot share the information of the virtual reality screen with the driver. When the driver and the passengers communicate, if the content of the virtual reality screen is involved, there will be communication barriers because the passengers cannot know the specific content of the virtual reality screen, and the interactivity is poor. At present, if you want the passengers in the front passenger seat or other passenger seats to also see the virtual reality screen seen by the driver, you need to set up AR-HUD on the windshield in front of the front passenger seat or other positions where the passengers can see the virtual reality screen, and each additional AR-HUD requires a set of vehicle-mounted reflectors, curved mirrors, extended mirrors and other AR-HUD-related hardware accessories, which are costly, occupy a lot of space, and are difficult to implement. The embodiment of the present application outputs the virtual reality image through a head-mounted display device, so that the passengers can share the virtual reality image with the driver by wearing a head-mounted display device. Compared with a full set of AR-HUD systems, the head-mounted display device can be freely carried, flexible and convenient, and has high scalability.
[0034] Exemplarily, the augmented reality system includes a head-mounted display device and an augmented reality head-up display, wherein the augmented reality head-up display refers to a display that combines augmented reality technology and head-up display function, and is used to project a first virtual reality image in a display area corresponding to the main driving seat of the vehicle, wherein the display area corresponding to the main driving seat should be located within the visible area of the driver sitting in the main driving seat, so that vehicle status information, vehicle driving related information, navigation information, and fault warning prompt information, etc. can be displayed in the form of virtual reality images within the driver's front sight range without the driver having to lower his head to look at the dashboard or the vehicle display screen; the head-mounted display device refers to a device that can achieve an augmented reality effect by sending optical signals to the eyes, including AR glasses, AR helmets, etc. The head-mounted display device is mounted on the vehicle and is used to output the virtual reality image after the perspective is converted according to the current perspective transformation relationship, so that the occupants in the vehicle can wear it when riding in the vehicle, thereby sharing the virtual reality image with the driver, that is, the virtual reality image can be converted from the driver's perspective to the perspective of the occupants wearing the head-mounted display device. , wherein the perspective conversion may include position conversion, specification conversion and / or shape conversion, etc. Position conversion refers to the conversion of the position of the virtual reality image relative to the augmented reality head-up display and the head-mounted display device; specification conversion refers to the conversion of the visual specifications of the virtual reality image relative to the augmented reality head-up display and the head-mounted display device; shape conversion refers to the conversion of the graphic shape and visible range of the virtual reality image relative to the augmented reality head-up display and the head-mounted display device. For example, the cube directly in front of the driver, the driver sees is located in the front direction, and the visible range is a square plane closest to the driver, while the passenger in the front passenger seat should see it in the left front direction, and this cube is farther away from the passenger in the front passenger seat, so the passenger in the front passenger seat sees a smaller cube, and the visible range is the square plane that the driver can see and another adjacent square plane closest to the front passenger seat. Due to the different perspectives, the virtual reality image seen by the passenger in the front passenger seat is not a square, but a parallelogram. For example, referring to Figure 2 , from the perspective of the driver in the main driving seat, the virtual reality image is located directly in front, and from the perspective of the passenger in the co-pilot seat, it should be located in the left front.
[0035] The head-mounted display device and the augmented reality head-up display can be communicatively connected, and the head-mounted display device and / or the augmented reality head-up display can control the output of the virtual reality image; the head-mounted display device and the augmented reality head-up display can also be communicatively connected to the central control system respectively, and the central control system can control the output of the virtual reality image.
[0036] In one practicable manner, the augmented reality system may further include a central control system, which refers to a device that centrally controls various functional modules in the augmented reality system. The central control system may be arranged on a vehicle and be communicatively connected with the augmented reality head-up display and the head-mounted display device. The central control system may be used to convert the perspective of the first virtual reality image to obtain a second virtual reality image.
[0037] In this embodiment, the augmented reality system includes a head mounted display device and an augmented reality head-up display, wherein the augmented reality head-up display is used to project a first virtual reality image in a display area corresponding to the main driving seat of the vehicle; the head mounted display device is mounted on the vehicle, and the head mounted display device is used to output a second virtual reality image, wherein the second virtual reality image is obtained by converting the perspective of the first virtual reality image. By configuring the augmented reality head-up display on the vehicle, the driver can view the driving information at any time without lowering his head. At the same time, by configuring the head mounted display device and outputting the display after converting the perspective according to the positional relationship between the head mounted display device and the augmented reality head-up display, the passengers in the vehicle can see the virtual reality picture seen by the driver through the augmented reality head-up display from their own perspective simply and conveniently by wearing the head mounted display device, and the virtual reality image seen is matched with the virtual reality image seen by the driver, thereby realizing the sharing of virtual reality images between the passengers and the driver, overcoming the technical defects that the virtual reality picture projected by the AR-HUD is limited to the driver, and the passengers in the co-pilot seat or other passenger seats cannot share the information of the virtual reality picture with the driver, and the interactivity is poor, thereby improving the interactivity of the augmented reality picture display.
[0038] Embodiment 2
[0039] Further, based on the above-mentioned embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment can be referred to the above introduction, and will not be described in detail later. On this basis, the embodiment of the present application provides an augmented reality display method, referring to Figure 3 , the augmented reality display method comprises the following steps:
[0040] Step S10, obtaining a first virtual reality image output by the augmented reality head-up display and a current viewing angle transformation relationship between the head mounted display device and the augmented reality head-up display;
[0041] The executor of the method of this embodiment can be an augmented reality display device, or an augmented reality display terminal device or server. This embodiment takes an augmented reality display device as an example. The augmented reality display device can be integrated in a terminal device such as a vehicle with data processing function, a vehicle-mounted terminal, a head-mounted display device, an augmented reality head-up display, a smart phone, a tablet computer, a computer, etc.
[0042] It should be noted that the first virtual reality image is a virtual reality image obtained by projecting the initial virtual reality image at the viewing angle corresponding to the augmented reality head-up display, that is, the actual form of the initial virtual reality image seen by the user through the augmented reality head-up display. For example, for a cube located directly in front of the main driving seat, the first virtual reality image can be a square, and the first virtual reality image can be a two-dimensional image or a three-dimensional image, etc.; the viewing angle transformation relationship refers to the transformation relationship between the second virtual reality image output and displayed by the head-mounted display device and the first virtual reality image output and displayed by the augmented reality head-up display, including translation transformation, size scaling and / or rotation transformation, etc., wherein the transformation relationship can be in the form of a function or a matrix, and the specific method of determining the viewing angle transformation relationship is similar to the prior art, which will not be elaborated here. It is easy to understand that the position of the augmented reality head-up display is usually fixed, so the perspective transformation relationship mainly changes with the position of the head-mounted display device. The position of the head-mounted display device can be detected in real time by a positioning device; it can also be determined according to user input information, for example, after the user wears the head-mounted display device, a vehicle position selection interface is first displayed through the head-mounted display device output, so that the user can select the current position in the car; it can also be determined according to the connection port of the head-mounted display device. For example, a connection port (such as a Type-C interface, etc.) that communicates with the augmented reality head-up display can be set at the position corresponding to each seat in the vehicle. When the user wears the head-mounted display device, the head-mounted display device is connected to the connection port corresponding to the seat where the user is sitting, and the position of the head-mounted display device can be determined. In the case where the position of the head-mounted display device is determined in real time, the perspective transformation relationship needs to be determined in real time according to the position of the head-mounted display device determined in real time; in the case where the position of the head-mounted display device is selected from a preset position, the perspective transformation relationship corresponding to each position can be predetermined, reducing the amount of calculation in the actual application process, shortening the perspective conversion time, and reducing the delay.
[0043] Exemplarily, before the first virtual reality image output by the augmented reality head-up display, the first virtual reality image to be output and displayed by the augmented reality head-up display is obtained. In this way, after the perspective conversion is performed on the first virtual display image, the second virtual reality image obtained after the perspective conversion can be output synchronously with the first virtual reality image.
[0044] Optionally, the perspective transformation relationship includes a position transformation relationship, and before the step of obtaining the first virtual reality image output by the augmented reality head-up display and the current perspective transformation relationship between the head-mounted display device and the augmented reality head-up display, the step further includes:
[0045] The coordinate system of the head mounted display device at at least one device position is jointly calibrated with the second coordinate system of the augmented reality head-up display to determine the position transformation relationship corresponding to each of the device positions.
[0046] In this embodiment, it should be noted that when the occupant wears the head-mounted display device, the head-mounted display device will swing with the occupant, and the occupant's swing may be out of sync with the vehicle's swing, so that the AR glasses and the vehicle will swing out of sync, causing the image seen by the human eye to shake and swing, causing the vestibular organ to feel the left and right swing, and constantly sending chaotic signals to the brain. The adjustment function of the inner ear cannot keep up, which will cause the driver to feel dizzy and affect driving safety. Therefore, the coordinate system of the head-mounted display device and the second coordinate system of the augmented reality head-up display can be jointly calibrated to synchronize the swing of the two. When the two reach a relatively static state, the swing intensity of the image seen by the driver through the head-mounted display device can be reduced. The joint calibration process can be performed before the augmented reality system is actually applied, for example, during the development phase of the augmented reality system, or before an occupant wears the head-mounted display device to drive a vehicle for the first time. This is because the joint calibration process requires sufficient test data and a certain amount of calculation. The calibration is completed before the augmented reality system is actually applied, and the conversion relationship is determined. In actual application, coordinate conversion can be performed directly based on the conversion relationship, which can greatly reduce the requirements for computing resources of the head-mounted display device or the vehicle. When the user needs to use the augmented reality system, the system can be used directly without spending too much time on calibration, and the information deviation before the calibration is completed can be effectively reduced.
[0047] Exemplarily, during the joint calibration process, the vehicle can be controlled to travel according to a preset route and scene. One or more routes and scenes can be selected according to actual needs. The more comprehensive the road conditions that may be encountered during the actual driving process covered by the scenes and routes, the more accurate the joint calibration is. Furthermore, during the driving process of the vehicle, sensor test data generated based on the movement of the vehicle can be collected by sensors mounted on the vehicle and sensors mounted on the head-mounted display device. The sensor test data can be collected once or multiple times according to actual needs. The more sensor test data collected, the more accurate the joint calibration is. As an example, the sensor can be an IMU (Inertial Measurement Unit). Furthermore, by matching the timestamps of the two sets of sensor data, the corresponding relationship between the two sets of sensor test data and the actual driving state of the vehicle can be determined. Furthermore, by introducing a reference coordinate system, the coordinates in the reference coordinate system corresponding to each of the two sets of sensor data can be determined. Since the augmented reality head-up display and the vehicle are relatively stationary, the transformation relationship between the coordinate system of the head-mounted display device and the second coordinate system of the augmented reality head-up display can be determined according to the coordinate deviation between the two, and the joint calibration is completed. Among them, when the head-mounted display device may be located at multiple device positions, the swaying changes of the occupants at different device positions may be different with the vehicle, so the coordinate system at each device position can be jointly calibrated respectively.
[0048] In an practicable manner, the step of jointly calibrating the coordinate system of the head mounted display device at at least one device position and the second coordinate system of the augmented reality head-up display to determine the position transformation relationship corresponding to each of the device positions includes: presetting the augmented reality head-up display coordinate system V, the expression form and definition relationship of the head mounted display device coordinate system B are as follows:
[0049]
[0050]
[0051] Through the reference coordinate system, the coordinate system V and the coordinate system are converted into the reference coordinate system coordinates, and then according to B = V*T vb , calculate T vb , where T vb Refers to the perspective transformation relationship between the coordinate system of the augmented reality head-up display and the coordinate system of the head-mounted display device;
[0052] When multiple sets of sensor data are available, matrix least squares fitting is performed to solve T vb =(V T V) -1 V T B.
[0053] Step S20, performing perspective conversion on the first virtual reality image according to the current perspective conversion relationship to obtain a second virtual reality image;
[0054] In this embodiment, it should be noted that the initial virtual reality image includes a two-dimensional image and a three-dimensional image. For a three-dimensional image, it needs to be projected onto a two-dimensional plane when outputting and displaying. As the viewing angle changes, the two-dimensional plane to be projected will also change accordingly, and the specifications, position, shape, etc. of the projected image will also change; for a two-dimensional image, as the viewing angle changes, the specifications, position, etc. of the projected image will also change. Viewpoint conversion refers to the conversion of the specifications, position, shape and / or visible range of the projected image as the viewing angle changes. The second virtual reality image is a virtual reality image obtained by projecting the initial virtual reality image at the viewing angle corresponding to the head-mounted display device, that is, the actual form of the initial virtual reality image seen by the user through the head-mounted display device.
[0055] Exemplarily, according to the current perspective transformation relationship, the initial virtual reality image is converted from a first virtual reality image projected at a perspective corresponding to the augmented reality head-up display to a second virtual reality image projected at a perspective corresponding to the head-mounted display device.
[0056] Step S30: outputting the second virtual reality image through the head mounted display device.
[0057] Exemplarily, the second virtual reality image can be aligned with the timestamp of the first virtual reality image, so that the second virtual reality image is output through the head-mounted display device in synchronization with the output of the first virtual reality image by the augmented reality head-up display, so that occupants in other seats in the vehicle except the driving seat can share information of the virtual reality screen synchronously with the driver.
[0058] In this embodiment, by acquiring the first virtual reality image output by the augmented reality head-up display and the current perspective transformation relationship between the head-mounted display device and the augmented reality head-up display, the first virtual reality image is transformed according to the current perspective transformation relationship to obtain the second virtual reality image, thereby realizing the perspective transformation of the first virtual reality image, and then the second virtual reality image is output through the head-mounted display device, thereby realizing the output and display of the second virtual display image obtained after the perspective transformation of the first virtual reality image through the head-mounted display device. In this way, no matter where the passengers in the vehicle are sitting, they can simply and conveniently see the virtual reality picture seen by the driver through the augmented reality head-up display from their own perspective by wearing the head-mounted display device, thereby realizing the sharing of the virtual reality picture between the passengers and the driver, overcoming the technical defects that the virtual reality picture projected by the AR-HUD is limited to the driver, and the passengers in the co-pilot seat or other passenger seats cannot share the information of the virtual reality picture with the driver, and the interactivity is poor, thereby improving the interactivity of the augmented reality picture display.
[0059] Embodiment 3
[0060] Further, refer to Figure 4 Based on the above embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. On this basis, the perspective transformation relationship includes a position transformation relationship, and the step of obtaining the current perspective transformation relationship between the head-mounted display device and the augmented reality head-up display includes:
[0061] Step S11, obtaining the current device position of the head mounted display device;
[0062] In this embodiment, it should be noted that, regardless of whether the initial virtual reality image is a two-dimensional image or a three-dimensional image, its position relative to the occupants or drivers in different seats will usually change, and in the field of in-vehicle virtual reality, the virtual reality image will have a certain impact on driving safety, driving decisions, etc. For example, if the position of the navigation virtual reality image is inaccurate, the driver may take the wrong road or fail to reach the destination while driving; if the position of the virtual reality image of the distance to the front vehicle is inaccurate, the accuracy of the distance measurement may be inaccurate, and the driver may obtain wrong distance information, which may lead to errors in the control and judgment of the vehicle speed, thereby affecting driving safety. The position transformation relationship refers to the transformation relationship of converting the position of the initial virtual reality image in the perspective of the augmented reality head-up display to the position in the perspective of the head-mounted display device, which can be a transformation matrix, a transformation function, etc.
[0063] Exemplarily, after the user gets in a vehicle and wears the head-mounted display device, the current device position of the head-mounted display device can be detected to obtain the detected current device position, wherein the current device position of the head-mounted display device can be detected by a positioning device, a connection port or user input, etc.; the current device position can be the real-time positioning position of the head-mounted display device, or it can be a pre-set position area, such as a seat position, including the co-pilot seat, the seat directly behind the main driver's seat, the middle seat of the back seat, the seat directly behind the co-pilot seat, etc.
[0064] Step S12: determining a current position transformation relationship between a first coordinate system of the head mounted display device and a second coordinate system of the augmented reality head-up display according to the current device position.
[0065] It should be noted that the first coordinate system refers to the virtual world coordinate system in which the head mounted display device displays the virtual image, and the second coordinate system refers to the virtual world coordinate system in which the augmented reality head-up display displays the virtual image.
[0066] Exemplarily, the current position transformation relationship between the first coordinate system and the second coordinate system is determined according to the relative position relationship between the current device position and the preset position of the augmented reality head-up display.
[0067] In an practicable manner, the current position transformation relationship is a current translation transformation relationship. In the actual application scenario of this implementation, the change in viewing angle is mainly caused by the change in the position of the passenger, and the change in the position of the passenger is mainly a spatial translation change. The translation transformation has a small amount of calculation, a fast processing speed, a fast viewing angle conversion speed, and a small delay in the output display of the virtual reality image. That is, other passengers in the car can receive the virtual reality image information known by the driver more synchronously.
[0068] Optionally, the step of determining, according to the current device position, a current position transformation relationship between a first coordinate system of the head mounted display device and a second coordinate system of the augmented reality head-up display comprises:
[0069] According to the mapping relationship between the current device position and the preset device position and position transformation relationship, the current position transformation relationship between the first coordinate system of the head mounted display device and the second coordinate system of the augmented reality head-up display is determined.
[0070] Exemplarily, the viewing angle transformation relationship corresponding to the device positions of multiple head-mounted display devices can be determined in advance, and a mapping relationship table between the device position and the position transformation relationship can be established. Then, the mapping relationship table between the preset device position and the position transformation relationship can be queried according to the current device position to determine the current position transformation relationship between the first coordinate system of the head-mounted display device and the second coordinate system of the augmented reality head-up display.
[0071] In one practicable manner, the device position may be determined based on the seat position, wherein the reference Figure 5 The seat position may include the co-pilot seat, the seat directly behind the main driver's seat ( Figure 5 middle passenger 1 position), the middle rear seat ( Figure 5 The middle passenger seat 2), the front passenger seat directly behind the seat ( Figure 5 When passengers board the vehicle in a standard manner, the optional seats are fixed, so the current position transformation relationship corresponding to each seat can be predetermined. When the perspective conversion is actually performed, the perspective conversion can be completed faster, and the delay of the virtual reality image output display will be smaller, that is, other passengers in the vehicle can receive the virtual reality image information known by the driver more synchronously.
[0072] Optionally, the device position includes a seat position, and the step of determining the current position transformation relationship between a first coordinate system of the head mounted display device and a second coordinate system of the augmented reality head-up display according to a mapping relationship between the current device position and a preset device position and position transformation relationship of the head mounted display device comprises:
[0073] In response to a user operation of selecting a current seat position, a current position transformation relationship between a first coordinate system of the head-mounted display device and a second coordinate system of the augmented reality head-up display is determined based on a mapping relationship between the current seat position and a preset seat position and position transformation relationship of the head-mounted display device.
[0074] Exemplarily, after detecting that the user is wearing a head mounted display device, the vehicle position selection interface can be firstly displayed through the head mounted display device output, so that the user can select the current seat position through the vehicle position selection interface. In response to the user operation of selecting the current seat position, the current position transformation relationship between the first coordinate system of the head mounted display device and the second coordinate system of the augmented reality head-up display is determined according to the mapping relationship between the current seat position and the preset seat position and position transformation relationship of the head mounted display device. Among them, the seat position includes the co-pilot seat, the seat directly behind the main driver's seat, the middle seat of the back seat, the seat directly behind the co-pilot seat, etc. When the passengers are riding in a vehicle in a standard manner, the optional seats are determined, so the current position transformation relationship corresponding to each seat can be predetermined. When the perspective conversion is actually performed, the perspective conversion can be completed faster, and the delay of the output display of the virtual reality image will be smaller, that is, other passengers in the vehicle can receive the virtual reality image information known by the driver more synchronously.
[0075] Optionally, the step of performing perspective conversion on the first virtual reality image according to the current perspective transformation relationship to obtain the second virtual reality image includes:
[0076] Step S21, obtaining a first display position of the first virtual reality image in the first coordinate system;
[0077] Step S22: converting the first display position into a second display position in the second coordinate system according to the current position transformation relationship, and obtaining a second virtual reality image output and displayed at the second display position.
[0078] Exemplarily, the first display position of the first virtual reality image in the first coordinate system can be obtained by synchronizing the coordinate system by means of a reference coordinate system or by means of an optical anchor point, and then the first display position is translated according to the current position transformation relationship, and the coordinates after the translation transformation are converted into the second coordinate system for expression, so that the second display position can be determined, that is, the display position of the second virtual reality image can be determined. Exemplarily, the first display position can be expressed as a coordinate value in the first coordinate system, and the second display position can be expressed as a coordinate value in the second coordinate system.
[0079] In this embodiment, the current position transformation relationship is determined by obtaining the current device position of the head-mounted display device, thereby realizing the perspective conversion of the virtual reality image seen by the driver, so that passengers in other seats in the car can see virtual reality images that are more in line with their own perspectives, and can improve the position accuracy of the virtual reality images seen by passengers in other seats in the car. The more accurate the position of the virtual reality image seen by the passengers, the more matching the information exchanged between the passengers and the driver, thereby improving the fluency of communication between the passengers and the driver.
[0080] Embodiment 4
[0081] Furthermore, the present application also provides an augmented reality display device, referring to Figure 6 , the augmented reality display device comprises:
[0082] An acquisition module 10 is used to acquire a first virtual reality image output by the augmented reality head-up display and a current viewing angle transformation relationship between the head mounted display device and the augmented reality head-up display;
[0083] A perspective conversion module 20, configured to perform perspective conversion on the first virtual reality image according to the current perspective conversion relationship to obtain a second virtual reality image;
[0084] The output module 30 is used to output the second virtual reality image through the head mounted display device.
[0085] Optionally, the acquisition module 10 is further used to:
[0086] Obtaining the current device position of the head mounted display device;
[0087] According to the current device position, a current position transformation relationship between a first coordinate system of the head mounted display device and a second coordinate system of the augmented reality head-up display is determined.
[0088] Optionally, the perspective conversion module 20 is further used for:
[0089] Acquire a first display position of the first virtual reality image in the first coordinate system;
[0090] According to the current position transformation relationship, the first display position is converted into a second display position in the second coordinate system, and a second virtual reality image output and displayed at the second display position is obtained.
[0091] Optionally, the acquisition module 10 is further used to:
[0092] According to the mapping relationship between the current device position and the preset device position and position transformation relationship, the current position transformation relationship between the first coordinate system of the head mounted display device and the second coordinate system of the augmented reality head-up display is determined.
[0093] Optionally, the acquisition module 10 is further used to:
[0094] In response to a user operation of selecting a current seat position, a current position transformation relationship between a first coordinate system of the head-mounted display device and a second coordinate system of the augmented reality head-up display is determined based on a mapping relationship between the current seat position and a preset seat position and position transformation relationship of the head-mounted display device.
[0095] Optionally, before the operation of acquiring the first virtual reality image output by the augmented reality head-up display and the current viewing angle transformation relationship between the head mounted display device and the augmented reality head-up display, the augmented reality display device further includes a calibration module, and the calibration module is used to:
[0096] The coordinate system of the head mounted display device at at least one device position is jointly calibrated with the second coordinate system of the augmented reality head-up display to determine the position transformation relationship corresponding to each of the device positions.
[0097] The augmented reality display device provided by the present invention adopts the augmented reality display method in the above embodiment, and solves the technical problem of poor interactivity of the prior art in displaying augmented reality images through an augmented reality head-up display. Compared with the prior art, the beneficial effects of the augmented reality display device provided by the embodiment of the present invention are the same as the beneficial effects of the augmented reality display method provided by the above embodiment, and other technical features in the augmented reality display device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0098] Embodiment 5
[0099] Furthermore, an embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the augmented reality display method in the above-mentioned embodiment.
[0100] Reference below Figure 7, which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as Bluetooth headsets, mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0101] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and arrays required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.
[0102] Typically, the following systems can be connected to the I / O interface: input devices including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices including, for example, a magnetic tape, a hard disk, etc.; and communication devices. The communication device can allow the electronic device to communicate with other devices wirelessly or by wire to exchange arrays. Although the electronic device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.
[0103] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0104] The electronic device provided by the present invention adopts the augmented reality display method in the above embodiment, and solves the technical problem of poor interactivity of the prior art in displaying augmented reality images through an augmented reality head-up display. Compared with the prior art, the beneficial effects of the electronic device provided by the embodiment of the present invention are the same as the beneficial effects of the augmented reality display method provided by the above embodiment, and other technical features in the electronic device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0105] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0106] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
[0107] Embodiment 6
[0108] Furthermore, the present embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the augmented reality display method in the above embodiment.
[0109] The computer-readable storage medium provided in the embodiment of the present invention may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
[0110] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.
[0111] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: obtains a first virtual reality image output by the augmented reality head-up display and a current perspective transformation relationship between the head-mounted display device and the augmented reality head-up display; performs perspective transformation on the first virtual reality image according to the current perspective transformation relationship to obtain a second virtual reality image; and outputs the second virtual reality image through the head-mounted display device.
[0112] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0113] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0114] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0115] The computer-readable storage medium provided by the present invention stores computer-readable program instructions for executing the above-mentioned augmented reality display method, which solves the technical problem of poor interactivity in displaying augmented reality images through augmented reality head-up displays in the prior art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiment of the present invention are the same as the beneficial effects of the augmented reality display method provided by the above-mentioned embodiment, and are not described in detail here.
[0116] Embodiment 7
[0117] Furthermore, the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned augmented reality display method when executed by a processor.
[0118] The computer program product provided by the present application solves the technical problem of poor interactivity in displaying augmented reality images through augmented reality head-up displays in the prior art. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present invention are the same as the beneficial effects of the augmented reality display method provided by the above embodiment, and will not be repeated here.
[0119] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.
Claims
1. An augmented reality system, characterized in that: The augmented reality system includes a head mounted display device and an augmented reality head up display, wherein: The augmented reality head-up display is used to project a first virtual reality image in a display area corresponding to the main driving seat of the vehicle; The head mounted display device is mounted on the vehicle, and is used to output a second virtual reality image, wherein the second virtual reality image is obtained by performing a perspective conversion on the first virtual reality image.
2. An augmented reality display method, characterized in that: The augmented reality display method is applied to the augmented reality system as claimed in claim 1, comprising the following steps: Acquire a first virtual reality image output by the augmented reality head-up display and a current viewing angle transformation relationship between the head mounted display device and the augmented reality head-up display; According to the current perspective transformation relationship, the first virtual reality image is transformed in perspective to obtain a second virtual reality image; The second virtual reality image is outputted through the head mounted display device.
3. The augmented reality display method according to claim 2, wherein: The perspective transformation relationship includes a position transformation relationship, and the step of obtaining the current perspective transformation relationship between the head mounted display device and the augmented reality head-up display includes: Obtaining the current device position of the head mounted display device; According to the current device position, a current position transformation relationship between a first coordinate system of the head mounted display device and a second coordinate system of the augmented reality head-up display is determined.
4. The augmented reality display method according to claim 3, characterized in that: The step of performing perspective conversion on the first virtual reality image according to the current perspective conversion relationship to obtain the second virtual reality image comprises: Acquire a first display position of the first virtual reality image in the first coordinate system; According to the current position transformation relationship, the first display position is converted into a second display position in the second coordinate system, and a second virtual reality image output and displayed at the second display position is obtained.
5. The augmented reality display method according to claim 3, characterized in that: The step of determining the current position transformation relationship between the first coordinate system of the head mounted display device and the second coordinate system of the augmented reality head-up display according to the current device position comprises: According to the mapping relationship between the current device position and the preset device position and position transformation relationship, the current position transformation relationship between the first coordinate system of the head mounted display device and the second coordinate system of the augmented reality head-up display is determined.
6. The augmented reality display method according to claim 5, characterized in that: The device position includes a seat position, and the step of determining the current position transformation relationship between a first coordinate system of the head mounted display device and a second coordinate system of the augmented reality head-up display according to the current device position and a mapping relationship between a preset device position and a position transformation relationship of the head mounted display device includes: In response to a user operation of selecting a current seat position, a current position transformation relationship between a first coordinate system of the head-mounted display device and a second coordinate system of the augmented reality head-up display is determined based on a mapping relationship between the current seat position and a preset seat position and position transformation relationship of the head-mounted display device.
7. The augmented reality display method according to claim 2, wherein: The perspective transformation relationship includes a position transformation relationship. Before the step of obtaining the first virtual reality image output by the augmented reality head-up display and the current perspective transformation relationship between the head-mounted display device and the augmented reality head-up display, the step further includes: The coordinate system of the head mounted display device at at least one device position is jointly calibrated with the second coordinate system of the augmented reality head-up display to determine the position transformation relationship corresponding to each of the device positions.
8. An augmented reality display device, characterized in that: The augmented reality display device comprises: An acquisition module, configured to acquire a first virtual reality image output by the augmented reality head-up display and a current viewing angle transformation relationship between the head mounted display device and the augmented reality head-up display; A perspective conversion module, configured to perform perspective conversion on the first virtual reality image according to the current perspective conversion relationship to obtain a second virtual reality image; An output module is used to output the second virtual reality image through the head mounted display device.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the augmented reality display method according to any one of claims 2 to 7.
10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which is stored a program for implementing the augmented reality display method, and the program for implementing the augmented reality display method is executed by a processor to implement the steps of the augmented reality display method as described in any one of claims 2 to 7.
Citation Information
Patent Citations
Lane markings HUD for operator's seat and assistance seat and display method thereof
CN107776490A
Multi-angle multi-content headup display system and method
CN111443489A
Method and system for associating device coordinate systems in multi-person AR system
CN114730212A
Augmented reality and virtual reality equipment, vehicle-mounted system and vehicle
CN115610348A
Method for representing at least one virtual element
DE102020208590A1
