Real-world object integration methods, smart cockpits, media, and smart cars

Abstract

The present specification discloses a real object virtual-real combination method, an intelligent cockpit, a medium and an intelligent automobile. The method comprises: collecting a real object in front of the vehicle, and determining a real coordinate position of the real object relative to a human eye coordinate system; determining a pixel coordinate position of the real object in a projection image plane of a projection display HUD based on the real coordinate position; wherein the projection image plane is an image plane formed by projecting an output image of the HUD into a front windshield; the projection image plane forms a corresponding virtual image plane after passing through the front windshield; constructing an auxiliary marking object for assisting in marking the real object based on the pixel coordinate position of the real object; and outputting the auxiliary marking object so that the auxiliary marking object is combined with the real object in the virtual image plane.

Description

Technical Field

[0001] This specification relates to the field of smart cockpit technology, and in particular to a method for virtual-real bonding of a real object, a smart cockpit, a medium, and a smart car. Background Technology

[0002] Among the various HUD (Head-up Display) technologies commonly used in the automotive industry, AR (Augmented Reality) HUD, which combines virtual reality with the actual driving experience to provide drivers with more intuitive, accurate, and diverse driving information, has become the future development direction of HUDs in the automotive industry.

[0003] In AR-HUD technology, 3D object virtual-real alignment algorithms are one of the most critical technologies. Existing 3D object virtual-real alignment algorithms introduce computer graphics and computer vision techniques into the virtual-real registration process, registering virtual objects with real objects by analyzing known templates and natural sequences in camera images. However, this registration method requires the use of templates and relies on manual calibration, resulting in unstable performance.

[0004] To date, the registration accuracy between virtual and real objects still needs improvement. Summary of the Invention

[0005] This specification provides a method for virtual-real object bonding, a smart cockpit, a medium, and a smart car, which solves or partially solves the technical problem of insufficient registration accuracy in existing 3D object virtual-real bonding algorithms.

[0006] To solve the above-mentioned technical problems, the first aspect of this specification provides a method for fitting real objects together with virtual objects, the method comprising:

[0007] Collect data on real objects in front of the vehicle and determine the actual coordinates of these real objects relative to the human eye coordinate system.

[0008] Based on the actual coordinate position, the pixel coordinate position of the real object in the projection image plane of the HUD is determined; wherein, the projection image plane is the image plane formed by projecting the output image of the HUD onto the windshield; the projection image plane forms a corresponding virtual image plane after passing through the windshield;

[0009] An auxiliary labeling object is constructed based on the pixel coordinates of the real object to help identify the real object;

[0010] Output the auxiliary marker object so that the auxiliary marker object fits the real object in the virtual image plane.

[0011] Preferably, the human eye coordinate system takes the ground projection point of the human eye in the driver's seat as the origin, the X-axis of the human eye coordinate system points from the driver's seat to the passenger seat, the Y-axis of the human eye coordinate system points from the driver's seat to the front of the vehicle, and the Z-axis of the human eye coordinate system is vertically upward.

[0012] Determining the true coordinate position of the real object relative to the human eye coordinate system specifically includes:

[0013] Based on the shape and size of the real object, determine the set of feature points used to characterize the real object;

[0014] Determine the true coordinate position of the feature point set relative to the human eye coordinate system;

[0015] The feature point set contains one or more feature points, each with a corresponding real coordinate position.

[0016] Preferably, determining the pixel coordinate position of the real object in the projection image plane of the HUD based on the real coordinate position specifically includes:

[0017] For a single feature point in the feature point set, the true coordinate position and HUD-related parameters of the single feature point are obtained; the HUD-related parameters include: downward viewing angle, the field of view height of the virtual image plane, the field of view width of the virtual image plane, the distance VID of the virtual image plane relative to the human eye, the image plane height of the projected image plane, and the image plane width of the projected image plane;

[0018] By inputting the true coordinates of the single feature point and the HUD-related parameters into the coordinate transformation formula, the pixel coordinates of the single feature point in the projected image plane are obtained.

[0019] Preferably, the coordinate transformation formula is as follows:

[0020]

[0021] Among them, s 横 s represents the horizontal coordinate position of the single feature point in the projected image plane. 纵 The pixel represents the vertical coordinate position of the single feature point in the projected image plane. w The pixel represents the image width of the projected image plane. h The image plane height is represented by x, the x-coordinate of a single feature point, the y-coordinate of a single feature point, the z-height of the single feature point relative to the ground, the h-height of the human eye relative to the ground, and the α-angle of the downward viewing angle. w The deg represents the field of view width of the virtual image plane.h This represents the field of view height of the virtual image plane.

[0022] Preferably, the step of constructing an auxiliary labeling object based on the pixel coordinates of the real object to assist in labeling the real object specifically includes:

[0023] The auxiliary labeling object is constructed in the output image of the HUD by selectively referencing the pixel coordinate positions of each feature point.

[0024] A second aspect of this specification provides an intelligent cockpit, comprising:

[0025] The acquisition module is used to acquire real objects in front of the vehicle and determine the real coordinate position of the real objects relative to the human eye coordinate system.

[0026] The determining module is used to determine the pixel coordinate position of the real object in the projection image plane of the projection display HUD based on the real coordinate position; wherein, the projection image plane is the image plane formed by projecting the output image of the HUD onto the windshield; the projection image plane forms a corresponding virtual image plane after passing through the windshield;

[0027] A construction module is used to construct an auxiliary labeling object for assisting in labeling the real object based on the pixel coordinates of the real object;

[0028] The output module is used to output the auxiliary marker object so that the auxiliary marker object fits the real object in the virtual image plane.

[0029] Preferably, the human eye coordinate system takes the ground projection point of the human eye in the driver's seat as the origin, the X-axis of the human eye coordinate system points from the driver's seat to the passenger seat, the Y-axis of the human eye coordinate system points from the driver's seat to the front of the vehicle, and the Z-axis of the human eye coordinate system is vertically upward.

[0030] The acquisition module is specifically used for:

[0031] Based on the shape and size of the real object, determine the set of feature points used to characterize the real object;

[0032] Determine the true coordinate position of the feature point set relative to the human eye coordinate system;

[0033] The feature point set contains one or more feature points, each with a corresponding real coordinate position.

[0034] Preferably, the determining module is specifically used for:

[0035] For a single feature point in the feature point set, the true coordinate position and HUD-related parameters of the single feature point are obtained; the HUD-related parameters include: downward viewing angle, the field of view height of the virtual image plane, the field of view width of the virtual image plane, the distance VID of the virtual image plane relative to the human eye, the image plane height of the projected image plane, and the image plane width of the projected image plane;

[0036] By inputting the true coordinates of the single feature point and the HUD-related parameters into the coordinate transformation formula, the pixel coordinates of the single feature point in the projected image plane are obtained.

[0037] Preferably, the coordinate transformation formula is as follows:

[0038]

[0039] Among them, s 横 s represents the horizontal coordinate position of the single feature point in the projected image plane. 纵 The pixel represents the vertical coordinate position of the single feature point in the projected image plane. w The pixel represents the image width of the projected image plane. h The image plane height is represented by x, the x-coordinate of a single feature point, the y-coordinate of a single feature point, the z-height of the single feature point relative to the ground, the h-height of the human eye relative to the ground, and the α-angle of the downward viewing angle. w The deg represents the field of view width of the virtual image plane. h This represents the field of view height of the virtual image plane.

[0040] Preferably, the output module is specifically used to: selectively refer to the pixel coordinate positions of each feature point to construct the auxiliary label object in the output image of the HUD.

[0041] A third aspect of this specification provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the above-described method.

[0042] A fourth aspect of this specification provides an intelligent vehicle, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method described above.

[0043] Through one or more embodiments of this specification, this specification has the following beneficial effects or advantages:

[0044] The solution described in this specification uses the real coordinates of a real object in the human eye coordinate system to calculate its pixel coordinates on the projection plane of the HUD. Based on this, an auxiliary marker object is constructed to help identify the real object, ensuring that the pixel coordinates of the constructed auxiliary marker object are consistent with those of the real object. Therefore, after outputting the auxiliary marker object, it can accurately fit the real object.

[0045] The above description is merely an overview of the technical solution in this specification. In order to better understand the technical means in this specification and to implement it in accordance with the contents of this specification, and to make the above and other objects, features and advantages of this specification more apparent and understandable, specific embodiments of this specification are given below. Attached Figure Description

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0047] Figure 1 A flowchart illustrating a method for bonding real and virtual objects according to embodiments of this specification is shown.

[0048] Figure 2 A schematic diagram of the imaging of a HUD according to an embodiment of this specification is shown;

[0049] Figure 3 A schematic diagram of a virtual image plane according to an embodiment of this specification is shown;

[0050] Figures 4-5 A mapping diagram of a single feature point in a virtual image plane according to an embodiment of this specification is shown;

[0051] Figure 6 A schematic diagram of a smart cockpit according to an embodiment of this specification is shown. Detailed Implementation

[0052] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0053] Figure 1 A flowchart illustrating the method for virtual-real object bonding provided in the embodiments of this specification is shown. See also... Figure 1The method for virtual-real object bonding provided in the embodiments of this specification includes the following steps S101 to S104:

[0054] S101, collect data on the real objects in front of the vehicle and determine the real coordinates of the objects relative to the human eye coordinate system.

[0055] Among them, real objects in front of the vehicle have a positive or negative effect on the vehicle's movement. For example, lane markings and driving signs that have a positive guiding effect on the vehicle's movement; landslides and falling rocks that obstruct the vehicle's movement due to extreme weather; and vehicles traveling in front of the vehicle that restrict the vehicle's movement; all of these are real objects in the real world.

[0056] In the human eye coordinate system, the origin is the projection point of the driver's eye onto the ground from the driver's seat. The X-axis points from the driver's seat to the passenger seat, the Y-axis points from the driver's seat to the front of the vehicle, and the Z-axis is vertically upward. The human eye coordinate system and the camera coordinate system share the same three axes. However, due to the different positions of the vehicle camera and the driver's eye, their origins differ. Therefore, the two coordinate systems can be converted using coordinate translation.

[0057] This vehicle uses an onboard camera to capture images of real objects in front of it. As is well known, onboard cameras use the vehicle's own camera coordinate system as a reference, while real objects use a world coordinate system. Therefore, by performing coordinate transformations and translations on the real objects from the world coordinate system, the true coordinates of the real objects relative to the human eye's coordinate system are obtained.

[0058] In one optional implementation, considering the shape and size variations of different real objects, each real object has a corresponding set of feature points. This set contains one or more feature points, the number of which depends on the shape and size of the real object. Each feature point has a corresponding real-world coordinate position. For example, the rear hood of a vehicle in front has four feature points, distributed at the four corners of the hood; lane markings have two feature points, located at both ends of the lane markings; if the real object is a small rock in the middle of the road, it has only one feature point. In determining the real-world coordinate position of a real object relative to the human eye coordinate system, the set of feature points used to represent the real object is determined based on its shape and size; and the real-world coordinate position of the feature point set relative to the human eye coordinate system is then determined.

[0059] S102, Based on the real coordinate position, determine the pixel coordinate position of the real object in the projection image plane of the HUD.

[0060] See Figure 2This is a schematic diagram of a HUD (Head-Up Display). The output image of the HUD is projected onto the windshield of the vehicle to form a projection image surface, which has a width of 1 pixel. w High pixel h The windshield can be wedge-shaped glass with a certain curvature and thickness. After the projected image passes through the windshield, a corresponding virtual image is formed at a distance. See also... Figures 2-3 This is a schematic diagram of a virtual image plane. A virtual image plane has width, height, field of view, and downward viewing angle. Figure 2 (as shown in the image) The distance VID of the virtual image plane relative to the human eye Figure 2 (As shown in the image). The field of view range includes the horizontal field of view (H-FOV) and the vertical field of view (V-FOV). The horizontal field of view (H-FOV) and the field of view width (deg) of the virtual image plane are also included. w Correspondingly, the vertical field of view (V-FOV) and the field of view height of the virtual image plane (deg) h Correspondingly. The downward angle, also known as the top angle, is the angle between the horizontal field of view (H-FOV) in the virtual image plane and the horizontal direction. It can also be seen as the angle formed between the human eye looking at the horizontal field of view (H-FOV) of the virtual image plane and the horizontal direction.

[0061] Since the feature point set contains one or more feature points, the true coordinate position and HUD related parameters of a single feature point are obtained for that single feature point in the feature point set.

[0062] HUD-related parameters include: downward viewing angle α, and the field of view height of the virtual image plane (deg). h The field of view width of the virtual image plane (deg) w The distance of the virtual image plane relative to the human eye (VID) and the image height (pix) of the projected image plane. h Image width (pixels) of the projected image plane w .

[0063] By inputting the true coordinates of the single feature point and the HUD-related parameters into the coordinate transformation formula, the pixel coordinates of the single feature point in the projected image plane are obtained.

[0064] Furthermore, the coordinate transformation formula is as follows:

[0065]

[0066] Among them, s 横 s represents the x-coordinate position of a single feature point in the projected image plane. 纵 The pixel represents the ordinate position of a single feature point in the projected image plane. w The pixel represents the width of the projected image plane. hThe image plane height is represented by x, the x-coordinate of a single feature point, y, the y-coordinate of a single feature point, z, the height of a single feature point relative to the ground, h, the height of the human eye relative to the ground, and α, the downward viewing angle. w The deg represents the field of view width of the virtual image plane. h This indicates the field of view height of the virtual image plane.

[0067] By inputting the real coordinates of any real object in front of the vehicle and the relevant parameters of the HUD into the coordinate transformation formula provided in this manual, the pixel coordinates of the real object in the projected image plane can be obtained, which is convenient and quick.

[0068] To facilitate understanding, the following example uses a single feature point to illustrate the transformation principle of the coordinate transformation formula. It is important to note that the single feature point in this example is equivalent to a single feature point.

[0069] See Figures 4-5 , is a mapping diagram of a single feature point in a virtual image plane.

[0070] Among the user's relevant parameters: E represents the human eye, G represents the ground projection point of the human eye, and h represents the height of the human eye relative to the ground, EG = h.

[0071] HUD-related parameters include: downward viewing angle α, and the field of view height of the virtual image plane (deg). h The field of view width of the virtual image plane (deg) w The distance of the virtual image plane relative to the human eye (VID) and the image height (pix) of the projected image plane. h Image width (pixels) of the projected image plane w .

[0072] In the virtual image plane, R and T represent the left and right edges of the virtual image plane, respectively, and RT forms the horizontal axis of the virtual image plane. Figure 5 The diagram illustrates the horizontal axis RT (which is the same length as the horizontal central axis of the virtual image plane), Q and P representing the upper and lower edges of the virtual image plane, respectively. PQ forms the vertical central axis of the virtual image plane, and O represents the pixel center point of the virtual image plane. RT and PQ intersect at O. Due to measurement limitations, the distance d0 between the pixel center point O's projection on the ground and the human eye's projection on the ground is determined to represent the distance of the human eye E relative to the virtual image plane. Notably, the line of sight EO from the human eye E to the pixel center point O of the virtual image plane is perpendicular to the virtual image plane and forms a downward viewing angle α with the horizontal line M.

[0073] The relevant parameters for a single feature point are: A represents a single feature point, whose true coordinates are (x, y, z), all of which are known quantities. Here, x represents its distance from the reference plane 10, y represents its distance from the ground projection point G of the human eye, z represents its height relative to the ground, and S is the intersection point of its line of sight with the human eye E in the virtual image plane. (For simplification...) Figure 4 , Figure 4 The height of A relative to the ground is z = 0.

[0074] See Figure 4 The processing logic for the coordinate transformation formula is as follows:

[0075] Step 41, construct reference plane 10. Reference plane 10 is formed by the human eye E, the human eye's ground projection point G, and the ground reference intersection point B. The ground reference intersection point B is formed by extending the line of sight EO to the ground.

[0076] Step 42: Determine the projection point A' of a single feature point A in the reference plane 10, and determine the distance between the projection point A' and the ground projection point G of the human eye.

[0077] In this context, a single feature point A, the human eye E, and the ground projection point G of the human eye form a real plane 11, in which EG⊥GA. Since the single feature point A encountered during the vehicle's movement is random, its position may not necessarily intersect the vertical or horizontal central axis of the virtual image plane. Therefore, it is necessary to determine the projection plane 12 formed by projecting the real plane 11 containing the single feature point A onto the reference plane 10, in order to calculate the ordinate position of the single feature point A in the projected image plane.

[0078] Projection plane 12 and reference plane 10 are the same plane, with projection plane 12 being a part of reference plane 10. In projection plane 12, there is a projection point A' with a single feature point A, and a projection point S' representing the intersection of lines of sight S. The projection point S' is also the intersection of the lines of sight between projection point A' and the human eye E in the virtual image plane. The projection point S' lies on the vertical central axis of the virtual image plane, and P, O, S', and Q are on a straight line. In projection plane 12, EG⊥GA'. The projection angle between the real plane 11 and projection plane 12 is β, which can be measured.

[0079] Specifically, the projected coordinates of the projection point A' are determined based on the true coordinates of a single feature point A. Specifically, (0, y, z), and the distance between the projection point A' and the ground projection point G (as seen from the human eye) is also the ordinate (y) of the projection point A'.

[0080] It is worth noting that if the intersection of the line of sight between a single feature point A and the human eye E in the virtual image plane is on the vertical central axis, then the single feature point A and its projection point A' are actually the same point, and there is no need to determine the projection plane.

[0081] Step 43: Using the distance y between the projection point A' and the ground projection point G of the human eye, the height h of the human eye, and the distance d0 of the human eye E relative to the virtual image plane, combined with the angular transformation relationship between the projection point A' in the projection plane 12 and the lower viewing angle α, determine the first distance OS' between the line-of-sight intersection projection point S' and the imaging center point O.

[0082] The virtual image distance is calculated using the distance d0 between the human eye E and the virtual image plane.

[0083] The angular transformation relationship between projection point A' within projection plane 12 and the downward viewing angle is as follows:

[0084] ∠GA'E=∠MEA',

[0085] ∠OES'=∠MEA'-α,

[0086] After merging the two, we get: ∠OES'=∠GA'E-α;

[0087] And by using the distance y and the height of the human eye h, we can obtain: Then we can further obtain:

[0088] The trigonometric relationships in △OES' are: After transformation, we get:

[0089] Step 44: Determine the second distance OP between the imaging center point O and the lower edge P.

[0090] Specifically, by using the trigonometric function relationships in △OEP, we can obtain:

[0091]

[0092] Step 45: Based on the first distance OS' and the second distance OP, and combining the ratio between the virtual image plane and the projected image plane: OP / OS' = op / os1', fit the ordinate of a single feature point A in the projected image plane.

[0093] Specifically, o represents the center point of the projected image plane, p represents the lower edge of the projected image plane, and op represents the distance between the center point o and its lower edge p. For ease of distinction, the projection point S' of the line-of-sight intersection of a single feature point A is transformed into point s1' on the projected image plane, and os1' represents the distance between point s1' formed by the transformation of the projection point S' of the line-of-sight intersection of a single feature point A on the projected image plane and the center point o of the image plane.

[0094]

[0095] Since the origin of the projection image plane is the upper left corner, the ordinate of a single feature point A in the projection image plane is:

[0096]

[0097] In practical applications, if we consider the height z of a single feature point A relative to the ground, then:

[0098]

[0099] It is worth noting that if the intersection of the line of sight between a single feature point A and the human eye E in the virtual image plane is on the vertical central axis, then the single feature point A and its projection point A' are actually the same point, and the ordinate of the single feature point A in the projection image plane can be calculated without determining the projection plane.

[0100] See Figure 5 ,exist Figure 4 Based on this, the horizontal axis RT, which is at the same height as the projection point S' of the intersection with the line of sight, is shown.

[0101] Step 51, construct reference plane 10. Reference plane 10 is formed by the human eye E, the human eye's ground projection point G, and the ground reference intersection point B. The ground reference intersection point B is formed by extending the line of sight EO to the ground.

[0102] Step 52: Determine the projection point A' of a single feature point A in the reference plane 10, determine the distance between the projection point A' and the ground projection point G of the human eye, the distance between the single feature point A and the projection point A', and the distance ES' between the human eye E and the projection point S' of the intersection of the line of sight.

[0103] In this context, a single feature point A, the human eye E, and the ground projection point G of the human eye form a real plane 11, in which EG⊥GA. Since the single feature point A encountered during the vehicle's movement is random, its position may not necessarily intersect the vertical or horizontal central axis of the virtual image plane. Therefore, it is necessary to determine the projection plane 12 formed by projecting the real plane 11 containing the single feature point A onto the reference plane 10, in order to calculate the horizontal coordinate position of the single feature point A in the projection image plane.

[0104] Projection plane 12 and reference plane 10 are the same plane, with projection plane 12 being a part of reference plane 10. In projection plane 12, there is a projection point A' with a single feature point A, and a projection point S' of the line-of-sight intersection point S. AA'⊥EA', therefore SS'⊥ES'. The projection point S' is also the line-of-sight intersection point between projection point A' and the human eye E in the virtual image plane. The distance ES' between the human eye E and the projection point S' can be measured. The projection point S' lies on the vertical central axis of the virtual image plane, with P, O, S', and Q on a straight line, and EG⊥GA'. Figure 4 The horizontal axis RT, which is at the same height as the projection point S' of the intersection with the line of sight, is shown. R, S, S', and T are located on a straight line.

[0105] The projected coordinates of the projection point A' are determined based on the true coordinates of a single feature point A. Specifically, A' is (0, y, z), and the distance between the projection point A' and the ground projection point G seen by the human eye is also the y-coordinate of the projection point A'. The distance between a single feature point A and the projection point A' is also the x-coordinate of the single feature point A.

[0106] The distance ES' between the human eye E and the projection point S' of the intersection of the line of sight can be measured.

[0107] It is worth noting that if the intersection of the line of sight between a single feature point A and the human eye E in the virtual image plane is on the vertical central axis, then the single feature point A and its projection point A' are actually the same point. The horizontal coordinate of the single feature point A is the center of the field of view width. By referring to the ratio between the virtual image plane and the projected image plane, the horizontal coordinate of the single feature point A in the projected image plane can be determined.

[0108] Step 53: Using the distance y between projection point A' and ground projection point G of the human eye, the distance x between a single feature point A and projection point A', the height of the human eye h, the distance ES' between the human eye E and the projection point S' of the line of sight intersection, and combining the angular transformation relationship between a single feature point A and projection plane 12, determine the third distance SS' between the line of sight intersection S and the projection point S' of the line of sight intersection.

[0109] It should be noted that since projection point A' is the point on reference plane 10 projected from a single feature point A, the distance between the human eye E and projection point A' is calculated. By combining the distance x between a single feature point A and the projection point A', we can obtain... because Combining the two yields:

[0110] Step 54: Determine the fourth distance S'R between the projection point S' of the line of sight intersection and the left edge point R.

[0111] Specifically, by using the trigonometric function relationships in △RES', we can obtain:

[0112] Step 55: Combining the ratio between the virtual image plane and the projected image plane: SS' / S'R=s1's1 / s1'r, fit the horizontal coordinate of a single feature point A in the projected image plane.

[0113] Where r represents the left edge of the projected image plane, s1'r represents half the width of the projected image plane, the line intersection point S of a single feature point A is transformed into point s1 after the projection image plane, and s1's1 represents the distance between the line intersection point S and the projection point S' of the line intersection point after the projection image plane.

[0114]

[0115] Since the origin of the projection image plane is the upper left corner, the x-coordinate of a single feature point A in the projection image plane is:

[0116]

[0117] In practical applications, the height z of a single feature point A relative to the ground needs to be taken into account, then:

[0118]

[0119] The above explains the principle of using coordinate transformation formulas to process the transformation of a single feature point.

[0120] In practical applications, a coordinate transformation formula that can characterize the coordinate positional relationship between a reference object and a virtual image plane, as well as the actual proportional relationship between the projected image plane and the virtual image plane, can be pre-constructed based on the real positional relationship between the reference object and the projected image plane. Then, this formula can be used to transform the coordinates of each feature point of the real object.

[0121] The solution described in this manual does not rely on templates or require manual calibration of natural sequences, making it simple and convenient to operate. In practical use, simply measuring the actual coordinates of the real object and combining this with known HUD parameters is sufficient to accurately locate the position of its corresponding auxiliary marker object.

[0122] S103, construct an auxiliary labeling object based on the pixel coordinates of the real object to help label the real object.

[0123] The auxiliary label objects are constructed within the HUD. These objects are used to represent real-world objects, and their shape and size are adapted to the real-world object. For example, if the real-world object is a lane line, the auxiliary label object is the line segment corresponding to that lane line. If the real-world object is the rear hood of a vehicle traveling in front, the auxiliary label object is an irregular quadrilateral with its four vertices located at the four corners of the rear hood.

[0124] Since a real object corresponds to one or more feature points, in this embodiment, auxiliary marker objects are constructed in the HUD output image by selectively referencing the pixel coordinates of each feature point. Specifically, all or some feature points are selected to construct auxiliary marker objects. Taking lane lines as an example, feature points at both ends of the lane lines can be selected to construct marker objects representing the lane lines, thereby enhancing the visual effect of the lane lines.

[0125] S104, output auxiliary marker objects so that the auxiliary marker objects fit the real objects in the virtual image plane.

[0126] After constructing auxiliary display objects, the HUD can combine other objects to be displayed in the output image, such as time, current vehicle speed, weather, instrument panel data, etc., and output and display them in the virtual image.

[0127] It should be noted that the registration process of real objects and virtual objects in AR-HUD technology provided in the embodiments of this specification will differ from the registration process of existing 3D object virtual-real object matching algorithms.

[0128] The traditional 3D object virtual-real alignment algorithm involves introducing computer graphics and computer vision techniques into the virtual-real alignment process. It registers virtual objects with real objects by analyzing known templates and natural sequences in camera images. This requires the use of templates and manual calibration of the natural sequences.

[0129] The real-object virtual-real alignment method provided in this specification utilizes the real object's true coordinates in the human eye coordinate system to calculate its pixel coordinates in the HUD's projection image plane. Based on this, an auxiliary marker object is constructed to help identify the real object, ensuring that the pixel coordinates of the constructed auxiliary marker object are consistent with those of the real object. Therefore, after outputting the auxiliary marker object, it can accurately align with the real object. This solution eliminates the need for templates and manual calibration of natural sequences to achieve accurate alignment of the auxiliary marker object with the real object.

[0130] Based on the same inventive concept, the following embodiments describe an intelligent cockpit, see below. Figure 6 ,include:

[0131] The acquisition module 601 is used to acquire real objects in front of the vehicle and determine the real coordinate position of the real objects relative to the human eye coordinate system.

[0132] The determining module 602 is used to determine the pixel coordinate position of the real object in the projection image plane of the projection display HUD based on the real coordinate position; wherein, the projection image plane is the image plane formed by projecting the output image of the HUD onto the windshield; the projection image plane forms a corresponding virtual image plane after passing through the windshield;

[0133] The construction module 603 is used to construct an auxiliary labeling object for assisting in labeling the real object based on the pixel coordinate position of the real object;

[0134] The output module 604 is used to output the auxiliary marker object so that the auxiliary marker object fits the real object in the virtual image plane.

[0135] In one optional implementation, the human eye coordinate system takes the ground projection point of the human eye in the driver's seat as the origin, the X-axis of the human eye coordinate system points from the driver's seat to the passenger seat, the Y-axis of the human eye coordinate system points from the driver's seat to the front of the vehicle, and the Z-axis of the human eye coordinate system is vertically upward.

[0136] The acquisition module 601 is specifically used for:

[0137] Based on the shape and size of the real object, determine the set of feature points used to characterize the real object;

[0138] Determine the true coordinate position of the feature point set relative to the human eye coordinate system;

[0139] The feature point set contains one or more feature points, each with a corresponding real coordinate position.

[0140] In one optional implementation, the determining module 602 is specifically used for:

[0141] For a single feature point in the feature point set, the true coordinate position and HUD-related parameters of the single feature point are obtained; the HUD-related parameters include: downward viewing angle, the field of view height of the virtual image plane, the field of view width of the virtual image plane, the distance VID of the virtual image plane relative to the human eye, the image plane height of the projected image plane, and the image plane width of the projected image plane;

[0142] By inputting the true coordinates of the single feature point and the HUD-related parameters into the coordinate transformation formula, the pixel coordinates of the single feature point in the projected image plane are obtained.

[0143] In one optional implementation, the coordinate transformation formula is as follows:

[0144]

[0145] Among them, s 横 s represents the horizontal coordinate position of the single feature point in the projected image plane. 纵 The pixel represents the vertical coordinate position of the single feature point in the projected image plane. w The pixel represents the image width of the projected image plane. h The image plane height is represented by x, the x-coordinate of a single feature point, the y-coordinate of a single feature point, the z-height of the single feature point relative to the ground, the h-height of the human eye relative to the ground, and the α-angle of the downward viewing angle. w The deg represents the field of view width of the virtual image plane. h This represents the field of view height of the virtual image plane.

[0146] In one optional implementation, the output module is specifically used to: selectively refer to the pixel coordinate positions of each feature point to construct the auxiliary label object in the output image of the HUD.

[0147] Based on the same inventive concept as in the foregoing embodiments, this specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0148] Based on the same inventive concept as in the foregoing embodiments, embodiments of this specification also provide an intelligent vehicle, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0149] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this specification is not directed to any particular programming language. It should be understood that the contents of this specification can be implemented using various programming languages, and the above descriptions of specific languages ​​are for the purpose of disclosing preferred embodiments of this specification.

[0150] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this specification may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0151] Similarly, it should be understood that, in order to streamline this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of this specification, various features of this specification are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed specification requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this specification.

[0152] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0153] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this specification and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0154] The various component embodiments of this specification can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components of the gateway, proxy server, or system according to embodiments of this specification. This specification can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such implementations of this specification can be stored on a computer-readable medium or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0155] It should be noted that the above embodiments are illustrative of this specification and not limiting of it, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This specification can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A method for real object virtual fitting, the method comprising: collecting a real object in front of a vehicle and determining a real coordinate position of the real object presented relative to a human eye coordinate system, specifically comprising: determining a feature point set for representing the real object according to a shape and size of the real object; and determining a real coordinate position of the feature point set presented relative to the human eye coordinate system, wherein the feature point set includes one or more feature points, and each feature point has a corresponding real coordinate position; determining a pixel coordinate position of the real object in a projection image plane of a HUD based on the real coordinate position; wherein the projection image plane is an image plane formed by projecting an output image of the HUD into a front windshield; the projection image plane forms a corresponding virtual image plane after passing through the front windshield; for a single feature point in the feature point set, obtaining a real coordinate position of the single feature point and a HUD related parameter; inputting the real coordinate position of the single feature point and the HUD related parameter into a coordinate transformation formula to obtain a pixel coordinate position of the single feature point in the projection image plane; wherein the coordinate transformation formula is as follows: wherein, denotes a horizontal coordinate position of the single feature point in the projection image plane, denotes a vertical coordinate position of the single feature point in the projection image plane, denotes an image plane width of the projection image plane, denotes an image plane height of the projection image plane, denotes a horizontal coordinate of a single feature point, denotes a vertical coordinate of a single feature point, denotes a height of the single feature point relative to the ground, denotes a height of the human eye relative to the ground, denotes a downward viewing angle, denotes a field of view angle width of the virtual image plane, denotes a field of view angle height of the virtual image plane; constructing an auxiliary marking object for assisting in marking the real object based on the pixel coordinate position of the real object; outputting the auxiliary marking object so that the auxiliary marking object fits the real object in the virtual image plane. 2.The method of claim 1, wherein the human eye coordinate system takes a ground projection point of a driver's eye as a coordinate origin, an X-axis direction of the human eye coordinate system points from the driver's seat to a front passenger's seat, a Y-axis of the human eye coordinate system points from the driver's seat to a front of the vehicle, and a Z-axis of the human eye coordinate system is a vertical upward direction. 3.The method of any one of claims 1-2, wherein the constructing the auxiliary marking object for assisting in marking the real object based on the pixel coordinate position of the real object specifically comprises: selectively referencing the pixel coordinate position of each feature point to construct the auxiliary marking object in an output image of the HUD. 4.An intelligent cockpit, comprising: a collecting module configured to collect a real object in front of a vehicle and determine a real coordinate position of the real object presented relative to a human eye coordinate system, specifically comprising: determining a feature point set for representing the real object according to a shape and size of the real object; and determining a real coordinate position of the feature point set presented relative to the human eye coordinate system; wherein the feature point set includes one or more feature points, and each feature point has a corresponding real coordinate position. determining, based on the real coordinate position, a pixel coordinate position of the real object in a projection image plane of a HUD; wherein the projection image plane is an image plane formed by projecting an output image of the HUD into a front windshield; the projection image plane forms a corresponding virtual image plane after passing through the front windshield; for a single feature point in the set of feature points, a real coordinate position of the single feature point and a HUD-related parameter are obtained; the real coordinate position of the single feature point and the HUD-related parameter are input into a coordinate transformation formula to obtain a pixel coordinate position of the single feature point in the projection image plane; wherein the coordinate transformation formula is as follows: wherein, denotes a horizontal coordinate position of the single feature point in the projection image plane, denotes a vertical coordinate position of the single feature point in the projection image plane, denotes an image plane width of the projection image plane, denotes an image plane height of the projection image plane, denotes a horizontal coordinate of a single feature point, denotes a vertical coordinate of a single feature point, denotes a height of the single feature point relative to the ground, denotes a height of the human eye relative to the ground, denotes a downward viewing angle, denotes a field of view angle width of the virtual image plane, denotes a field of view angle height of the virtual image plane; constructing, based on the pixel coordinate position of the real object, an auxiliary marking object for assisting in marking the real object; outputting the auxiliary marking object so that the auxiliary marking object is attached to the real object in the virtual image plane.

5. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method of any one of claims 1-3.

6. An intelligent vehicle comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method of any one of claims 1-3 when executing the program.

Images