Anti-shake method for virtual scene picture, vehicle-mounted equipment and computer readable storage medium

Abstract

The invention is suitable for the technical field of terminals, and particularly relates to an anti-shake method for a virtual scene picture, vehicle-mounted equipment and a computer readable storage medium. In the method, a vehicle obtains a first pose and driving data of the vehicle at a first moment and first point cloud data corresponding to an environment where the vehicle is located, and determines a first target position corresponding to a target object at the first moment according to the first point cloud data. And then, the vehicle predicts a second pose of the vehicle at the target moment according to the driving data and the first point cloud data, and determines a second target position corresponding to the target object at the target moment according to the first pose, the second pose and the first target position, so that the vehicle can draw a virtual scene picture corresponding to the target moment according to the second target position. And at the target moment, the virtual scene picture corresponding to the target moment is displayed in the virtual image area, so that at the target moment, the virtual scene picture displayed in the virtual image area fits the real scene environment where the vehicle is actually located, and the user experience is improved.

Description

Technical Field

[0001] The present application belongs to the field of terminal technology, and in particular relates to a virtual scene image anti-shake method, a vehicle-mounted device, and a computer-readable storage medium. Background Art

[0002] At present, augmented reality headup display (AR HUD) technology can be widely used in vehicles. Among them, AR HUD can be used to combine virtual scenes such as navigation guide lines with the real environment where the vehicle is located and project them into the virtual image area for presentation, so that the driver can see the navigation guide lines combined with the real environment through the front windshield of the vehicle, thereby assisting the driver to drive the vehicle efficiently and safely. When projecting virtual scenes through AR HUD, the vehicle generally generates virtual scenes such as navigation guide lines based on the real environment (such as road surface and / or lane lines, etc.) where the vehicle is located, as perceived by the perception system in the vehicle, in combination with the navigation application, and then uses an optical machine to project virtual scenes such as navigation guide lines through multiple layers of reflection, and finally projects them into the virtual image area for presentation.

[0003] Since there is generally a long delay from the time the perception system perceives the real environment where the vehicle is located to the time the virtual scene images such as the generation of navigation guide lines are projected into the virtual image area for presentation, the vehicle posture and / or the real environment where the vehicle is located may change during this delay, especially when the vehicle needs to pass through bumpy areas such as speed bumps, manhole covers or potholes. As a result, the virtual scene images presented in the virtual image area do not match the real environment where the vehicle is actually located, affecting the user experience. Summary of the invention

[0004] The embodiments of the present application provide a method for stabilizing a virtual scene image, a vehicle-mounted device, and a computer-readable storage medium, which can solve the problem that the virtual scene image displayed in the virtual image area does not fit the real scene environment where the vehicle is actually located, affecting the user experience.

[0005] In a first aspect, an embodiment of the present application provides a method for stabilizing a virtual scene image, comprising:

[0006] Acquire a first pose of a vehicle, driving data of the vehicle, and first point cloud data corresponding to an environment where the vehicle is located at a first moment;

[0007] Determine, according to the first point cloud data, a first target position corresponding to a target object at the first moment, wherein the target object is an object for reference by the vehicle when drawing a virtual scene, and the target object is an object in the environment where the vehicle is currently located;

[0008] Determining, based on the driving data and the first point cloud data, a second posture of the vehicle at a target time, the target time being later than the first time;

[0009] A second target position corresponding to the target object at the target time is determined according to the first posture, the second posture and the first target position, so that the vehicle draws a virtual scene corresponding to the target time according to the second target position corresponding to the target object.

[0010] In the anti-shake method of the virtual scene image provided above, during the driving process of the vehicle, the vehicle can obtain the posture of the vehicle at the first moment (i.e., the first posture), the driving data of the vehicle at the first moment, and the first point cloud data corresponding to the environment where the vehicle is located at the first moment, and determine the first target position corresponding to the target object at the first moment based on the first point cloud data. Subsequently, the vehicle can predict the second posture of the vehicle at the target moment based on the driving data and the first point cloud data, and determine the second target position corresponding to the target object at the target moment based on the first posture, the second posture and the first target position, so that the vehicle can draw the virtual scene image corresponding to the target moment based on the second target position corresponding to the target object at the target moment, and can display the virtual scene image corresponding to the target moment in the virtual image area at the target moment, so that at the target moment, the virtual scene image displayed by the vehicle is consistent with the real scene environment where the vehicle is actually located, reducing the display deviation of the virtual scene image and improving the user experience.

[0011] It should be understood that the first moment may refer to any moment of acquiring the first pose, the driving data and the first point cloud data. Optionally, the first moment may be the current moment or a moment earlier than the current moment, and the target moment is later than the current moment.

[0012] Exemplarily, the first point cloud data includes point cloud data corresponding to static objects in the environment where the vehicle is located, and the target object is a static object in the environment where the vehicle is currently located.

[0013] Optionally, the target object includes one or more of a road surface, a lane line, a fence and a curb.

[0014] Exemplarily, the driving data includes driving speed and driving direction.

[0015] Optionally, the driving data at the first moment may be driving data directly sensed by a perception system, or may be driving data determined based on point cloud data acquired between a certain moment before the first moment and the first moment.

[0016] Exemplarily, the first target position and the second target position are positions of the target object in a vehicle body coordinate system.

[0017] Exemplarily, the vehicle body coordinate system is a coordinate system established with a preset position on the vehicle as the origin, or is a coordinate system established with a position that satisfies a preset position relationship with the preset position as the origin.

[0018] Optionally, the preset position may be the center of mass of the vehicle or the center point of the rear axle of the vehicle.

[0019] Optionally, the position that satisfies the preset position relationship with the preset position may be a projection point of the center of mass of the vehicle on the ground, or may be a projection point of the center point of the rear axle of the vehicle on the ground.

[0020] In some embodiments, the method further comprises:

[0021] Acquire a first reference position corresponding to the vehicle at the first moment, where the first reference position is a position of a first position on the vehicle in a target coordinate system at the first moment, and the target coordinate system is a body coordinate system of the vehicle at the first moment;

[0022] The determining, based on the driving data and the first point cloud data, a second posture of the vehicle at a target time, comprises:

[0023] A second posture of the vehicle at the target time is determined according to the first reference position, the driving data and the first point cloud data.

[0024] In the anti-shake method for virtual scene images provided in this embodiment, the vehicle can determine the second posture of the vehicle based on the position change of a certain part of the vehicle between the first moment and the target moment. That is, the vehicle can determine the position change of the part relative to the first moment at the target moment based on the first position, driving data and first point cloud data corresponding to the part at the first moment, thereby determining the second posture of the vehicle at the target moment.

[0025] Exemplarily, the first position on the vehicle includes at least one of a position corresponding to a front left tire of the vehicle, a position corresponding to a front right tire of the vehicle, a position corresponding to a rear left tire of the vehicle, and a position corresponding to a rear right tire of the vehicle.

[0026] Optionally, the position corresponding to the front left tire of the vehicle is the position where the front left tire of the vehicle contacts the ground. For example, the position corresponding to the front left tire of the vehicle may be the position of the center point of the portion where the front left tire of the vehicle contacts the ground. Similarly, the position corresponding to the front right tire of the vehicle may be the position of the center point of the portion where the front right tire of the vehicle contacts the ground. The position corresponding to the rear left tire of the vehicle may be the position of the center point of the portion where the rear left tire of the vehicle contacts the ground. The position corresponding to the rear right tire of the vehicle may be the position of the center point of the portion where the rear right tire of the vehicle contacts the ground.

[0027] In a possible implementation, determining the second posture of the vehicle at the target time according to the first reference position, the driving data, and the first point cloud data includes:

[0028] Determine, based on the first reference position, the driving data, and the first point cloud data, a candidate vehicle position that the vehicle will travel through within a preset time, wherein the preset time is greater than or equal to a time interval between the target time and the first time;

[0029] Determine a candidate position and a first candidate time of the vehicle at the candidate vehicle position, and associate and save the candidate position and the first candidate time corresponding to the candidate vehicle position;

[0030] A second candidate time that matches the target time is determined from the first candidate time, and a candidate posture corresponding to the second candidate time is determined as a second posture of the vehicle at the target time.

[0031] In the anti-shake method of the virtual scene image provided by the implementation, after obtaining the first reference position, driving data and first point cloud data corresponding to the vehicle, the vehicle can predict in advance the candidate time and candidate posture of the vehicle arriving at each candidate vehicle position, and associate and save the candidate time and candidate posture corresponding to each candidate vehicle position. After determining the target time according to the first time of the vehicle and the preset time delay, the vehicle can match the target time from the candidate time, thereby determining the second posture of the vehicle at the target time.

[0032] Optionally, the second candidate time is the time among the first candidate time that is closest to the target time.

[0033] Optionally, determining a candidate position and posture of the vehicle at the candidate vehicle position includes:

[0034] Determining, based on the candidate vehicle position, the posture of the vehicle at the candidate vehicle position;

[0035] According to the posture, a rotation matrix of the vehicle at the candidate vehicle position is determined, and a candidate posture of the vehicle at the candidate vehicle position is determined according to the rotation matrix.

[0036] It can be understood that the posture of the vehicle can be represented by the pitch angle, roll angle and navigation angle of the vehicle.

[0037] Exemplarily, when the first reference position corresponding to the vehicle includes multiple of the position corresponding to the front left tire of the vehicle, the position corresponding to the front right tire of the vehicle, the position corresponding to the rear left tire of the vehicle, and the position corresponding to the rear right tire of the vehicle, for each first candidate moment, the candidate vehicle positions include multiple positions, each of which may be the position where the first position on the vehicle may be after a certain moment, and the candidate vehicle position refers to the position in the target coordinate system, and the target coordinate system is the body coordinate system of the vehicle at the first moment;

[0038] The determining of the candidate position and the first candidate time of the vehicle at the candidate vehicle position includes:

[0039] For each first candidate moment, the position of the origin of the vehicle coordinate system at the first candidate moment in the vehicle coordinate system at the first moment is determined based on the multiple candidate vehicle positions corresponding to the vehicle, and the candidate posture of the vehicle at the first candidate moment is determined based on the position of the origin of the vehicle coordinate system at the first candidate moment in the vehicle coordinate system at the first moment.

[0040] That is to say, when the first position on the vehicle includes the positions of multiple of the vehicle's front left tire, front right tire, rear left tire and rear right tire in the vehicle's body coordinate system at the first moment, the vehicle can determine the vehicle's second posture at the target moment based on these multiple positions, driving data and the first point cloud data.

[0041] For example, when the first position on the vehicle includes the position of the vehicle's front left tire in the body coordinate system at the first moment, the position of the vehicle's front right tire in the body coordinate system at the first moment, the position of the vehicle's rear left tire in the body coordinate system at the first moment, and the position of the vehicle's rear right tire in the body coordinate system at the first moment, the vehicle can determine various candidate vehicle positions that the vehicle's front left tire may have traveled through within a preset time based on the position of the vehicle's front left tire in the body coordinate system at the first moment, the driving data, and the first point cloud data, and determine various candidate vehicle positions that the vehicle's front right tire may have traveled through within a preset time based on the position of the vehicle's front right tire in the body coordinate system at the first moment, the driving data, and the first point cloud data. The position of the vehicle's rear left tire in the vehicle body coordinate system at the first moment, the driving data and the first point cloud data are used to determine the candidate vehicle positions that the vehicle's rear left tire may have traveled through within a preset time, and the candidate vehicle positions that the vehicle's rear right tire may have traveled through within a preset time are determined based on the position of the vehicle's rear right tire in the vehicle body coordinate system at the first moment, the driving data and the first point cloud data, and the time when the vehicle's front left tire arrives at the corresponding candidate vehicle positions, the time when the vehicle's front right tire arrives at the corresponding candidate vehicle positions, the time when the vehicle's rear left tire arrives at the corresponding candidate vehicle positions, and the time when the vehicle's rear right tire arrives at the corresponding candidate vehicle positions can be predicted.

[0042] Subsequently, the vehicle can determine the candidate posture corresponding to the vehicle at each candidate moment based on the candidate vehicle position that the vehicle's front left tire may reach, the candidate vehicle position that the vehicle's front right tire may reach, the candidate vehicle position that the vehicle's rear left tire may reach, and the candidate vehicle position that the vehicle's rear right tire may reach at the same candidate moment.

[0043] Optionally, when it is determined that the candidate vehicle position corresponding to the front left tire, the candidate vehicle position corresponding to the front right tire, the candidate vehicle position corresponding to the rear left tire, and the candidate vehicle position corresponding to the rear right tire at the same candidate moment are located in the same plane, the vehicle can determine the candidate posture corresponding to the vehicle at the candidate moment based on the plane. For example, the position of the origin of the body coordinate system at the candidate moment in the body coordinate system at the first moment can be determined based on the plane, and the posture (such as pitch angle, roll angle, and navigation angle) corresponding to the vehicle at the candidate moment can be determined based on the position corresponding to the origin of the body coordinate system, so as to convert the posture into a rotation matrix according to the conversion relationship between the Euler angle and the rotation matrix, and obtain the rotation matrix of the vehicle at the candidate moment, so that the candidate posture corresponding to the vehicle at the candidate moment can be determined based on the rotation matrix obtained by the conversion.

[0044] Exemplarily, after obtaining the position of the vehicle's front left tire in the body coordinate system at the first moment, the position of the vehicle's front right tire in the body coordinate system at the first moment, the position of the vehicle's rear left tire in the body coordinate system at the first moment, the position of the vehicle's rear right tire in the body coordinate system at the first moment, the driving data and the first point cloud data, the vehicle can determine the position of the vehicle's front left tire at the target moment, the position of the vehicle's front right tire at the target moment, the position of the vehicle's rear left tire at the target moment, and the position of the vehicle's rear right tire at the target moment based on the position of the vehicle's front left tire in the body coordinate system at the first moment, the position of the vehicle's front right tire in the body coordinate system at the first moment, the position of the vehicle's rear left tire in the body coordinate system at the first moment, the position of the vehicle's rear right tire in the body coordinate system at the first moment, the driving data (such as driving speed and driving direction, etc.), a preset time delay and the first point cloud data. Subsequently, the vehicle can determine the second posture of the vehicle at the target moment based on the position of the vehicle's front left tire at the target moment, the position of the vehicle's front right tire at the target moment, the position of the vehicle's rear left tire at the target moment, and the position of the vehicle's rear right tire at the target moment.

[0045] For example, when it is determined that the position of the vehicle's front left tire at the target time, the position of the vehicle's front right tire at the target time, the position of the vehicle's rear left tire at the target time, and the position of the vehicle's rear right tire at the target time are located in the same plane, the vehicle can determine the position of the origin of the vehicle coordinate system at the target time in the vehicle coordinate system at the first moment based on the plane, and can determine the corresponding posture (such as pitch angle, roll angle, and navigation angle) of the vehicle at the target time based on the position of the origin of the vehicle coordinate system at the target time in the vehicle coordinate system at the first moment, so as to transform the posture into a rotation matrix according to the conversion relationship between Euler angles and rotation matrices, and obtain the rotation matrix of the vehicle at the target time, so as to obtain the second posture corresponding to the vehicle at the target time according to the rotation matrix obtained by the conversion.

[0046] In some embodiments, the candidate vehicle position includes a bumpy position. Optionally, the target time may be a bumpy time corresponding to a bumpy position.

[0047] In the anti-shake method of the virtual scene image provided in this embodiment, the mismatch between the virtual scene image displayed in the virtual image area and the real scene environment where the vehicle is actually located often occurs when the vehicle needs to pass through bumpy locations such as speed bumps, manhole covers or potholes. When the vehicle is always driving on a flat road, the mismatch between the virtual scene image displayed in the virtual image area and the real scene environment where the vehicle is actually located is generally not obvious. In order to reduce the amount of calculation of the vehicle and reduce the power consumption of the vehicle, the vehicle can only predict the second posture of the vehicle when it is in a bumpy position, so as to determine the second target position corresponding to the target object when the vehicle reaches the bumpy position based on the second posture at the bumpy position, thereby drawing the virtual scene image corresponding to the bumpy position based on the second target position, so that when the vehicle reaches the bumpy position, the virtual scene image displayed in the virtual image area is consistent with the real scene environment where the vehicle is actually located.

[0048] Therefore, after obtaining the first reference position, driving data and first point cloud data corresponding to the vehicle, the vehicle can determine the position (i.e., bump position) that may cause the virtual scene to not fit the real scene environment where the vehicle is actually located based on the first point cloud data, that is, it can identify bump positions such as speed bumps, manhole covers or potholes based on the first point cloud data. Subsequently, the vehicle can predict the time when the vehicle reaches each bump position (i.e., bump time) based on the first reference position, driving data (such as driving speed and driving direction, etc.) and bump position.

[0049] Then, the vehicle can determine the candidate posture of the vehicle at the bump position, and can associate and save the bump moment corresponding to the bump position with the candidate posture. After determining the target moment according to the first moment and the preset delay, the vehicle can determine the bump moment that matches the target moment from the bump moments, and determine the candidate posture corresponding to the determined bump moment as the second posture of the vehicle at the target moment. That is, the vehicle only needs to predict the candidate posture of the bump position, which can reduce the number of candidate posture predictions made by the vehicle, increase the speed of determining the second posture, and improve the user experience.

[0050] In a possible implementation, determining the second posture of the vehicle at the target time according to the first reference position, the driving data, and the first point cloud data includes:

[0051] Determine, according to the first reference position, the driving data, the preset time delay and the first point cloud data, a second reference position corresponding to the vehicle at the target time, wherein the second reference position is a position of the first position on the vehicle in the target coordinate system at the target time;

[0052] A second posture of the vehicle at the target time is determined based on the second reference position.

[0053] In the anti-shake method for virtual scene images provided by this implementation, after obtaining the first reference position, driving data and first point cloud data corresponding to the vehicle, the vehicle can directly predict the second reference position corresponding to the vehicle after a preset delay based on the first reference position, driving speed, driving direction and first point cloud data, that is, determine the second reference position where the first position on the vehicle may be located after the preset delay, so that the second posture of the vehicle at the target time can be determined directly based on the second reference position corresponding to the vehicle.

[0054] Exemplarily, determining, according to the second reference position, a second posture of the vehicle at the target time includes:

[0055] determining the posture of the vehicle at the target time according to the second reference position;

[0056] According to the posture, a rotation matrix of the vehicle at the target time is determined, and according to the rotation matrix, a second posture of the vehicle at the target time is determined.

[0057] In some embodiments, the target time is determined based on the first time and a preset delay.

[0058] Optionally, the preset delay includes a perception delay and / or a rendering delay, the perception delay is a delay in acquiring the first point cloud data, and the rendering delay is a delay in rendering the virtual scene image.

[0059] It is understandable that when the first point cloud data acquired at the first moment is point cloud data converted from point cloud data acquired at a certain moment before the first moment, the preset delay may not include the perception delay, that is, the preset delay at this time may only include the rendering delay. In other words, the preset delay may be equal to the rendering delay, or the preset delay may be equal to the sum of the perception delay and the rendering delay.

[0060] In a possible implementation, determining the second posture of the vehicle at the target time according to the first reference position, the driving data, and the first point cloud data includes:

[0061] Determine second point cloud data from the first point cloud data according to the first reference position and the driving data, the second point cloud data being all or part of the first point cloud data, and the second point cloud data being point cloud data corresponding to a position on a road surface that the vehicle may travel on within a preset time;

[0062] A second posture of the vehicle at the target time is determined according to the first reference position, the driving data and the second point cloud data.

[0063] In the anti-shake method for virtual scene images provided by this implementation, the point cloud data that the vehicle may travel over is generally the point cloud data corresponding to the road surface. Therefore, after obtaining the first point cloud data, the vehicle can delete the point cloud data corresponding to the non-road surface from the first point cloud data, and can determine the second posture of the vehicle based on the second point cloud data after deleting the point cloud data corresponding to the non-road surface, which can reduce the influence of the point cloud data corresponding to the non-road surface on the determination of the second posture, thereby improving the speed and efficiency of determining the second posture.

[0064] In a second aspect, an embodiment of the present application provides an anti-shake device for a virtual scene image, comprising:

[0065] A first pose acquisition module, used to acquire, at a first moment, a first pose of a vehicle, driving data of the vehicle, and first point cloud data corresponding to an environment where the vehicle is located;

[0066] A first target position determination module, used to determine a first target position corresponding to a target object at the first moment according to the first point cloud data, wherein the target object is an object for reference by the vehicle when drawing a virtual scene, and the target object is an object in the environment where the vehicle is currently located;

[0067] A second posture determination module, configured to determine a second posture of the vehicle at a target time according to the driving data and the first point cloud data, wherein the target time is later than the first time;

[0068] A second target position determination module is used to determine a second target position of the target object corresponding to the target moment according to the first posture, the second posture and the first target position, so that the vehicle draws a virtual scene corresponding to the target moment according to the second target position corresponding to the target object.

[0069] Exemplarily, the first point cloud data includes point cloud data corresponding to static objects in the environment where the vehicle is located, and the target object is a static object in the environment where the vehicle is currently located.

[0070] Optionally, the target object includes one or more of a road surface, a lane line, a fence and a curb.

[0071] Exemplarily, the driving data includes driving speed and driving direction.

[0072] Exemplarily, the first target position and the second target position are positions of the target object in a body coordinate system of the vehicle.

[0073] In some embodiments, the apparatus further comprises:

[0074] A first reference position acquisition module, used to acquire a first reference position corresponding to the vehicle at the first moment, wherein the first reference position is a position of a first position on the vehicle in a target coordinate system at the first moment, and the target coordinate system is a body coordinate system of the vehicle at the first moment;

[0075] The second posture determination module is specifically used to determine the second posture of the vehicle at the target time according to the first reference position, the driving data and the first point cloud data.

[0076] In a possible implementation, the second posture determination module is also used to determine a candidate vehicle position that the vehicle will pass within a preset time based on the first reference position, the driving data and the first point cloud data, and the preset time is greater than or equal to the time interval between the target moment and the first moment; determine the candidate posture and the first candidate moment of the vehicle at the candidate vehicle position, and associate and save the candidate posture and the first candidate moment corresponding to the candidate vehicle position; determine a second candidate moment that matches the target moment from the first candidate moment, and determine the candidate posture corresponding to the second candidate moment as the second posture of the vehicle at the target moment.

[0077] Optionally, the candidate vehicle position includes a bumpy position.

[0078] Optionally, the second posture determination module is also used to determine the posture of the vehicle at the candidate vehicle position based on the candidate vehicle position; determine the rotation matrix of the vehicle at the candidate vehicle position based on the posture, and determine the candidate posture of the vehicle at the candidate vehicle position based on the rotation matrix.

[0079] Optionally, the second candidate time is the time among the first candidate time that is closest to the target time.

[0080] Exemplarily, the first position on the vehicle includes at least one of a position corresponding to a front left tire of the vehicle, a position corresponding to a front right tire of the vehicle, a position corresponding to a rear left tire of the vehicle, and a position corresponding to a rear right tire of the vehicle.

[0081] Optionally, the position corresponding to the front left tire of the vehicle is the position where the front left tire of the vehicle contacts the ground, the position corresponding to the front right tire of the vehicle is the position where the front right tire of the vehicle contacts the ground, the position corresponding to the rear left tire of the vehicle is the position where the rear left tire of the vehicle contacts the ground, and the position corresponding to the rear right tire of the vehicle is the position where the rear right tire of the vehicle contacts the ground.

[0082] In a possible implementation, the second posture determination module is also used to determine a second reference position corresponding to the vehicle at the target moment based on the first reference position, the driving data and the first point cloud data, the second reference position being the position of the first position on the vehicle in the target coordinate system at the target moment; and determine a second posture of the vehicle at the target moment based on the second reference position.

[0083] Exemplarily, the second posture determination module is also used to determine the posture of the vehicle at the target moment based on the second reference position; determine the rotation matrix of the vehicle at the target moment based on the posture; and determine the second posture of the vehicle at the target moment based on the rotation matrix.

[0084] In some embodiments, the target time is determined based on the first time and a preset delay.

[0085] Optionally, the preset delay includes a perception delay and / or a rendering delay, the perception delay is a delay in acquiring the first point cloud data, and the rendering delay is a delay in rendering the virtual scene image.

[0086] In a possible implementation, the second posture determination module is also used to determine second point cloud data from the first point cloud data based on the first reference position and the driving data, the second point cloud data being all or part of the first point cloud data, and the second point cloud data being point cloud data corresponding to a position on the road surface that the vehicle may have traveled on within a preset time; determine the second posture of the vehicle at the target time based on the first reference position, the driving data and the second point cloud data.

[0087] In a third aspect, an embodiment of the present application provides a vehicle-mounted device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the vehicle-mounted device implements the anti-shake method for the virtual scene image described in any one of the first aspects above.

[0088] In a fourth aspect, an embodiment of the present application provides a vehicle, comprising the vehicle-mounted device described in the third aspect above.

[0089] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer implements the anti-shake method for a virtual scene image as described in any one of the first aspects above.

[0090] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a vehicle-mounted device, the vehicle-mounted device executes the anti-shake method for a virtual scene image as described in any one of the above-mentioned first aspects.

[0091] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 is a structural schematic diagram of a vehicle provided in an embodiment of the present application;

[0093] Figure 2 is a schematic flow chart of a method for stabilizing a virtual scene image provided by an embodiment of the present application;

[0094] Figure 3 is an example diagram of a vehicle body coordinate system provided in an embodiment of the present application;

[0095] Figure 4 This is an example of an application scenario provided by the embodiment of the present application. Figure 1 ;

[0096] Figure 5 This is an example of an application scenario provided by the embodiment of the present application. Figure 2 . DETAILED DESCRIPTION

[0097] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0098] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0099] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0100] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0101] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0102] In addition, the “plurality” mentioned in the embodiments of the present application should be interpreted as two or more.

[0103] The steps involved in the anti-shake method for the virtual scene provided in the embodiment of the present application are only examples, not all steps are steps that must be performed, or not all information or messages are required, and can be increased or reduced as needed during use. The same step or steps or messages with the same function in the embodiment of the present application can be referenced and learned from each other in different embodiments.

[0104] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art will appreciate that, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0105] AR HUD technology can be widely used in vehicles. Among them, AR HUD can combine virtual scenes such as navigation guide lines with the real environment where the vehicle is located, and then project them into the virtual image area for presentation, so that the driver can see the navigation guide lines combined with the real environment through the front windshield of the vehicle, thereby assisting the driver to drive the vehicle efficiently and safely. When projecting virtual scenes through AR HUD, the vehicle generally generates virtual scenes such as navigation guide lines based on the real environment (such as road surface and / or lane lines, etc.) where the vehicle is located, as perceived by the perception system in the vehicle, and combined with the navigation application, and then uses an optical machine to reflect the virtual scenes such as navigation guide lines through multiple layers, and finally projects them into the virtual image area for presentation.

[0106] Since there is generally a long delay from the time the perception system perceives the real environment where the vehicle is located to the time the virtual scene images such as the generation of navigation guide lines are projected into the virtual image area for presentation, the vehicle posture and / or the real environment where the vehicle is located may change during this delay, especially when the vehicle needs to pass through bumpy areas such as speed bumps, manhole covers or potholes. As a result, the virtual scene images presented in the virtual image area do not match the real environment where the vehicle is actually located, affecting the user experience.

[0107] For instance, taking a speed bump as an example, when a vehicle passes over a speed bump, the vehicle needs to quickly go uphill and then downhill. When the vehicle goes uphill, due to the time delay, the virtual scenes such as the navigation guide lines presented in the virtual image area are generally drawn and displayed based on the real environment perceived by the vehicle when it is on a flat road. Due to the change in the vehicle's posture when going uphill, the virtual scenes such as the navigation guide lines drawn and displayed based on the real environment perceived on a flat road will not fit the real environment where the vehicle is currently located, resulting in a poor user experience.

[0108] Similarly, when the vehicle is going downhill, due to time delay, the virtual scenes such as the navigation guide lines presented in the virtual image area are generally drawn and displayed based on the real environment perceived by the vehicle when going uphill. Due to the change in the vehicle's posture when going downhill, the virtual scenes such as the navigation guide lines drawn and displayed based on the real environment perceived when going uphill will not be consistent with the real environment where the vehicle is currently located, resulting in a poor user experience.

[0109] To solve the above problems, the embodiment of the present application provides a method for anti-shaking a virtual scene image, a vehicle-mounted device and a computer-readable storage medium. In the method, during the driving process of the vehicle, the vehicle can obtain the first posture of the vehicle, the driving data of the vehicle and the first point cloud data corresponding to the environment in which the vehicle is located at the first moment, and can determine the first target position corresponding to the target object at the first moment according to the first point cloud data, the target object is the object for reference by the vehicle to draw the virtual scene image, and the target object is the object in the environment in which the vehicle is currently located. Subsequently, the vehicle can determine the second posture of the vehicle at the target moment according to the driving data and the first point cloud data, and can determine the second target position corresponding to the target object at the target moment according to the first posture, the second posture and the first target position, the target moment is later than the first moment, so that the vehicle can draw the virtual scene image corresponding to the target moment according to the second target position corresponding to the target object, and can display the virtual scene image corresponding to the target moment in the virtual image area at the target moment, so that at the target moment, the virtual scene image displayed by the vehicle fits the real scene environment in which the vehicle is currently located, reduces the display deviation of the virtual scene image such as the navigation guide line, improves the user experience, and has strong ease of use and practicality.

[0110] The following first introduces the vehicle involved in the embodiment of the present application. Figure 1 , Figure 1 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application is shown. It is understandable that the vehicle may be an internal combustion engine vehicle using an engine as a power source, a hybrid vehicle using an engine and an electric motor as a power source, or an electric vehicle using an electric motor as a power source, etc. The embodiment of the present application does not impose any restrictions on the specific type of the vehicle.

[0111] like Figure 1 As shown, vehicle 100 may include multiple subsystems, such as a control system 110 , a perception system 120 , a communication system 130 , a navigation system 140 , an entertainment system 150 , a drive system 160 , and a computer system 170 .

[0112] Optionally, each system may include multiple components. Each system and component of the vehicle 100 may be interconnected in a wired or wireless manner.

[0113] It should be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the vehicle 100. In other embodiments, the vehicle 100 may include more or fewer systems or components than those shown in the figure.

[0114] The control system 110 may further include a vehicle controller 111 for controlling a power system of the vehicle 100 , and a steering system 112 , a throttle 113 , and a braking system 114 for controlling the vehicle 100 .

[0115] The vehicle controller 111 can be used to coordinate and control the power battery and engine 161 of the vehicle 100 to improve the power performance of the vehicle 100 .

[0116] The steering system 112 may be used to adjust the forward direction of the vehicle 100. For example, the steering system 112 may be a steering wheel system.

[0117] The throttle 113 may be used to control the operating speed of the engine 161 to control the speed of the vehicle 100 .

[0118] The braking system 114 can be used to control the deceleration of the vehicle 100. For example, the braking system 114 can use friction to slow down the wheels 164.

[0119] The perception system 120 may include several sensors for sensing the surrounding environment information of the vehicle 100. For example, the perception system 120 may include a global positioning system 121 (such as a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS), and / or a satellite based augmentation system (SBAS), etc.), an inertial measurement unit (IMU) 122, a laser radar 123, a millimeter wave radar 124, an ultrasonic radar 125, and a camera 126, etc. The perception system 120 may also include sensors for monitoring the internal systems of the vehicle 100 (such as an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). The vehicle 100 may obtain the current location, direction, speed, and surrounding environment information of the vehicle 100 through the perception system 120.

[0120] The global positioning system 121 can be used to estimate the position and posture of the vehicle 100, that is, it can be used to estimate the geographical location and direction of the vehicle.

[0121] The inertial measurement unit 122 may sense, among other things, the position and orientation changes of the vehicle 100 based on inertial acceleration. In some embodiments, the inertial measurement unit 122 may be a combination of an accelerometer and a gyroscope.

[0122] The laser radar 123 can use lasers to sense objects in the environment of the vehicle 100. In some embodiments, the laser radar 123 can include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. In some embodiments, the laser radar 123 can be used to obtain point cloud data corresponding to the environment of the vehicle 100.

[0123] The millimeter wave radar 124 can use radio signals to sense objects in the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the millimeter wave radar 124 can also be used to sense the speed and / or the direction of travel of the object.

[0124] The ultrasonic radar 125 may sense objects around the vehicle 100 using ultrasonic signals.

[0125] The camera 126 can be used to capture static images or videos of the surrounding environment of the vehicle 100. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.

[0126] In some embodiments, the vehicle 100 may include 1 or N cameras 126 , where N is a positive integer greater than 1.

[0127] In other embodiments, the camera 126 may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc. Exemplarily, the camera 126 may also acquire point cloud data corresponding to the environment where the vehicle 100 is located.

[0128] The communication system 130 may include a wireless communication system that can communicate wirelessly with one or more devices using a communication network. For example, the wireless communication system can communicate using 3G cellular communications, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communications, such as LTE, or 5G cellular communications. For example, the wireless communication system can communicate with a wireless local area network (WLAN) using WiFi. In some embodiments, the wireless communication system can communicate using an infrared link, Bluetooth, or ZigBee.

[0129] The navigation system 140 may include a map service provided by a map provider, thereby providing navigation of a driving route for the vehicle 100. The navigation system 140 may be used in conjunction with the vehicle's global positioning system 121, inertial measurement unit 122, etc. The map service provided by the map provider may be a two-dimensional map or a high-precision map.

[0130] The entertainment system 150 may include a display device 151, an audio module 152, a speaker 152A, a receiver 152B, a microphone 152C, an earphone jack 152D, and the like.

[0131] The entertainment system 150 can implement audio functions such as music playing and recording through the audio module 152, the speaker 152A, the receiver 152B, the microphone 152C, the headphone jack 152D, and the application processor.

[0132] The audio module 152 can be used to convert digital audio information into analog audio signal output, and can also be used to convert analog audio input into digital audio signals. The audio module 152 can also be used to encode and decode audio signals.

[0133] The speaker 152A, also called a "horn", is used to convert an audio electrical signal into a sound signal. The vehicle 100 can listen to music or listen to hands-free calls through the speaker 152A.

[0134] The receiver 152B, also called a "handset", is used to convert audio electrical signals into sound signals. When the vehicle 100 receives a call or voice message, the voice can be received by the receiver 152B.

[0135] Microphone 152C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak through microphone 152C to input the sound signal into microphone 152C. The vehicle 100 can be provided with at least one microphone 152C. In other embodiments, the vehicle 100 can be provided with two microphones 152C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the vehicle 100 can also be provided with three, four or more microphones 152C to realize the collection of sound signals, noise reduction, identification of sound sources, and directional recording functions, etc.

[0136] The earphone interface 152D is used to connect a wired earphone and can be a universal serial bus (USB) interface, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0137] In some examples, the user's voice signal can also be obtained through a microphone, and the user's voice signal can be analyzed to enable the user to control the vehicle 100 in certain ways, such as adjusting the temperature in the vehicle, playing music, etc.

[0138] The display device 151 can be used to display images, videos, etc.

[0139] Optionally, the display device 151 is an actual display device, such as a central control screen. The display device 151 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the vehicle 100 may include 1 or N display devices 151, where N is a positive integer greater than 1.

[0140] Optionally, the display device 151 may be a display device formed by projection technology onto an object, for example, by projection onto the front windshield of the vehicle through holographic projection or head up display (HUD) technology, with the projection area on the front windshield serving as the display device. The embodiment of the present application does not specifically limit the type of display device.

[0141] The drive system 160 may include components that provide power movement for the vehicle 100. In one embodiment, the drive system 160 may include an engine 161, an energy source 162, a transmission system 163, and wheels 164. The engine 161 may be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 161 converts the energy source 162 into mechanical energy.

[0142] The energy source 162 may include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, other sources of electricity, etc. The energy source 162 may also provide power to other systems of the vehicle 100 .

[0143] The transmission system 163 can transmit mechanical power from the engine 161 to the wheels 164. The transmission system 163 can include a gearbox, a differential, and a drive shaft. Among them, the drive shaft can include one or more shafts that can be coupled to one or more wheels 164. In some examples, the transmission system 163 can also include other devices, such as a clutch.

[0144] Some or all functions of the vehicle 100 are controlled by a computer system 170. The computer system 170 may include at least one processor 171. The processor 171 may include one or more processing units, for example, the processor 171 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0145] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0146] A memory may also be provided in the processor 171 for storing instructions and data. In some examples, the memory in the processor 171 is a cache memory. The memory may store instructions or data that the processor 171 has just used or cyclically used. If the processor 171 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 171, and thus improves the efficiency of the system.

[0147] In some embodiments, the processor 171 can control some functions of the vehicle 100 based on information obtained by various subsystems (such as the perception system 120, the navigation system 140, etc.).

[0148] For example, the processor 171 can analyze the information obtained by each subsystem, draw virtual scenes such as navigation guide lines, and display the virtual scenes such as navigation guide lines on the front windshield of the vehicle 100.

[0149] For example, the processor 171 can obtain data perceived by the perception system 120, such as the current posture, driving speed, driving direction of the vehicle 100 and point cloud data corresponding to the surrounding environment of the vehicle 100, and can determine the position corresponding to the target object (hereinafter referred to as the first target position) based on the point cloud data corresponding to the surrounding environment of the vehicle 100. The target object can be a static object such as a lane line that is referenced when the vehicle 100 draws a virtual scene (such as a navigation guide line, etc.).

[0150] Subsequently, the vehicle 100 can predict the relative position of the vehicle 100 at the target moment based on the data sensed by the perception system 120, and can predict the second target position corresponding to the target object at the target moment based on the current position, relative position and first target position corresponding to the target object of the vehicle 100, so that a virtual scene image (such as a navigation guide line, etc.) corresponding to the target moment can be drawn according to the second target position corresponding to the target object at the target moment, and the virtual scene image corresponding to the target moment can be displayed in the virtual image area at the target moment, so that at the target moment, the virtual scene images such as the navigation guide line displayed in the virtual image area are consistent with the real scene environment where the vehicle 100 is actually located.

[0151] The anti-shake method for the virtual scene image provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings and specific application scenarios.

[0152] See also Figure 2 , Figure 2 FIG. 1 is a schematic flow chart of a method for stabilizing a virtual scene image provided by an embodiment of the present application. The method can be applied to the vehicle 100 described above. Figure 2 As shown, the method may include:

[0153] S201. Acquire a first point cloud data corresponding to a first posture of a vehicle, driving data of the vehicle, and an environment where the vehicle is located at a first moment.

[0154] S202. Determine, based on the first point cloud data, a first target position corresponding to a target object at a first moment, where the target object is an object referenced by the vehicle for drawing a virtual scene, and the target object is an object in the environment where the vehicle is currently located.

[0155] S203. Determine a second posture of the vehicle at a target time according to the driving data and the first point cloud data, where the target time is later than the first time.

[0156] S204: Determine a second target position corresponding to the target object at the target moment according to the first posture, the second posture and the first target position.

[0157] In an embodiment of the present application, during the driving process of the vehicle, the vehicle can obtain the posture of the vehicle at the first moment (i.e., the first posture), the driving data of the vehicle at the first moment, and the first point cloud data corresponding to the environment where the vehicle is located at the first moment, and determine the first target position corresponding to the target object at the first moment based on the first point cloud data. Subsequently, the vehicle can predict the second posture of the vehicle at the target moment based on the driving data and the first point cloud data, and can determine the second target position corresponding to the target object at the target moment based on the first posture, the second posture and the first target position, so that the vehicle can draw the virtual scene corresponding to the target moment based on the second target position corresponding to the target object at the target moment, and can display the virtual scene corresponding to the target moment in the virtual image area at the target moment, so that at the target moment, the virtual scene displayed by the vehicle is consistent with the real scene environment where the vehicle is actually located, reducing the display deviation of the virtual scene and improving the user experience.

[0158] It is understandable that the first moment may refer to any moment when the first pose, driving data, and first point cloud data are obtained, and the present application embodiment does not impose any limitation on this. The first moment may be the current moment or a moment earlier than the current moment.

[0159] Exemplarily, the first pose of the vehicle may refer to the pose of the vehicle in the world coordinate system at the first moment. For example, it may refer to the pose of the center of mass of the vehicle in the world coordinate system at the first moment. It should be understood that the pose may include the position and attitude of the vehicle.

[0160] Exemplarily, the vehicle's driving data may include the vehicle's driving speed and driving direction, etc.

[0161] Exemplarily, the first point cloud data corresponding to the environment where the vehicle is located may include one or more of the point cloud data corresponding to the road surface in front of the vehicle, the point cloud data corresponding to the lane line, the point cloud data corresponding to the fence, the point cloud data corresponding to the curb, and the point cloud data corresponding to other vehicles or obstacles on the road surface. Among them, the first point cloud data may refer to a set of vectors of each object in the vehicle body coordinate system. The first point cloud data contains three-dimensional coordinates, which are used to represent the position in the vehicle body coordinate system.

[0162] Optionally, a perception system may be provided in the vehicle, and the vehicle may obtain the first posture of the vehicle at the first moment, the vehicle's driving data (such as driving speed and driving direction, etc.), and the first point cloud data corresponding to the environment in which the vehicle is located through the perception system. It should be understood that the point cloud data corresponding to the environment in which the vehicle is located collected by the perception system may be data in a coordinate system corresponding to the perception system (such as a radar coordinate system). Therefore, after obtaining the point cloud data collected by the perception system, the vehicle may convert the point cloud data collected by the perception system into the vehicle body coordinate system at the first moment to obtain the above-mentioned first point cloud data.

[0163] It should be noted that the embodiment of the present application does not impose any restrictions on the specific process of converting the point cloud data collected by the perception system into the vehicle body coordinate system to obtain the first point cloud data, which can be determined according to the actual scenario.

[0164] Optionally, the first point cloud data acquired at the first moment may be point cloud data acquired by the perception system at the first moment, or may be point cloud data acquired by the perception system at a certain moment before the first moment.

[0165] It should be noted that when the point cloud data acquired at the first moment is point cloud data acquired by the perception system at a certain moment before the first moment, the point cloud data acquired at that moment can be converted into point cloud data corresponding to the first moment based on the relationship between the first moment and that moment.

[0166] Optionally, the driving data at the first moment may be driving data directly sensed by a perception system, or may be driving data determined based on point cloud data acquired between a certain moment before the first moment and the first moment.

[0167] Exemplarily, the target object may refer to an object that is referenced when the vehicle draws a virtual scene (such as a navigation guide line and / or a guide arrow, etc.). For example, the target object may include one or more static objects such as a road surface, a lane line, a fence, and a curb.

[0168] Optionally, the first target position corresponding to the target object and the second target position corresponding to the target object may refer to the position of the target object in the vehicle body coordinate system, for example, may refer to the coordinates of the target object in the vehicle body coordinate system. That is, the first target position corresponding to the target object may refer to the coordinates of the target object in the vehicle body coordinate system at the first moment. The second target position corresponding to the target object may refer to the coordinates of the target object in the vehicle body coordinate system at the target moment.

[0169] In the embodiment of the present application, the body coordinate system can be specifically determined according to the actual scenario. For example, the origin of the body coordinate system can be determined as the center of mass of the vehicle according to the actual scenario, the X-axis of the body coordinate system is parallel to the ground and points to the front of the vehicle, the Z-axis of the body coordinate system passes through the center of mass of the vehicle and points upward, and the Y-axis of the body coordinate system points to the left side of the driver. For example, the origin of the body coordinate system can be determined as the center of mass of the vehicle according to the actual scenario, the X-axis of the body coordinate system is parallel to the ground and points to the front of the vehicle, the Z-axis of the body coordinate system passes through the center of mass of the vehicle and points downward, and the Y-axis of the body coordinate system points to the right side of the driver. For example, the origin of the body coordinate system can be determined as the projection point of the center point of the rear axle of the vehicle on the ground according to the actual scenario, the X-axis of the body coordinate system is parallel to the ground and points to the front of the vehicle, the Z-axis of the body coordinate system passes through the center of mass of the vehicle and points downward, and the Y-axis of the body coordinate system points to the right side of the driver. For example, Figure 3 As shown, the origin of the vehicle coordinate system can be determined as the projection point of the vehicle's center of mass on the ground according to the actual scenario, the X-axis of the vehicle coordinate system is parallel to the ground and points to the front of the vehicle, the Z-axis of the vehicle coordinate system passes through the vehicle's center of mass and points upward, the Y-axis of the vehicle coordinate system points to the left of the driver, and so on.

[0170] The following is an example in which the origin of the vehicle coordinate system is the projection point of the center point of the rear axle of the vehicle on the ground, the X-axis of the vehicle coordinate system is parallel to the ground and points to the front of the vehicle, the Z-axis of the vehicle coordinate system passes through the center of mass of the vehicle and points upward, and the Y-axis of the vehicle coordinate system points to the left side of the driver.

[0171] In some embodiments, the first target position corresponding to the target object can be determined based on the first point cloud data corresponding to the environment in which the vehicle is currently located. For example, after the vehicle obtains the first point cloud data, it can perform target recognition based on the first point cloud data to identify the target object and determine the first target position corresponding to the target object. In other words, the vehicle can determine the first point cloud data corresponding to the target object based on the first point cloud data, and can determine the first target position corresponding to the target object based on the first point cloud data corresponding to the target object (e.g., the three-dimensional coordinates contained in the first point cloud data).

[0172] It is understandable that the embodiment of the present application does not impose any restrictions on the method of performing target recognition based on the first point cloud data and determining the first target position corresponding to the target object, which can be specifically determined according to the actual scene. In addition, the embodiment of the present application performs target recognition based on the first point cloud data to determine the target object and the first target position corresponding to the target object is only explained as an example and should not be understood as a limitation on the embodiment of the present application. The embodiment of the present application can also perform target object recognition and determination of the first target position corresponding to the target object in other ways. For example, an image can be collected by a camera, and the target object and the first target position corresponding to the target object can be determined by image recognition, and so on.

[0173] The following will describe S203, determining the second position of the vehicle at the target time according to the driving data and the first point cloud data. The process of posture is described in detail.

[0174] Optionally, the second posture may be a relative posture. For example, the second posture may be a relative posture between the vehicle at the target moment and the vehicle at the first moment.

[0175] Exemplarily, the target time can be determined according to the first time and the preset time delay. For example, the target time can be the time obtained by adding the preset time delay to the first time.

[0176] In some embodiments, the preset delay may be the time required from the perception system starting to perceive the relevant data corresponding to the real scene environment where the vehicle is located to the generation of the virtual scene image projected into the virtual image area for presentation. Exemplarily, the preset delay may include the perception delay and / or rendering delay of the vehicle.

[0177] Optionally, the perception delay may refer to the delay of the vehicle sensing data, that is, the delay of the vehicle's processor acquiring the data perceived by the perception system. The perception delay may be determined based on the timestamp in the data. For example, when the perception system acquires data A, the acquisition time (e.g., time T0) may be added to data A. After the vehicle's processor acquires data A, the perception delay may be determined based on the acquisition time in data A and the time when the processor acquires data A. For example, if the processor acquires data A perceived by the perception system at time T1, the processor may determine the perception delay to be (T1-T0).

[0178] Optionally, the rendering delay may refer to the delay of the processor in the vehicle to render the virtual scene, that is, the time required for the processor to render the virtual scene after acquiring the relevant data perceived by the perception system. For example, when the processor acquires the relevant data perceived by the perception system at time T2, assuming that the processor renders the virtual scene at time T3 based on the relevant data perceived by the perception system, the processor may determine the rendering delay as (T3-T2).

[0179] In one example, the perception delay and rendering delay can be determined in real time, that is, the perception delay can be determined in real time based on the relevant data collected by the perception system, and the rendering delay can be determined in real time based on the picture rendering process performed by the processor. In another example, the perception delay and rendering delay can be obtained by prior testing, that is, the perception delay and rendering delay can be fixed values ​​obtained by prior testing. In another example, the perception delay can be determined in real time, and the rendering delay can be a fixed value obtained by prior testing. In another example, the perception delay can be a fixed value obtained by prior testing, and the rendering delay can be determined in real time.

[0180] It is understandable that the preset time delay is generally small, for example, generally a few hundred milliseconds or tens of milliseconds, etc., that is, the second posture corresponding to the target moment required to be determined in the embodiment of the present application is generally the posture of the vehicle after a short time of a few hundred milliseconds or tens of milliseconds. During the high-speed driving of the vehicle, the movement of the vehicle in such a short time can be approximated as a straight-line motion. The following will be exemplified by taking the movement of the vehicle as a straight-line motion as an example.

[0181] In some embodiments, the vehicle may obtain a first reference position corresponding to the vehicle at a first moment, and may determine a second posture of the vehicle at a target moment based on the first reference position, the driving data, and the first point cloud data. The first reference position may be a position of a first position on the vehicle in a target coordinate system at the first moment, and the target coordinate system may be a body coordinate system of the vehicle at the first moment.

[0182] Optionally, the first position on the vehicle may include the position of at least one of the front left tire, the front right tire, the rear left tire, and the rear right tire of the vehicle in the vehicle body coordinate system at the first moment.

[0183] For example, the first position on the vehicle may include coordinates of a front left tire of the vehicle in the vehicle body coordinate system at the first moment, and / or coordinates of a front right tire of the vehicle in the vehicle body coordinate system at the first moment.

[0184] For example, the first position on the vehicle may include coordinates of a rear left tire of the vehicle in the vehicle body coordinate system at the first moment, and / or coordinates of a rear right tire of the vehicle in the vehicle body coordinate system at the first moment.

[0185] For example, the first position on the vehicle may include the coordinates of the vehicle's front left tire in the vehicle's body coordinate system at the first moment, the coordinates of the vehicle's front right tire in the vehicle's body coordinate system at the first moment, the coordinates of the vehicle's rear left tire in the vehicle's body coordinate system at the first moment, and the coordinates of the vehicle's rear right tire in the vehicle's body coordinate system at the first moment.

[0186] Optionally, the coordinates of the front left tire of the vehicle in the vehicle body coordinate system at the first moment may be the coordinates of the center point of the ground contact portion of the front left tire in the vehicle body coordinate system at the first moment. The coordinates of the front right tire of the vehicle in the vehicle body coordinate system at the first moment may be the coordinates of the center point of the ground contact portion of the front right tire in the vehicle body coordinate system at the first moment.

[0187] Similarly, the coordinates of the rear left tire of the vehicle in the vehicle body coordinate system at the first moment may be the coordinates of the center point of the portion of the rear left tire in contact with the ground in the vehicle body coordinate system at the first moment. The coordinates of the rear right tire of the vehicle in the vehicle body coordinate system at the first moment may be the coordinates of the center point of the portion of the rear right tire in contact with the ground in the vehicle body coordinate system at the first moment.

[0188] It should be understood that the vehicle can obtain the coordinates of the front left tire and / or the front right tire of the vehicle in the body coordinate system at the first moment based on the calibration relationship between the front left tire and / or the front right tire and the body coordinate system. Similarly, the vehicle can obtain the coordinates of the rear left tire and / or the rear right tire of the vehicle in the body coordinate system at the first moment based on the calibration relationship between the rear left tire and / or the rear right tire and the body coordinate system to obtain the first reference position corresponding to the vehicle.

[0189] In some embodiments, when the first position on the vehicle includes the position of one of the vehicle's front left tire, front right tire, rear left tire, and rear right tire in the vehicle's body coordinate system at a first moment, the vehicle can determine the vehicle's second posture at the target moment based on the vehicle's corresponding first reference position, driving data, and first point cloud data.

[0190] In one possible implementation, after obtaining the first reference position, driving data and first point cloud data corresponding to the vehicle, the vehicle can determine the various positions that the vehicle may pass within a preset time (hereinafter referred to as candidate vehicle positions) based on the first reference position, driving data (such as driving speed and driving direction, etc.) and the first point cloud data, and can predict the time when the vehicle arrives at each candidate vehicle position (hereinafter referred to as candidate time).

[0191] For each candidate vehicle position, the vehicle can determine the relative posture of the vehicle at the candidate vehicle position (hereinafter referred to as the candidate posture), and can associate and save the candidate time corresponding to the candidate vehicle position with the candidate posture. Therefore, after determining the target time according to the first time and the preset delay, the vehicle can determine the candidate time that matches the target time from the candidate time, and determine the candidate posture corresponding to the determined candidate time as the second posture of the vehicle at the target time.

[0192] That is to say, after obtaining the first reference position, driving data and first point cloud data corresponding to the vehicle, the vehicle can predict in advance the candidate time and candidate posture of the vehicle arriving at each candidate vehicle position, and associate and save the candidate time and candidate posture corresponding to each candidate vehicle position. After determining the target time according to the first time and the preset time delay, the vehicle can match the target time from the candidate time, thereby determining the second posture of the vehicle at the target time.

[0193] Optionally, the moment among the candidate moments that matches the target moment may be the moment among the candidate moments closest to the target moment, that is, the vehicle may determine the candidate posture corresponding to the candidate moment closest to the target moment as the second posture of the vehicle.

[0194] It is understandable that the candidate vehicle position may refer to the position where the first position on the vehicle may be located at a certain moment. For example, the candidate vehicle position may refer to the position that one of the front left tire, front right tire, rear left tire and rear right tire of the vehicle may reach, determined based on the vehicle body coordinate system at the first moment. For example, when the first position on the vehicle is the position corresponding to the front left tire of the vehicle, the candidate vehicle position may be the position that the front left tire of the vehicle may reach. For example, when the first position on the vehicle is the position corresponding to the rear right tire of the vehicle, the candidate vehicle position may be the position that the rear right tire of the vehicle may reach.

[0195] Exemplarily, the preset time may be a preset time. The preset time may be greater than or equal to the time interval between the target moment and the first moment. For example, a technician may set the preset time to any value such as 1 second, 50 milliseconds, or 10 milliseconds according to the actual scenario.

[0196] Exemplarily, the preset time may be a time determined according to a preset time delay. The preset time may be greater than or equal to the preset time delay. For example, the vehicle may directly determine the preset time delay as the preset time, or may determine a time (hereinafter referred to as time A) according to the preset time delay as the preset time, wherein time A may be greater than or equal to the preset time delay. For example, when the preset time delay is 100 milliseconds, the preset time may be determined as 100 milliseconds, or the preset time may be determined as any value such as 200 milliseconds, 30 milliseconds, or 1 second.

[0197] Optionally, for each candidate vehicle position, the vehicle can determine the posture of the vehicle at the candidate vehicle position, and can convert the posture into a rotation matrix according to the conversion relationship between the Euler angle and the rotation matrix to obtain the rotation matrix of the vehicle at the candidate moment, so as to determine the converted rotation matrix as the candidate posture of the vehicle at the candidate moment, that is, the candidate posture of the vehicle at the candidate moment can be characterized by the rotation matrix. The posture can be represented by the pitch angle, the roll angle and the navigation angle.

[0198] For example, see Figure 4 , Figure 4 The application scenario provided by the embodiment of the present application is shown Figure 1 This application scenario is exemplified by taking the reference position (eg, the first reference position or the second reference position) corresponding to the vehicle as the position of the front left tire of the vehicle in the vehicle body coordinate system as an example.

[0199] like Figure 4 As shown, it is assumed that after obtaining the first reference position (x1, y1, 0), driving data and first point cloud data corresponding to the vehicle at the first moment, the vehicle determines that the candidate vehicle position A that the vehicle (i.e., the front left tire) may have traveled through includes position A1 (x2, y2, z2) based on the first reference position, driving data and first point cloud data. At this time, the vehicle can determine the candidate posture of the vehicle when the front left tire is at position A1 and the candidate time when the front left tire reaches position A1 based on position A1.

[0200] Assume that the vehicle can determine the position A2 (x3, y3, 0) based on the first point cloud data. Therefore, the vehicle can determine that the pitch angle θ of the vehicle when the front left tire is at position A1 is arctan [z2 / (x2-x3)], and the roll angle and navigation angle are both 0. At this time, the vehicle can perform a rotation matrix conversion on the pitch angle, roll angle and navigation angle of the vehicle when the front left tire is at position A1 based on the conversion relationship between the Euler angle and the rotation matrix, and obtain the candidate posture of the vehicle when the front left tire is at position A1.

[0201] In addition, the vehicle can determine the time t when the front left tire reaches the position A1 according to the first reference position, the first point cloud data and the driving data (such as driving speed and driving direction, etc.) corresponding to the vehicle, and can determine the candidate time when the front left tire reaches the position A1 according to the first time and time t, that is, the candidate time = first time + time t. Subsequently, the vehicle can associate and save the candidate time and candidate posture corresponding to the position A1.

[0202] like Figure 4 As shown, time t = time used for going uphill + time used for going on flat land. Among them, time used for going uphill = (z2 / sinθ / V), time used for going on flat land = d / V, d is the distance traveled on flat land, and V is the driving speed of the vehicle.

[0203] It is understandable that the flat ground driving distance d can be determined based on the first reference position and the first point cloud data corresponding to the vehicle. For example, the vehicle can determine the position A2 (x3, y3, 0) based on the first point cloud data, and can determine the flat ground driving distance d based on the position A2 and the first reference position, that is, the flat ground driving distance d = |x3-x1|.

[0204] Next, the vehicle is determined based on the first reference position, the driving data and the first point cloud data to determine the vehicle within the preset time. The process of identifying candidate vehicle locations that may have been traveled through is described in detail.

[0205] Exemplarily, after obtaining the first point cloud data, the vehicle can determine, from the first point cloud data, point cloud data (hereinafter referred to as second point cloud data) that the vehicle may have traveled through within a preset time based on the first reference position and driving data corresponding to the vehicle, and determine candidate vehicle positions that the vehicle may have traveled through based on the second point cloud data.

[0206] It is understandable that the candidate vehicle position that the vehicle may travel through may refer to the position where the first position on the vehicle may be located at a certain moment. For example, the candidate vehicle position may refer to a position where one of the front left tire, the front right tire, the rear left tire, and the rear right tire of the vehicle may travel through based on the body coordinate system at the first moment. For example, when the first reference position is the position of the center point of the ground contact portion of the front left tire of the vehicle in the body coordinate system, the candidate vehicle position may be the position where the front left tire of the vehicle may travel through. For example, when the first reference position is the position of the center point of the ground contact portion of the front right tire of the vehicle in the body coordinate system, the candidate vehicle position may be the position where the front right tire of the vehicle may travel through. For example, when the first reference position is the position of the center point of the ground contact portion of the rear left tire of the vehicle in the body coordinate system, the candidate vehicle position may be the position where the rear left tire of the vehicle may travel through. For example, when the first reference position is the position of the center point of the ground contact portion of the rear right tire of the vehicle in the body coordinate system, the candidate vehicle position may be the position where the rear right tire of the vehicle may travel through.

[0207] See also Figure 5 , Figure 5 The application scenario provided by the embodiment of the present application is shown Figure 2 This application scenario uses the first reference position as the position of the front left tire of the vehicle in the vehicle body coordinate system or the position of the front right tire of the vehicle in the vehicle body coordinate system as an example to illustrate the position where the vehicle may travel.

[0208] like Figure 5 As shown, it is assumed that the position A of the front left tire of the vehicle in the vehicle body coordinate system is (x, y, 0), the position B of the front right tire of the vehicle in the vehicle body coordinate system is (x, -y, 0), the length of the tread of the tire (including the front left tire and the front right tire) is R, and the driving speed of the vehicle is Vm / s. The tread of the tire may refer to the area of ​​the tire that contacts the ground.

[0209] Within the preset time Ts, the second point cloud data (hereinafter referred to as the second point cloud data A) that the front left tire of the vehicle may pass through can be Figure 5The point cloud data contained in the rectangular area A shown in (a) in . That is to say, within the preset time Ts, the coordinates of the second point cloud data A that the front left tire of the vehicle may pass through can be located within the rectangular area A. Based on the position A, driving speed, tread and preset time, the vehicle can determine that the coordinates of the four vertices of the rectangular area A are (x, y+R / 2, 0), (x, yR / 2, 0), (x+V*T, y+R / 2, 0) and (x+V*T, yR / 2, 0). Therefore, the vehicle can obtain the second point cloud data A from the first point cloud data based on the coordinates of the second point cloud data A, and can determine the candidate vehicle position that the left tire may have passed before the preset time based on the second point cloud data A. For example, as Figure 5 As shown in (a) in FIG. 1 , each position on the center line 501 of the rectangular area A can be determined as a candidate vehicle position that the front left tire may travel over within a preset time.

[0210] Within the preset time Ts, the second point cloud data (hereinafter referred to as the second point cloud data B) that the front right tire of the vehicle may pass through can be Figure 5 The point cloud data contained in the rectangular area B shown in (b) in . That is to say, within the preset time Ts, the coordinates of the second point cloud data B that the front right tire of the vehicle may pass through can be located within the rectangular area B. Based on the position B, driving speed, tread and preset time, the vehicle can determine that the coordinates of the four vertices of the rectangular area B are (x, -y+R / 2, 0), (x, -yR / 2, 0), (x+V*T, -y+R / 2, 0) and (x+V*T, -yR / 2, 0). Therefore, the vehicle can obtain the second point cloud data B from the first point cloud data based on the coordinates of the second point cloud data B, and can determine the candidate vehicle position that the front right tire may pass through within the preset time based on the second point cloud data B. For example, Figure 5 As shown in (b) in FIG. 1 , each position on the center line 502 of the rectangular area B can be determined as a candidate vehicle position that the front right tire may travel over within a preset time.

[0211] In one example, the road surface on which the vehicle travels may not always be flat, and there may be speed bumps, manhole covers, or potholes on the road surface. Among them, the coordinates of the point cloud data corresponding to the speed bumps, manhole covers, or potholes on the Z axis are often not 0. In other words, the coordinates of the second point cloud data that the vehicle may have traveled on may not always be 0 on the Z axis. In order to ensure the accuracy of the second point cloud data and accurately determine the position of the candidate vehicle, when determining the second point cloud data that the front left tire of the vehicle may have traveled on within a preset time, the coordinates of the second point cloud data on the Z axis may not be restricted.

[0212] That is, the vehicle can determine that within the preset time, the coordinates of the second point cloud data that the front left tire of the vehicle may have passed through are located at [x, x+V*T] on the X-axis and [yR / 2, y+R / 2] on the Y-axis. Therefore, the vehicle can determine the point cloud data whose coordinates on the X-axis are located at [x, x+V*T] and whose coordinates on the Y-axis are located at [yR / 2, y+R / 2] from the first point cloud data, and use the determined point cloud data as the second point cloud data that the front left tire may have passed through.

[0213] Similarly, the vehicle can determine that within a preset time, the coordinates of the second point cloud data that the front right tire of the vehicle may pass through on the X-axis are located at [x, x+V*T], and the coordinates on the Y-axis may be located at [-yR / 2, -y+R / 2]. Therefore, the vehicle can determine the point cloud data whose coordinates on the X-axis are located at [x, x+V*T] and whose coordinates on the Y-axis are located at [-yR / 2, -y+R / 2] from the first point cloud data, and use the determined point cloud data as the second point cloud data that the front right tire may pass through.

[0214] In one example, the point cloud data that the vehicle may have traveled through is generally the point cloud data corresponding to the road surface. Therefore, after determining the second point cloud data, the vehicle can delete the point cloud data corresponding to the non-road surface from the second point cloud data (hereinafter the point cloud data corresponding to the non-road surface may be referred to as the third point cloud data), and can determine the candidate vehicle positions that the vehicle may have traveled through based on the second point cloud data after deleting the third point cloud data, which can reduce the impact of the point cloud data corresponding to the non-road surface on the determination of the candidate vehicle positions, thereby improving the speed and efficiency of determining the candidate vehicle positions.

[0215] It is understandable that the vehicle can determine the third point cloud data based on the coordinates of the point cloud data on the Z axis. For example, when the coordinates of the point cloud data on the Z axis are greater than a preset threshold, the point cloud data can be determined to be the third point cloud data. The preset threshold can be specifically determined according to the actual scenario, and the embodiments of the present application do not impose any restrictions on this.

[0216] In another example, after determining the second point cloud data, or after deleting the third point cloud data from the second point cloud data, the vehicle may fit the second point cloud data into a smooth shape to facilitate determining the candidate vehicle positions that the vehicle may have traveled through based on the second point cloud data, thereby improving the speed and efficiency of determining the candidate vehicle positions.

[0217] In another possible implementation, after obtaining the first reference position, driving data and first point cloud data corresponding to the vehicle, the vehicle can determine the position of the first position on the vehicle at the target time (hereinafter referred to as the second reference position) based on the first reference position, driving data (such as driving speed and driving direction, etc.), a preset time delay and the first point cloud data, and directly determine the second posture of the vehicle at the target time based on the second reference position.

[0218] That is to say, after obtaining the first reference position, driving data and first point cloud data corresponding to the vehicle, the vehicle can directly predict the second reference position corresponding to the vehicle after a preset delay based on the first reference position, driving speed, driving direction and first point cloud data, that is, determine the second reference position where the first position of the vehicle may be after the preset delay, so that the second posture of the vehicle at the target time can be determined directly based on the second reference position corresponding to the vehicle.

[0219] It is understood that the second reference position may refer to the position that the front left tire or the front right tire of the vehicle may reach after a preset time delay, determined based on the vehicle body coordinate system at the first moment. Alternatively, the second reference position may refer to the position that the rear left tire or the rear right tire of the vehicle may reach after a preset time delay, determined based on the vehicle body coordinate system at the first moment. In other words, the second reference position refers to the position in the vehicle body coordinate system at the first moment.

[0220] Optionally, the vehicle can determine the posture of the vehicle at the target moment (such as pitch angle, roll angle and navigation angle) based on the second reference position, and can transform the posture (such as pitch angle, roll angle and navigation angle) into a rotation matrix based on the conversion relationship between Euler angles and the rotation matrix to obtain the rotation matrix of the vehicle at the target moment, so as to determine the converted rotation matrix as the second posture of the vehicle at the target moment.

[0221] In another possible implementation, the mismatch between the virtual scene displayed in the virtual image area and the real scene environment where the vehicle is actually located often occurs when the vehicle needs to pass through a bumpy location such as a speed bump, a manhole cover or a pothole. When the vehicle is always driving on a flat road, the mismatch between the virtual scene displayed in the virtual image area and the real scene environment where the vehicle is actually located is generally not obvious. In order to reduce the amount of calculation of the vehicle and reduce the power consumption of the vehicle, the vehicle can only predict the second posture of the vehicle when it is at a bumpy location, so as to determine the second target position corresponding to the target object when the vehicle reaches the bumpy location based on the second posture at the bumpy location, and then draw the virtual scene corresponding to the bumpy location based on the second target position, so that when the vehicle reaches the bumpy location, the virtual scene displayed in the virtual image area is consistent with the real scene environment where the vehicle is actually located.

[0222] That is to say, when the vehicle needs to pass through bumpy locations such as speed bumps, manhole covers or potholes, the vehicle can determine the vehicle's second posture based on the vehicle's corresponding first reference position, driving data and first point cloud data, and can predict the second target position of the target object at the target moment based on the vehicle's first posture, second posture and the first target position corresponding to the target object, so that the virtual scene corresponding to the target moment can be drawn based on the second target position of the target object at the target moment, and the virtual scene corresponding to the target moment can be displayed in the virtual image area at the target moment, so that when the vehicle drives to the bumpy position, the virtual scene displayed in the virtual image area is consistent with the real scene environment where the vehicle is actually located.

[0223] Therefore, after obtaining the first reference position, driving data and first point cloud data corresponding to the vehicle, the vehicle can determine the position (hereinafter referred to as the bump position) that may cause the virtual scene to not fit the real scene environment where the vehicle is actually located based on the first point cloud data, that is, the bump position such as speed bumps, manhole covers or potholes can be identified based on the first point cloud data. Subsequently, the vehicle can predict the time when the vehicle reaches each bump position (hereinafter referred to as the bump moment) based on the first reference position, driving data (such as driving speed and driving direction, etc.) and bump position.

[0224] For each bump position, the vehicle can determine the candidate posture of the vehicle at the bump position, and can associate and save the bump moment corresponding to the bump position with the candidate posture. After determining the target moment according to the first moment and the preset delay, the vehicle can determine the bump moment that matches the target moment from the bump moments, and determine the candidate posture corresponding to the determined bump moment as the second posture of the vehicle at the target moment. That is, the vehicle only needs to predict the candidate posture of the bump position, which can reduce the number of candidate posture predictions made by the vehicle, increase the speed of determining the second posture, and improve the user experience.

[0225] It should be understood that when there is no match to the bumpy moment corresponding to the target moment, the vehicle does not need to determine the second posture of the vehicle, and thus does not need to determine the second target position corresponding to the target object. At this time, the vehicle can directly draw the virtual scene according to the first target position corresponding to the target object.

[0226] In other embodiments, when the first reference position corresponding to the vehicle includes the positions of multiple ones of the vehicle's front left tire, front right tire, rear left tire and rear right tire in the vehicle's body coordinate system at the first moment, the vehicle can determine the vehicle's second posture at the target moment based on these multiple positions, driving data and first point cloud data.

[0227] For the sake of convenience of description, the following will take the first reference position corresponding to the vehicle including the position of the front left tire of the vehicle in the body coordinate system at the first moment (hereinafter referred to as position C1), the position of the front right tire of the vehicle in the body coordinate system at the first moment (hereinafter referred to as position C2), the position of the rear left tire of the vehicle in the body coordinate system at the first moment (hereinafter referred to as position C3) and the position of the rear right tire of the vehicle in the body coordinate system at the first moment (hereinafter referred to as position C4) as an example for exemplary explanation.

[0228] In one possible implementation, the vehicle can determine various positions that the front left tire of the vehicle may have traveled through within a preset time (hereinafter referred to as candidate vehicle position D1) based on position C1, driving data, and the first point cloud data, determine various positions that the front right tire of the vehicle may have traveled through within a preset time (hereinafter referred to as candidate vehicle position D2) based on position C2, driving data, and the first point cloud data, determine various positions that the rear left tire of the vehicle may have traveled through within a preset time (hereinafter referred to as candidate vehicle position D3) based on position C3, driving data, and the first point cloud data, and determine various positions that the rear left tire of the vehicle may have traveled through within a preset time (hereinafter referred to as candidate vehicle position D4) based on position C4, driving data, and The first point cloud data determines the various positions that the rear right tire of the vehicle may pass within a preset time (hereinafter referred to as candidate vehicle positions D4), and can predict the time when the front left tire of the vehicle reaches each candidate vehicle position D1 (hereinafter referred to as candidate time E1), predict the time when the front right tire of the vehicle reaches each candidate vehicle position D2 (hereinafter referred to as candidate time E2), predict the time when the rear left tire of the vehicle reaches each candidate vehicle position D3 (hereinafter referred to as candidate time E3), and predict the time when the rear right tire of the vehicle reaches each candidate vehicle position D4 (hereinafter referred to as candidate time E4).

[0229] Subsequently, the vehicle can determine, based on each candidate time E1 and the corresponding candidate vehicle position D1, each candidate time E2 and the corresponding candidate vehicle position D2, each candidate time E3 and the corresponding candidate vehicle position D3, and each candidate time E4 and the corresponding candidate vehicle position D4, the candidate vehicle position D1' that the front left tire of the vehicle may reach, the candidate vehicle position D2' that the front right tire of the vehicle may reach, the candidate vehicle position D3' that the rear left tire of the vehicle may reach, and the candidate vehicle position D4' that the rear right tire of the vehicle may reach at each same candidate time (hereinafter referred to as E5), and can determine the candidate posture corresponding to the vehicle at the candidate time E5 based on the candidate vehicle position D1', the candidate vehicle position D2', the candidate vehicle position D3', and the candidate vehicle position D4', thereby determining the candidate posture corresponding to the vehicle at each candidate time within the preset time.

[0230] Optionally, when it is determined that the candidate vehicle position D1', candidate vehicle position D2', candidate vehicle position D3' and candidate vehicle position D4' corresponding to the candidate time E5 are located in the same plane, the vehicle can determine the candidate posture corresponding to the vehicle at the candidate time E5 based on the plane. For example, the position of the origin of the body coordinate system at the candidate time E5 in the body coordinate system of the first moment can be determined based on the plane, and the posture (such as pitch angle, roll angle and navigation angle) corresponding to the vehicle at the candidate time E5 can be determined based on the position corresponding to the origin of the body coordinate system, so as to perform a rotation matrix conversion on the posture (such as pitch angle, roll angle and navigation angle) according to the conversion relationship between Euler angles and rotation matrices, and obtain the rotation matrix of the vehicle at the candidate time E5, so as to determine the candidate posture corresponding to the vehicle at the candidate time E5 based on the rotation matrix obtained by the conversion.

[0231] It can be understood that after the same plane is determined according to the candidate vehicle position D1', the candidate vehicle position D2', the candidate vehicle position D3' and the candidate vehicle position D4', the position of each point in the plane can be determined according to the candidate vehicle position D1', the candidate vehicle position D2', the candidate vehicle position D3' and the candidate vehicle position D4'. Therefore, when the origin of the vehicle body coordinate system at the first moment is the projection point of the preset position of the vehicle on the ground, the vehicle can determine the position corresponding to the origin of the vehicle body coordinate system at the candidate moment E5 according to the plane. When the origin of the vehicle body coordinate system at the first moment is not the projection point of the preset position of the vehicle on the ground, the vehicle can determine the projection point of the origin on the ground, and can determine the position corresponding to the projection point according to the plane, so that the position corresponding to the origin of the vehicle body coordinate system at the candidate moment E5 can be determined according to the conversion relationship between the projection point and the origin. Among them, the position corresponding to the origin of the vehicle body coordinate system refers to the position in the vehicle body coordinate system at the first moment.

[0232] It should be noted that the embodiment of the present application does not impose any restrictions on the specific method of determining the position corresponding to the origin of the vehicle coordinate system at the candidate time E5 based on the plane, and can be determined specifically according to the actual scenario.

[0233] That is to say, after obtaining the first reference position, driving data and first point cloud data corresponding to the vehicle, the vehicle can predict in advance the candidate time and candidate posture of the vehicle arriving at each candidate vehicle position, and associate and save the candidate time and candidate posture corresponding to each candidate vehicle position. After determining the target time according to the first time and the preset time delay, the vehicle can match the target time from the candidate time, thereby determining the second posture of the vehicle at the target time.

[0234] In another possible implementation, after obtaining the position C1 corresponding to the front left tire of the vehicle, the position C2 corresponding to the front right tire of the vehicle, the position C3 corresponding to the rear left tire of the vehicle, the position C4 corresponding to the rear right tire of the vehicle, the driving data and the first point cloud data, the vehicle can determine the position of the front left tire of the vehicle at the target time (hereinafter referred to as the second reference position F1), the position of the front right tire of the vehicle at the target time (hereinafter referred to as the second reference position F2), the position of the rear left tire of the vehicle at the target time (hereinafter referred to as the second reference position F3), and the position of the rear right tire of the vehicle at the target time (hereinafter referred to as the second reference position F4) according to the position C1, the position C2, the position C3, the position C4, the driving data (such as the driving speed and the driving direction, etc.), the preset delay and the first point cloud data. Subsequently, the vehicle can determine the second posture of the vehicle at the target time according to the second reference position F1, the second reference position F2, the second reference position F3 and the second reference position F4.

[0235] For example, when it is determined that the second reference position F1, the second reference position F2, the second reference position F3 and the second reference position F4 are located in the same plane, the vehicle can determine the position of the origin of the vehicle coordinate system at the target moment in the vehicle coordinate system at the first moment based on the plane, and can determine the pitch angle, roll angle and navigation angle corresponding to the vehicle at the target moment based on the position of the origin of the vehicle coordinate system at the target moment in the vehicle coordinate system at the first moment, and perform rotation matrix transformation on the pitch angle, roll angle and navigation angle according to the conversion relationship between the Euler angle and the rotation matrix to obtain the rotation matrix of the vehicle at the target moment, so that the second posture corresponding to the vehicle at the target moment can be obtained according to the rotation matrix obtained by the transformation.

[0236] It should be noted that the specific contents of determining the candidate vehicle position D1, candidate vehicle position D2, candidate vehicle position D3 and candidate vehicle position D4, the specific contents of determining the candidate time E1, candidate time E2, candidate time E3 and candidate time E4, and the specific contents of determining the second reference position F1, second reference position F2, second reference position F3 and second reference position F4 can be referred to the above-mentioned relevant descriptions and will not be repeated here.

[0237] Next, S204 is used to determine the target position at the target time according to the first posture, the second posture and the first target position. The process of positioning the second target position corresponding to the target object is described in detail.

[0238] It is understandable that the coordinates of the target object (e.g., static objects such as lane lines, fences, or curbs) in the world coordinate system remain unchanged. The vehicle can predict the coordinates of the target object in the vehicle coordinate system (i.e., the second target position) at the target time based on the characteristic that the coordinates of the target object in the world coordinate system remain unchanged. That is, the second target position of the target object in the vehicle coordinate system at the target time.

[0239] It should be noted that at a certain moment, the coordinates of the target object in the world coordinate system can be determined based on the coordinates of the target object in the vehicle coordinate system and the position and posture of the vehicle at that moment. For example, the coordinates of the target object in the world coordinate system = the coordinates of the target object in the vehicle coordinate system at a certain moment * the position and posture of the vehicle at that moment.

[0240] Therefore, when determining the first position of the vehicle at the first moment (e.g., T wa ), the first target position corresponding to the target object (e.g. P a ), and the relative position of the vehicle at the target time (i.e., the second position, such as T ab ), the vehicle can determine the target posture of the vehicle at the target time (e.g., T wb ), so that the second target position (e.g., P) corresponding to the target object at the target moment can be determined according to the first pose, the first target position, and the target pose. b ).

[0241] Optionally, the target position T of the vehicle at the target time wb =T wa *T ab . From the characteristic that the coordinates of the target object in the world coordinate system remain unchanged, we can know that: T wa *P a =T wb *P b , T wb =T wa *T ab , therefore, T wa *P a =T wa *T ab *P b , so that the second target position P can be determined b =(T wa *T ab ) -1 *T wa *P a .

[0242] That is to say, after determining the first position of the vehicle at the first moment and the first target position of the target object in the vehicle body coordinate system at the first moment, if you want to determine the second target position of the target object in the vehicle body coordinate system at the target moment, you only need to determine the relative position (i.e., the second position) between the target moment and the first moment. After determining the second position of the vehicle, the vehicle can determine the coordinates of the target object in the vehicle body coordinate system at the target moment (i.e., the second target position) according to the first position, the second position, and the first target position corresponding to the target object, so that the vehicle can draw a virtual scene according to the second target position corresponding to the target object at the target moment, and can display the virtual scene in the virtual image area at the target moment, so that at the target moment, the virtual scene displayed in the virtual image area fits the real scene environment where the vehicle is currently located, thereby improving the user experience.

[0243] It should be noted that the anti-shake method for the virtual scene image provided in the embodiment of the present application can be a function provided by an application (such as application A) installed in the vehicle, and the drawing and display of the virtual scene image according to the second target position corresponding to the target object can be a system function of the vehicle.

[0244] For example, during the driving process of the vehicle, the perception system in the vehicle can collect the first posture of the vehicle at the first moment, the driving data of the vehicle, and the first point cloud data corresponding to the environment in which the vehicle is located. Application A can obtain the first posture, driving data, and first point cloud data from the perception system, and can determine the first target position corresponding to the target object at the first moment based on the first point cloud data. In addition, application A can also obtain the first reference position stored in the vehicle. Subsequently, application A can determine the second posture of the vehicle at the target moment based on the first reference position, driving data, and first point cloud data, and determine the second target position corresponding to the target object at the target moment based on the first posture of the vehicle, the second posture, and the first target position corresponding to the target object. Finally, the computer system in the vehicle can draw a virtual scene according to the second target position of the target object at the target moment, and can display the virtual scene corresponding to the target moment in the virtual image area at the target moment.

[0245] It should be understood that the anti-shake method for the virtual scene image may be an original function of the application A, or may be an updated function after a certain update of the application A, and the embodiment of the present application does not impose any limitation on this.

[0246] In some embodiments, the virtual scene picture displayed by the vehicle in the virtual image area at the target time can be a virtual scene picture obtained by compensating the original virtual scene picture according to the second target position. In other words, the vehicle can draw a virtual scene picture (hereinafter referred to as the original virtual scene picture) according to the first target position corresponding to the target object. After determining the second target position corresponding to the target object, the vehicle can compensate the original virtual scene picture according to the first target position and the second target position corresponding to the target object to obtain the target virtual scene picture corresponding to the target time, and can display the target virtual scene picture in the virtual image area at the target time, so that at the target time, the virtual scene picture displayed in the virtual image area is consistent with the real scene environment where the vehicle is currently located, reducing the display deviation of the virtual scene picture and improving the user experience.

[0247] It should be noted that the embodiment of the present application does not impose any restrictions on the specific method of compensating the original virtual scene image according to the first target position and the second target position corresponding to the target object to obtain the target virtual scene image corresponding to the target moment, and can be determined as any existing compensation method according to the actual scene.

[0248] In other embodiments, the virtual scene picture displayed by the vehicle in the virtual image area at the target time may be a virtual scene picture directly drawn according to the second target position. That is, after determining the second target position corresponding to the target object, the vehicle may draw the virtual scene picture corresponding to the target time according to the second target position corresponding to the target object, and may display the virtual scene picture corresponding to the target time in the virtual image area at the target time, so that at the target time, the virtual scene picture displayed in the virtual image area is consistent with the real scene environment where the vehicle is currently located, thereby reducing the display deviation of the virtual scene picture and improving the user experience.

[0249] In a possible implementation, the posture of the vehicle is not only affected by the road, but also by the suspension or tire pressure of the vehicle. Therefore, in order to improve the accuracy of the second posture prediction, in an embodiment of the present application, a prediction model can be trained based on data such as the road surface, driving speed, suspension, tire pressure, seat elasticity, and driving habits, so as to predict the second posture of the vehicle at the target time through the trained prediction model. That is, the second posture can be predicted by a prediction model that considers more influencing factors, which not only improves the accuracy of the second posture prediction, but also allows the second posture to be predicted in a wider range of scenarios, making the prediction of the second posture more robust, improving the user experience, and having strong ease of use and practicality.

[0250] Optionally, the prediction model may be a network model constructed based on a machine learning algorithm such as a random forest, and the specific model may be determined according to the actual scenario.

[0251] For example, data such as each posture of the vehicle during driving and the road surface, driving speed, suspension, tire pressure, seat elasticity, and driving habits in a period of time before each posture occurs can be obtained, and the prediction model can be trained using these data and the corresponding posture, so that the prediction model learns the relationship between the posture and these data.

[0252] For example, the abnormal posture of the vehicle when it passes over bumpy locations such as speed bumps, manhole covers or potholes, as well as data such as the road surface, speed, suspension, tire pressure, seat elasticity and driving habits in a period of time before the abnormal posture occurs can be obtained, and these data and the corresponding abnormal posture can be used to train the prediction model, so that the prediction model learns the relationship between the abnormal posture and these data.

[0253] Therefore, during the driving process of the vehicle, the vehicle can obtain data such as road surface, speed, suspension, tire pressure, seat elasticity and driving habits collected by the perception system, and can predict the second posture of the vehicle at the target moment through the prediction model and the obtained data.

[0254] In another possible implementation, for abnormal road surfaces such as gravel roads or muddy roads, the virtual scene images displayed in the virtual image area may cause dizziness and other problems in the user, or for other scenes that are not suitable for displaying virtual scenes, the vehicle can directly display the virtual scene images in the form of 2D navigation prompts, voice or text, so as to provide users with better driving guidance and improve user experience by changing the interaction mode. Alternatively, the vehicle can first output prompt information, which is used to prompt the user to switch the interaction mode. The vehicle can display the virtual scene images according to the interaction mode switched by the user.

[0255] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0256] Corresponding to the anti-shake method for a virtual scene image described in the above embodiment, an embodiment of the present application further provides an anti-shake device for a virtual scene image, and each module of the device may correspond to each step of implementing the anti-shake method for a virtual scene image.

[0257] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0258] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0259] An embodiment of the present application also provides a vehicle-mounted device, which may include at least one memory, at least one processor, and a computer program stored in the at least one memory and executable on the at least one processor. When the processor executes the computer program, the vehicle-mounted device implements the steps in any of the above-mentioned method embodiments.

[0260] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer implements the steps in any of the above method embodiments.

[0261] An embodiment of the present application provides a computer program product. When the computer program product is executed on a vehicle-mounted device, the vehicle-mounted device implements the steps in any of the above-mentioned method embodiments.

[0262] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may at least include: any entity or device that can carry the computer program code to the device / vehicle-mounted equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable storage media cannot be electric carrier signals and telecommunication signals.

[0263] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0264] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0265] In the embodiments provided in the present application, it should be understood that the disclosed devices / vehicle-mounted equipment and methods can be implemented in other ways. For example, the device / vehicle-mounted equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0266] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0267] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for stabilizing a virtual scene image, characterized in that: include: Acquire a first pose of a vehicle, driving data of the vehicle, and first point cloud data corresponding to an environment where the vehicle is located at a first moment; Determine, according to the first point cloud data, a first target position corresponding to a target object at the first moment, wherein the target object is an object for reference by the vehicle when drawing a virtual scene, and the target object is an object in the environment where the vehicle is currently located; Determine, based on the driving data and the first point cloud data, a second posture of the vehicle at a target time, the target time being later than the first time; A second target position corresponding to the target object at the target time is determined according to the first posture, the second posture and the first target position, so that the vehicle draws a virtual scene corresponding to the target time according to the second target position corresponding to the target object.

2. The method according to claim 1, characterized in that The first point cloud data includes point cloud data corresponding to a static object in the environment where the vehicle is located, and the target object is a static object in the environment where the vehicle is currently located.

3. The method according to claim 2, characterized in that The target object includes one or more of a road surface, a lane line, a fence and a curb.

4. The method according to any one of claims 1 to 3, characterized in that The driving data includes driving speed and driving direction.

5. The method according to any one of claims 1 to 4, characterized in that The first target position and the second target position are positions of the target object in a body coordinate system of the vehicle.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Acquire a first reference position corresponding to the vehicle at the first moment, where the first reference position is a position of a first position on the vehicle in a target coordinate system at the first moment, and the target coordinate system is a body coordinate system of the vehicle at the first moment; The determining, based on the driving data and the first point cloud data, a second posture of the vehicle at a target time, comprises: A second posture of the vehicle at the target time is determined according to the first reference position, the driving data and the first point cloud data.

7. The method according to claim 6, characterized in that The determining, based on the first reference position, the driving data, and the first point cloud data, a second posture of the vehicle at the target time comprises: Determine, based on the first reference position, the driving data, and the first point cloud data, a candidate vehicle position that the vehicle will travel through within a preset time, wherein the preset time is greater than or equal to a time interval between the target time and the first time; Determine a candidate position and a first candidate time of the vehicle at the candidate vehicle position, and associate and save the candidate position and the first candidate time corresponding to the candidate vehicle position; A second candidate time that matches the target time is determined from the first candidate time, and a candidate posture corresponding to the second candidate time is determined as a second posture of the vehicle at the target time.

8. The method according to claim 7, characterized in that The candidate vehicle positions include a bumpy position.

9. The method according to claim 7 or 8, characterized in that: The determining a candidate position of the vehicle at the candidate vehicle position includes: Determining, based on the candidate vehicle position, the posture of the vehicle at the candidate vehicle position; According to the posture, a rotation matrix of the vehicle at the candidate vehicle position is determined, and a candidate posture of the vehicle at the candidate vehicle position is determined according to the rotation matrix.

10. The method according to any one of claims 7 to 9, characterized in that The second candidate time is the time closest to the target time among the first candidate time.

11. The method according to any one of claims 6 to 10, characterized in that The first position on the vehicle includes at least one of a position corresponding to a front left tire of the vehicle, a position corresponding to a front right tire of the vehicle, a position corresponding to a rear left tire of the vehicle, and a position corresponding to a rear right tire of the vehicle.

12. The method according to claim 11, characterized in that The position corresponding to the front left tire of the vehicle is the position where the front left tire of the vehicle contacts the ground, the position corresponding to the front right tire of the vehicle is the position where the front right tire of the vehicle contacts the ground, the position corresponding to the rear left tire of the vehicle is the position where the rear left tire of the vehicle contacts the ground, and the position corresponding to the rear right tire of the vehicle is the position where the rear right tire of the vehicle contacts the ground.

13. The method according to claim 6, characterized in that The determining, based on the first reference position, the driving data, and the first point cloud data, a second posture of the vehicle at the target time comprises: Determine, according to the first reference position, the driving data and the first point cloud data, a second reference position corresponding to the vehicle at the target time, wherein the second reference position is a position of the first position on the vehicle in the target coordinate system at the target time; A second posture of the vehicle at the target time is determined based on the second reference position.

14. The method according to claim 13, characterized in that The step of determining, based on the second reference position, a second posture of the vehicle at the target time, comprises: determining the posture of the vehicle at the target time according to the second reference position; According to the posture, a rotation matrix of the vehicle at the target time is determined, and according to the rotation matrix, a second posture of the vehicle at the target time is determined.

15. The method according to any one of claims 1 to 14, characterized in that The target time is determined according to the first time and a preset delay.

16. The method according to claim 15, characterized in that The preset delay includes a perception delay and / or a rendering delay, the perception delay is a delay in acquiring the first point cloud data, and the rendering delay is a delay in rendering the virtual scene image.

17. The method according to any one of claims 6 to 14, characterized in that The determining, based on the first reference position, the driving data, and the first point cloud data, a second posture of the vehicle at the target time comprises: Determine second point cloud data from the first point cloud data according to the first reference position and the driving data, the second point cloud data being all or part of the first point cloud data, and the second point cloud data being point cloud data corresponding to a position on a road surface that the vehicle may travel on within a preset time; A second posture of the vehicle at the target time is determined according to the first reference position, the driving data and the second point cloud data.

18. An in-vehicle device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the vehicle-mounted device implements the anti-shake method for a virtual scene image according to any one of claims 1 to 17.

19. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a computer, the computer is enabled to implement the anti-shake method for a virtual scene image according to any one of claims 1 to 17.

Images