Road element rendering method and device, vehicle and electronic equipment

Abstract

The invention discloses a road element rendering method and device, a vehicle and electronic equipment, and relates to the technical field of auxiliary driving. The method comprises the steps of obtaining a current image frame collected by a vehicle and real position data corresponding to a to-be-rendered real road element; performing gradient fitting calculation based on the current image frame to obtain a gradient fitting result; determining rendering position data by using the gradient fitting result and the real position data; and rendering the virtual road elements to an augmented reality HUD field of view range according to the rendering position data. The technical problem of poor driving experience caused by poor fitness of virtual road elements obtained by rendering in related technologies and real road elements is solved.

Description

Technical Field

[0001] This application relates to the field of assisted driving technology, and more particularly, to a method, device, vehicle, and electronic device for rendering road elements. Background Art

[0002] During the process of driving a vehicle, the driver usually has to check the map information from time to time. In this process, line-of-sight focusing is required, and frequent line-of-sight focusing is likely to cause visual fatigue. To solve this problem, in related technologies, the Head Up Display (HUD) technology is combined with the Augmented Reality (AR) display technology, and the driving information of the vehicle is projected onto the display screen in the form of an image. However, the HUD image generated by the AR display technology does not fit well with the actual scene, which is likely to cause visual interference, interfere with the driver's driving, and result in poor driving safety.

[0003] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of this application provide a method, device, vehicle, and electronic device for rendering road elements, so as to at least solve the technical problem that the virtual road elements rendered in related technologies do not fit well with the real road elements, resulting in a poor driving experience.

[0005] According to one aspect of the embodiments of this application, a method for rendering road elements for an in-vehicle augmented reality HUD is provided, including: obtaining a current image frame collected by the vehicle and real position data corresponding to real road elements to be rendered, where the display content of the current image frame includes real road elements, and the real position data is used to represent the real position of the real road elements in the target area; performing slope fitting calculation based on the current image frame to obtain a slope fitting result, where the slope fitting result is used to represent the road slope trend in the target area; using the slope fitting result and the real position data to determine rendering position data, where the rendering position data is used to represent the position to be rendered of the virtual road elements corresponding to the real road elements in the field of view of the in-vehicle augmented reality HUD; and rendering the virtual road elements into the field of view of the in-vehicle augmented reality HUD according to the rendering position data.

[0006] Optionally, performing slope fitting calculation based on the current image frame to obtain a slope fitting result includes: performing visual slope recognition on the current image frame to obtain slope recognition data; and performing fitting processing on the slope recognition data to obtain a slope fitting result.

[0007] Optionally, perform visual slope recognition on the current image frame to obtain slope recognition data, including: dividing the current image frame into an image grid respectively to determine the positions of multiple grid points; using the positions of the multiple grid points and a visual recognition model to perform slope recognition on the current image frame to obtain slope recognition data, where the slope recognition data includes multiple sets of slope scatter data, the first data dimension of the multiple sets of slope scatter data is the horizontal coordinate components of the positions of the multiple grid points, and the second data dimension of the multiple sets of slope scatter data is the slope values corresponding to the positions of the multiple grid points.

[0008] Optionally, perform fitting processing on the slope recognition data to obtain a slope fitting result, including: using a fitting algorithm to perform hash value fitting calculation on the slope recognition data to obtain the slope fitting result.

[0009] Optionally, the above road element rendering method further includes: obtaining the slope fitting result corresponding to a reference image frame, where the reference image frame is the previous image frame adjacent to the current image frame in time sequence, and the slope fitting result is obtained by performing slope fitting calculation on the reference image frame; performing inter-frame smoothing calculation on the slope fitting result and the slope fitting result to obtain a smoothing calculation result; using the smoothing calculation result to update the slope fitting result.

[0010] Optionally, use the slope fitting result and the real position data to determine the rendering position data, including: according to the real position data, determine the first horizontal coordinate component of the real road element in the vehicle body coordinate system; use the slope fitting result and the first horizontal coordinate component to calculate and obtain the rendering position data.

[0011] Optionally, according to the real position data, determining the first horizontal coordinate component of the real road element in the vehicle body coordinate system includes: converting the real position data from the world coordinate system to the vehicle body coordinate system to obtain the converted real position data; performing horizontal component extraction on the converted real position data to obtain the first horizontal coordinate component.

[0012] Optionally, the rendering position data includes the position coordinates of the position to be rendered in the augmented reality HUD field of view, and the position coordinates include a second horizontal coordinate component and a height coordinate component. Using the slope fitting result and the first horizontal coordinate component to calculate and obtain the rendering position data includes: determining the second horizontal coordinate component according to the first horizontal coordinate component and a preset rendering mapping relationship; using the second horizontal coordinate component and the slope fitting result to perform height solution calculation to obtain the height coordinate component.

[0013] According to another aspect of the embodiments of the present application, there is also provided a road element rendering device, including: an acquisition module, configured to acquire a current image frame collected by a vehicle and real position data corresponding to a real road element to be rendered, wherein the display content of the current image frame includes the real road element, and the real position data is used to represent the real position of the real road element in the target area; a calculation module, configured to perform slope fitting calculation based on the current image frame to obtain a slope fitting result, wherein the slope fitting result is used to represent the road slope trend in the target area; a determination module, configured to determine rendering position data by using the slope fitting result and the real position data, wherein the rendering position data is used to represent the position to be rendered of the virtual road element corresponding to the real road element in the augmented reality HUD field of view of the vehicle; and a rendering module, configured to render the virtual road element into the augmented reality HUD field of view according to the rendering position data.

[0014] According to another aspect of the embodiments of the present application, there is also provided a vehicle, including: a visual acquisition module, a calculation module, and an augmented reality rendering module, wherein the visual acquisition module is configured to acquire a current image frame corresponding to the vehicle and real position data corresponding to a real road element to be rendered, the display content of the current image frame includes the real road element, and the real position data is used to represent the real position of the real road element in the target area; the calculation module is configured to perform slope fitting calculation based on the current image frame to obtain a slope fitting result, and determine rendering position data by using the slope fitting result and the real position data, the slope fitting result is used to represent the road slope trend in the target area, and the rendering position data is used to represent the position to be rendered of the virtual road element corresponding to the real road element in the augmented reality HUD field of view of the vehicle; and the augmented reality rendering module is configured to render the virtual road element into the augmented reality HUD field of view according to the rendering position data.

[0015] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory storing an executable program; a processor, configured to run the program, wherein when the program runs, it executes the road element rendering method according to any one of the above.

[0016] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, the computer-readable storage medium includes a stored executable program, wherein when the executable program runs, it controls the device where the storage medium is located to execute the road element rendering method according to any one of the above.

[0017] According to another aspect of the embodiments of the present application, there is also provided a computer program product, including a computer program, which implements the road element rendering method according to any one of the above when executed by a processor.

[0018] In an embodiment of the present application, a current image frame collected by a vehicle and real position data corresponding to real road elements to be rendered are obtained, where the display content of the current image frame includes real road elements, and the real position data is used to represent the real positions of the real road elements in a target area; slope fitting calculation is performed based on the current image frame to obtain a slope fitting result, where the slope fitting result is used to represent the road slope trend in the target area; the rendering position data is determined by using the slope fitting result and the real position data, where the rendering position data is used to represent the to-be-rendered positions of virtual road elements corresponding to the real road elements in the augmented reality HUD field of view of the vehicle; and the virtual road elements are rendered into the augmented reality HUD field of view according to the rendering position data.

[0019] It is easy to note that in the embodiment of the present application, by collecting the current image frame and the real position data, performing slope fitting calculation on the current image frame to obtain a slope fitting result that can more accurately represent the road slope trend, and using the slope fitting result and the real position data, the correlation between the rendering position data and the real position data can be ensured. That is to say, rendering the virtual road elements corresponding to the real road elements into the position corresponding to the rendering position data in the augmented reality HUD field of view will make the real road elements and the virtual road elements more fitting in the field of view of the vehicle driver. Thus, the embodiment of the present application achieves the purpose of rendering the virtual road elements into a position with a stronger correlation with the real position data in the augmented reality HUD field of view to ensure the fitting degree of the real road elements and the virtual road elements in the field of view of the vehicle driver, thereby realizing the technical effect of improving the correlation between the rendering position data of the virtual road elements and the real position data and enhancing the fitting degree of the real road elements and the virtual road elements in the field of view of the vehicle driver, and further solving the technical problem that the fitting degree between the virtual road elements rendered in the related art and the real road elements is poor, resulting in a poor driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and the related descriptions of the embodiments are used to explain the present application, and do not constitute an improper limitation to the present application. In the drawings:

[0021] Figure 1 is a hardware structure block diagram of an optional computing terminal for implementing a road element rendering method according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of a road element rendering method according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an optional road element rendering method according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of a method for identifying a smooth road slope according to an embodiment of the present application;

[0025] Figure 5 is a structural block diagram of a road element rendering device according to an embodiment of the present application;

[0026] Figure 6 is a structural block diagram of a vehicle according to an embodiment of the present application. Detailed implementation manners

[0027] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only include some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] According to an embodiment of the present application, a method embodiment of a road element rendering method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a sequence different from that here.

[0030] First, the operating environment of the above method embodiment will be exemplarily described. Figure 1 is an optional hardware structural block diagram of a computing terminal for implementing a road element rendering method according to an embodiment of the present application, as Figure 1As shown, the computing terminal 10 (e.g., a computer terminal, a mobile intelligent terminal, a vehicle terminal, or a cloud computing virtual terminal, etc.) may include: one or more processors 102 (e.g., may include processors 102a, 102b, ……, 102n), a memory 104 for storing data, and a transmission device 106 for implementing communication functions. Among them, the processor 102 may include, but is not limited to, processing components such as a microcontroller unit (MCU) or a field programmable gate array (FPGA).

[0031] The above computing terminal 10 may further include: a display device 110, an input / output interface 108, a universal serial bus (USB) port (which can be one of the ports of the computer bus and is not shown in the figure), a network interface (not shown in the figure), a power supply (not shown in the figure), and a camera (not shown in the figure).

[0032] It should be noted that one or more of the processors 102 and / or other data processing circuits in the above computing terminal 10 may be embodied in whole or in part as software, hardware, firmware, or any combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be wholly or partially incorporated into any one of the other elements in the computing terminal 10 (or mobile device).

[0033] The memory 104 can be used to store software programs and modules of application software, such as the program instructions and data storage devices corresponding to the road element rendering method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above road element rendering method. The memory 104 may include a high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories may be connected to the vehicle terminal through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of a vehicle terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC) and a network interface. The network adapter can be connected to other network devices through a base station to communicate with the Internet. The transmission device 106 can perform data communication in a wired and / or wireless network connection manner. In one example, the transmission device 106 can be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0035] The input / output interface can be connected to the input / output devices corresponding to the computing terminal 10 to implement input / output functions. The input / output devices may include, but are not limited to: cursor control devices, keyboards, displays, etc. The above input / output devices can be built into the computing terminal 10 or external external devices of the computing terminal 10.

[0036] Those of ordinary skill in the art can understand that Figure 1 The structure of the computing terminal 10 shown is only schematic and does not strictly limit the structure of the above computing terminal 10. For example, the computing terminal 10 may further include more or fewer components than those shown in Figure 1 or the computing terminal 10 may have different types of components from those shown in Figure 1 shown.

[0037] Under the above operating environment, an embodiment of the present application provides a road element rendering method. The road element rendering method can be used to provide an assisted driving function for a preset application scenario. The above preset application scenarios may include the following scenarios in the vehicle field: commuting autonomous driving scenario, artificial intelligence (AI) driving scenario for household cars, automatic parking assist (APA) scenarios (such as memory parking for own parking spaces in the garage, intelligent parking for designated parking spaces in the parking lot, etc.), intelligent navigation assist (Navigation Guided Pilot, NGP) scenarios in urban areas or highway areas. In addition, the above preset application scenarios may further include, but are not limited to: assisted transportation scenarios for intelligent driving trucks or driverless trucks in the logistics transportation field, assisted farming scenarios for autonomous agricultural vehicles in the agricultural machinery field, assisted flight scenarios for drones, and assisted operation scenarios for intelligent robots (such as cleaning robots, service robots, delivery robots, etc.).

[0038] When the above preset application scenario is a scenario in other fields except the vehicle field, those skilled in the art should be able to understand that the vehicle in the above road element rendering method can be replaced with other objects (such as, agricultural machinery, drones, robots, etc.). On this basis, in the embodiments of the present application, taking the field of assisted driving technology as an example, the specific implementation manners of the above road element rendering method are described by way of example.

[0039] Under the above operating environment, the embodiments of the present application provide a road element rendering method as Figure 2 shown, Figure 2 which is a flowchart of a road element rendering method according to an embodiment of the present application. As Figure 2 shown, the method includes the following implementation steps:

[0040] Step S201, obtain the current image frame collected by the vehicle and the real position data corresponding to the real road elements to be rendered, where the display content of the current image frame includes the real road elements, and the real position data is used to represent the real position of the real road elements in the target area;

[0041] Step S202, perform slope fitting calculation based on the current image frame to obtain a slope fitting result, where the slope fitting result is used to represent the road slope trend in the target area;

[0042] Step S203, use the slope fitting result and the real position data to determine the rendering position data, where the rendering position data is used to represent the position to be rendered of the virtual road elements corresponding to the real road elements in the augmented reality HUD field of view of the vehicle;

[0043] Step S204, render the virtual road elements to the augmented reality HUD field of view according to the rendering position data.

[0044] The above vehicle can be a vehicle with assisted driving functions. The vehicle can also be a vehicle with autonomous driving functions. The above current image frame can be obtained through a vision sensor (such as, an in-vehicle camera).

[0045] The above target area can be the road section area where the vehicle is currently located. The target area can be used to represent the area within the driver's visual range. The specific range of the target area can be set based on the human eye's viewing distance. For example, in the corresponding scenario of vehicle driving, considering the need for safe driving, the specific range of the target area can be set to a meters from the current position of the vehicle to the front of the vehicle.

[0046] The above-mentioned real road elements may include, but are not limited to: road markings (such as lane dividers, stop lines, zebra crossings, guiding arrows, etc.), traffic signs, traffic lights, pedestrian guardrails, trees, street lights, pedestrians, surrounding vehicles. The above-mentioned real position data may at least include horizontal dimension position data. The real position data may also include, but are not limited to: height and depth dimension position data. The real position data can be obtained by using on-vehicle radar sensors (such as vision radar, lidar). The real position data can also be obtained by using high-precision maps or standard-precision maps. The real position data can be represented by using data in multiple coordinate axis systems, and the coordinate axis systems may include, but are not limited to: the vehicle body coordinate system in the vehicle dimension, the world coordinate system (such as rectangular coordinate system, cylindrical coordinate system, polar coordinate system, etc.). In particular, lidar can be used to collect data on the real road elements to be rendered in the target area, so as to obtain the real position data in the vehicle body coordinate system corresponding to the real road elements.

[0047] The above-mentioned vehicle body coordinate system can be a coordinate system with the vehicle itself as a reference. In this vehicle body coordinate system, the X-axis can be defined as pointing forward of the vehicle, the Y-axis as pointing to the left of the vehicle, and the Z-axis as perpendicular to the ground and pointing upward. The data in the vehicle body coordinate system can directly associate the road elements around the vehicle with the vehicle, and can more accurately determine the position information of the road elements relative to the vehicle.

[0048] In an exemplary application scenario, Figure 3 is a schematic diagram of an optional road element rendering method according to an embodiment of the present application. As Figure 3 shown, the road element rendering method may include a slope sensing and recognition module. In this slope sensing and recognition module, a vision sensor is used to perform real-time acquisition on the target area at a preset shooting frequency to obtain the current image frame. The above-mentioned shooting frequency can be adjusted according to actual needs. The shooting frequency can also be automatically adjusted according to the pitch angle of the vehicle.

[0049] Still in the above application scenario, the above-mentioned real position data is represented by using data in the vehicle body coordinate system. By using lidar to collect the real positions of the real road elements to be rendered in the target area, the real position data in the vehicle body coordinate system can be directly obtained. Compared with the related art for obtaining the real position data in the world dimension coordinate system, the solution of the present application for obtaining the real position data in the vehicle body coordinate system does not require additional coordinate transformation processing, can improve the efficiency of obtaining the real position data, and the real position data has a higher correlation with the vehicle driving state and can more accurately reflect the environment where the vehicle is currently located.

[0050] The above slope fitting result can be used to assist in determining the rendering position data. The slope fitting result can include multiple fitting parameters, which can be used to characterize the parameters of the fitting function corresponding to the ramp in the target area. The slope fitting result can be obtained by using the slope fitting method. The slope fitting method can include, but is not limited to: slope identification method, slope calculation method, fitting calculation method, smoothing processing method, noise processing method. By using the slope fitting method, the ramp included in the acquired current image frame is identified and fitting calculation is performed to obtain a fitting function that can characterize the slope trend of the ramp included in the target area. Based on the multiple fitting parameters of the fitting function, the slope fitting result is obtained. Compared with the solution in the related art that directly applies the acquired slope information to determine the display position of the virtual road element, the slope fitting result obtained by performing slope fitting calculation in this application can more accurately characterize the road slope trend in the target area, providing data support for subsequent determination of the rendering position data with stronger relevance to the real position data.

[0051] The above rendering position data can include multiple virtual position data of the virtual road element in the augmented reality HUD field of view of the vehicle. The virtual position data can include, but is not limited to: virtual horizontal dimension position data, virtual height and low dimension position data, virtual depth dimension position data. By combining and using the real position data in the vehicle body coordinate system and the slope fitting result for solution calculation, the rendering position data is obtained. It is easy to understand that since the above real position data has a higher relevance to the vehicle, and the above slope fitting result can more accurately characterize the road slope trend in the target area, by combining and using the real position data and the slope fitting result, the obtained rendering position data also has a stronger relevance to the real position data.

[0052] The above virtual road element can be generated based on the corresponding real road element in the target area. The virtual road element can include, but is not limited to: virtual light carpet, virtual arrow indication, virtual road marking, virtual pedestrian. The above virtual light carpet corresponds to the lane in the real road element. The above augmented reality HUD field of view can be determined according to the augmented reality display area, and the augmented reality display area can include, but is not limited to: HUD display, AR device display area. The augmented reality display area is configured in the vehicle. For example, the above augmented reality display area can be configured inside the front windshield of the vehicle. According to the rendering position data with stronger relevance to the real position data, the virtual road element can be rendered to a position with stronger relevance to the real position data in the augmented reality HUD field of view, enhancing the fitting degree of the real road element and the virtual road element in the driver's field of view of the vehicle, enabling the driver to obtain key road information without shifting the line of sight, thereby improving the safety and convenience of vehicle driving.

[0053] Through the technical solutions provided in the above steps S201 to S204, in the embodiments of the present application, by collecting the current image frame and the real position data, performing slope fitting calculation on the current image frame, a slope fitting result that can more accurately represent the slope trend of the road is obtained. By using the slope fitting result and the real position data, the correlation between the rendered position data and the real position data can be ensured. That is to say, based on the rendered position data, the virtual road elements corresponding to the real road elements are rendered to the position corresponding to the rendered position data in the augmented reality HUD field of view, which will make the real road elements and the virtual road elements fit better in the field of view of the vehicle driver. Thus, the embodiments of the present application achieve the purpose of rendering the virtual road elements to a position in the augmented reality HUD field of view with a stronger correlation with the real position data to ensure the fitting degree of the real road elements and the virtual road elements in the field of view of the vehicle driver. Therefore, the technical effect of improving the correlation between the rendered position data of the virtual road elements and the real position data and enhancing the fitting degree of the real road elements and the virtual road elements in the field of view of the vehicle driver is achieved, and further the technical problem that the fitting degree between the virtual road elements rendered in the related art and the real road elements is poor, resulting in a poor driving experience, is solved.

[0054] As an alternative implementation, in the above step S202, when performing slope fitting calculation based on the current image frame to obtain the slope fitting result, the following execution steps may further be included:

[0055] Step S221: Perform visual slope recognition on the current image frame to obtain slope recognition data;

[0056] Step S222: Perform fitting processing on the slope recognition data to obtain the slope fitting result.

[0057] The above slope recognition data can be used to represent the slope of the target area. The slope recognition data may include a slope value and ramp length data. The slope recognition data can be obtained by using a deep learning model. By using the deep learning model, visual recognition is performed on the slope-related information included in the current image frame to obtain the slope recognition data.

[0058] It should be noted that in order to improve the accuracy of the slope recognition data, the following processing may further be performed during the above visual slope recognition process: image preprocessing, feature extraction processing, and slope value prediction.

[0059] In the above fitting process, a fitting algorithm can be used to perform fitting processing on multiple sets of slope scatter data included in the slope recognition data, and a slope fitting result is obtained. Compared with the solution in the related art that directly applies the obtained slope information to determine the display position of virtual road elements, in the embodiment of the present application, the slope fitting result used to characterize the road slope trend in the target area is more accurate, ensuring that stronger rendering position data can be obtained subsequently in relation to the real position data.

[0060] Through the technical solutions provided in the above steps S221 to S222, for the current image frame in the present application, visual slope recognition is first performed, and then fitting processing is performed. The obtained slope fitting result can more accurately characterize the road slope trend in the target area, thereby providing data support for determining stronger rendering position data in relation to the real position data subsequently.

[0061] As an optional implementation manner, in the above step S221, when performing visual slope recognition on the current image frame to obtain slope recognition data, the following execution steps may further be included:

[0062] Step S2211: Perform image grid division on the current image frame respectively to determine the positions of multiple grid points;

[0063] Step S2212: Use the positions of multiple grid points and a visual recognition model to perform slope recognition on the current image frame to obtain slope recognition data, where the slope recognition data includes multiple sets of slope scatter data, the first data dimension of the multiple sets of slope scatter data is the horizontal coordinate component of the positions of multiple grid points, and the second data dimension of the multiple sets of slope scatter data is the slope value corresponding to the positions of multiple grid points.

[0064] The above image grid division can be performed according to the following process: Divide according to a preset row-column ratio, and the display content of the current image frame is multiple divided areas (i.e., grids), obtaining multiple grid points, thereby determining the grid point position corresponding to each grid point among the multiple grid points and determining the positions of multiple grid points. The above row-column ratio can be adjusted according to actual needs. The above grid points can be used to represent the center points of the above divided areas. The above grid point positions can at least include horizontal coordinate components. The grid point positions can also include but are not limited to: depth coordinate components, height coordinate components.

[0065] It should be noted that the above image grid division can also be implemented using an image segmentation algorithm. The above image segmentation algorithm can include but is not limited to: a pixel-based segmentation algorithm, a convolutional neural network-based segmentation algorithm.

[0066] The above visual recognition model may include, but is not limited to: a recurrent neural network visual recognition model, a convolutional neural network visual recognition model, and a long short-term memory network visual recognition model. The above slope scatter data may include data of multiple dimensions. The above first data dimension may be used to characterize the horizontal position distribution of grid point positions in the target area. The above second data dimension may be used to characterize the road inclination degree corresponding to the grid point positions. The above slope value may be obtained by using the visual recognition model. The slope value may also be obtained by using a high-precision radar sensor.

[0067] In an exemplary application scenario, for the display content of the current image frame, it is divided according to a preset row-column ratio to obtain multiple grid point positions corresponding to the current image frame. Using the visual recognition model, slope recognition is performed on the horizontal coordinate component (denoted as S) of any grid point position among the multiple grid point positions corresponding to the current image frame to obtain the slope value (denoted as P) corresponding to the grid point position. Furthermore, the horizontal coordinate component S of the grid point position is used as the first data dimension, and the slope value P corresponding to the grid point position is used as the second data dimension to form the slope scatter data corresponding to the grid point position. The slope scatter data can be represented by the binary data group (S, P). Similarly, multiple groups of slope scatter data corresponding to the multiple grid point positions corresponding to the current image frame can be obtained, and the multiple groups of slope scatter data are used as the slope recognition data corresponding to the current image frame.

[0068] Through the technical solutions provided in the above steps S2211 to S2212, in the embodiments of the present application, the current image frame is respectively subjected to image grid division to obtain multiple grid point positions, and then the slope value corresponding to each grid point position is recognized by using the visual recognition model, so that the slope scatter data corresponding to each grid point position in the current image frame can be obtained, and the slope data of the target area is collected more comprehensively, improving the accuracy of the slope recognition data.

[0069] As an optional implementation manner, in the above step S222, when performing fitting processing on the slope recognition data to obtain a slope fitting result, the following execution steps may further be included:

[0070] Step S2221, using a fitting algorithm to perform hash value fitting calculation on the slope recognition data to obtain a slope fitting result.

[0071] The above fitting algorithms may include, but are not limited to: the least squares method, spline interpolation method, machine learning methods based on probability theory, locally weighted regression, and Kalman filtering. Using the fitting algorithm, hash value fitting calculations are performed on a partial set of slope scatter data within a preset distance range in the slope identification data to obtain a function that matches this partial set of slope scatter data, and multiple fitting parameters corresponding to this function are obtained, thereby obtaining a slope fitting result, which can more accurately characterize the road slope trend. The above preset distance range can be set in combination with the requirements of the fitting algorithm.

[0072] In an exemplary application scenario, still as shown in FIG. 3, the road element rendering method may further include a slope smoothing module. In this slope smoothing module, the least squares method is selected as the fitting algorithm. A partial set of slope scatter data belonging to a preset distance range (e.g., 100 m) is selected from the multiple sets of slope scatter data included in the slope identification data, and the least squares method is used to perform hash value fitting calculations on this partial set of slope scatter data, and a quadratic function that matches this partial set of slope scatter data is fitted, and multiple fitting parameters corresponding to this quadratic function are obtained. Based on these multiple fitting parameters and this quadratic function, a slope fitting result is obtained.

[0073] In another alternative application scenario, the above slope identification data may be multiple grid point positions, and these grid point positions can be obtained by respectively performing image grid division on the current image frame using an image segmentation algorithm. Using the fitting algorithm, hash value fitting calculations are performed on the multiple grid point positions included in the slope identification data, and a fitting slope function that matches these multiple grid point positions is fitted, and multiple fitting slope parameters corresponding to this fitting slope function are obtained. Based on these multiple fitting slope parameters and this fitting slope function, a slope fitting result is obtained.

[0074] Through the technical solution provided in the above step S2221, in the embodiments of the present application, hash value fitting calculations are performed on multiple sets of slope scatter data in the slope identification data using the fitting algorithm, and the obtained slope fitting result can more accurately characterize the road slope trend, improving the accuracy of the data for determining the display position of the virtual road element.

[0075] As an alternative implementation manner, the above road element rendering method may further include the following execution steps:

[0076] Step S205, obtaining a slope fitting result corresponding to a reference image frame, where the reference image frame is the previous image frame adjacent in time sequence to the current image frame, and the slope fitting result is obtained by performing slope fitting calculations on the reference image frame;

[0077] Step S206, performing inter-frame smoothing calculations on the slope fitting result and the slope fitting result to obtain a smoothing calculation result;

[0078] Step S207, update the slope fitting result using the smoothing calculation result.

[0079] The above reference image frame can be obtained based on video stream data. After acquiring multiple image frames collected in real time by a vision sensor, the multiple image frames are combined according to the timing information corresponding to each image frame in the multiple image frames to obtain the above video stream data, and the video stream data is stored in the system storage area. The system obtains the video stream data from the system storage area, automatically selects the previous image frame adjacent in timing to the current image frame from the video stream data, and uses the previous image frame adjacent in timing as the reference image frame. The above slope fitting result can include multiple fitting reference parameters. For the method of obtaining the slope fitting result corresponding to the reference image frame, reference can be made to the introduction of obtaining the slope fitting result by performing slope fitting calculation on the current image frame in the previous text, which will not be elaborated here.

[0080] The above smoothing calculation result can be used to characterize the continuous change trend of the road slope. The smoothing calculation result can include multiple smoothing fitting parameters. The smoothing calculation result can be obtained using a smoothing algorithm. The above smoothing algorithm can include but is not limited to: exponential smoothing algorithm, weighted average algorithm, recursive filtering algorithm, smoothing algorithm based on polynomial fitting.

[0081] In an exemplary application scenario, still as Figure 3 shown, the exponential smoothing algorithm is selected as the smoothing algorithm. In the slope smoothing module, using the exponential smoothing algorithm, an inter-frame smoothing calculation is performed on any one of the multiple fitting parameters included in the slope fitting result corresponding to the current image frame and the fitting reference parameter corresponding to this fitting parameter included in the slope fitting result corresponding to the reference image frame to obtain the smoothing fitting parameter corresponding to this fitting parameter. Similarly, the smoothing fitting parameters corresponding to each of the multiple fitting parameters can be obtained. Based on the multiple smoothing fitting parameters corresponding to the multiple fitting parameters respectively, the smoothing calculation result is obtained. Furthermore, the slope fitting result is updated using the smoothing calculation result, and the smoothness of the slope fitting result is better, which can avoid the poor stability of virtual road element rendering caused by inter-frame jitter.

[0082] When the vehicle is driving, there may be a situation of up and down jitter. At this time, when the pitch angle of the vehicle changes violently, the slope value of the target area in front of the vehicle recognized by the vehicle will also change significantly. However, in fact, the slope value of the target area is fixed. That is to say, when the vehicle jitters up and down, there is a large error between the slope value collected by the vehicle and the real slope value. If the virtual road elements (such as virtual light carpets) on the augmented reality HUD field of view are directly rendered based on the slope value collected by the vehicle, the expected effect cannot be achieved. Specifically, on the one hand, a large slope value error may lead to a poor fit between the virtual road elements rendered in the driver's field of view and the real road elements. On the other hand, the violent change of the slope value caused by the jitter will cause problems such as deformation (for example, the display length of the virtual light carpet becomes shorter) and jitter of the virtual road elements rendered in real time. Even, it may cause the virtual road elements that could originally be mapped to the augmented reality field of view to exceed the augmented reality HUD field of view (for example, the jitter of the vehicle causes the virtual light carpet that should originally be displayed within the field of view to be mapped outside the field of view, visually manifested as the virtual light carpet being invisible to the driver), and cannot provide a stable driving guidance for the driver.

[0083] In an exemplary application scenario, Figure 4 is a schematic diagram of a method for stably identifying the road slope according to an embodiment of the present application. As Figure 4 shown, the process of the vehicle jittering up and down during driving can include three states. Among them, vehicle state 1 means that the vehicle is in a horizontal state (for example, driving on a flat road), vehicle state 2 means that the vehicle is in an upward jitter state (understood as the vehicle maintaining a certain upward viewing angle), and vehicle state 3 means that the vehicle is in a downward jitter state (understood as the vehicle maintaining a certain downward viewing angle).

[0084] Since both slope recognition and slope fitting calculation are based on the visual data collected by the vehicle (such as the current image frame), when the vehicle is in a horizontal state, the slope fitting result obtained through slope recognition and slope fitting calculation will be highly consistent with the real road slope. As Figure 4 shown, at time t1 (corresponding to image frame n), the vehicle is in vehicle state 1, and the slope fitting result obtained according to the foregoing method steps can be characterized as curve Z1. For example, the slope fitting result includes curve parameters a1, b1, and c1 corresponding to curve Z1. As described above, compared with the related art solution that directly recognizes the slope value by relying on sensors, the curve Z1 obtained in the embodiment of the present application can more accurately characterize the trend of the road slope.

[0085] Furthermore, when the vehicle is in vehicle state 2, the upward viewing angle of the vehicle may cause the road slope value corresponding to the slope fitting result to be greater than the real road slope value. As Figure 4As shown, at time t2 (corresponding to image frame n+1), the vehicle is in vehicle state 2, and the slope fitting result obtained according to the foregoing method steps can be characterized as curve Z2. For example, the slope fitting result includes curve parameters a2, b2, and c2 corresponding to curve Z2. At this time, if the height to be rendered of the virtual road element is calculated according to the road slope trend corresponding to curve Z2, it will not only cause the virtual road element not to fit the real road element, but also show severe jitter of the virtual road element in the dynamic effect, and may even cause the virtual road element to jitter upward out of the field of view of the augmented reality HUD. To this end, curve Z2 and curve Z1 are smoothed to obtain curve Z2'. Specifically, the curve parameters a1, b1, and c1 in the slope fitting result obtained for image frame n are subjected to one-to-one smoothing filtering calculation with the curve parameters a2, b2, and c2 in the slope fitting result obtained for image frame n+1. For example, the preset smoothing coefficient 0.01 is multiplied by (a2 - a1) to obtain curve parameter a2'; the preset smoothing coefficient 0.01 is multiplied by (b2 - b1) to obtain curve parameter b2'; the preset smoothing coefficient 0.01 is multiplied by (c2 - c1) to obtain curve parameter c2'. Curve Z2' can be determined according to curve parameters a2', b2', and c2'. The smoothing calculation result obtained by performing inter-frame smoothing calculation on image frame n and image frame n+1 can include the above curve parameters a2', b2', and c2'. As Figure 4 shown, compared with curve Z2, curve Z2' fits curve Z1 better. Calculating the position to be rendered of the virtual road element based on curve Z2' can ensure anti-shake and smooth display of the virtual road element.

[0086] Furthermore, when the vehicle is in vehicle state 3, the downward viewing angle of the vehicle may cause the road slope value corresponding to the slope fitting result to be less than the real road slope value. As Figure 4As shown, at time t3 (corresponding to image frame n + 2), the vehicle is in vehicle state 3, and the slope fitting result obtained according to the foregoing method steps can be characterized as curve Z3. For example, the slope fitting result includes curve parameters a3, b3, and c3 corresponding to curve Z3. At this time, if the height to be rendered of the virtual road element is calculated according to the road slope trend corresponding to curve Z3, it will not only cause the virtual road element not to fit the real road element, but also show severe jitter of the virtual road element in the dynamic effect, and may even cause the virtual road element to jitter downward out of the field of view of the augmented reality HUD. To this end, curve Z3 and curve Z2' are smoothed to obtain curve Z3'. Specifically, the curve parameters a2', b2', and c2' in the smoothing calculation result obtained for image frame n + 1 are subjected to one-to-one smoothing filtering calculation with the curve parameters a3, b3, and c3 in the slope fitting result obtained for image frame n + 2. For example, multiplying the preset smoothing coefficient 0.01 by (a3 - a2') gives curve parameter a3'; multiplying the preset smoothing coefficient 0.01 by (b3 - b2') gives curve parameter b3'; multiplying the preset smoothing coefficient 0.01 by (c3 - c2') gives curve parameter c3'. Curve Z3' can be determined according to curve parameters a3', b3', and c3'. The smoothing calculation result obtained by performing inter-frame smoothing calculation on image frame n + 1 and image frame n + 2 may include the following curve parameters a3', b3', and c3'. As Figure 4 shown, compared with curve Z3, curve Z3' fits curve Z2' better. Calculating the position to be rendered of the virtual road element based on curve Z3' can ensure anti-shake and smooth display of the virtual road element.

[0087] It should be noted that the above Figure 4 shown road slope stable recognition method is only an example. In other application scenarios, an inter-frame smoothing calculation method different from the above process may also be adopted, and the embodiments of the present application do not limit this.

[0088] It is easy to notice that in the embodiments of the present application, the slope fitting result is obtained by performing visual recognition and fitting calculation on the current image frame collected by the vehicle, and the rendering position of the virtual road element (such as, virtual light carpet) in the field of view of the augmented reality HUD is calculated by using the slope fitting result, which can ensure that the virtual road element and the corresponding real road element (such as, the lane corresponding to the light carpet) are fitted in the driver's field of view. In addition, in the process of slope fitting in the embodiments of the present application, the inter-frame smoothing process is performed on the slope fitting results corresponding to different image frames, ensuring that the slope fitting result is smooth and stable in the time dimension. Therefore, in the case of the jitter of the self-vehicle during driving, it can be ensured that the virtual road elements rendered in the field of view of the augmented reality HUD are stable and anti-shake. In summary, the embodiments of the present application can achieve the visual anti-shake of the virtual road elements on the basis of ensuring the fitting degree between the virtual road elements and the real road elements, and ensure that the virtual road elements can be stably displayed in the field of view of the augmented reality HUD during vehicle driving, providing a stable driving guidance for the driver.

[0089] Through the technical solutions provided in the above steps S205 to S207, in the embodiments of the present application, the reference image frame is automatically obtained, and the slope fitting result corresponding to the reference image frame and the slope fitting result are used for inter-frame smoothing calculation, and the obtained smoothing calculation result is used to update the slope fitting result, which can enhance the smoothness of the slope fitting result and achieve the effect of inter-frame anti-shake, so as to ensure that the virtual road elements can be stably rendered into the field of view of the augmented reality HUD subsequently, enhancing the stability of the virtual road element rendering.

[0090] As an alternative implementation manner, in the above step S203, when using the slope fitting result and the real position data to determine the rendering position data, the following execution steps may further be included:

[0091] Step S231: Determine the first horizontal coordinate component of the real road element in the vehicle body coordinate system according to the real position data;

[0092] Step S232: Calculate the rendering position data by using the slope fitting result and the first horizontal coordinate component.

[0093] The above first horizontal coordinate component can be determined according to the real position data. According to the real position data corresponding to the real road element, determine the horizontal relative position data of the real road element relative to the vehicle's horizontal direction, and use this horizontal relative position data as the first horizontal coordinate component of the real road element in the vehicle body coordinate system.

[0094] Convert the first horizontal coordinate component corresponding to the real road element into the coordinate component of the virtual road element corresponding to the real road element within the augmented reality HUD field of view range. Combine this coordinate component with the slope fitting result that can more accurately represent the road slope trend to calculate the rendering position data of the virtual road element corresponding to the real road element. The relevance between this rendering position data and the real position data is stronger, and using this rendering position data can enhance the fitting degree between the real road element and the virtual road element in the vehicle driver's field of view.

[0095] Through the technical solutions provided in the above steps S231 to S232, in the embodiments of the present application, according to the real position data corresponding to the real road element, determine the first horizontal coordinate component of the real road element in the vehicle body coordinate system. This first horizontal coordinate component has a higher relevance to the vehicle and can more accurately reflect the relative position relationship between the real road element and the vehicle. Further, by combining the use of this first horizontal coordinate component and the slope fitting result, the rendering position data with a stronger relevance to the real position data can be calculated, thereby improving the fitting degree between the real road element and the virtual road element in the vehicle driver's field of view.

[0096] As an alternative implementation manner, in the above step S231, determining the first horizontal coordinate component of the real road element in the vehicle body coordinate system according to the real position data includes:

[0097] Step S2311, convert the real position data from the world coordinate system to the vehicle body coordinate system to obtain the converted real position data;

[0098] Step S2312, extract the horizontal component from the converted real position data to obtain the first horizontal coordinate component.

[0099] The above world coordinate system can be a fixed three-dimensional coordinate system. This world coordinate system can be used to describe the absolute position of an object in the real world. By using mathematical coordinate transformation methods (such as coordinate rotation, coordinate translation, etc.), the data in the world coordinate system can be converted into the data in the vehicle body coordinate system.

[0100] In an exemplary application scenario, the above-mentioned real position data can be obtained by using a high-precision map or a standard-precision map. The real position data includes multiple coordinate components (such as horizontal coordinate components, altitude coordinate components, etc.). The real position data corresponding to the real road elements in the world coordinate system is obtained by using a high-precision map, and the real position data in the world coordinate system is converted to the vehicle body coordinate system by using a mathematical coordinate conversion method to obtain the converted real position data. Further, horizontal component extraction is performed on the converted real position data, and a first horizontal coordinate component is extracted from the multiple converted coordinate components included in the converted real position data. That is to say, the implementation manner of obtaining position data based on a standard-precision or high-precision map is also included in the technical solution of this application.

[0101] It should be noted that the above-mentioned real position data can also be collected by using a lidar. The lidar is used to collect data on the real road elements in the target area to obtain the real position data of the real road elements in the world coordinate system corresponding to the sensor, and the real position data in the world coordinate system corresponding to the sensor is converted to the real position data in the vehicle body coordinate system by using a mathematical coordinate conversion method.

[0102] Through the technical solutions provided in the above steps S2311 to S2312, in the embodiments of this application, the real position data in the world coordinate system is converted into the real position data in the vehicle body coordinate system. The converted real position data has a higher relevance to the vehicle. Further, a first horizontal coordinate component is extracted from the converted real position data. The first horizontal coordinate component has a higher relevance to the vehicle and can more accurately reflect the relative position relationship in the horizontal direction between the real road element corresponding to the real position data and the vehicle.

[0103] As an optional implementation manner, the rendering position data includes the position coordinates of the position to be rendered within the field of view range of the augmented reality HUD. The position coordinates include a second horizontal coordinate component and an altitude coordinate component. In the above step S232, when calculating the rendering position data by using the slope fitting result and the first horizontal coordinate component, the following execution steps may further be included:

[0104] Step S2321: Determine the second horizontal coordinate component according to the first horizontal coordinate component and a preset rendering mapping relationship;

[0105] Step S2322: Perform altitude solution calculation by using the second horizontal coordinate component and the slope fitting result to obtain the altitude coordinate component.

[0106] The above second horizontal coordinate component can be used to characterize the horizontal position of the virtual road element in the field of view of the augmented reality HUD. The above height coordinate component can be used to characterize the height position of the virtual road element in the field of view of the augmented reality HUD. The above slope fitting result may further include a vehicle coordinate fitting function in the vehicle body coordinate system. The vehicle coordinate fitting function can be used to characterize the mapping relationship between the second horizontal coordinate component and the height coordinate component. By substituting the second horizontal coordinate component corresponding to the virtual road element into the vehicle coordinate fitting function, the height coordinate component corresponding to the virtual road element can be obtained.

[0107] The above rendering mapping relationship can be a conversion rule between the first horizontal coordinate component and the second horizontal coordinate component. The rendering mapping relationship can be stored in the vehicle-mounted memory. The rendering mapping relationship can be obtained through a mapping experiment on the first horizontal coordinate component. The rendering mapping relationship can also be obtained by using a mapping network model, which can include but is not limited to: fully connected neural network model, convolutional neural network model, recurrent neural network model, long short-term memory network model, generative adversarial network model, deep residual network model.

[0108] In an exemplary application scenario, still as Figure 3 shown, the vehicle coordinate fitting function included in the slope fitting result is F(x), where x can be used to represent the second horizontal coordinate component. For the first horizontal coordinate component corresponding to any real road element, the preset rendering mapping relationship in the vehicle-mounted memory is called, and using this rendering mapping relationship, the first horizontal coordinate component is subjected to rendering mapping processing to obtain the second horizontal coordinate component corresponding to the real road element (denoted as x1). Further, taking the second horizontal coordinate component x1 as the independent variable of the vehicle coordinate fitting function in the slope fitting result and substituting it into the vehicle coordinate fitting function F(x) in the slope fitting result for height solving calculation, the height coordinate component F(x1) corresponding to the real road element is obtained.

[0109] Still in the above application scenario, the second horizontal coordinate component corresponding to the real road element and the height coordinate component corresponding to the real road element are used as the rendering position data corresponding to the real road element and transmitted to the rendering module. The rendering module renders the virtual road element corresponding to the real road element to the position corresponding to the rendering position data in the field of view of the augmented reality HUD according to the rendering position data.

[0110] Through the technical solutions provided in the above steps S2321 to S2322, in the embodiments of the present application, using the preset rendering mapping relationship, the first horizontal coordinate component in the vehicle coordinate system corresponding to the real road element is converted to the second horizontal coordinate component at the rendering position of the virtual road element corresponding to the real road element in the augmented reality area of the vehicle. Further, using the second horizontal coordinate component and the slope fitting result for height solution calculation, the height coordinate component at the rendering position of the virtual road element corresponding to the real road element in the augmented reality area of the vehicle can be obtained more accurately. This height coordinate component has a stronger correlation with the real position data, which can enhance the fitting degree of the real road element and the virtual road element in the driver's field of view of the vehicle, and avoid the situation of low driving safety caused by visual interference due to the low fitting degree of the real road element and the virtual road element in the driver's field of view of the vehicle.

[0111] According to the embodiments of the present application, a road element rendering device is also provided. Please refer to Figure 5 . The device includes:

[0112] An acquisition module 501, configured to acquire the current image frame collected by the vehicle and the real position data corresponding to the real road element to be rendered, where the display content of the current image frame includes the real road element, and the real position data is used to represent the real position of the real road element in the target area;

[0113] A calculation module 502, configured to perform slope fitting calculation based on the current image frame to obtain a slope fitting result, where the slope fitting result is used to represent the road slope trend in the target area;

[0114] A determination module 503, configured to use the slope fitting result and the real position data to determine the rendering position data, where the rendering position data is used to represent the rendering position of the virtual road element corresponding to the real road element in the augmented reality HUD field of view of the vehicle;

[0115] A rendering module 504, configured to render the virtual road element to the augmented reality HUD field of view according to the rendering position data.

[0116] It should be noted here that the above acquisition module 501, calculation module 502, determination module 503, and rendering module 504 correspond to steps S201 to S204 in the embodiments. The functions of the four modules and the corresponding steps are the same in terms of the implemented examples and application scenarios, but are not limited to the content disclosed in the foregoing embodiments.

[0117] Optionally, the above calculation module 502 is further configured to: perform visual slope recognition on the current image frame to obtain slope recognition data; and perform fitting processing on the slope recognition data to obtain a slope fitting result.

[0118] Optionally, the above calculation module 502 is further configured to: perform image grid division on the current image frame respectively to determine the positions of multiple grid points; use the positions of the multiple grid points and the visual recognition model to perform slope recognition on the current image frame to obtain slope recognition data, where the slope recognition data includes multiple groups of slope scatter data, the first data dimension of the multiple groups of slope scatter data is the horizontal coordinate component of the positions of the multiple grid points, and the second data dimension of the multiple groups of slope scatter data is the slope value corresponding to the positions of the multiple grid points.

[0119] Optionally, the above calculation module 502 is further configured to: perform hash value fitting calculation on the slope recognition data by using a fitting algorithm to obtain a slope fitting result.

[0120] Optionally, the above road element rendering device further includes an update module 504 (not shown in the figure), configured to obtain the slope fitting result corresponding to a reference image frame, where the reference image frame is the previous image frame adjacent to the current image frame in time sequence, and the slope fitting result is obtained by performing slope fitting calculation on the reference image frame; perform inter-frame smoothing calculation on the slope fitting result and the slope fitting result to obtain a smoothing calculation result; use the smoothing calculation result to update the slope fitting result.

[0121] Optionally, the above determination module 503 is further configured to: determine the first horizontal coordinate component of the real road element in the vehicle body coordinate system according to the real position data; use the slope fitting result and the first horizontal coordinate component to calculate and obtain the rendering position data.

[0122] Optionally, the above determination module 503 is further configured to: convert the real position data from the world coordinate system to the vehicle body coordinate system to obtain the converted real position data; perform horizontal component extraction on the converted real position data to obtain the first horizontal coordinate component.

[0123] Optionally, the rendering position data includes the position coordinates of the position to be rendered in the augmented reality HUD field of view, and the position coordinates include a second horizontal coordinate component and a height coordinate component. The above rendering module 504 is further configured to: determine the second horizontal coordinate component according to the first horizontal coordinate component and a preset rendering mapping relationship; use the second horizontal coordinate component and the slope fitting result to perform height solution calculation to obtain the height coordinate component.

[0124] According to another aspect of the embodiments of the present application, a vehicle is further provided. Please refer to Figure 6, the vehicle includes: a visual acquisition module 601, a calculation module 602, and an augmented reality rendering module 603. Among them, the visual acquisition module 601 is used to acquire the current image frame corresponding to the vehicle and the real position data corresponding to the real road elements to be rendered. The display content of the current image frame includes the real road elements, and the real position data is used to represent the real position of the real road elements in the target area; the calculation module 602 is used to perform slope fitting calculation based on the current image frame to obtain a slope fitting result, and use the slope fitting result and the real position data to determine the rendering position data. The slope fitting result is used to represent the road slope trend in the target road area, and the rendering position data is used to represent the position to be rendered of the virtual road elements corresponding to the real road elements in the augmented reality HUD field of view of the vehicle; the augmented reality rendering module 603 is used to render the virtual road elements into the augmented reality HUD field of view according to the rendering position data.

[0125] It should be noted that the various modules mentioned in the above system embodiments can be implemented by software, hardware, or a combination of software and hardware. For example, when implementing the above modules in hardware, each module can be set in the same processor, or each module can be set in different processors in any combination. For another example, the above modules can be hardware components or software components stored in a memory (for example, memory 104) and processed by one or more processors (for example, processors 102a, 102b,..., 102n). The above modules can also run as part of a system in a computing terminal 10 as shown in Figure 1 Figure 10.

[0126] According to another aspect of the embodiments of the present application, an electronic device is further provided, including a memory storing an executable program; a processor for running the program, where the program, when running, executes the road element rendering method of any one of the above.

[0127] According to another aspect of the embodiments of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored executable program, where, when the executable program runs, it controls the device where the storage medium is located to execute the road element rendering method of any one of the above.

[0128] Optionally, the above computer storage medium may include, but is not limited to: a hard disk drive (HDD), a solid state drive (SSD), a USB flash drive, an optical disc, a memory card, a cloud storage medium, and a network-attached storage (NAS), etc.

[0129] Optionally, the above computer-readable storage medium may be configured to store a computer program for performing the following steps: obtaining a current image frame and real position data collected by a vehicle, where the display content of the current image frame includes real road elements to be rendered within a target area, and the real position data is used to represent the real positions of the real road elements within the target area; performing slope fitting calculation based on the current image frame to obtain a slope fitting result, where the slope fitting result is used to represent the road slope trend within the target area; using the slope fitting result and the real position data to determine rendering position data, where the rendering position data is used to represent the positions to be rendered of virtual road elements corresponding to the real road elements within the augmented reality HUD field of view of the vehicle; and rendering the virtual road elements into the augmented reality HUD field of view according to the rendering position data.

[0130] According to an embodiment of the present application, there is also provided a computer program product. The computer program product includes a computer program that can implement the above road element rendering method when executed by a processor.

[0131] Optionally, the above computer program product may provide a road element rendering service based on the above road element rendering method.

[0132] Optionally, in this embodiment, the above computer program product may be a set of instructions and codes pre-written according to the above road element rendering method. The computer program product can run on various different computer platforms, including personal computers, servers, mobile devices, etc.

[0133] Optionally, in this embodiment, the instructions and codes corresponding to the computer program product are used to implement the following method steps: obtaining a current image frame collected by a vehicle and real position data corresponding to real road elements to be rendered, where the display content of the current image frame includes real road elements to be rendered within a target area, and the real position data is used to represent the real positions of the real road elements within the target area; performing slope fitting calculation based on the current image frame to obtain a slope fitting result, where the slope fitting result is used to represent the road slope trend within the target area; using the slope fitting result and the real position data to determine rendering position data, where the rendering position data is used to represent the positions to be rendered of virtual road elements corresponding to the real road elements within the augmented reality HUD field of view of the vehicle; and rendering the virtual road elements into the augmented reality HUD field of view according to the rendering position data.

[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0135] It should be noted that for the above method embodiments, for the sake of simple description, the technical solutions in the method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of actions in the described action combinations. Because according to this application, some of the above steps can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification of this application are preferred embodiments, and the actions and modules involved are not necessarily essential for implementing the technical solutions of this application.

[0136] In the above-mentioned multiple embodiments of this application, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0137] In several embodiments provided by this application, it should be understood that the disclosed technical content can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of multiple modules can be a logical function division, and there can be any other possible division methods when actually implemented in an application scenario; furthermore, multiple modules (or units or components in a module) can be combined with each other and integrated into another system. For example, some features in the method embodiments described above can be ignored or skipped.

[0138] It should be noted that in the above-mentioned embodiments, the modules, components, or units described as separate components can be physically separated or physically integrated. The components shown as modules or units can be physical modules or physical units, or virtual modules or virtual units. That is to say, multiple modules or multiple units can be in the same location, or distributed to multiple locations or multiple spaces. In an application scenario, according to the actual needs of the scenario, some or all of the multiple modules or multiple units can be selected to implement the technical solutions of the embodiments of this application, so as to achieve the corresponding technical purposes.

[0139] In particular, for an integrated functional module or functional unit, if it is implemented in the form of a software functional unit and sold or used as an independent product, the module or unit can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application.

[0140] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A road element rendering method for vehicle-mounted augmented reality HUD, characterized in that: include: Acquire a current image frame collected by the vehicle and real position data corresponding to a real road element to be rendered, wherein the display content of the current image frame includes the real road element, and the real position data is used to represent the real position of the real road element in the target area; Performing a slope fitting calculation based on the current image frame to obtain a slope fitting result, wherein the slope fitting result is used to characterize the road slope trend in the target area; Determine rendering position data using the slope fitting result and the real position data, wherein the rendering position data is used to represent a position to be rendered of a virtual road element corresponding to the real road element in the augmented reality HUD field of view of the vehicle; According to the rendering position data, the virtual road element is rendered into the field of view of the augmented reality HUD.

2. The road element rendering method according to claim 1, characterized in that: Performing a slope fitting calculation based on the current image frame to obtain the slope fitting result includes: Performing visual slope recognition on the current image frame to obtain slope recognition data; The slope recognition data is fitted to obtain the slope fitting result.

3. The road element rendering method according to claim 2, characterized in that: Performing visual slope recognition on the current image frame to obtain the slope recognition data includes: Performing image grid division on the current image frame respectively to determine positions of multiple grid points; The slope of the current image frame is identified by utilizing the multiple grid point positions and the visual recognition model to obtain the slope identification data, wherein the slope identification data includes multiple groups of slope scatter point data, a first data dimension of the multiple groups of slope scatter point data is the horizontal coordinate component of the multiple grid point positions, and a second data dimension of the multiple groups of slope scatter point data is the slope values ​​corresponding to the multiple grid point positions.

4. The road element rendering method according to claim 2, characterized in that: Performing fitting processing on the slope recognition data to obtain the slope fitting result includes: A fitting algorithm is used to perform hash value fitting calculation on the slope recognition data to obtain the slope fitting result.

5. The road element rendering method according to claim 4, characterized in that: The road element rendering method further includes: Obtaining a slope fitting result corresponding to a reference image frame, wherein the reference image frame is a previous image frame that is adjacent to the current image frame in time sequence, and the slope fitting result is obtained by performing a slope fitting calculation on the reference image frame; Performing inter-frame smoothing calculation on the slope fitting result and the slope fitting result to obtain a smoothing calculation result; The slope fitting result is updated using the smoothing calculation result.

6. The road element rendering method according to any one of claims 1 to 5, characterized in that: Determining the rendering position data using the slope fitting result and the actual position data includes: Determine, according to the actual position data, a first horizontal coordinate component of the actual road element in the body coordinate system of the vehicle; The rendering position data is calculated using the slope fitting result and the first horizontal coordinate component.

7. The road element rendering method according to claim 6, characterized in that: Determining the first horizontal coordinate component of the actual road element in the vehicle body coordinate system according to the actual position data includes: Converting the actual position data from the world coordinate system to the vehicle body coordinate system to obtain converted actual position data; The horizontal component of the converted actual position data is extracted to obtain the first horizontal coordinate component.

8. The road element rendering method according to claim 6, characterized in that: The rendering position data includes the position coordinates of the position to be rendered in the field of view of the augmented reality HUD, the position coordinates include a second horizontal coordinate component and a height coordinate component, and the rendering position data is calculated using the slope fitting result and the first horizontal coordinate component, including: Determining the second horizontal coordinate component according to the first horizontal coordinate component and a preset rendering mapping relationship; The height coordinate component is obtained by performing a height solution calculation using the second horizontal coordinate component and the slope fitting result.

9. A road element rendering device, characterized in that: include: An acquisition module, used for acquiring a current image frame collected by the vehicle and real position data corresponding to a real road element to be rendered, wherein the display content of the current image frame includes the real road element, and the real position data is used for representing the real position of the real road element in the target area; A calculation module, configured to perform a slope fitting calculation based on the current image frame to obtain a slope fitting result, wherein the slope fitting result is used to characterize a road slope trend in the target area; A determination module, configured to determine rendering position data using the slope fitting result and the real position data, wherein the rendering position data is used to represent a position to be rendered of a virtual road element corresponding to the real road element in a field of view of an augmented reality HUD of the vehicle; A rendering module is used to render the virtual road element into the field of view of the augmented reality HUD according to the rendering position data.

10. A vehicle, characterized in that: include: Visual acquisition module, computing module and augmented reality rendering module, among which, The visual acquisition module is used to acquire a current image frame corresponding to the vehicle and real position data corresponding to the real road element to be rendered, the display content of the current image frame includes the real road element, and the real position data is used to represent the real position of the real road element in the target area; The calculation module is used to perform a slope fitting calculation based on the current image frame to obtain a slope fitting result, and to determine rendering position data using the slope fitting result and the real position data, wherein the slope fitting result is used to characterize the road slope trend in the target area, and the rendering position data is used to characterize the position to be rendered of the virtual road element corresponding to the real road element in the augmented reality HUD field of view of the vehicle; The augmented reality rendering module is used to render the virtual road element to the augmented reality HUD field of view according to the rendering position data.

11. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the road element rendering method described in any one of claims 1 to 7 is executed when the program is run.

Images