A rendering method, device, electronic device, and storage medium for a vehicle model

Abstract

The present application discloses a rendering method, device, electronic device and storage medium for a vehicle model, relating to the technical field of image processing, and is used to make the constructed vehicle model more conform to the real environment. The method includes: acquiring images captured by a plurality of cameras with different perspectives configured for the vehicle; mapping the images captured by the plurality of cameras to an ambient light model to generate an ambient light panoramic image; wherein, the ambient light model includes a hemisphere model and a round-bottom surround view model, the hemisphere model is used to construct the sky scene of the ambient light, and the round-bottom surround view model is used to construct the ground scene and the scene around the vehicle body of the ambient light; the ambient light panoramic image is used to represent the ambient light information around the vehicle; sampling the ambient light panoramic image based on the vehicle light reflection vector to generate an ambient light texture map; wherein, the ambient light texture map is used to represent the ambient light information reflected by the surface of the vehicle; rendering the vehicle model based on the ambient light texture map.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to a method, apparatus, electronic device, and storage medium for rendering a vehicle model. Background Art

[0002] Panoramic surround view is a technology for displaying panoramic images around a moving device. It uses multiple cameras installed around the moving device to collect image data of the surrounding scene of the moving device, and then maps the image data collected by the multiple cameras to a fixed surround view model to obtain a panoramic image.

[0003] Vehicle driving assistance systems are often equipped with in-vehicle surround view systems that utilize panoramic surround view technology to facilitate drivers in observing the surrounding environment when driving or parking, for example. With the popularization of in-vehicle surround view, users' demand for the realism of surround view is also increasing. Among them, if the reflection of the vehicle's surrounding environment is missing in the rendering of the vehicle model in the surround view, the realism of the vehicle model will be greatly reduced.

[0004] The traditional method for rendering a vehicle model is to project a static scene image onto a pre-constructed surround view model to obtain the rendered vehicle model. The pre-constructed surround view model used in this method is usually a bowl-shaped model. When the viewing angle of the vehicle changes, the ambient light rendering of some positions of the vehicle cannot be achieved at some viewing angles, reducing the realism of the global ambient light rendering of the vehicle. Moreover, this method uses a static scene image, which is not the current actual scene and lacks the realism of real-time ambient light rendering of the vehicle model. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for rendering a vehicle model, which is used to make the constructed vehicle model more conform to the real environment.

[0006] To achieve the above technical objectives, this application adopts the following technical solutions:

[0007] In a first aspect, an embodiment of this application provides a method for rendering a vehicle model, and the method includes:

[0008] Obtain images captured by multiple cameras with different perspectives configured on the vehicle;

[0009] Map the images captured by the multiple cameras to an ambient light model to generate an ambient light panoramic image; wherein, the ambient light model includes a hemisphere model and a round-bottom surround view model. The hemisphere model is used to construct the sky scene of the ambient light, and the round-bottom surround view model is used to construct the ground scene and the scene around the vehicle body of the ambient light; the ambient light panoramic image is used to represent the ambient light information around the vehicle;

[0010] Sample an ambient light panoramic image based on the vehicle's light reflection vector to generate an ambient light map; wherein, the vehicle's light reflection vector is used to represent the reflection angle of the vehicle's surface to light; the ambient light map is used to represent the ambient light information reflected by the vehicle's surface.

[0011] Render the vehicle model based on the ambient light map.

[0012] The technical solution provided by this application at least brings the following beneficial effects: The ambient light model includes a hemisphere model and a round-bottom panoramic view model, covering the sky scene, ground scene, and the scenes around the vehicle body of the ambient light, covering all angles of the vehicle, realizing global ambient light rendering of the vehicle. Even if the panoramic view angle is switched, there will be no problem that the ambient light rendering cannot be achieved at some positions of the vehicle. On this basis, mapping the images captured by the cameras from each perspective of the vehicle to the ambient light model to generate an ambient light panoramic image can ensure that the rendering of the vehicle model is carried out in real time based on the environment around the vehicle, with a stronger sense of reality.

[0013] In a possible implementation manner, mapping the images captured by multiple cameras to the ambient light model to generate an ambient light panoramic image includes: mapping the images captured by multiple cameras to the round-bottom panoramic view model to generate a first ambient light image; wherein, the first ambient light image is used to represent the ambient light information in the ground scene and the scenes around the vehicle body of the vehicle; mapping the image captured by the camera of the target perspective to the hemisphere model to generate a second ambient light image; wherein, the target perspective includes the perspectives that are the same as and / or opposite to the forward direction of the vehicle; the second ambient light image is used to represent the ambient light information in the sky scene of the vehicle; splicing the first ambient light image and the second ambient light image to generate an ambient light panoramic image.

[0014] In a possible implementation manner, mapping the image captured by the camera of the target perspective to the hemisphere model to generate a second ambient light image includes: based on the driving speed of the vehicle, intercepting the local image to be mapped to the hemisphere model from the current image and historical images captured by the camera of the target perspective; mapping the local image to the hemisphere model to generate a second ambient light image.

[0015] In a possible implementation manner, mapping the images captured by multiple cameras to the ambient light model to generate an ambient light panoramic image includes: determining the mapping relationship between the images captured by multiple cameras and the ambient light model based on the internal and external parameters of each of the multiple cameras; based on the mapping relationship between the images captured by multiple cameras and the ambient light model, mapping the images captured by multiple cameras to the ambient light model to generate an ambient light panoramic image.

[0016] In a possible implementation, the round-bottom panoramic view model is determined as follows: obtaining the number of cameras configured for the vehicle, the parameters of the cameras, and the dimensions of the vehicle; based on the number of cameras configured for the vehicle, the parameters of the cameras, and the dimensions of the vehicle, adjusting the model parameters of the round-bottom panoramic view model to generate the round-bottom panoramic view model.

[0017] In a second aspect, the present application provides a rendering device for a vehicle model, including:

[0018] An acquisition module, configured to acquire images captured by a plurality of cameras with different perspectives configured for the vehicle;

[0019] A processing module, configured to map the images captured by the plurality of cameras to an ambient light model to generate an ambient light panoramic image; wherein, the ambient light model includes a hemisphere model and a round-bottom panoramic view model, the hemisphere model is used to construct the sky scene of the ambient light, and the round-bottom panoramic view model is used to construct the ground scene and the scene around the vehicle body of the ambient light; the ambient light panoramic image is used to represent the ambient light information around the vehicle;

[0020] The processing module is further configured to sample the ambient light panoramic image based on the vehicle light reflection vector to generate an ambient light texture map; wherein, the vehicle light reflection vector is used to represent the reflection angle of the vehicle's surface to the light; the ambient light texture map is used to represent the ambient light information reflected by the vehicle's surface;

[0021] The processing module is further configured to render the vehicle model based on the ambient light texture map.

[0022] In a possible implementation, the processing module is specifically configured to: map the images captured by the plurality of cameras to the round-bottom panoramic view model to generate a first ambient light image; wherein, the first ambient light image is used to represent the ambient light information in the ground scene and the scene around the vehicle body of the vehicle; map the images captured by the cameras at the target perspective to the hemisphere model to generate a second ambient light image; wherein, the target perspective includes perspectives that are the same as and / or opposite to the forward direction of the vehicle; the second ambient light image is used to represent the ambient light information in the sky scene of the vehicle; splice the first ambient light image and the second ambient light image to generate an ambient light panoramic image.

[0023] In a possible implementation, the processing module is specifically configured to: based on the driving speed of the vehicle, intercept local images to be mapped to the hemisphere model from the current image and the historical images captured by the cameras at the target perspective; map the local images to the hemisphere model to generate a second ambient light image.

[0024] In a possible implementation, the processing module is specifically configured to: determine the mapping relationship between the images captured by multiple cameras and the ambient light model based on the internal and external parameters of each of the multiple cameras; map the images captured by the multiple cameras to the ambient light model based on the mapping relationship between the images captured by the multiple cameras and the ambient light model, and generate an ambient light panoramic image.

[0025] In a possible implementation, the obtaining module is further configured to obtain the number of cameras configured in the vehicle, the parameters of the cameras, and the size of the vehicle; the processing module is further configured to adjust the model parameters of the round-bottom surround view model based on the number of cameras configured in the vehicle, the parameters of the cameras, and the size of the vehicle, and generate a round-bottom surround view model.

[0026] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement any one of the vehicle model rendering methods provided in the first aspect above.

[0027] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer instruction is stored, and when the computer instruction is executed by a processor, it implements any one of the vehicle model rendering methods provided in the first aspect above.

[0028] In a fifth aspect, the present application provides a computer program product, including a computer instruction, and when the computer instruction is executed by a processor, it implements any one of the vehicle model rendering methods provided in the first aspect above.

[0029] For the specific descriptions of the second to fifth aspects and their various implementation manners in the present application, reference may be made to the detailed descriptions in the first aspect and its various implementation manners; and, for the beneficial effects of the second to fifth aspects and their various implementation manners, reference may be made to the beneficial effect analysis in the first aspect and its various implementation manners, which will not be elaborated herein.

[0030] These aspects or other aspects of the present application will be more clearly understood in the following description. Description of the Drawings

[0031] Figure 1 It is a schematic structural diagram of a vehicle model rendering system applicable to a vehicle model rendering method provided by an embodiment of the present application;

[0032] Figure 2 It is a schematic hardware composition diagram of a computing device provided by an embodiment of the present application;

[0033] Figure 3 It is a flowchart of a vehicle model rendering method provided by an embodiment of the present application;

[0034] Figure 4Schematic diagram of the application scenario of a rendering method for a vehicle model provided by an embodiment of the present application Figure 1 ;

[0035] Figure 5 Schematic diagram of the application scenario of a rendering method for a vehicle model provided by an embodiment of the present application Figure 2 ;

[0036] Figure 6 Schematic diagram of the application scenario of a rendering method for a vehicle model provided by an embodiment of the present application Figure 3 ;

[0037] Figure 7 Schematic diagram of the application scenario of a rendering method for a vehicle model provided by an embodiment of the present application Figure 4 ;

[0038] Figure 8 Schematic logic diagram of the rendering process of a vehicle model provided by an embodiment of the present application;

[0039] Figure 9 Schematic diagram of the application scenario of a rendering method for a vehicle model provided by an embodiment of the present application Figure 5 ;

[0040] Figure 10 Logic diagram of a rendering method for a vehicle model provided by an embodiment of the present application;

[0041] Figure 11 Schematic diagram of the rendering effect of a vehicle model provided by an embodiment of the present application;

[0042] Figure 12 Schematic diagram of the structure of a rendering device for a vehicle model provided by an embodiment of the present application. Detailed implementation manners

[0043] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0044] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner. The terms "first" and "second" are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present application, unless otherwise stated, the meaning of "a plurality" is two or more.

[0045] Panoramic surround view is a technology for displaying panoramic images around a moving device. It uses multiple cameras installed around the moving device to collect image data of the surrounding scene of the moving device, and then maps the image data collected by the multiple cameras to a fixed surround view model to obtain a panoramic image.

[0046] Vehicle driving assistance systems are often equipped with in-vehicle surround view systems that utilize panoramic surround view technology to facilitate drivers in observing the surrounding environment when driving or parking, for example. With the popularization of in-vehicle surround view, users' demand for the realism of the surround view is also increasing. Among them, if the reflection of the surrounding environment of the vehicle is missing in the rendering of the vehicle model, the realism of the vehicle model will be greatly reduced.

[0047] The traditional method for rendering a vehicle model is to project a static scene image onto a pre-constructed surround view model to obtain a rendered vehicle model. The pre-constructed surround view model used in this method is usually a bowl-shaped model. When the surround view angle of the vehicle changes, the ambient light rendering of certain positions of the vehicle cannot be achieved at some surround view angles, reducing the realism of the global ambient light rendering of the vehicle. Moreover, this method uses a static scene image, which is not the current actual scene and lacks the realism of real-time ambient light rendering of the vehicle model.

[0048] In response to this, the embodiments of the present application provide a method for rendering a vehicle model. Images captured by cameras at various perspectives of the vehicle are obtained and mapped onto an ambient light model. The ambient light model includes a hemispherical model and a round-bottom surround view model, which can include the ambient light, sky scene, ground scene, and the scene around the vehicle body, covering all angles of the vehicle and achieving global ambient light rendering of the vehicle. Even when the surround view angle is changed, the problem that the ambient light rendering of certain positions of the vehicle cannot be achieved will not occur. On this basis, the images captured by cameras at various perspectives of the vehicle are mapped onto the ambient light model to generate an ambient light panoramic image, which can ensure that the rendering of the vehicle model is carried out in real time based on the surrounding environment of the vehicle and has a stronger sense of realism.

[0049] Please refer toFigure 1 , which shows the rendering system of the vehicle model applicable to the vehicle model rendering method provided by this application. As Figure 1 shown, the vehicle model rendering system 1 includes cameras 10 with multiple different shooting perspectives and an electronic device 20.

[0050] Among them, a communication connection is established between the camera 10 and the electronic device 20 in a direct or indirect manner. It should be understood that the connection method can be a wireless connection, such as a Bluetooth connection, a wireless fidelity (Wi-Fi) connection, etc.; or, the connection method can also be a wired connection, such as an optical fiber connection, etc., which is not limited herein. For example, the camera 10 and the electronic device 20 are connected through a wireless local area network.

[0051] In some embodiments, multiple cameras 10 are respectively installed at different positions of the vehicle, so that each camera 10 can capture images from different perspectives. For example, if a camera 10 is installed at the front, rear, left, and right of the vehicle respectively, the front camera can capture the scene in front of the vehicle, the left camera can capture the scene on the left side of the vehicle, the right camera can capture the scene on the right side of the vehicle, and the rear camera can capture the scene behind the vehicle.

[0052] In practical applications, to ensure the surround view effect of the vehicle, the shooting ranges of all surround view cameras deployed on the vehicle should cover 360° around the vehicle.

[0053] Specifically, the camera 10 can be a fish-eye camera, a monocular camera, a binocular camera, a trinocular camera, a wide-angle camera, etc., and the specific implementation of the camera 10 is not limited in this application. In practical applications, the camera 10 is usually a fish-eye camera.

[0054] In some embodiments, the electronic device 20 is used to receive the images captured by each camera 10, map the images captured by each camera 10 to the ambient light model, and generate an ambient light panoramic image. Among them, the ambient light model includes a hemisphere model and a round-bottom surround view model. The hemisphere model is used to construct the sky scene of the ambient light, and the round-bottom surround view model is used to construct the ground scene and the scene around the vehicle body of the ambient light; the ambient light panoramic image is used to represent the ambient light information around the vehicle. Furthermore, the electronic device 20 samples the ambient light panoramic image based on the vehicle light reflection vector to generate an ambient light map, and finally the electronic device 20 renders the vehicle model based on the ambient light map.

[0055] During the rendering process of the vehicle model, the electronic device 20 adjusts the model parameters of a preset ambient light model based on the number of cameras 10 configured in the vehicle, the parameters of the cameras, and the size of the vehicle, and generates an ambient light model. It should be understood that during the driving process of the vehicle, abnormal situations such as camera failures may occur. If the ambient light model is fixed, the abnormal situations will have an adverse effect on the ambient light rendering effect of the vehicle model. Therefore, to ensure the ambient light rendering effect of the vehicle model, the electronic device 20 generates the ambient light model in real time during the ambient light rendering process of the vehicle model. Alternatively, if it can be ensured that the number of cameras 10 configured in the vehicle, the parameters of the cameras, and the size of the vehicle do not change, the ambient light model can also be pre-constructed, and the subsequent image mapping process can be directly based on the pre-constructed ambient light model during the ambient light rendering process of the vehicle model.

[0056] In some embodiments, the electronic device 20 may include a display for presenting the rendered vehicle model to the user. For example, the display may be a liquid crystal display, an organic light-emitting diode (OLED) display, etc. The specific type, size, and resolution of the display are not limited.

[0057] Optionally, the electronic device 20 is connected to a display device and sends the fused multi-view stitching image to the display device, so that the display device presents the multi-view stitching image to the user. The display device may be a liquid crystal display, an organic light-emitting diode (OLED) display. The specific type, size, and resolution of the display device are not specifically limited. Those skilled in the art can understand that the display device can be changed in terms of performance and configuration as needed.

[0058] In some embodiments, the electronic device 20 may be a device or component having the above-mentioned image processing capabilities, such as a computer, a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The electronic device 20 may also be other devices with processing functions, such as circuits, components, or software modules. The embodiments of the present application do not impose any restrictions on this.

[0059] In some embodiments, the electronic device 20 may be a single server, a server cluster, or the electronic device 20 may also be a terminal device, such as a personal computer (PC), a laptop, a mobile device, a tablet computer, a notebook computer, a vehicle-mounted terminal, etc. The embodiments of the present application do not limit the specific form of the electronic device 20.

[0060] In some embodiments, multiple cameras 10 and the electronic device 20 are both installed on a moving target. For example, the camera 10 is a vehicle-mounted camera with different shooting perspectives on a vehicle, and the electronic device 20 is a vehicle-mounted terminal on the vehicle.

[0061] Optionally, the electronic device 20 may not be installed in the vehicle. The camera 10 transmits the captured images to the electronic device 20, and the electronic device 20 performs ambient light rendering of the vehicle model based on each image. Then, the electronic device transmits the rendered vehicle model to a display device for displaying the vehicle model.

[0062] In some embodiments, the camera 10 and the electronic device 20 may be Figure 1 independent devices as shown, or the electronic device 20 may also be integrated with a camera 10. The embodiments of the present application do not make specific limitations on this.

[0063] The hardware structure of the above-mentioned electronic device 20 includes Figure 2 the components included in the computing device shown. Taking the Figure 2 computing device shown as an example, the hardware structure of the electronic device 20 will be introduced below.

[0064] As Figure 2 shown, the computing device may include a processor 301, a memory 302, a communication interface 303, and a bus 304. The processor 301, the memory 302, and the communication interface 303 may be connected through the bus 304.

[0065] The processor 301 is the control center of the computing device and may be a single processor or a collective term for multiple processing elements. For example, the processor 301 may be a general-purpose central processing unit (CPU), or other general-purpose processors. Among them, the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0066] As an embodiment, the processor 301 may include one or more CPUs, such as Figure 2 the CPU 0 and CPU 1 shown in

[0067] The memory 302 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or can also be an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium, or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0068] In a possible implementation, the memory 302 can exist independently of the processor 301. The memory 302 can be connected to the processor 301 through the bus 304 and is used to store instructions or program code. When the processor 301 calls and executes the instructions or program code stored in the memory 302, the model deployment method provided by the embodiments of the present application can be implemented.

[0069] In another possible implementation, the memory 302 can also be integrated with the processor 301.

[0070] The communication interface 303 is used for the computing device to be connected to other devices through a communication network, and the communication network can be an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 303 can include a receiving unit for receiving data and a sending unit for sending data.

[0071] The bus 304 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0072] It should be noted that Figure 2 the structure shown in the figure does not constitute a limitation on the computing device, except Figure 2In addition to the components shown, the computing device may include more or fewer components than shown, or combine certain components, or have a different component arrangement.

[0073] The implementation manners of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0074] The rendering method of the vehicle model provided by the embodiments of the present application can be executed by the above-mentioned electronic device 20.

[0075] As Figure 3 shown, the embodiments of the present application provide a rendering method of a vehicle model, and the method includes the following steps:

[0076] S101. Obtain images captured by multiple cameras at different perspectives of the vehicle configuration.

[0077] Exemplarily, as Figure 4 shown, 4 cameras are deployed on the vehicle, namely a front camera, a left camera, a right camera, and a rear camera. If the cameras are fish-eye cameras, the images captured by these 4 cameras can be as Figure 4 shown.

[0078] The cameras establish a communication connection with the electronic device, and the images captured by the cameras are transmitted to the electronic device in real time.

[0079] S102. Map the images captured by the multiple cameras to an ambient light model to generate an ambient light panoramic image.

[0080] Among them, the ambient light model includes a hemisphere model and a round-bottom panoramic view model. The hemisphere model is used to construct the sky scene of the ambient light, and the round-bottom panoramic view model is used to construct the ground scene and the scene around the vehicle body of the ambient light; the ambient light panoramic image is used to characterize the ambient light information around the vehicle.

[0081] In some embodiments, the mapping process of step S102 can be performed on the GPU.

[0082] The specific schematic diagram of the ambient light model jointly constructed by the hemisphere model and the round-bottom panoramic view model can be as Figure 5 shown.

[0083] Specifically, the formula adopted by the hemisphere model is: , . Among them, R is used to represent the radius of the sphere of the hemisphere model, is used to represent the world coordinates corresponding to a model point in the hemisphere model.

[0084] The formula adopted by the round-bottom panoramic view model is: , , , . Among them, x is used to represent the world coordinates corresponding to a model point in the surround view model, a is used to represent the bottom radius of the surround view model, h is used to represent the height of the surround view model, and r is used to represent the model surface curvature of the surround view model.

[0085] In some embodiments, the bottom-round surround view model is determined as follows: obtaining the number of cameras configured on the vehicle, the parameters of the cameras, and the dimensions of the vehicle; adjusting the model parameters of the bottom-round surround view model based on the number of cameras configured on the vehicle, the parameters of the cameras, and the dimensions of the vehicle, and generating the bottom-round surround view model.

[0086] The internal parameter of the camera determines how the image captured by the camera is corrected for distortion. The number of cameras determines the positions where the cameras are distributed, which in turn determines the change in the external parameters of the cameras. The external parameters of the cameras and the dimensions of the vehicle jointly determine the values of the bottom radius a of the bottom-round surround view model and the height h of the bottom-round surround view model. In this process, it is necessary to ensure that the value of a can make the bottom area of the bottom-round surround view model cover the ground area that the user wants to observe in the surround view image of the vehicle.

[0087] In some embodiments, mapping the images captured by multiple cameras to the ambient light model to generate an ambient light panoramic image can be specifically implemented as follows: determining the mapping relationship between the images captured by multiple cameras and the ambient light model based on the internal and external parameters of each of the multiple cameras; mapping the images captured by multiple cameras to the ambient light model based on the mapping relationship between the images captured by multiple cameras and the ambient light model, and generating the ambient light panoramic image.

[0088] As Figure 6 shown, an embodiment of the present application provides a mapping schematic diagram. In the image, the camera coordinate system is the coordinate system of the camera that collects image data, the pixel coordinate system is the coordinate system of the imaging of the image data, the distance between the origin of the camera coordinate system and the imaging plane corresponding to the pixel coordinate system is the same as the camera focal length, and the world coordinate system is the coordinate system where the surround view model is located. The mapping process from the camera coordinate system to the pixel coordinate system and the world coordinate system is shown in the figure. After obtaining the image data, according to the internal and external parameters of the camera and the spatial coordinate distribution of the adjusted bottom-round surround view model, the distortion correction algorithm and the Zhang's calibration method for the imaging of a pinhole camera can be used to map the image data (generally the original image data collected by a fish-eye camera) to the surround view model. Optionally, the coordinates of the adjusted bottom-round surround view model in the world coordinate system can be first converted to the coordinates in the camera coordinate system of the camera, and then further converted to the pixel coordinates in the pixel coordinate system, so as to realize the mapping of the three-dimensional space coordinate points to the two-dimensional image pixel point coordinates.

[0089] In some embodiments, mapping the images captured by multiple cameras to an ambient light model to generate an ambient light panoramic image may be specifically implemented as follows: mapping the images captured by multiple cameras to a round-bottom surround-view model to generate a first ambient light image, where the first ambient light image is used to characterize the ambient light information in the ground scene and the scene around the vehicle body; mapping the image captured by the camera at the target viewing angle to a hemisphere model to generate a second ambient light image, where the target viewing angle includes the viewing angles that are the same as and / or opposite to the forward direction of the vehicle; the second ambient light image is used to characterize the ambient light information in the sky scene of the vehicle; splicing the first ambient light image and the second ambient light image to generate an ambient light panoramic image.

[0090] As Figure 7 shown, an embodiment of the present application provides a splicing schematic diagram. After sequentially completing the construction and image mapping of the front surround-view model, rear surround-view model, left surround-view model, and right surround-view model on the GPU (Graphics Processing Unit), image fusion is performed by means such as overlapping coverage, overlapping transparency blending, and overlapping color blending. For example, for the overlapping area of the four-way surround-view model images, image fusion is performed using a gradient transparency value to quickly output a complete ambient light panoramic image under the currently specified viewing angle.

[0091] In some embodiments, mapping the image captured by the camera at the target viewing angle to a hemisphere model to generate a second ambient light image may be specifically implemented as follows: based on the driving speed of the vehicle, extracting a partial image to be mapped to the hemisphere model from the current image and historical images captured by the camera at the target viewing angle; mapping the partial image to the hemisphere model to generate a second ambient light image. In practical applications, this process can be performed on the CPU.

[0092] Surround-view cameras are generally installed around the vehicle, but there is generally no camera that can capture images above the vehicle roof. Since in reality, there is a lighting effect on the vehicle roof, we need to construct an image above the vehicle roof based on the surround-view camera images around the vehicle. Because when the vehicle is moving, the vehicle front-view camera can capture the images that can appear above the vehicle roof, so the images above the vehicle can be spliced by using the historical images captured by the vehicle front-view camera and / or the vehicle rear-view camera. The driving speed of the vehicle determines how far the vehicle has advanced between adjacent historical images, and thus determines which areas of the historical images are used for splicing the image above the vehicle roof.

[0093] During the process of mapping to obtain the second ambient light image, mapping operations can also be performed based on the external camera parameters (camera installation position and angle) and internal camera parameters (distortion parameters). During the image splicing and mapping processes, internal camera parameter distortion correction is involved, and the external parameters are involved when generating the lighting model.

[0094] Since the images involved in the surround view model are directly captured by the cameras deployed on the vehicle, during the process of mapping the images to the surround view model to obtain the ambient light panoramic image, there is no need to determine the historical images captured by the cameras.

[0095] S103. Sample the ambient light panoramic image based on the vehicle light reflection vector to generate an ambient light map.

[0096] Among them, the vehicle light reflection vector is used to represent the reflection angle of the vehicle's surface to the light; the ambient light map is used to represent the ambient light information reflected by the vehicle's surface.

[0097] Specifically, the electronic device first obtains the vehicle light reflection vectors; these vectors usually come from the 3D model of the vehicle, and each point or surface has its corresponding normal vector, so the reflection direction can be determined.

[0098] The electronic device then prepares the ambient light panoramic image: This is usually a 360-degree panoramic image representing the light environment around the vehicle. The electronic device calculates the intersection points of the vehicle light reflection vectors and the ambient light panoramic image: For each point on the vehicle, the vehicle light reflection vector is calculated according to its normal vector. The electronic device maps the reflection vector onto the panoramic image to find the corresponding pixel points. The electronic device samples the ambient light. On the panoramic image, at the pixel points found by the electronic device according to the reflection vector, the color value of this point is read as the reflected light color. The electronic device generates the ambient light map. The electronic device stores the reflected light color of each point in a map, and this map is the ambient light map. The ambient light map can be used for real-time rendering to simulate the ambient light reflected by the vehicle's surface.

[0099] Considering performance, preprocessing can be performed on the ambient light map, such as using the mipmap technology to optimize the rendering effect at different distances.

[0100] During the real-time rendering process, the generated ambient light map is used to perform texture mapping and rendering on the vehicle model to simulate the real lighting effect.

[0101] Through the above steps, we can effectively utilize the vehicle light reflection vectors and the panoramic environment image to generate a high-quality ambient light map, thereby enhancing the realism and immersion of vehicle rendering.

[0102] S104. Render the vehicle model based on the ambient light map.

[0103] When specifically implemented, the rendering method can adopt an image-based lighting (IBL) scheme. The specific implementation process of this scheme can be as Figure 8 shown:

[0104] First, the electronic device calculates the normal vectors and reflection light directions of the vertices of the vehicle model, calculates the base color of the vehicle model, sets parameters such as the metallicity and roughness of the vehicle model material, samples the ambient light map, and then samples the bidirectional reflectance distribution function (BRDF) map, and finally outputs the calculated color. Among them, the BRDF map is a pre-calculated lookup table, which can be as Figure 9 shown.

[0105] Figure 3 The technical solution shown above brings at least the following beneficial effects: The ambient light model includes a hemisphere model and a round-bottom panoramic view model, which includes the sky scene, ground scene, and the scenes around the vehicle body of the ambient light, covering all angles of the vehicle, realizing global ambient light rendering of the vehicle. Even when the panoramic view angle is switched, there will be no problem that the ambient light rendering cannot be achieved at some positions of the vehicle. On this basis, mapping the images captured by the cameras at each perspective of the vehicle onto the ambient light model to generate an ambient light panoramic image can ensure that the rendering of the vehicle model is carried out in real time based on the environment around the vehicle, with a stronger sense of reality.

[0106] The following introduces the rendering method of the vehicle model provided by this application from the perspective of the overall process:

[0107] As Figure 10 shown, the electronic device first obtains the real-time images captured by each camera deployed on the vehicle. The electronic device also obtains the internal and external parameters of each camera, constructs an ambient light model, and then constructs an ambient light panoramic map, corrects the distortion of the fisheye images captured by the cameras, and finally performs the ambient light rendering process of the vehicle model to obtain the rendered vehicle model.

[0108] When the vehicle model rendering process does not refer to the ambient light, the display effect of the vehicle model can be as shown in the left figure of Figure 11 . After rendering the vehicle model using the vehicle model rendering method of this application, the display effect of the vehicle model can be as shown in the right figure of Figure 11 . The vehicle model rendering method of this application takes into account the display effect of the top of the vehicle.

[0109] The above mainly introduced the solution provided by the embodiments of the present application from the perspective of methods. To implement the above functions, it includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described function, but such implementation should not be considered to exceed the scope of the present application.

[0110] As Figure 12 shown, the embodiments of the present application also provide a rendering device for a vehicle model, which is used for the rendering method of the vehicle model shown in the above method embodiments. The rendering device 400 for the vehicle model includes: an acquisition module 401 and a processing module 402.

[0111] Among them, the acquisition module is used to acquire images taken by multiple cameras with different perspectives of the vehicle configuration; the processing module 402 is used to map the images taken by the multiple cameras to the ambient light model to generate an ambient light panoramic image; among them, the ambient light model includes a hemisphere model and a round-bottom surround-view model, the hemisphere model is used to construct the sky scene of the ambient light, and the round-bottom surround-view model is used to construct the ground scene and the scene around the vehicle body of the ambient light; the ambient light panoramic image is used to represent the ambient light information around the vehicle; the processing module 402 is further used to sample the ambient light panoramic image based on the vehicle light reflection vector to generate an ambient light texture map; among them, the vehicle light reflection vector is used to represent the reflection angle of the vehicle surface to the light; the ambient light texture map is used to represent the ambient light information reflected by the vehicle surface; the processing module 402 is further used to render the vehicle model based on the ambient light texture map.

[0112] In a possible implementation manner, the processing module 402 is specifically used for: mapping the images taken by the multiple cameras to the round-bottom surround-view model to generate a first ambient light image; among them, the first ambient light image is used to represent the ambient light information in the ground scene and the scene around the vehicle body of the vehicle; mapping the image taken by the camera of the target perspective to the hemisphere model to generate a second ambient light image; among them, the target perspective includes the perspective that is the same as and / or opposite to the forward direction of the vehicle; the second ambient light image is used to represent the ambient light information in the sky scene of the vehicle; splicing the first ambient light image and the second ambient light image to generate an ambient light panoramic image.

[0113] In another possible implementation manner, the processing module 402 is specifically configured to: extract a local image to be mapped onto the hemispherical model from the current image and the historical image captured by the camera from the target perspective based on the driving speed of the vehicle; map the local image onto the hemispherical model to generate a second ambient light image.

[0114] In another possible implementation manner, the processing module 402 is specifically configured to: determine the mapping relationship between the images captured by multiple cameras and the ambient light model based on the internal and external parameters of each of the multiple cameras; map the images captured by the multiple cameras onto the ambient light model based on the mapping relationship between the images captured by the multiple cameras and the ambient light model to generate an ambient light panoramic image.

[0115] In another possible implementation manner, the obtaining module 401 is further configured to obtain the number of cameras configured in the vehicle, the parameters of the cameras, and the size of the vehicle; the processing module 402 is further configured to adjust the model parameters of the round-bottom surround view model based on the number of cameras configured in the vehicle, the parameters of the cameras, and the size of the vehicle to generate a round-bottom surround view model.

[0116] It should be noted that Figure 12 the division of the modules herein is illustrative, merely a logical function division, and there may be other division methods in actual implementation. For example, two or more functions may also be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or in the form of software function modules.

[0117] Another embodiment of the present application further provides an electronic device, which includes a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions. Wherein, when the processor executes the computer instructions, the electronic device is caused to execute each step performed by the electronic device in the method flow shown in the above method embodiment.

[0118] In actual implementation, the obtaining module 401 and the processing module 402 may be implemented by the processor of the electronic device calling the computer program code in the memory. The specific execution process may refer to the description in the above vehicle model rendering method section, and will not be elaborated here.

[0119] Another embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored, and when the computer instructions run on a computer device, the computer device is caused to execute each step performed by the electronic device in the method flow shown in the above method embodiment.

[0120] In another embodiment of the present application, a computer program product is further provided. The computer program product includes computer instructions. When the computer instructions run on a computer device, the computer device is caused to execute each step performed by the electronic device in the method flow shown in the above method embodiment.

[0121] In the above embodiment, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer execution instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more media integrated therein. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), etc.

[0122] As described above, it is only the specific implementation manner of the present application. Those skilled in the art of this technology can think of changes or substitutions according to the specific implementation manner provided by the present application, and all should be covered within the protection scope of the present application.

Claims

1. A rendering method for a vehicle model, characterized in that, The method includes: Obtaining images captured by multiple cameras with different perspectives configured on the vehicle; Mapping the images captured by the multiple cameras to an ambient light model to generate an ambient light panoramic image; wherein, the ambient light model includes a hemisphere model and a round-bottom surround view model, the hemisphere model is used to construct the sky scene of the ambient light, and the round-bottom surround view model is used to construct the ground scene and the surrounding scene of the vehicle body; the ambient light panoramic image is used to represent the ambient light information around the vehicle; Sampling the ambient light panoramic image based on the vehicle light reflection vector to generate an ambient light texture map; wherein, the vehicle light reflection vector is used to represent the reflection angle of the vehicle's surface to light; the ambient light texture map is used to represent the ambient light information reflected by the vehicle's surface; Rendering the vehicle model based on the ambient light texture map; Wherein, the mapping of the images captured by the multiple cameras to the ambient light model to generate an ambient light panoramic image includes: Mapping the images captured by the multiple cameras to the round-bottom surround view model to generate a first ambient light image; wherein, the first ambient light image is used to represent the ambient light information in the ground scene and the surrounding scene of the vehicle body; Based on the driving speed of the vehicle, extracting local images to be mapped to the hemisphere model from the current image and historical images captured by the camera from the target perspective; Mapping the local images to the hemisphere model to generate a second ambient light image; wherein, the target perspective includes the perspectives that are the same as and / or opposite to the forward direction of the vehicle; the second ambient light image is used to represent the ambient light information in the sky scene of the vehicle; Stitching the first ambient light image and the second ambient light image to generate the ambient light panoramic image.

2. The method according to claim 1, wherein The mapping of the images captured by the multiple cameras to the ambient light model to generate an ambient light panoramic image includes: Determining the mapping relationship between the images captured by the multiple cameras and the ambient light model based on the internal and external parameters of each of the multiple cameras; Based on the mapping relationship between the images captured by the multiple cameras and the ambient light model, mapping the images captured by the multiple cameras to the ambient light model to generate an ambient light panoramic image.

3. The method according to claim 1, wherein The round-bottom surround view model is determined by the following method: Obtaining the number of cameras configured on the vehicle, the parameters of the cameras, and the size of the vehicle; Adjusting the model parameters of the round-bottom surround view model based on the number of cameras configured on the vehicle, the parameters of the cameras, and the size of the vehicle to generate the round-bottom surround view model.

4. A rendering device for a vehicle model, characterized in that, It includes: An acquisition module, configured to obtain images captured by multiple cameras with different perspectives configured on the vehicle; A processing module, configured to map the images captured by the multiple cameras to an ambient light model to generate an ambient light panoramic image; wherein, the ambient light model includes a hemisphere model and a round-bottom surround view model, the hemisphere model is used to construct the sky scene of the ambient light, and the round-bottom surround view model is used to construct the ground scene and the surrounding scene of the vehicle body; the ambient light panoramic image is used to represent the ambient light information around the vehicle; The processing module is further configured to sample the ambient light panoramic image based on the vehicle light reflection vector to generate an ambient light map; wherein, the vehicle light reflection vector is used to characterize the reflection angle of the vehicle's surface to light; and the ambient light map is used to characterize the ambient light information reflected by the vehicle's surface. The processing module is further configured to render the vehicle model based on the ambient light map. Specifically, the processing module is configured to: map the images captured by the multiple cameras onto the round-bottom panoramic view model to generate a first ambient light image; wherein, the first ambient light image is used to characterize the ambient light information in the ground scene and the scene around the vehicle body; based on the driving speed of the vehicle, extract local images to be mapped onto the hemispherical model from the current image and the historical image captured by the camera at the target perspective; map the local images onto the hemispherical model to generate a second ambient light image; wherein, the target perspective includes perspectives that are the same as and / or opposite to the forward direction of the vehicle; the second ambient light image is used to characterize the ambient light information in the sky scene of the vehicle; and splice the first ambient light image and the second ambient light image to generate the ambient light panoramic image.

5. The apparatus according to claim 4, wherein Specifically, the processing module is configured to: determine the mapping relationship between the images captured by the multiple cameras and the ambient light model based on the internal and external parameters of each of the multiple cameras; and map the images captured by the multiple cameras onto the ambient light model based on the mapping relationship between the images captured by the multiple cameras and the ambient light model to generate an ambient light panoramic image. The acquisition module is further configured to acquire the number of cameras configured on the vehicle, the parameters of the cameras, and the size of the vehicle; and the processing module is further configured to adjust the model parameters of the round-bottom panoramic view model based on the number of cameras configured on the vehicle, the parameters of the cameras, and the size of the vehicle to generate the round-bottom panoramic view model.

6. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-3.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When the computer instruction is executed by the processor, it implements the steps of the method according to any one of claims 1-3.

8. A computer program product comprising computer instructions, characterized in that, When the computer instruction is executed by the processor, it implements the steps of the method according to any one of claims 1-3.

Images