Environment reconstruction image processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an environment reconstruction image processing method and device, electronic equipment and a medium, and relates to the field of intelligent driving. The method comprises the following steps: performing semantic analysis on a style instruction input by a user based on an intelligent cabin to obtain a first label and a second label, inputting the first label into a material generation model, and generating virtual material data corresponding to a virtual object through the material generation model; determining display position coordinates of the virtual material data based on a display area indicated by the second label in the environment reconstruction image; and in the environment reconstruction image, rendering the virtual material data based on the display position coordinates of the virtual material data to obtain quasi material data of the target environment reconstruction image. A user can create the virtual object in the intelligent cabin in a customized manner by inputting the style instruction, and replace a real object in the environment reconstruction image, so that the customized display of a specific object is realized, and the flexibility of material style switching is improved.

Description

An environmental reconstruction image processing method, apparatus, electronic device, and storage medium Technical Field

[0001] This disclosure relates to the field of intelligent driving, and in particular to an image processing method, apparatus, electronic device, and storage medium based on environment reconstruction. Background Technology

[0002] The vehicle's smart cockpit can display SR (Surrounding Reality) images on a screen. SR images are generated by the vehicle's sensors collecting data on the vehicle's surrounding environment, allowing users to observe the vehicle's surroundings through SR images.

[0003] In existing technologies, when users want to change the style of materials in SR images, the smart cockpit cannot customize the style display for specific objects, which limits the ability to switch material styles in SR images. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides an environment reconstruction image processing method, apparatus, electronic device, and storage medium to solve the problem that smart cockpits cannot customize the display of specific objects in the environment reconstruction image, resulting in limitations in switching material styles.

[0005] The first aspect of this disclosure provides an environment reconstruction image processing method, comprising: performing semantic parsing on a style command input by a user based on a smart cockpit to obtain a first label and a second label, wherein the style command is used to instruct the display of virtual material data corresponding to a virtual object in an environment reconstruction image corresponding to the smart cockpit, the first label is used to describe the characteristics of the virtual object, and the second label indicates the position of the virtual object in the environment reconstruction image; inputting the first label into a material generation model and generating virtual material data corresponding to the virtual object through the material generation model; determining the display position coordinates of the virtual material data based on the display area indicated by the second label in the environment reconstruction image; and rendering the virtual material data in the environment reconstruction image based on the display position coordinates of the virtual material data to obtain a target environment reconstruction image.

[0006] A second aspect of this disclosure provides an environment reconstruction image processing apparatus, comprising: a label determination module, configured to perform semantic parsing on a style command input by a user based on a smart cockpit to obtain a first label and a second label, wherein the style command is used to instruct the display of virtual material data corresponding to a virtual object in an environment reconstruction image corresponding to the smart cockpit, the first label is used to describe the characteristics of the virtual object, and the second label indicates the position of the virtual object in the environment reconstruction image; a material generation module, configured to input the first label into a material generation model and generate virtual material data corresponding to the virtual object through the material generation model; a material positioning module, configured to determine the display position coordinates of the virtual material data based on the display area indicated by the second label in the environment reconstruction image; and an image processing module, configured to render the virtual material data in the environment reconstruction image based on the display position coordinates of the virtual material data to obtain a target environment reconstruction image.

[0007] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program for performing the environment reconstruction image processing method provided in the above embodiments.

[0008] A fourth aspect of this disclosure provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the environment reconstruction image processing method provided in the above embodiments.

[0009] The environmental reconstruction image processing method, apparatus, electronic device, and medium provided in the above embodiments of this disclosure, through semantic parsing of user-input style instructions, can accurately obtain a first label describing the characteristics of a virtual object and a second label indicating the position of the virtual object in the environmental reconstruction image, thereby transforming the user's abstract style instructions into concrete, executable labels. By inputting the first label into a material generation model to generate corresponding virtual material data, virtual objects are created in a personalized manner according to the user-input style instructions. Then, the display position coordinates of the virtual material data are determined based on the display area indicated by the second label, ensuring that the virtual object can be accurately positioned in the environmental reconstruction image as desired by the user. Finally, the target environmental reconstruction image is obtained by rendering the virtual material data in the environmental reconstruction image according to the display position coordinates. Therefore, the technical solution provided in the above embodiments of this disclosure can customize the creation of virtual material data for virtual objects based on user-input style instructions, thereby replacing the real material data in the environmental reconstruction image with virtual material data. This allows specific objects to be displayed as virtual objects specified by style instructions in the environmental reconstruction image, achieving customized display of specific objects and improving the flexibility of material style switching. Attached Figure Description

[0010] Figure 1 is a vehicle system architecture provided in an exemplary embodiment of the present disclosure; Figure 2 is a flowchart of an environmental reconstruction image processing method provided in an exemplary embodiment of the present disclosure; Figure 3 is a flowchart of step 203 of the environmental reconstruction image processing method provided in an exemplary embodiment of the present disclosure; Figure 4 is a flowchart of step 204 of the environmental reconstruction image processing method provided in an exemplary embodiment of the present disclosure; Figure 5 is a flowchart after step 2031 of the environmental reconstruction image processing method provided in an exemplary embodiment of the present disclosure; Figure 6 is a flowchart after obtaining the target environmental reconstruction image of the environmental reconstruction image processing method provided in an exemplary embodiment of the present disclosure; Figure 7 is a flowchart after step S204 of the environmental reconstruction image processing method provided in an exemplary embodiment of the present disclosure; Figure 8 is a flowchart of step 2031 of the environmental reconstruction image processing method provided in an exemplary embodiment of the present disclosure; Figure 9 is a structural diagram of an environmental reconstruction image processing method apparatus provided in an exemplary embodiment of the present disclosure; Figure 10 is a structural diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0011] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0012] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0013] Application Overview: The intelligent cockpit of a vehicle is the core area that integrates various intelligent interactive functions inside the vehicle. The intelligent cockpit can provide users with a convenient, comfortable and safe driving experience by integrating display, control, perception and other technologies.

[0014] The environmental reconstruction and display function is an important interactive feature of the smart cockpit. It uses vehicle sensors, such as onboard cameras and radar, to collect data on the external environment and then displays a virtual or augmented reality image of the vehicle's surroundings in real time on the smart cockpit's interface, such as the instrument panel and central control screen. Users can intuitively understand road conditions around the vehicle within the smart cockpit, such as road markings, pedestrians, other vehicles, and obstacles, reducing blind spots and improving driving safety and convenience.

[0015] The reconstructed environment display image can include various material styles. In different scenarios, users can switch the material style of the reconstructed environment display image to obtain a visual experience that better suits their personal preferences or scenario needs. For example, when driving long distances, users may want the reconstructed environment display image to present a soft and soothing style to reduce visual fatigue; while at night or in complex road conditions, users may want to switch the reconstructed environment display image to a nighttime material style to reduce the display brightness of the reconstructed environment display image and avoid excessive brightness difference between the reconstructed environment display image and the nighttime environment, thereby improving driving safety.

[0016] However, traditional methods of adjusting material styles have a global nature. For example, once a user selects a certain material style, all material data in the reconstructed environment image will uniformly apply that material style. It is impossible to customize the style display for specific objects or specific areas in the reconstructed environment image, which results in limitations in switching material styles.

[0017] Figure 1 illustrates an exemplary vehicle system architecture provided by this disclosure. As shown in Figure 1, the vehicle system in this disclosure may include: an vehicle processing module 110, a processor 120, and a memory 130. The vehicle processing module 110 is used to collect user input control commands based on the smart cockpit, and to parse, convert, and transmit the control commands.

[0018] The processor 120, as the core computing unit of the vehicle system, is connected to the vehicle processing module 110 and the memory 130 via an internal bus or other communication methods. It is responsible for receiving parsed control commands from the vehicle processing module 110 and controlling the smart cockpit to perform corresponding interactive actions based on these commands. The memory 130 stores various types of data from the smart cockpit, allowing the processor 120 to retrieve the corresponding data at any time when responding to control commands.

[0019] For example, the vehicle-mounted processing module 110 may include an audio acquisition device 111, a text acquisition device 112, a communication device 113, and a neural network device 114, etc.

[0020] The audio acquisition device 111 is used to acquire voice control commands issued by the user in the smart cockpit, such as "switch the navigation interface to dark mode" or "adjust the ambient lighting in the car to blue." Since the processor 120 cannot directly respond to audio control commands, the vehicle processing module can also convert the voice control commands to convert analog audio signals into digital audio signals that can be processed by the processor 120.

[0021] The text acquisition device 112 is used to acquire text control commands input by the user based on the smart cockpit. For example, the smart cockpit includes a display screen, which can be a touch screen, allowing the user to directly input text control commands by clicking, swiping, or handwriting on the screen. For instance, if the user manually inputs "Turn on AC mode" on the display screen, the text acquisition device 112 can acquire the text "Turn on AC mode" as the text input command.

[0022] For example, the smart cockpit includes physical buttons, allowing users to input text control commands by clicking these buttons. For instance, a user selects a specific function option using the buttons to generate a text control command.

[0023] For example, after receiving a control command, the processor 120 can parse the format of the control command. If the control command is a voice control command, the processor 120 needs to first convert the voice control command into command text, and then perform semantic recognition on the command text to parse the command intent of the voice control command, thereby executing the corresponding interactive action. If the control command is a text control command, the processor 120 does not need to perform voice-to-text conversion, but directly extracts the command text from the text control command and executes the subsequent interactive process.

[0024] The communication device 113 can interact with the processor 120, memory 130, and other in-vehicle devices (such as audio systems, air conditioning controllers, navigation modules, etc.) within the smart cockpit to transmit control commands and related data. For example, the communication device 113 can also remotely interact with the smart cockpit's server, such as receiving system update commands and map data update packages from the server, or uploading user driving habit data and vehicle status information to the server, to achieve continuous optimization and upgrading of the smart cockpit's functions.

[0025] The communication device 113 can support multiple communication protocols, such as CAN bus protocol and Ethernet protocol, which can ensure the stability and real-time performance of data transmission with different types of vehicle equipment, enabling the intelligent cockpit system to work together efficiently.

[0026] The neural network device 114 performs deep semantic parsing on the acquired audio control commands or text control commands to obtain parsed control commands. For example, when a user says a vague voice control command such as "make this brighter," the neural network device 114 can combine the current cabin environment (such as current light intensity, time, etc.) and the user's historical operating habits to accurately understand whether the user intends to adjust the brightness of the interior lights or the display screen, and convert the parsed control command into a format that the processor 120 can recognize, so that the processor 120 can respond to the control command and increase the brightness of the interior lights or the display screen.

[0027] In some embodiments, the neural network device 114 can also be used to generate material data, for example, to generate material data specified by the control command according to the control command input by the user, and to send the material data to the controller 120 for application, and to send it to the memory 130 for storage.

[0028] Figure 2 is a flowchart illustrating an exemplary embodiment of the environmental reconstruction image processing method provided by this disclosure. This embodiment can be applied to vehicles, electronic devices, and in-vehicle electronic devices. The following embodiment uses an in-vehicle electronic device as the execution subject for illustrative purposes. As shown in Figure 2, it includes the following steps: S201: Perform semantic parsing on the style instructions input by the user based on the smart cockpit to obtain a first label and a second label.

[0029] In some embodiments, style instructions are used to indicate the style of virtual material data corresponding to virtual objects displayed in the reconstructed environment image corresponding to the smart cockpit. Virtual objects are objects that do not exist in the reconstructed environment image, and virtual material data are visual element data used to construct virtual objects, such as the 3D model, texture map, color parameters, dynamic effect parameters, etc. of the virtual objects.

[0030] In some embodiments, the style instruction is an instruction that is input as a real object that actually exists in the environment reconstruction image. For example, if there is an electric vehicle in the environment reconstruction image, the style instruction can be "turn the electric vehicle on the road into a cheetah", where "electric vehicle" is a real object in the environment reconstruction image and "cheetah" is a virtual object.

[0031] In some embodiments, style instructions may be input without referencing real objects that actually exist in the environment reconstructed image; for example, "Generate an Eiffel Tower on the left," where "Eiffel Tower" is a virtual object.

[0032] In some embodiments, style commands can be input via voice or text. For voice input, the user can activate the smart cockpit's audio acquisition device by speaking a specified wake-up word, enabling the device to capture subsequent voice control commands issued by the user within the smart cockpit. These voice control commands constitute the user-input style commands. For example, if the user says, "Hello, Xiao X, turn the electric vehicles on the road into cheetahs," "Hello, Xiao X" is the wake-up word, and the subsequent phrase "turn the electric vehicles on the road into cheetahs" constitutes the voice-based style command. Exemplarily, the smart cockpit can integrate a voice-app linkage function, through which an application can respond to style commands input by the user via voice.

[0033] For text input, users can input text control commands via the smart cockpit's touchscreen display, physical buttons, or connected external input devices (such as in-vehicle keyboards, mobile terminals, etc.). These text control commands are the user-defined style commands. For example, the smart cockpit can integrate intelligent chatbots, such as IM Chatbots, which are intelligent chatbots integrated into instant messaging (IM) platforms and can interact with users through text input.

[0034] In some embodiments, the display screen of the smart cockpit can display an interactive interface, such as a human-machine interface (HMI). A designated area of ​​the interactive interface can display a text input box, through which the user can input text content, such as "turn electric cars on the road into cheetahs". The smart cockpit will then use the controller to parse the input text content into specific style instructions.

[0035] In some embodiments, the intelligent cockpit can perform semantic parsing of style instructions using a neural network device to extract a first label and a second label from the style instructions. The first label describes the characteristics of the virtual object. Taking a "cheetah" as an example, the first label could describe the characteristics of the "cheetah," such as its color, posture, and size. The second label indicates the location of the virtual object in the reconstructed environmental image.

[0036] S202: Input the first label into the material generation model, and generate virtual material data corresponding to the virtual object through the material generation model.

[0037] In some embodiments, the memory may also store a material generation model, which is used to generate virtual material data based on an input first tag. For example, the material generation model may be an AI-Generated Content (AIGC) generation model, which can automatically generate virtual material data based on an input text-formatted first tag using artificial intelligence technology.

[0038] In some embodiments, the target material data can be generated by invoking a "text-to-element" generation model through the vehicle's high-performance computing unit. For example, the high-performance computing unit can be an embedded neural network processor (NPU).

[0039] In some embodiments, based on the input format of the material generation model, the style instructions need to be formatted according to the text format during the semantic parsing process. The memory may also store a speech processing model for the in-vehicle environment, which can convert the voice input style instructions into a first label in text format.

[0040] For example, the speech processing model can include acoustic and text models, such as an end-to-end automatic speech recognition (ASR) model based on deep neural networks. For example, the deep neural network can be a connectionist temporal classification (CTC) model or an attention model. This automatic speech recognition model can effectively suppress noise interference from wind, road, and in-vehicle conversations generated while driving, ensuring accurate text conversion of style instructions to obtain the first label even in high signal-to-noise ratio environments.

[0041] S203: Determine the display position coordinates of the virtual material data based on the display area indicated by the second label in the environmental reconstruction image.

[0042] In some embodiments, the display area is a specific region in the reconstructed environment image used to display virtual objects. After obtaining the second label, it is necessary to first determine the boundary information of the specific display area pointed to by the second label in the two-dimensional or three-dimensional coordinate system of the reconstructed environment image. For example, when the second label is "electric vehicle on the road", the region where the "electric vehicle" is located in the reconstructed environment image can be located by image recognition technology, and the pixel coordinates of the upper left and lower right corners of the region can constitute the boundary of the display area. For example, if the style instruction is "generate an iron tower on the left", then "left" in the style instruction will be recognized as the second label. The second label can represent a preset proportion of the left side of the reconstructed environment image in the horizontal dimension, such as the left 1 / 3 of the reconstructed environment image, and combine it with information such as the ground height in the vertical dimension to determine the approximate range of the display area.

[0043] The display position coordinates are the anchor point coordinates of the virtual object. After determining the boundary of the display area, the display position coordinates of the virtual material data in the coordinate system of the reconstructed environment image can be calculated based on the size parameters of the virtual object and the preset layout rules of the reconstructed environment image (such as centering, bottom alignment with the ground, etc.). For example, the display position coordinates are the center point coordinates of the display area, or the display position coordinates can be determined based on the relative positional relationship between the reference point of the virtual object's 3D model and the boundary of the display area.

[0044] S204: In the environment reconstruction image, the virtual material data is rendered based on the display position coordinates of the virtual material data to obtain the target environment reconstruction image.

[0045] Rendering virtual asset data is the process of fusing and displaying the generated virtual asset data with the reconstructed environment image. The intelligent cockpit can use a rendering engine to load the asset framework corresponding to the virtual asset data of virtual objects into the three-dimensional spatial coordinate system corresponding to the reconstructed environment image, based on the display position coordinates.

[0046] After loading is complete, the material framework can be rendered using the preset texture maps, color parameters and other attribute parameters of the virtual object, so that the virtual material data can be naturally integrated with other material data in the environment reconstruction image to obtain the target environment reconstruction image.

[0047] In some embodiments, after rendering the virtual material data, edge blending processing can be performed on the virtual material data and other material data in the reconstructed environment image to reduce the incongruity between the virtual material data and other material data in the reconstructed environment image.

[0048] The method provided in this disclosure can perform semantic parsing on user-input style commands to obtain a first label describing a virtual object and a second label indicating the position of the virtual object in an environment reconstruction image. It then generates virtual material data corresponding to the virtual object using a material generation model, determines the display position coordinates of the virtual material data in the environment reconstruction image using the second label, and renders the virtual material data in the environment reconstruction image according to the display position coordinates to obtain a target environment reconstruction image. Therefore, the method provided in this disclosure allows for the creation of custom virtual material data for virtual objects by directly inputting style commands. This virtual material data replaces the real material data in the environment reconstruction image, enabling the display of a specific object as the virtual object specified by the style command in the environment reconstruction image. This achieves customized display of specific objects and improves the flexibility of material style switching.

[0049] Figure 3 is a flowchart illustrating step 203 of the environment reconstruction image processing method provided in an exemplary embodiment of this disclosure.

[0050] As shown in Figure 3, based on the embodiment shown in Figure 2 above, step S203 may include the following steps: S2031: Perform road recognition on the environment reconstruction image to obtain the road area and non-road area of ​​the environment reconstruction image.

[0051] In some embodiments, the implementation of virtual material data displayed by the smart cockpit differs for different display areas of the reconstructed environment image. For example, the memory may store a road recognition model, which can perform road recognition on the reconstructed environment image to identify road areas and non-road areas. The road areas and non-road areas correspond to two different display areas of the reconstructed environment image, respectively.

[0052] For example, the road area can be an area for vehicles to travel on, such as urban roads, highways, and rural roads. The road area can have features such as continuous road surface texture, lane lines, and traffic signs. The non-road area can include other scene elements besides the road, such as buildings on both sides of the road, green belts, trees, and the sky. By performing pixel-level classification processing on the reconstructed environment image through a road recognition model, the boundary range of the road area can be accurately delineated, thereby distinguishing between the road area and the non-road area.

[0053] S2032: In response to the display area being located in the road area of ​​the environment reconstruction image, determine the display position coordinates of the virtual material data based on the center point coordinates of the real material data of the road area.

[0054] Depending on the display area, virtual material data can be displayed in different ways. For example, for road areas, it's crucial to avoid generating virtual material data directly from scratch, as this could cause vehicles to misidentify the generated virtual material data as real objects on the road, affecting driving judgment. Therefore, when the display area of ​​the virtual material data is located within the road area of ​​the reconstructed environment image, a replacement display method can be used, replacing the real material data in the reconstructed environment image with virtual material data. Based on this, the position coordinates of the real material data in the reconstructed environment image are the display position coordinates of the virtual material data.

[0055] In some embodiments, during the semantic parsing of style instructions, a neural network device can also be used to determine the display intent information of the style instructions, so as to determine whether the display mode indicated by the style instructions is a replacement display or a generation display.

[0056] When the display area is located in the road area, if the style command indicates that the display mode is generated from scratch, the smart cockpit can generate a prompt message and display the prompt message on the smart cockpit's display screen. The prompt message is used to remind the user that virtual objects cannot be generated from scratch in the road area, thereby guiding the user to re-enter the style command.

[0057] In some embodiments, the real-world source data is generated by the vehicle acquiring real-time environmental information about its surroundings using a data acquisition device such as an onboard camera. For example, if the vehicle's camera detects multiple electric vehicles on the road the vehicle is traveling on, the corresponding location in the reconstructed environment image can display the real-world source data for the electric vehicle.

[0058] For example, for the style instruction "turn electric vehicles on the road into cheetahs", the "electric vehicles" need to be replaced by the virtual object "cheetah". Therefore, the position of the "electric vehicles" in the environment reconstruction image is the same as the position of the virtual object "cheetah" in the environment reconstruction image.

[0059] In some embodiments, the smart cockpit can select a specific coordinate within the real-world material data as the display position coordinates for the virtual material data. For example, the center point coordinates of the real-world material data can be selected. Based on the principle of replacing road areas, the center point coordinates of the real-world material data are the same as the display position coordinates of the virtual material data; that is, the center point coordinates of the real-world material data are the same as the display position coordinates of the virtual material data.

[0060] For example, the intelligent cockpit can obtain the bounding rectangle of the real material data in the reconstructed environmental image by performing image detection on the real material data. The coordinates of the top left and bottom right vertices of the bounding rectangle can be obtained by the detection algorithm, and then the coordinates of the center point of the rectangle can be calculated based on the coordinates of the top left and bottom right vertices.

[0061] The method provided in this disclosure, by performing road region recognition on the reconstructed environment image, can accurately distinguish between road areas and non-road areas, providing a foundation for subsequent positioning processing. When the display area indicated by the second label is located in a road area, the display position coordinates of the virtual material data can be determined based on the center point coordinates of the real material data within the road area, enabling the virtual material data to accurately cover or replace the original real material data. Therefore, this positioning method based on the location of real material data is not only accurate but also achieves seamless integration of virtual material data and the reconstructed environment image, enhancing the realism and immersion of virtual object display in the reconstructed environment image, and improving the accuracy and naturalness of customized replacement displays.

[0062] Figure 4 is a flowchart illustrating step 204 of the environment reconstruction image processing method provided in an exemplary embodiment of this disclosure.

[0063] As shown in Figure 4, based on the embodiment shown in Figure 3 above, step S204 may include the following steps: S2041: Based on the display position coordinates of the virtual material data, create a first temporary layer on top of the real material data of the environment reconstruction image.

[0064] In some embodiments, the intelligent cockpit can create a first temporary layer on top of the reconstructed environment image based on the display position coordinates. This first temporary layer is used to temporarily render virtual material data, allowing the virtual material data to be overlaid on the reconstructed environment image and cover the corresponding real material data. The size of the first temporary layer can be dynamically adjusted based on the display area of ​​the virtual material data and the display area of ​​the real material data in the reconstructed environment image, thereby ensuring that the first temporary layer can fully display the virtual material data while covering the originally displayed real material data.

[0065] For example, for the style instruction "turn electric cars on the road into cheetahs," the intelligent cockpit can create a first temporary layer on top of each "electric car" in the reconstructed environment image, based on the display position coordinates. By creating an independent temporary layer on top of the real material data, direct modification of the real material data in the reconstructed environment image can be avoided. This facilitates subsequent operations on displaying, adjusting, or removing virtual material data, improving the flexibility and security of display processing. Thus, when adjustments to the position, size, or posture of virtual material data are needed, only the corresponding operation needs to be performed on the first temporary layer, without affecting the real material data in the reconstructed environment image.

[0066] S2042: In the environment reconstruction image, render the virtual material data in the first temporary layer to obtain the target environment reconstruction image.

[0067] In some embodiments, the intelligent cockpit can perform layered rendering of the environment reconstruction image and the first temporary layer through the vehicle's rendering module. That is, the rendering module can render the environment reconstruction image on the layer corresponding to the environment reconstruction image, and render virtual material data on the first temporary layer.

[0068] For example, the rendering module can run a rendering pipeline, which is a processing flow consisting of multiple rendering stages, such as vertex shading, geometry shading, and pixel shading, so as to render the virtual material data in the first temporary layer and obtain the target environment reconstruction image.

[0069] Since the first temporary layer is located above the environment reconstruction image, after the virtual material data is rendered in the first temporary layer, the virtual material data will cover the real material data in the environment reconstruction image. For example, the virtual material data of "cheetah" rendered in the first temporary layer covers the real material data of "electric car" in the environment reconstruction image, so as to achieve the purpose of replacing the real material data in the environment reconstruction image with virtual material data.

[0070] The method provided in this disclosure can create a first temporary display layer on top of the real material data of the reconstructed environment image based on the display position coordinates of the virtual material data, and render the virtual material data on the first temporary display layer. This layered rendering method allows the newly generated virtual material data to effectively cover the underlying real material data, thereby achieving the purpose of replacing real objects with virtual objects. Furthermore, by creating a temporary display layer, the display state of virtual objects can be flexibly manipulated, such as adjusting their position and posture, facilitating the updating and clearing of virtual objects, and avoiding direct modification of the reconstructed environment image data, effectively improving the efficiency and visual effect of custom display implementation.

[0071] Figure 5 is a flowchart illustrating the environmental reconstruction image processing method after step 2031 provided in an exemplary embodiment of this disclosure.

[0072] As shown in Figure 5, based on the embodiment shown in Figure 4 above, after step S2031, the following steps may also be included: S2033: In response to the display area being located in a non-road area of ​​the environment reconstruction image, in the non-road area of ​​the environment reconstruction image, the display position coordinates of the virtual material data are determined based on the center point coordinates of the target area.

[0073] For example, since the vehicle does not travel in off-road areas, virtual material data can be generated directly from scratch in off-road areas without affecting the vehicle's driving decisions. Therefore, when the display area of ​​the virtual material data is located in the off-road area of ​​the reconstructed environment image, the virtual material data can be generated directly and displayed in the reconstructed environment image.

[0074] For example, for the style instruction "generate an iron tower on the left grass," a virtual object "iron tower" needs to be generated on the real object "grass." Through semantic recognition of the style instruction, it can be determined that the second label "left grass" is located in a non-road area. The smart cockpit can then determine the target area within this non-road area based on the second label. The target area is the region indicated by the second label in the non-road area. For instance, a non-road area can include "grass" on both sides of a road; therefore, the "left grass" among the "grass" on both sides is the target area. This avoids positional deviations when generating virtual material data in non-road areas, ensuring that the virtual object is accurately displayed in the area expected by the user.

[0075] After determining the target area, the intelligent cockpit can obtain the coordinates of the center point of the target area, which is the same as the display position coordinates of the virtual material data. Therefore, the center point coordinates can be used as the display position coordinates of the virtual material data. For example, if "the grass on the left" is taken as the target area, the intelligent cockpit can use image segmentation or region detection algorithms to determine the boundary range of "the grass on the left" in the reconstructed environment image, and then calculate the center coordinates of "the grass on the left," using these center coordinates as the display position coordinates of the virtual material data for the virtual object "Eiffel Tower." For example, the target area can also be an irregular shape; therefore, the intelligent cockpit can use image segmentation or region detection algorithms to determine the center point of the target area based on the boundary range of the irregular shape.

[0076] S2034: Based on the display position coordinates of the virtual material data, create a second temporary layer on top of the target area of ​​the environment reconstruction image.

[0077] In some embodiments, during the generation of virtual objects in non-road areas, to ensure that the virtual objects can be fully displayed in the target area, the intelligent cockpit can create a second temporary layer above the target area based on the display position coordinates. The size of the second temporary layer needs to be dynamically adjusted according to the display area of ​​the target area in the reconstructed environment image and the display area of ​​the virtual material data in the reconstructed environment image. This ensures that the second temporary layer can fully display the virtual material data while reducing the regional display restrictions of the target area on the virtual material data, allowing the virtual objects to be displayed within or beyond the target area.

[0078] For example, for the style instruction "Generate an iron tower on the left grass", the smart cockpit can create a second temporary layer on top of "left grass" based on the displayed location coordinates in the environment reconstruction image, which can cover "left grass".

[0079] For example, in non-road areas, style instructions can not only instruct the generation of virtual objects in the reconstructed environment image, but also implement alternative display methods such as those for road areas. See the disclosed embodiments in steps S2041-S2042 above, which will not be repeated here. In addition to the disclosed embodiments described above, style instructions can also be used to instruct the generation of virtual images such as weather, scenery, and scenes in the reconstructed environment image. This disclosure does not specifically limit the object format generated in non-road areas.

[0080] In some embodiments, if a first temporary layer and a second temporary layer are created simultaneously in the reconstructed environmental image, the intelligent cockpit can determine the layer relationship between the first temporary layer and the second temporary layer based on the time order in which the temporary layers are created. For example, the temporary layer created later can be located above the temporary layer created earlier.

[0081] S2035: In the environment reconstruction image, render virtual material data in the second temporary layer to obtain the target environment reconstruction image.

[0082] In some embodiments, the intelligent cockpit can perform layered rendering of the reconstructed environment image and the second temporary layer through the vehicle's rendering module. The implementation method of layered rendering can refer to the implementation method of step S2042, which will not be repeated here.

[0083] The method provided in this disclosure, after identifying a non-road area, responds to the display area being located in the non-road area by determining the display position coordinates of virtual material data in the non-road area based on the center point coordinates of the target area indicated by the second label. This allows for the generation of virtual objects in the non-road area. Furthermore, by creating a second temporary display layer on top of the target area and rendering the virtual material data, the virtual objects can be naturally added to the non-road area of ​​the reconstructed environment image. This enriches the display content of the reconstructed environment image while avoiding interference with information in the road area. Therefore, this differentiated display method for different road and non-road areas improves the application scenarios and flexibility of customized display of reconstructed environment images, meeting users' needs to add personalized virtual objects in different locations.

[0084] Figure 6 is a schematic diagram of the process after obtaining a target environment reconstruction image using an exemplary embodiment of the present disclosure.

[0085] Referring to Figure 6, based on the embodiments shown in Figures 4 and 5, after obtaining the target environment reconstruction image, the following steps may also be included: S2043: Based on the user's restoration command input to the smart cockpit, remove the virtual material data rendered in the first temporary layer and / or the virtual material data rendered in the second temporary layer from the target environment reconstruction image.

[0086] In some embodiments, when a user wants to restore the initial reconstructed environmental image, they can input a restoration command into the smart cockpit. For example, the restoration command may include voice input and text input methods; specific details can be found in the input methods for style commands in the foregoing disclosed embodiments, and will not be repeated here.

[0087] For example, the smart cockpit can also display functional controls on the human-machine interface (HMI) for restoring the reconstructed environmental image, and users can trigger restoration commands by clicking on the functional controls on the display screen.

[0088] In some embodiments, when the smart cockpit receives a restoration command, it identifies the first temporary layer and / or the second temporary layer that have been created in the target environment reconstruction image, clears the virtual material data rendered and displayed by the first temporary layer and / or the second temporary layer, and then removes the first temporary layer and / or the second temporary layer, thereby restoring the target environment reconstruction image to the original environment reconstruction image.

[0089] For example, taking the "cheetah" rendered in the first temporary layer as covering the "electric car" in the reconstructed environment image, the smart cockpit can respond to the restoration command, clear the virtual material data of the "cheetah" in the first temporary layer, and then remove the first temporary layer, so that the "electric car" covered by the "cheetah" in the reconstructed environment image can be displayed again, thereby realizing the state restoration of the reconstructed environment image.

[0090] For example, if virtual material data rendered by a second temporary layer also exists, such as the "Iron Tower" generated in the "left grass", the intelligent cockpit will also clear the "Iron Tower" in the second temporary layer, so that the "left grass" that was originally covered by the "Iron Tower" can be displayed again in the environment reconstruction image.

[0091] In some embodiments, during the removal of the first temporary layer and the second temporary layer, the smart cockpit can determine the removal order of the first temporary layer and the second temporary layer. For example, the smart cockpit can obtain the layer relationship between the first temporary layer and the second temporary layer, and remove the first temporary layer and the second temporary layer in a top-to-bottom order. In cases where there are multiple first temporary layers or multiple second temporary layers, the temporary layers can also be removed sequentially with reference to the embodiments disclosed above, which will not be elaborated further here.

[0092] In some embodiments, if a user only wants to remove one or a type of temporary layer, they can do so through more specific restoration commands, such as the voice command "Remove the iron tower on the left grass". The smart cockpit will then only clean up the virtual material data and remove the layer for the second temporary layer of the "iron tower", while retaining the contents of the remaining first temporary layers and / or the remaining second temporary layers.

[0093] For example, the restore command could also be "restore all objects in the road". The smart cockpit would then only clean up the virtual objects in the first temporary layer corresponding to the road area and remove the layers, while retaining the content of the second temporary layer.

[0094] The method provided in this disclosure, after generating a target environment reconstruction image, removes virtual material data rendered on a first temporary display layer and / or a second temporary display layer based on a user-inputted restore command. This provides users with a convenient one-click restore operation, quickly clearing the virtual material data of the target environment reconstruction image and restoring the original environment reconstruction image to its state before adding custom virtual objects. Therefore, this restore mechanism enhances the flexibility and controllability of user interaction with the smart cockpit. Users can restore to the original environment reconstruction image at any time as needed, improving user experience and the flexibility of restoring the material style of the environment reconstruction image.

[0095] Figure 7 is a flowchart illustrating the environmental reconstruction image processing method after step S204 provided in an exemplary embodiment of this disclosure.

[0096] Referring to Figure 7, after step S204, the following steps may also be included: S2044: Based on the vehicle's positioning data, the vehicle's display position coordinates and the vehicle's motion trend information are determined by reconstructing the image in the target environment.

[0097] In some embodiments, the vehicle's onboard system may further include a Global Navigation Satellite System (GNSS) and a Visual-Inertial Odometry (VIO). The GNSS can acquire the vehicle's positioning data in real time. For example, the positioning data may include the vehicle's current latitude and longitude information, enabling the vehicle to determine its location based on the positioning data. The VIO can fuse image data acquired by visual sensors and motion sensor data acquired by an Inertial Measurement Unit (IMU), whereby the motion sensor data may include information such as the vehicle's speed, acceleration, and steering angle.

[0098] For example, after acquiring the vehicle's positioning data, the smart cockpit can use a processor to combine the coordinate system of the reconstructed target environment image and map the positioning data onto the reconstructed target environment image to determine the vehicle's display position coordinates within the image. For example, the smart cockpit can combine motion sensor data collected by a visual inertial odometry system and use a processor to calculate the vehicle's motion trend information. This motion trend information reflects changes in the vehicle's direction of travel and position over a future period. For instance, when the vehicle is accelerating in a straight line, the motion trend information can be represented as continuous forward movement along the current direction of travel; when the vehicle is turning, the motion trend information can be reflected as a trajectory shift towards the turning direction.

[0099] S2045: Based on the vehicle's display position coordinates, the virtual material data's display position coordinates, and the vehicle's motion trend information, determine the relative motion trend between the vehicle and the virtual material data.

[0100] In some embodiments, the smart cockpit can calculate the relative positional relationship between the vehicle's displayed position coordinates and the virtual material data in the target environment's reconstructed image coordinate system by comparing the vehicle's displayed position coordinates with the virtual material data's displayed position coordinates. For example, if the vehicle's displayed position coordinates are (X1, Y1) and the virtual material data's displayed position coordinates are (X2, Y2), the lateral and longitudinal distances between the two can be calculated using the coordinate difference, thereby determining whether the virtual material data is located in front of, behind, to the left of, or to the right of the vehicle.

[0101] In some embodiments, when the vehicle is in motion, the smart cockpit can combine the vehicle's motion trend information with the relative positional relationship between the vehicle and the virtual material data to determine the relative motion trend between the vehicle and the virtual material data. For example, when the vehicle's motion trend information is moving at a speed of 5 m / s in the current direction, the smart cockpit can predict that the relative distance between the vehicle and the virtual material data will shorten by 5 meters after 1 second. If the virtual material data does not have motion attributes, the distance between the virtual material data and the vehicle may change from the current "10 meters ahead" to "5 meters ahead".

[0102] S2046: Based on the relative motion trend, adjust the display position coordinates of the first temporary layer and / or the second temporary layer in the target environment reconstructed image.

[0103] In some embodiments, based on relative motion trends, the position of the virtual material data changes during vehicle movement. For example, when there is a speed difference between the vehicle and the virtual material data, the virtual material data will undergo relative motion in the target environment reconstruction image. Therefore, the smart cockpit can adjust the display position coordinates of the first temporary layer and / or the second temporary layer in the target environment reconstruction image according to the relative motion trends.

[0104] For example, taking the virtual material data "Cheetah" corresponding to the real material data "electric vehicle" displayed in the first temporary layer as an example, the "Cheetah" can be located in the Y-axis direction of the vehicle's motion coordinate system, moving at a speed of 5 m / s and maintaining a distance of 10 m from the vehicle. At this time, the display position coordinates of the "Cheetah" are (0, 10), that is, the display position coordinates of the first temporary layer in the environment reconstruction image are (0, 10). The vehicle travels along the Y-axis at a speed of 10 m / s. At this time, after 1 second, the "Cheetah" moves 5 m along the Y-axis, and the vehicle moves 10 m along the Y-axis. Then, the relative distance between the "Cheetah" and the vehicle is shortened by 5 m. In other words, visually, the "Cheetah" moves behind the vehicle. Therefore, the smart cockpit can adjust the display position coordinates of the first temporary layer corresponding to the "Cheetah" from (0, 10) to (0, 5) to accurately present the relative motion state of the "Cheetah" in the target environment reconstruction image, so that the target environment reconstruction image viewed by the user in the smart cockpit is more in line with the real motion perception, and enhances the user's immersive experience.

[0105] It should be noted that in the above example, the vehicle in the reconstructed image of the target environment is used as the reference point of the motion coordinate system, i.e., the origin of the motion coordinate system. The Y-axis of the motion coordinate system is the road direction in which the vehicle travels forward, and the X-axis is the lateral direction passing through the left and right sides of the vehicle. Other embodiments of this disclosure can adaptively adjust the reference point of the coordinate system according to the user's driving habits and road layout. This disclosure does not limit the way the motion coordinate system is set.

[0106] The method provided in this disclosure determines the relative positional relationship between the vehicle and virtual material data based on vehicle positioning data and motion trend information, and adjusts the display position coordinates of the temporary layer in the target environment reconstruction image accordingly. This dynamic adjustment avoids drifting or unnatural jumping of virtual objects in the target environment reconstruction image, maintains the coordinate anchorage of virtual objects in the target environment reconstruction image, significantly improves the realism and stability of virtual object display in the environment reconstruction image under dynamic driving scenarios, and enhances the visual presentation effect of the smart cockpit when the vehicle is in motion.

[0107] Figure 8 is a flowchart illustrating step 2031 of the environment reconstruction image processing method provided in an exemplary embodiment of this disclosure.

[0108] Referring to Figure 8, based on the embodiment shown in Figure 4, step S2031 may include the following steps: S20311: Reconstruct the image based on the environment, perform rasterization processing through a rasterization neural network to obtain multiple image grids.

[0109] In some embodiments, the intelligent cockpit can rasterize the reconstructed environment image using a rasterized neural network to divide the reconstructed environment image into multiple image grids of preset sizes. Each image grid has road features representing a local part of the reconstructed environment image, such as road curvature, slope, lane line type, road surface material, and details such as the presence of traffic signs and markings.

[0110] For example, a rasterized neural network can be a rasterized semantic occupancy grid, which can perform semantic classification and occupancy state prediction for each pixel in an environment reconstruction image, thereby discretizing the continuous image space into a regular image grid.

[0111] S20312: Based on the pixel difference between multiple pixels in multiple image grids, perform road recognition on the environment reconstruction image to determine the road area and non-road area of ​​the environment reconstruction image.

[0112] In some embodiments, the intelligent cockpit can use a processor to calculate pixel difference parameters such as RGB color difference, grayscale difference, or gradient magnitude between adjacent pixels within each image grid, and set a pixel difference threshold to distinguish between road areas and non-road areas. For example, the image grid of a road area typically has a relatively uniform pixel distribution, with small pixel differences between adjacent pixels and a smooth overall change; while in non-road areas, such as sidewalks on both sides of the road, vegetation-covered areas, and obstacle edges, the differences between pixels in the image grid are often large, and the pixel values ​​change drastically.

[0113] By calculating the pixel differences of multiple image grids, road recognition can be performed on the reconstructed environment image. This allows for the selection of image grid sets whose pixel differences conform to the distribution pattern of road features. The areas covered by these image grid sets are marked as road areas, while the remaining areas are marked as non-road areas.

[0114] In some embodiments, the intelligent cockpit can combine semantic labels of image grids to verify and correct road recognition results, thereby improving the accuracy of the division between road and non-road areas.

[0115] For example, the rasterized semantic occupancy network can also generate semantic labels corresponding to image grids, such as roads, sidewalks, and buildings. Each image grid corresponds to one or more semantic labels. For example, the semantic labels can represent the category of environmental elements corresponding to the image grid. For instance, if the semantic label of an image grid is "road-lane line-dashed line", it indicates that the image grid belongs to a road area. If the semantic label of an image grid is "sidewalk-vegetation", it indicates that the image grid belongs to a non-road area.

[0116] The method provided in this disclosure divides the reconstructed environment image into multiple image grids through rasterization processing, and identifies road areas and non-road areas based on the pixel differences between pixels. This identification method can achieve pixel-level accurate region division and improve the accuracy of road area and non-road area identification.

[0117] Figure 9 is a schematic diagram of an environment reconstruction image processing method apparatus provided in an exemplary embodiment of the present disclosure. The apparatus 900 of this embodiment can be used to implement the corresponding method embodiments of the present disclosure. Referring to Figure 9, the apparatus 900 may include: a tag determination module 910, a material generation module 920, a material positioning module 930, and an image processing module 940.

[0118] The label determination module 910 performs semantic parsing on the style commands input by the user based on the smart cockpit, obtaining a first label and a second label. The style command instructs the display of virtual material data corresponding to the virtual object in the reconstructed environment image corresponding to the smart cockpit. The first label describes the characteristics of the virtual object, and the second label indicates the position of the virtual object in the reconstructed environment image. The material generation module 920 inputs the first label into the material generation model and generates virtual material data corresponding to the virtual object through the material generation model. The material positioning module 930 determines the display position coordinates of the virtual material data based on the display area indicated by the second label in the reconstructed environment image. The image processing module 940 renders the virtual material data in the reconstructed environment image based on the display position coordinates of the virtual material data, obtaining the target environment reconstructed image.

[0119] In some embodiments, the material positioning module 930 is further configured to perform road recognition on the environment reconstruction image to obtain the road area and non-road area of ​​the environment reconstruction image; in response to the display area being located in the road area of ​​the environment reconstruction image, the display position coordinates of the virtual material data are determined according to the center point coordinates of the real material data of the road area, wherein the center point coordinates of the real material data are the same as the display position coordinates of the virtual material data.

[0120] In some embodiments, the image processing module 940 is further configured to create a first temporary layer on top of the real material data of the environment reconstruction image based on the display position coordinates of the virtual material data; and to render the virtual material data in the first temporary layer in the environment reconstruction image to obtain the target environment reconstruction image.

[0121] In some embodiments, the material positioning module 930 is further configured to, in response to the display area being located in a non-road area of ​​the environment reconstruction image, determine the display position coordinates of the virtual material data based on the center point coordinates of the target area within the non-road area of ​​the environment reconstruction image. The target area is the area indicated by the second label in the non-road area, and the center point coordinates of the target area are the same as the display position coordinates of the virtual material data. The image processing module is further configured to, based on the display position coordinates of the virtual material data, create a second temporary layer on top of the target area of ​​the environment reconstruction image; and render the virtual material data in the second temporary layer within the environment reconstruction image to obtain the target environment reconstruction image.

[0122] In some embodiments, the image processing module 940 is further configured to remove virtual material data rendered in the first temporary layer and / or the second temporary layer from the target environment reconstruction image based on the user's restoration command input to the smart cockpit.

[0123] In some embodiments, the image processing module 940 is further configured to determine the display position coordinates of the vehicle and the motion trend information of the vehicle in the target environment reconstructed image based on the vehicle's positioning data; determine the relative motion trend of the vehicle and the virtual material data based on the display position coordinates of the vehicle, the display position coordinates of the virtual material data, and the motion trend information of the vehicle; and adjust the display position coordinates of the first temporary layer and / or the second temporary layer in the target environment reconstructed image based on the relative motion trend.

[0124] In some embodiments, the material positioning module 930 is further configured to perform rasterization processing on the reconstructed environment image through a rasterized neural network to obtain multiple image grids; and to perform road recognition on the reconstructed environment image based on the pixel differences between multiple pixels in the multiple image grids to obtain and determine the road area and non-road area of ​​the reconstructed environment image.

[0125] The beneficial technical effects corresponding to the exemplary embodiment of this device 900 can be found in the corresponding beneficial technical effects in the exemplary method section above, and will not be repeated here.

[0126] Exemplary Electronic Device Figure 10 is a structural diagram of an electronic device provided in an embodiment of this disclosure.

[0127] As shown in Figure 10, the electronic device 1000 includes at least one processor 1001 and a memory 1002.

[0128] The processor 1001 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.

[0129] The memory 1002 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1001 may execute one or more computer program instructions to implement the environment reconstruction image processing methods and / or other desired functions of the various embodiments of this disclosure described above.

[0130] In one example, the electronic device 1000 may also include an input device 1003 and an output device 1004, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0131] The input device 1003 may also include, for example, a keyboard, a mouse, etc.

[0132] The output device 1004 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0133] Of course, for simplicity, Figure 10 only shows some of the components of the electronic device 1000 that are relevant to this disclosure, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1000 may include any other suitable components depending on the specific application.

[0134] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the environment reconstruction image processing methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0135] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0136] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the environmental reconstruction image processing methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0137] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0138] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0139] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An environment reconstruction image processing method, comprising: Semantic parsing is performed on the style commands input by the user based on the smart cockpit to obtain a first label and a second label. The style commands are used to instruct the display of virtual material data corresponding to virtual objects in the environment reconstruction image corresponding to the smart cockpit. The first label is used to describe the characteristics of the virtual object, and the second label indicates the position of the virtual object in the environment reconstruction image. The first label is input into the material generation model, and the virtual material data corresponding to the virtual object is generated through the material generation model. Based on the display area indicated by the second label in the environment reconstruction image, the display position coordinates of the virtual material data are determined. In the environment reconstruction image, the virtual material data is rendered based on the display position coordinates of the virtual material data to obtain the target environment reconstruction image.

2. The environmental reconstruction image processing method according to claim 1, wherein, The step of determining the display position coordinates of the virtual material data based on the display area indicated by the second tag in the reconstructed environment image includes: performing road recognition on the reconstructed environment image to obtain the road area and the non-road area of ​​the reconstructed environment image; in response to the display area being located in the road area of ​​the reconstructed environment image, determining the display position coordinates of the virtual material data according to the center point coordinates of the real material data of the road area, wherein the center point coordinates of the real material data are the same as the display position coordinates of the virtual material data.

3. The environmental reconstruction image processing method according to claim 2, wherein, The step of rendering the virtual material data based on the display position coordinates of the virtual material data in the reconstructed environment image to obtain the target environment reconstructed image includes: creating a first temporary layer on top of the real material data of the reconstructed environment image based on the display position coordinates of the virtual material data; and rendering the virtual material data on the first temporary layer in the reconstructed environment image to obtain the target environment reconstructed image.

4. The environmental reconstruction image processing method according to claim 3, wherein, After performing road recognition on the reconstructed environment image to obtain the road area and non-road area of ​​the reconstructed environment image, the method further includes: in response to the display area being located in the non-road area of ​​the reconstructed environment image, determining the display position coordinates of the virtual material data based on the center point coordinates of the target area in the non-road area of ​​the reconstructed environment image, wherein the target area is the area indicated by the second label in the non-road area, and the center point coordinates of the target area are the same as the display position coordinates of the virtual material data; creating a second temporary layer on top of the target area of ​​the reconstructed environment image based on the display position coordinates of the virtual material data; and rendering the virtual material data on the second temporary layer in the reconstructed environment image to obtain the target reconstructed environment image.

5. The environmental reconstruction image processing method according to claim 4, wherein, After rendering the virtual material data based on the display position coordinates of the virtual material data in the reconstructed environment image to obtain the target environment reconstructed image, the method further includes: removing the virtual material data rendered in the first temporary layer and / or the virtual material data rendered in the second temporary layer from the target environment reconstructed image based on the restoration command input by the user to the smart cockpit.

6. The environmental reconstruction image processing method according to claim 4, wherein, After rendering the virtual material data based on its display position coordinates in the reconstructed environment image to obtain the target environment reconstructed image, the method further includes: determining the display position coordinates of the vehicle and its motion trend information in the target environment reconstructed image based on the vehicle's positioning data; determining the relative motion trend between the vehicle and the virtual material data based on the vehicle's display position coordinates, the virtual material data's display position coordinates, and the vehicle's motion trend information; and adjusting the display position coordinates of the first temporary layer and / or the second temporary layer in the target environment reconstructed image based on the relative motion trend.

7. The environmental reconstruction image processing method according to claim 2, wherein, The step of performing road recognition on the reconstructed environment image to obtain the road region and the non-road region of the reconstructed environment image includes: performing rasterization processing on the reconstructed environment image through a rasterized neural network to obtain multiple image grids; and performing road recognition on the reconstructed environment image based on the pixel differences between multiple pixels in the multiple image grids to determine the road region and the non-road region of the reconstructed environment image.

8. An environmental reconstruction image processing apparatus, comprising: A label determination module is used to perform semantic parsing on style commands input by the user based on the smart cockpit, to obtain a first label and a second label. The style command instructs the display of virtual material data corresponding to a virtual object in the reconstructed environment image corresponding to the smart cockpit. The first label describes the characteristics of the virtual object, and the second label indicates the position of the virtual object in the reconstructed environment image. A material generation module is used to input the first label into a material generation model and generate virtual material data corresponding to the virtual object through the material generation model. A material positioning module is used to determine the display position coordinates of the virtual material data based on the display area indicated by the second label in the reconstructed environment image. An image processing module is used to render the virtual material data in the reconstructed environment image based on the display position coordinates of the virtual material data to obtain a target environment reconstructed image.

9. A computer-readable storage medium storing a computer program for performing the environment reconstruction image processing method as described in any one of claims 1-7.

10. An electronic device, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the environment reconstruction image processing method as described in any one of claims 1-7.

Images