Panoramic image generation method and device of vehicle, vehicle and storage medium

Abstract

The application discloses a panoramic image generation method and device of a vehicle, the vehicle and a storage medium. The method comprises the following steps: acquiring a first image frame and a plurality of second image frames in a driving process of the vehicle. The first image frame is an environment image frame captured by a head-mounted display device in the vehicle at a first time. The plurality of second image frames are environment image frames captured by a plurality of image acquisition devices of the vehicle at the first time. A first panoramic image frame is generated based on the first image frame and the plurality of second image frames. The first panoramic image frame is used for transmission to the head-mounted display device for display. In this way, when the panoramic image is spliced based on the environment images captured by the plurality of image acquisition devices in the driving process of the vehicle, the environment image captured by the head-mounted display device in the vehicle effectively eliminates the blind area, so that the panoramic image without the blind area which is closer to the actual environment can be obtained, thereby improving the visual experience of the user.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology for vehicles, and more specifically, to a method, apparatus, vehicle, and storage medium for generating panoramic images of a vehicle. Background Technology

[0002] With the booming development of the automotive industry, the application of intelligent driving systems is becoming increasingly widespread. These systems can collect images of the vehicle's surroundings using onboard environmental perception cameras, providing users with a comprehensive and immersive visual experience. However, in related technologies, the fixed shooting angles of multiple cameras on a vehicle make it difficult to capture a 360° panoramic view of the vehicle's surroundings during operation. This results in blind spots in the final panoramic display image, thus affecting the user's visual experience. Summary of the Invention

[0003] This application proposes a method, apparatus, vehicle, and storage medium for generating panoramic images of vehicles to improve upon the aforementioned deficiencies.

[0004] In a first aspect, embodiments of this application provide a method for generating panoramic images of a vehicle, comprising: acquiring a first image frame and a plurality of second image frames during the vehicle's driving process, wherein the first image frame is an environmental image frame captured by a head-mounted display device inside the vehicle at a first moment, and the plurality of second image frames are environmental image frames captured by a plurality of image acquisition devices of the vehicle at a first moment; generating a first panoramic image frame based on the first image frame and the plurality of second image frames, wherein the first panoramic image frame is used to transmit to the head-mounted display device for display.

[0005] In one optional embodiment, generating a first panoramic image frame based on the first image frame and the plurality of second image frames includes: generating a second panoramic image frame at a first moment based on the plurality of second image frames; determining a region in the second panoramic image frame that does not contain image pixels as a blind area region in the second panoramic image frame; and using the first image frame to fill the blind area region in the second panoramic image frame to obtain the first panoramic image frame.

[0006] In an optional embodiment, before filling the blind area in the second panoramic image frame with the first image frame to obtain the first panoramic image frame, the method further includes: acquiring a panoramic image frame generated at a time adjacent to the first time moment as a third panoramic image frame; identifying a first target image region in the third panoramic image frame that matches the blind area; and filling a portion of the image region in the blind area with the first target image region. The step of filling the blind area in the second panoramic image frame with the first image frame to obtain the first panoramic image frame includes: filling a specified image region in the blind area with the first image frame to obtain the first panoramic image frame, wherein the specified image region is other image regions in the blind area besides the specified portion of the image region.

[0007] In an optional embodiment, the step of using the first target image region to fill a portion of the image region in the blind spot region includes: determining the second pose information of the vehicle at a first moment based on the first pose information and first motion parameters of the vehicle at a moment adjacent to the first moment; obtaining the relative pose relationship between the second pose information and the first pose information; determining the first coordinate information of each image pixel in the first target image region in the portion of the image region in the blind spot region based on the relative pose relationship; and filling the portion of the image region based on the first coordinate information of each image pixel in the first target image region.

[0008] In an optional embodiment, the step of using the first image frame to fill a designated image region in the blind zone to obtain the first panoramic image frame includes: determining a second target image region in the first image frame that matches the designated image region in the blind zone; acquiring target pose information of the head-mounted display device at a first moment; determining second coordinate information of each image pixel in the second target image region in the designated image region in the blind zone based on the target pose information; and filling the designated image region based on the second target image region and the second coordinate information of each image pixel to obtain the first panoramic image frame.

[0009] In an optional embodiment, determining the second target image region in the first image frame that matches the designated image region in the blind zone region includes: acquiring edge pixels surrounding the blind zone region in the second panoramic image frame; performing feature similarity matching between the edge pixels of the blind zone region and image pixels in the first image frame; acquiring multiple pixels in the first image frame whose feature similarity to the edge pixels of the blind zone region is greater than a preset similarity threshold, as multiple target pixels; and acquiring the image region enclosed by the multiple target pixels, as the second target image region.

[0010] In an optional embodiment, after generating a first panoramic image frame based on the first image frame and the plurality of second image frames, the method further includes: obtaining the relative pose relationship between the head-mounted display device and the vehicle at a first moment; performing three-dimensional mapping on the first panoramic image frame based on the relative pose relationship to obtain a three-dimensional image frame corresponding to the first panoramic image frame; superimposing a preset virtual component on the three-dimensional image frame to obtain a target three-dimensional image frame, the target three-dimensional image frame being used to transmit to the head-mounted display device for display.

[0011] Secondly, embodiments of this application provide a panoramic image generation device for a vehicle. The device includes: an image frame acquisition module, used to acquire a first image frame and a plurality of second image frames during the vehicle's driving process, wherein the first image frame is an environmental image frame captured by a head-mounted display device inside the vehicle at the current moment, and the plurality of second image frames are environmental image frames captured by a plurality of image acquisition devices of the vehicle at a first moment; and a panoramic image frame generation module, used to generate a first panoramic image frame based on the first image frame and the plurality of second image frames, wherein the first panoramic image frame is used to transmit to the head-mounted display device for display.

[0012] Thirdly, embodiments of this application also provide a vehicle, including: one or more processors; a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the methods described above.

[0013] Fourthly, embodiments of this application also provide a computer-readable storage medium storing program code that can be invoked by a processor to execute the above-described method.

[0014] This application provides a method, apparatus, vehicle, and storage medium for generating panoramic images of a vehicle. In this application, a first image frame and multiple second image frames are acquired during the vehicle's driving process. The first image frame is an environmental image frame captured at a specific moment by a head-mounted display device inside the vehicle, and the multiple second image frames are environmental image frames captured at the same moment by multiple image acquisition devices within the vehicle. Based on the first image frame and the multiple second image frames, a first panoramic image frame is generated and transmitted to the head-mounted display device for display. Thus, when stitching panoramic images based on environmental images captured by multiple image acquisition devices during vehicle operation, the environmental images captured by the head-mounted display device inside the vehicle effectively eliminate blind spots, resulting in a more realistic, blind-spot-free panoramic image, thereby improving the user's visual experience.

[0015] Other features and advantages of the embodiments of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the embodiments of this application. The objects and other advantages of the embodiments of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram of a panoramic image generation system provided in an embodiment of this application is shown.

[0018] Figure 2 A schematic flowchart of a method for generating panoramic images of a vehicle according to an embodiment of this application is shown.

[0019] Figure 3 This application shows Figure 2 A flowchart illustrating step S220 as a sub-step in one embodiment.

[0020] Figure 4 A flowchart illustrating a method for generating panoramic images of a vehicle according to another embodiment of this application is shown.

[0021] Figure 5 This application shows Figure 4 A flowchart illustrating a sub-step of step S360 in one embodiment.

[0022] Figure 6 This application shows Figure 4 A flowchart illustrating a sub-step of step S370 in one embodiment.

[0023] Figure 7 This application shows Figure 5 A flowchart illustrating a sub-step of step S371 in one embodiment.

[0024] Figure 8 A structural block diagram of a vehicle panoramic image generation apparatus according to an embodiment of this application is shown.

[0025] Figure 9 A structural block diagram of a vehicle provided in an embodiment of this application is shown.

[0026] Figure 10 A structural block diagram of a computer-readable storage medium provided in an embodiment of this application is shown. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. The components of the embodiments of the present application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are within the scope of protection of the present application.

[0028] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] Please see Figure 1 , Figure 1 A schematic diagram of a panoramic image generation system provided in an embodiment of this application is shown. The panoramic image generation system 10 may include a vehicle 100 and a head-mounted display device 120. Multiple image acquisition devices 110 with different image acquisition angles are disposed on the exterior of the vehicle 100, and the head-mounted display device 120 is disposed inside the vehicle 100. All of the multiple image acquisition devices 110 and the head-mounted display device 120 are connected to the vehicle 100 through communication.

[0030] Optionally, multiple image acquisition devices 110 can capture 360° real-time images of the external environment while the vehicle is in motion. These multiple image acquisition devices 110 externally mounted on the vehicle 100 may include at least a front-view camera, a left front-side camera, a left rear-side camera, a right front-side camera, a right rear-side camera, and a rear camera; no limitation is imposed here. Simultaneously, while the vehicle is in motion, the vehicle 100 can acquire video streams of real-time environmental images captured by the head-mounted display device 120 used by the user, within the camera's field of view.

[0031] Optionally, the vehicle 100 can acquire a first video stream captured by the head-mounted display device 120 and multiple second video streams captured by multiple image acquisition devices 110 through the intelligent driving domain control device, and process each frame of the multiple video streams, including at least brightness and color equalization processing and grayscale processing, converting each frame of the image into image data that can be processed and analyzed by the cockpit domain control device. The intelligent driving domain control device is also used to transmit image data to the cockpit domain control device via Ethernet communication or Low-Voltage Differential Signaling (LVDS) communication.

[0032] In this embodiment, the cockpit domain control device acquires a first image frame and multiple second image frames during the vehicle's driving process. The first image frame is an environmental image frame captured by the head-mounted display device 120 inside the vehicle at the current moment, and the multiple second image frames are environmental image frames captured by multiple image acquisition devices 110 of the vehicle at the first moment. This generates a 360° panoramic environmental image of the vehicle's exterior for transmission to the head-mounted display device for display. In other words, the cockpit domain control device can generate a first panoramic image frame based on the first image frame and multiple second image frames, and the first panoramic image frame is used for transmission to the head-mounted display device for display.

[0033] Please see Figure 2 , Figure 2 A schematic flowchart of a method for generating panoramic images of a vehicle according to an embodiment of this application is shown. The following will be combined with... Figure 2 The method for generating panoramic images of a vehicle provided in this application is described in detail. For the method of generating panoramic images of a vehicle, please refer to [link / reference needed]. Figure 2 This may include the following steps:

[0034] Step S210: Acquire a first image frame and multiple second image frames of the vehicle during its driving process. The first image frame is an environmental image frame captured by the head-mounted display device inside the vehicle at the first moment, and the multiple second image frames are environmental image frames captured by multiple image acquisition devices of the vehicle at the first moment.

[0035] In this embodiment, the head-mounted display device installed inside the vehicle and the multiple image acquisition devices installed outside the vehicle are both used to capture the real-time environmental scene during the vehicle's driving process. In order to generate a 360° panoramic environmental image of the vehicle's exterior at the first moment, it is necessary to acquire the environmental image frames captured by the head-mounted display device inside the vehicle at the first moment and the environmental image frames captured by the multiple image acquisition devices of the vehicle at the first moment during the vehicle's driving process, that is, to acquire the first image frame and multiple second image frames.

[0036] The first moment can be the current moment, that is, the first image frame and multiple second image frames are images generated in real time when the head-mounted display device and multiple image acquisition devices set outside the vehicle capture the real scene of the environment; the first moment can also be a historical moment, that is, after the head-mounted display device and multiple image acquisition devices set outside the vehicle capture the real scene of the environment for a preset time period, multiple image frames of any moment in the first video stream and multiple second video streams captured within the preset time period.

[0037] Step S220: Based on the first image frame and multiple second image frames, generate a first panoramic image frame, which is used to transmit to the head-mounted display device for display.

[0038] In this embodiment, based on a first image frame captured by a head-mounted display device and multiple second image frames captured by multiple image acquisition devices, blind spot filling and panoramic image stitching are performed on the multiple second image frames captured by the multiple image acquisition devices to obtain a first panoramic image frame, which is then transmitted to the head-mounted display device for display. Because blind spot filling eliminates blind areas during the generation of the first panoramic image frame, a panoramic image closer to the actual environment can be obtained.

[0039] When the first moment is the current moment, the first panoramic image frame is a panoramic image generated in real time based on the first image frame at the current moment and multiple second image frames, thereby reducing the delay in displaying the first panoramic image frame at the current moment on the head-mounted display device; when the first moment is a historical moment, the first panoramic image frame is a panoramic image generated based on the first video stream captured within a preset time period and multiple image frames from multiple second video streams, for any moment within the preset time period, corresponding to the first image frame and multiple second image frames, thereby ensuring the smoothness of the generated panoramic image video stream.

[0040] In some implementation methods, please refer to Figure 3 Step S220 may include the contents of steps S221 to S223:

[0041] Step S221: Generate a second panoramic image frame at the first moment based on multiple second image frames.

[0042] In this embodiment, during the process of eliminating blind spots to generate the first panoramic image frame, multiple second image frames captured by multiple image acquisition devices at a first moment of the vehicle's driving environment are first stitched together to generate the second panoramic image frame at the first moment. Because the image acquisition angle of each fixed external image acquisition device is fixed, the area captured by the multiple image acquisition devices during vehicle movement is difficult to cover the 360° panoramic environment outside the vehicle, thus creating blind spots. Consequently, the second panoramic image frame generated by stitching together multiple second image frames contains blind spot areas.

[0043] Specifically, after installing multiple image acquisition devices with different image acquisition angles fixed to the vehicle on its exterior, the intrinsic parameter data, extrinsic parameter data, and distortion data of each image acquisition device can be obtained by calibrating each image acquisition device.

[0044] First, distortion correction is performed on the second image frames corresponding to each image acquisition device based on the distortion data of each device, eliminating imaging distortion. Since the extrinsic parameters of each image acquisition device are different (i.e., the height and shooting angle of multiple devices are different), all images need to be projected onto the same coordinate system to fuse the images captured by multiple devices to obtain a panoramic image. Therefore, based on the rotation matrix and translation vector in the extrinsic parameters, the coordinate transformation parameters of each image acquisition device's camera coordinate system relative to the world coordinate system can be obtained. According to the coordinate transformation parameters of each image acquisition device relative to the world coordinate system, the multiple distortion-corrected second image frames are stitched together in the world coordinate system to obtain the first panoramic image frame. This world coordinate system is established with a point in the vehicle as the origin. After transforming each second image frame to the world coordinate system, image fusion is also required for the overlapping areas in the second image frames. This involves fusing regions composed of image pixels with the same coordinate information in adjacent second image frames to generate the first panoramic image frame based on the multiple second image frames fused from the overlapping areas.

[0045] Step S222: Determine the regions in the second panoramic image frame that do not contain image pixels, and designate them as blind areas in the second panoramic image frame.

[0046] Optionally, the blind zone is the area in the second panoramic image frame that does not contain any image pixels. It's important to distinguish this from the case where, when the image acquisition device captures an area containing a black object, that area will contain image pixels in the corresponding second panoramic image frame. That is, a blind zone without image pixels may appear as a black area in the second panoramic image frame, but not every black area in the second panoramic image frame is a blind zone. Therefore, areas in the second panoramic image frame that do not contain image pixels are defined as the blind zone in the second panoramic image frame.

[0047] In this embodiment, during the acquisition of the blind zone region in the second panoramic image frame, a perspective transformation is first performed on the second panoramic image frame based on the camera's extrinsic data to obtain a panoramic top-down view corresponding to the second panoramic image frame. The blind zone region is then determined based on this panoramic top-down view. Converting the three-dimensional panoramic image into a two-dimensional panoramic top-down view allows for more efficient and faster determination of the blind zone region within a two-dimensional scene.

[0048] Step S223: Using the first image frame, fill the blind area in the second panoramic image frame with the image to obtain the first panoramic image frame.

[0049] In this embodiment, a first image frame is obtained by capturing the real-time environment of the vehicle during its driving process using a head-mounted display device. This first image frame is then used to fill in the blind spots in the second panoramic image frame to obtain the first panoramic image frame. This results in the first panoramic image frame being obtained based on the second panoramic image frame after eliminating the blind spots, making it a panoramic image that is closer to the actual environment.

[0050] In other embodiments, during the process of eliminating blind areas to generate a first panoramic image frame, the regions in each second image frame that do not contain image pixels can be first determined as blind areas in each second image frame; then, the first image frame is used to fill the blind areas in each second image frame to obtain multiple blind-filled second image frames; finally, the first panoramic image frame is generated based on the multiple blind-filled second image frames.

[0051] In this embodiment, when filling the blind area in the second panoramic image frame, the filling is based on the panoramic top view of the two-dimensional scene corresponding to the second panoramic image frame. After the blind area filling is completed, the panoramic top view is subjected to inverse perspective transformation based on the camera's extrinsic data, and then converted into the first panoramic image frame in the three-dimensional scene.

[0052] It should be noted that during the process of generating the first panoramic image frame based on the first image frame and multiple second image frames, the panoramic image can be stitched together first based on the multiple second image frames, and then blind spot filling can be performed based on the stitched second panoramic image frame; alternatively, blind spot filling can be performed first on the multiple second image frames, and then panoramic image stitching can be performed based on the stitched second image frames. No restriction is placed on this. In the process of blind spot filling based on the stitched second panoramic image frame, it is not necessary to determine the blind spot area for each of the multiple second image frames, nor is it necessary to use the first image frame to fill the blind spot area of ​​each second image frame individually. This simplifies the data processing for blind spot filling, improves the generation speed of the first panoramic image frame, and thus reduces the latency of the head-mounted display device displaying the first panoramic image frame at the first moment when the panoramic image frame generation module and the head-mounted display device capture the real-time environmental scene during vehicle movement, thereby improving the user's immersive panoramic display experience.

[0053] Optionally, after the step of generating the first panoramic image frame based on the first image frame and multiple second image frames, the steps of this application may further include: obtaining the relative pose relationship between the head-mounted display device and the vehicle at a first moment, that is, obtaining the pose information of the head-mounted display device at the first moment in a world coordinate system established based on a point in the vehicle at the first moment as the origin of the coordinate system.

[0054] Furthermore, based on the relative pose relationship, a 3D mapping is performed on the first panoramic image frame to obtain the corresponding 3D image frame. This embodiment, based on the relative pose relationship between the head-mounted display device and the vehicle, can obtain the coordinate information of each image pixel in the first panoramic image frame in a world coordinate system established with a point on the head-mounted display device as the origin. This allows for the 3D mapping of the first panoramic image frame to obtain the corresponding 3D image frame, thereby converting the panoramic image obtained through panoramic stitching and blind spot filling into a virtual image that conforms to the user's viewing angle while wearing the head-mounted display device.

[0055] Furthermore, preset virtual components are superimposed on the 3D image frame to obtain a target 3D image frame, which is then transmitted to a head-mounted display device for display. By superimposing preset virtual components on the 3D image frame, including but not limited to control components, environmental information display components, and vehicle operation information display components superimposed on the virtual screen, environmental and vehicle operation information are displayed in the virtual screen from the user's perspective. Simultaneously, the user can control the relevant functions of the device through the control components via preset interactive methods, thus obtaining the target 3D image frame for transmission to the head-mounted display device for display.

[0056] Optionally, the head-mounted display device is used to display based on the received target 3D image frame, and the head-mounted display device can display a portion of the image area in the target 3D image frame corresponding to the user's head posture when wearing the head-mounted display device. That is, when the user turns their head, the display view of the target 3D image frame obtained by the user through the head-mounted display device will follow the user's head rotation, thereby improving the user's immersive visual experience.

[0057] Based on the above method, the panoramic image obtained after panoramic stitching and blind spot filling is converted into a virtual screen that conforms to the perspective of the user wearing a head-mounted display device. By superimposing preset virtual components on the three-dimensional image frame, interactive control components and information related to driving data are displayed in the virtual screen from the user's perspective, so that the user can immerse himself in the virtual and real scene of vehicle operation by wearing a head-mounted display device.

[0058] In the embodiments of this application, a first image frame and multiple second image frames are acquired during the vehicle's driving process. The first image frame is an environmental image frame captured by the vehicle's in-vehicle head-mounted display device at a first moment, and the multiple second image frames are environmental image frames captured by multiple image acquisition devices of the vehicle at a first moment. Based on the first image frame and the multiple second image frames, a first panoramic image frame is generated and transmitted to the head-mounted display device for display. Thus, when stitching panoramic images based on environmental images captured by multiple image acquisition devices during vehicle movement, the environmental images captured by the vehicle's in-vehicle head-mounted display device effectively eliminate blind spots, resulting in a more realistic, blind-spot-free panoramic image, thereby improving the user's visual experience.

[0059] Please see Figure 4 , Figure 4 A flowchart illustrating a method for generating panoramic images of a vehicle according to another embodiment of this application is shown, including the following steps:

[0060] Step S310: Acquire a first image frame and multiple second image frames of the vehicle during its driving process. The first image frame is an environmental image frame captured by the head-mounted display device inside the vehicle at the first moment, and the multiple second image frames are environmental image frames captured by multiple image acquisition devices of the vehicle at the first moment.

[0061] Step S320: Generate a second panoramic image frame at the first moment based on multiple second image frames.

[0062] Step S330: Determine the regions in the second panoramic image frame that do not contain image pixels as blind areas in the second panoramic image frame.

[0063] In this embodiment, the specific implementation of steps S310 to S330 can be found in the content of the foregoing embodiments, and will not be repeated here.

[0064] Step S340: Obtain the panoramic image frame generated at the time adjacent to the first time moment, and use it as the third panoramic image frame.

[0065] In this embodiment, a third panoramic image frame is generated by acquiring panoramic image frames generated at times adjacent to the first time, specifically the panoramic image frames of the previous and subsequent times adjacent to the first time. The panoramic image frames at times adjacent to the first time are panoramic image frames generated by stitching together environmental image frames captured by multiple image acquisition devices of the vehicle at times adjacent to the first time.

[0066] Step S350: Confirm the first target image region in the third panoramic image frame that matches the blind zone region.

[0067] In this embodiment, edge pixels surrounding the blind zone in the second panoramic image frame are acquired, and feature similarity matching is performed between these edge pixels and image pixels in the third panoramic image frame. This yields multiple pixels in the third panoramic image frame whose feature similarity to the edge pixels of the blind zone is greater than a preset similarity threshold. The image region enclosed by these multiple pixels is then acquired as the first target image region in the third panoramic image frame that matches the blind zone. The blind zone is the region in the second panoramic image frame that does not contain any image pixels.

[0068] Step S360: Using the first target image region, fill a portion of the image region in the blind area with the image.

[0069] In this embodiment, the first target image region in the third panoramic image frame that matches the blind zone region is used to fill a portion of the image region in the blind zone region. At this time, the panoramic image frame generated based on the environmental image frames captured by multiple image acquisition devices at a time adjacent to the first time moment only fills a portion of the image region in the second panoramic image frame. For the other image regions in the blind zone region (i.e., the designated image regions), in addition to the portion of the image regions, the first image frame captured by the wearable device at the first time moment is also used to fill the designated image regions.

[0070] In some implementation methods, please refer to Figure 5 Step S360 may include the contents of steps S361 to S364:

[0071] Step S361: Determine the second pose information of the vehicle at the first moment based on the first pose information of the vehicle at the moment adjacent to the first moment and the first motion parameters.

[0072] In this embodiment, based on the first pose information and first motion parameters of the vehicle at a time adjacent to the first moment, the second pose information of the vehicle at the first moment can be determined. Specifically, based on the vehicle's velocity and heading angle during the time interval between the first moment and the time adjacent to the first moment, the relative displacement and relative heading angle between the vehicle at the first moment and the vehicle at the time adjacent to the first moment can be obtained, serving as the vehicle's first motion parameters.

[0073] The first motion parameter can be the motion parameters of the vehicle at a time adjacent to the first time, including at least the velocity, acceleration, heading angle, and acceleration of the heading angle at the time adjacent to the first time. Based on the motion parameter, the second pose information of the vehicle at the first time can be estimated.

[0074] Optionally, the first motion parameter can also be the motion parameter of the vehicle during the time period consisting of the first moment and the moment adjacent to the first moment, including at least the vehicle's speed and heading angle during the time period. By integrating the vehicle's speed and heading angle during the time period, the relative displacement and relative heading angle between the vehicle at the first moment and the vehicle at the moment adjacent to the first moment can be obtained, thereby obtaining the vehicle's second pose information at the first moment.

[0075] Step S362: Obtain the relative pose relationship between the second pose information and the first pose information.

[0076] Optionally, based on the vehicle's second pose information at the first moment and the vehicle's first pose information at a moment adjacent to the first moment, the relative pose relationship between the second pose information and the first pose information is obtained.

[0077] Step S363: Based on the relative pose relationship, determine the first coordinate information of each image pixel in the first target image region in the partial image region of the blind zone region.

[0078] In this embodiment, based on the above relative pose relationship, the coordinate information of each image pixel in the third panoramic image frame in the world coordinate system established based on the vehicle at the first moment can be obtained, thereby enabling the determination of the first coordinate information of each image pixel in the first target image region in the third panoramic image frame in a portion of the image region in the blind zone region that matches the first target image region.

[0079] Step S364: Based on the first coordinate information of each image pixel in the first target image region, perform image filling on a portion of the image region.

[0080] Optionally, based on the first coordinate information of each image pixel in the first target image region in a portion of the image region in the blind zone region, image filling is performed on a portion of the image region, thereby completing a panoramic image frame generated based on environmental image frames captured by multiple image acquisition devices at a time adjacent to the first time, and image filling is performed on a portion of the image region in the second panoramic image frame.

[0081] Step S370: Using the first image frame, fill the specified image area in the blind zone to obtain the first panoramic image frame. The specified image area is the image area in the blind zone other than the partial image area.

[0082] In this embodiment, the panoramic image frame generated based on the environmental image frames captured by multiple image acquisition devices at a time adjacent to the first time moment only fills a portion of the image area in the second panoramic image frame. For the other image areas in the blind zone (i.e., the designated image areas), the designated image areas are filled using the first image frame captured by the wearable device at the first time moment. Thus, through two blind zone fillings, the final generated first panoramic image frame can be a panoramic image that is closer to the actual environment.

[0083] In some implementation methods, please refer to Figure 6 Step S370 may include the contents of steps S371 to S374:

[0084] Step S371: Determine a second target image region in the first image frame that matches a specified image region in the blind zone region.

[0085] In this embodiment, by performing feature similarity matching between the edge pixels of the blind zone and the image pixels in the first image frame, a second target image region in the first image frame that matches a specified image region in the blind zone can be determined.

[0086] Specifically, please refer to Figure 7 Step S371 may include the contents of steps S371-1 to S371-4:

[0087] Step S371-1: Obtain the edge pixels surrounding the blind area in the second panoramic image frame.

[0088] In this embodiment, it is first necessary to obtain the edge pixels of the specified image region surrounding the blind zone region from multiple image pixels of the second image frame.

[0089] Step S371-2: Perform feature similarity matching between the edge pixels of the blind area and the image pixels in the first image frame.

[0090] Optionally, feature similarity matching is performed between the edge pixels of the specified image region in the blind zone and the image pixels in the first image frame captured by the wearable device at the first moment, so as to retrieve multiple pixels in the first image frame that are similar to the edge pixels of the specified image region in the blind zone.

[0091] Step S371-3: Obtain multiple pixels in the first image frame whose feature similarity with the edge pixels of the blind area is greater than a preset similarity threshold, and use them as multiple target pixels.

[0092] In this embodiment, multiple pixels in the feature pixels of the first image frame whose similarity to the edge pixels of a specified image region in the blind zone is greater than a preset similarity threshold are obtained and used as multiple target pixels that match the edge pixel features of the specified image region in the blind zone.

[0093] Step S371-4: Obtain the image region enclosed by multiple target pixels as the second target image region.

[0094] In this embodiment, the image region enclosed by multiple target pixels that match the edge pixel features of the specified image region in the blind zone is used as the second target image region in the first image frame that matches the specified image region in the blind zone.

[0095] Step S372: Obtain the target pose information of the head-mounted display device at the first moment.

[0096] In this embodiment, feature matching is performed between the first image frame captured by the wearable device at the first moment and the second panoramic image frame to establish the correspondence between the matched pixels, thereby obtaining the target pose information of the head-mounted display device at the first moment in the world coordinate system established based on the vehicle at the first moment.

[0097] Step S373: Based on the target pose information, determine the second coordinate information of each image pixel in the second target image region within the specified image region of the blind zone region.

[0098] In this embodiment, based on the target pose information, the coordinate information of each image pixel in the first image frame in the world coordinate system established based on the vehicle at the first moment can be obtained, thereby enabling the determination of the second coordinate information of each image pixel in the second target image region in the first image frame in the specified image region of the blind zone region that matches the second target image region.

[0099] Step S374: Based on the second target image region and the second coordinate information of each image pixel, fill the specified image region to obtain the first panoramic image frame.

[0100] Optionally, based on the second coordinate information of each image pixel in the second target image region in the specified image region of the blind zone region, the specified image region is filled with images, thereby completing the blind zone filling of the specified image region based on the first image frame captured by the wearable device at the first moment, so that the final generated first panoramic image frame can be a panoramic image that is closer to the actual environment.

[0101] In this embodiment, when filling the blind area in the second panoramic image frame, not only is a portion of the image area in the second panoramic image frame filled using panoramic image frames generated from environmental image frames captured by multiple image acquisition devices at times adjacent to the first time, but also, for the other image areas in the blind area (i.e., designated image areas), the designated image areas are filled using the first image frame captured by the wearable device at the first time. Thus, through these two blind area image fillings, the final generated first panoramic image frame becomes a panoramic image closer to the actual environment, improving the aesthetics of the image display in the head-mounted display device and enhancing the user's immersive visual experience.

[0102] Please refer to Figure 8 , Figure 8 This diagram illustrates a structural block diagram of a panoramic image generation device for a vehicle according to an embodiment of this application. The panoramic image generation device 400 may include an image frame acquisition module 410 and a panoramic image frame generation module 420.

[0103] The image frame acquisition module 410 is used to acquire a first image frame and multiple second image frames of the vehicle during its driving process. The first image frame is an environmental image frame captured by the head-mounted display device inside the vehicle at the current moment, and the multiple second image frames are environmental image frames captured by multiple image acquisition devices of the vehicle at the first moment.

[0104] The panoramic image frame generation module 420 is used to generate a first panoramic image frame based on a first image frame and multiple second image frames. The first panoramic image frame is used to transmit to a head-mounted display device for display.

[0105] In some implementations, the panoramic image frame generation module 420 may include a blind spot filling module, which is specifically used to: generate a second panoramic image frame at a first moment based on multiple second image frames; determine the region in the second panoramic image frame that does not contain image pixels as the blind spot region in the second panoramic image frame; and fill the blind spot region in the second panoramic image frame with the first image frame to obtain a first panoramic image frame.

[0106] Optionally, before using the first image frame to fill the blind area in the second panoramic image frame to obtain the first panoramic image frame, the panoramic image frame generation module 420 is further configured to: acquire a panoramic image frame generated at a time adjacent to the first time as the third panoramic image frame; identify a first target image region in the third panoramic image frame that matches the blind area; and use the first target image region to fill a portion of the image region in the blind area.

[0107] Furthermore, the blind spot filling module in the panoramic image frame generation module 420 can be specifically used to: determine the second pose information of the vehicle at the first moment based on the first pose information and the first motion parameters of the vehicle at the moment adjacent to the first moment; obtain the relative pose relationship between the second pose information and the first pose information; determine the first coordinate information of each image pixel in the first target image region in the partial image region of the blind spot region based on the relative pose relationship; and fill the partial image region based on the first coordinate information of each image pixel in the first target image region.

[0108] In some implementations, the blind spot filling module may also be specifically used to: determine a second target image region in the first image frame that matches a specified image region in the blind spot region; acquire the target pose information of the head-mounted display device at a first moment; determine the second coordinate information of each image pixel in the second target image region in the specified image region of the blind spot region based on the target pose information; and fill the specified image region with image based on the second target image region and the second coordinate information of each image pixel to obtain a first panoramic image frame.

[0109] Optionally, the blind spot filling module can also be specifically used for: obtaining edge pixels surrounding the blind spot region in the second panoramic image frame; performing feature similarity matching between the edge pixels of the blind spot region and the image pixels in the first image frame; obtaining multiple pixels in the first image frame whose feature similarity with the edge pixels of the blind spot region is greater than a preset similarity threshold, as multiple target pixels; and obtaining the image region enclosed by the multiple target pixels, as the second target image region.

[0110] Optionally, the panoramic image generation device 400 may further include a three-dimensional image frame generation module, used to obtain the relative pose relationship between the head-mounted display device and the vehicle at a first moment; based on the relative pose relationship, perform three-dimensional mapping on the first panoramic image frame to obtain a three-dimensional image frame corresponding to the first panoramic image frame; superimpose a preset virtual component on the three-dimensional image frame to obtain a target three-dimensional image frame, and the target three-dimensional image frame is used to transmit to the head-mounted display device for display.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0112] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0113] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0114] In summary, the solution provided in this application acquires a first image frame and multiple second image frames during the vehicle's driving process. The first image frame is an environmental image frame captured by the vehicle's internal head-mounted display device at a first moment, and the multiple second image frames are environmental image frames captured by multiple image acquisition devices of the vehicle at a first moment. Based on the first image frame and the multiple second image frames, a first panoramic image frame is generated and transmitted to the head-mounted display device for display. Thus, when stitching panoramic images based on environmental images captured by multiple image acquisition devices during vehicle operation, the environmental images captured by the vehicle's internal head-mounted display device effectively eliminate blind spots, resulting in a more realistic, blind-spot-free panoramic image, thereby improving the user's visual experience.

[0115] Please refer to Figure 9 , Figure 9 The diagram shows a structural block diagram of a vehicle 500 according to an embodiment of this application. The above-described method provided in this embodiment of the application can be executed by the vehicle 500.

[0116] The vehicle 500 in this application embodiment may include one or more of the following components: processor 501, memory 502, and one or more application programs, wherein the one or more application programs may be stored in memory 502 and configured to be executed by one or more processors 501, and the one or more programs are configured to perform the methods as described in the foregoing method embodiments.

[0117] Processor 501 may include one or more processing cores. Processor 501 connects to various parts within the vehicle 500 using various interfaces and lines, and performs various functions and processes data of the vehicle 500 by running or executing instructions, programs, code sets, or instruction sets stored in memory 502, and by calling data stored in memory 502. Optionally, processor 501 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 501 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the aforementioned modem can also be integrated into processor 501 and implemented using a separate communication chip.

[0118] The memory 502 may include random access memory (RAM) or read-only memory (ROM). The memory 502 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 502 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the vehicle 500 during use (such as the various correspondences described above).

[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0120] In the several embodiments provided in this application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or other forms.

[0121] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0122] Please refer to Figure 10 , Figure 10 A structural block diagram of a computer-readable storage medium 700 provided in an embodiment of this application is shown. The computer-readable storage medium 700 stores program code 610, which can be called by a processor to execute the methods described in the above method embodiments.

[0123] The computer-readable storage medium 600 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 600 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 600 has storage space for program code 610 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 610 may be compressed, for example, in a suitable form.

[0124] In some embodiments, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the steps in the above-described method embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for generating a panoramic image of a vehicle, characterized in that, The method includes: The vehicle acquires a first image frame and multiple second image frames during its driving process. The first image frame is an environmental image frame captured by the head-mounted display device inside the vehicle at a first moment, and the multiple second image frames are environmental image frames captured by multiple image acquisition devices of the vehicle at a first moment. Based on the plurality of second image frames, a second panoramic image frame at the first moment is generated; The regions in the second panoramic image frame that do not contain image pixels are identified as blind areas in the second panoramic image frame. Using the first image frame, the blind area in the second panoramic image frame is filled with an image to obtain a first panoramic image frame, which is then transmitted to the head-mounted display device for display.

2. The method according to claim 1, characterized in that, Before filling the blind area in the second panoramic image frame with the first image frame to obtain the first panoramic image frame, the method further includes: A panoramic image frame generated at a time adjacent to the first time is acquired as a third panoramic image frame, wherein the panoramic image frame is generated by stitching together environmental image frames captured by multiple image acquisition devices of the vehicle at a time adjacent to the first time. Confirm the first target image region in the third panoramic image frame that matches the blind zone region; Using the first target image region, a portion of the image region in the blind area is filled with an image; The step of filling the blind area in the second panoramic image frame with the first image frame to obtain the first panoramic image frame includes: Using the first image frame, a designated image region in the blind zone is filled with an image to obtain the first panoramic image frame. The designated image region is the other image region in the blind zone besides the partially defined image region.

3. The method according to claim 2, characterized in that, The step of using the first target image region to fill a portion of the image region in the blind area includes: The second pose information of the vehicle at the first moment is determined based on the first pose information and the first motion parameters of the vehicle at the moment adjacent to the first moment. Obtain the relative pose relationship between the second pose information and the first pose information; Based on the relative pose relationship, determine the first coordinate information of each image pixel in the first target image region in the partial image region of the blind zone region; Based on the first coordinate information of each image pixel in the first target image region, the partial image region is filled with images.

4. The method according to claim 2, characterized in that, The step of using the first image frame to fill a designated image region in the blind spot region to obtain the first panoramic image frame includes: Determine a second target image region in the first image frame that matches the specified image region in the blind zone region; Obtain the target pose information of the head-mounted display device at the first moment; Based on the target pose information, determine the second coordinate information of each image pixel in the second target image region within the specified image region of the blind zone region; Based on the second target image region and the second coordinate information of each image pixel, the specified image region is filled to obtain the first panoramic image frame.

5. The method according to claim 4, characterized in that, Determining the second target image region in the first image frame that matches the designated image region in the blind zone region includes: Obtain the edge pixels surrounding the blind zone region in the second panoramic image frame; The edge pixels of the blind area are matched with the image pixels in the first image frame by feature similarity. Multiple pixels in the first image frame whose edge pixel feature similarity with the blind area is greater than a preset similarity threshold are obtained as multiple target pixels. The image region enclosed by the plurality of target pixels is obtained as the second target image region.

6. The method according to any one of claims 1 to 5, characterized in that, After obtaining the first panoramic image frame, the method further includes: Obtain the relative pose of the head-mounted display device to the vehicle at a first moment; Based on the relative pose relationship, the first panoramic image frame is 3D mapped to obtain the 3D image frame corresponding to the first panoramic image frame. A preset virtual component is superimposed on the three-dimensional image frame to obtain a target three-dimensional image frame, which is used to transmit to the head-mounted display device for display.

7. A panoramic image generation device for a vehicle, characterized in that, The device includes: The image frame acquisition module is used to acquire a first image frame and multiple second image frames of the vehicle during its driving process. The first image frame is an environmental image frame captured by the head-mounted display device inside the vehicle at the current moment, and the multiple second image frames are environmental image frames captured by multiple image acquisition devices of the vehicle at the first moment. A panoramic image frame generation module is used to generate a second panoramic image frame at a first moment based on the plurality of second image frames; determine the region in the second panoramic image frame that does not contain image pixels as the blind area region in the second panoramic image frame; and use the first image frame to fill the blind area region in the second panoramic image frame to obtain a first panoramic image frame, which is used to transmit to the head-mounted display device for display.

8. A vehicle, characterized in that, The vehicles include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images